Embedded 3D Vision

High-resolution 3D data directly from the camera.

If large volumes or several object views are to be checked automatically by 3D cameras, e.g. on constantly running production lines in the automotive industry, high-resolution 3D result data must be generated and processed quickly. Stereo camera systems with large 5 MP sensors and variable baselines provide the ideal output data. However, interfaces and CPU power can quickly form the bottleneck in such high-performance 3D applications.

The challenge: to reduce data rates and performance requirements on system components without compromising data quality. At the same time, the systems should be space-saving and efficient. Embedded 3D cameras of the Ensenso XR series with integrated data processing are therefore the next logical step.

In machine vision applications with 3D cameras that work according to the principle of spatial vision (stereo vision), camera images are processed with high resolution and frame rates in order to make result data available to further processes as quickly as possible. The calculation of the three-dimensional data, so-called “point clouds”, from the image material of the stereo cameras requires several complex process steps, which were previously performed by powerful industrial PCs (IPCs). With increasing demands regarding the quality and speed of these result data, modern 3D stereo cameras, such as the Ensenso X series, use high-resolution 2D cameras with Gigabit Ethernet interfaces. However, the transmission of the 2D output data to the processing IPC requires optimally utilizing the network bandwidth in order to avoid time delays or data loss. In addition to that, the processing power of the IPC hardware must constantly increase in order not to restrict the overall system.

By using high-quality components, the performance of such 3D camera systems can be further increased. Thanks to exchangeable 2D cameras, the flexible design of the Ensenso X series is not tied to specific data interfaces and sensor resolutions and can continue to grow with the requirements for speed, object sizes, and quality. But high-resolution, fast GigE cameras, specially shielded cables, high-performance network technology, and powerful PC hardware are simply too expensive for some applications. In addition, sufficient space must be available for these peripherals.

With the new XR camera series, Ensenso is pursuing a different approach. According to the principle of “Internet of Things” (IoT), each individual component in a “distributed system” carries out a specific task and produces results that can be used directly by other systems. In the case of a 3D camera, these are three-dimensional coordinates of pixels of a real object.

Onboard 3D Processing

An SoC (System-on-Chip) integrated into the Ensenso XR projector unit allows the camera to execute the 3D processes itself, including stereo analysis. After correction of the lens distortion, the 2D output images are transferred to an axis-parallel stereo system by a virtual rotation of the cameras (rectification), which greatly simplifies all subsequent analyses. Then, the highly optimized matching algorithms for static or moving scenes search the recorded image pairs for corresponding pixels. Due to the different perspectives of the cameras, these pixels are also subject to different horizontal shifts in the image plane, known as “disparity”. Due to the geometric relationships in the parallel stereo system, this disparity represents a measure of the spatial depth of a 3D point in millimeters after the application of beam sets and knowledge of known system parameters such as focal lengths, pixel sizes and the base length of the stereo system.

These time- and computation-intensive pixel operations are performed in parallel by a supporting FPGA in the camera. This allows a 3D data rate to be realized which is comparable to that of an Ensenso X system that performs stereo analysis on a desktop PC with an Intel Core i7 Quad CPU.

“By combining integrated data processing with FlexView2 technology, accurate and fast acquisition of image details goes hand in hand,” explains Martin Hennemann, Product Manager and 3D image processing specialist at IDS

Benefit Of “Embedded” Functioning

In combination with FlexView2 technology, XR36 series models are able to process up to 16 fast sequential images for the 3D dataset of a static scene without the additional delay of transferring the raw data to the host PC. With the displacement of the projector pattern by FlexView2, different 3D points result in each image pair, which contributes to a very high-resolution 3D representation.

Due to the relocation of the computation-intensive processes into the camera, these no longer have to be carried out by powerful industrial PCs. In addition, the transfer of 3D result data instead of high-resolution 2D raw data reduces the network load. With fast, direct memory access between image acquisition and processing, this application for high-resolution 3D data offers enormous advantages in terms of result rate and bandwidth reduction compared to external processing on an industrial PC.

Multi-camera systems in particular benefit from the resource-saving features of the new XR series. If raw data from several high-resolution 2D cameras have to be transported over the network, bandwidth bottlenecks can quickly occur, resulting in frame rate drops that have a negative impact on overall performance. Here, the early evaluation and simultaneous data reduction of the XR series excels with reliable result rates, less computing power of the peripheral components and thus also fewer space requirements. A 3D application with Ensenso XR cameras is much easier to scale to the required requirements.

To further reduce the data rate, the camera only transmits the disparity map. The 16-bit 1-channel image is considerably smaller than a complete “point cloud”, which is a 32-bit RDB image with color overlay. The simple conversion can be carried out by the Ensenso SDK without much computing load on the industrial PC.

New Independence

The higher autonomy of the Ensenso XR stereo camera compared to other 3D cameras becomes an important selection criterion for 3D applications not only in terms of speed. Reduced performance requirements on network peripherals and IPC hardware simplify the entire construction of a 3D application and reduce costs, especially in multi-camera systems.

In addition to the wired Gigabit Ethernet connection, an additional WiFi interface allows temporary access to data and parameters during setup and maintenance, which is very useful where there is difficult or cost-intensive cabling. Furthermore, the new Ensenso XR projector unit has an integrated front light. When in use, it supports the calibration of the working environment or improves the image quality of the 2D camera images if the ambient light is insufficient or no external lighting is available.

The many improvements in the Ensenso XR series open up new fields of application for 3D camera technology. Integrated data processing is the next logical step in optimizing data exchange.

About IDS Imaging Development Systems GmbH:

The industrial camera manufacturer IDS Imaging Development Systems GmbH develops high-performance, easy-to-use USB, GigE and 3D cameras with a wide spectrum of sensors and variants. The almost unlimited range of applications covers multiple non-industrial and industrial sectors e.g. in the field of equipment, plant and mechanical engineering. In addition to its successful CMOS cameras, the company expands its portfolio with vision app-based sensors and cameras. The novel image processing platform IDS NXT is freely programmable and extremely versatile. Since its foundation in 1997 as a two-man company, IDS has developed into a globally leading, independent, ISO-certified family business with approximately 280 employees. The headquarters in Obersulm, Germany, is used for development as well as production. With subsidiaries in the USA, Japan, South Korea, and the UK as well as further offices, IDS is represented internationally.

For more information: