Environments where robots are being applied are becoming more complex and dynamix, which requires the development of more flexible and advanced vision and localization functions. An essential capability of robots operating in uncontrolled environments is identifying their surrounding. One solution to achieve this is the use of Time of Flight (ToF) imaging sensors with spot and flood illumination options. 

ToF sensors enable robotic applications to image their environment and create a 3D map of their surrounding. Letting the robot know what objects are close as well as gain detailed information about their size, shape and edges. For example, a robotic vacuum cleaner could use this information to understand whether it can drive underneath a sofa or cabinet and clean there as well. 

This article explores how ToF imaging sensors work, their applications and benefits for robotics.

Introducing ToF 3D Image Sensors and Working Principles

ToF is an active measurement, meaning that modulated light is emitted by a laser source next to the ToF imager, and the ToF sensor receives the light reflected from the objects in the scene. Infrared (IR) light, e.g at 940nm, is usually used in these applications since it is invisible to humans. ToF imaging sensors currently available on the market are based on two measurement principles: direct and indirect ToF.

Direct ToF measurement

For direct ToF measurements, the laser source emits short laser pulses, and the ToF sensor measures the delay between the emitted and the reflected laser pulse. The time delay between the pulse emission and the detection in the sensor (time of flight) is converted into a distance measurement to rebuild a 3D scene. The imaging sensors for direct ToF measurements is more complex as fast detectors (SPADs) are needed and on-chip histograms have to be calculated to enable the necessary peak finding correlating with object surfaces.

The simple mechanism of direct ToF sensor. Image credit: [1]

Indirect ToF measurement

For indirect ToF measurements, the laser source, usually vertical-cavity surface-emitting lasers (VCSEL), emits modulated infrared (IR) light with a wavelength between 850 and 940 nm. VCSEL is used over LEDs because they present better results when operating with modulation frequencies of 80 to 100 MHz and have a narrow optical spectrum2.  The ToF sensor captures the reflected signal through its lens and each pixel compares the emitted modulated signal with the captured from received from the scene. The fast global shutter ToF pixel collects the amount of light overlapping with the emitted signal and the amount arriving later. From this information the phase shift between both signals is then determined. For indirect ToF so called phase images are the output of the sensor. Depending on sensor and application 2, 3 and 4 phase-based approaches are used. The advantage of indirect ToF is that no on-chip processing is required.

Detectors collect light alternately from the object of interest. Image credit: [1]

ToF sensors output data

In both measurement methods, the measured data is converted on-chip into a digital representation, phase images for indirect ToF and histograms followed by peak finding for direct ToF. Then, the representation is transferred to an application processor (AP) as raw 3D data either via MIPI-CSI-2 or parallel (PIF) interfaces. By processing the raw data, the AP provides two images, a depth map and an amplitude image. The depth map contains the depth information for each pixel that is the distance at this pixel between the object and the camera. The amplitude image contains the amplitude information, representing the brightnessof the object at that position. Sophisticated systems can also provide additional images, such as a confidence image indicating each pixel's quality measures.

Time-of-Flight block diagram. Image credit: Infineon.

Depending on the application, a ToF sensor can be combined with an RGB image. In this case, algorithms can combine the RGB and depth images to obtain 3D representations of the scene and its objects containing both depth and color information. This allows advanced application functions such as object recognition to e.g. let the robot decide if the object is e.g. a toy that should not be touched by driving with a safety distance around it or a cabinet that should cleaned closeby. Or face authentication identifying persons based on their facial biometrics in a reliable and secure way.

Benefits of ToF Sensors

ToF sensors replace contact and proximity sensors, featuring a high accuracy that allows the detection of small objects thanks to the high number of pixels available. Another advantage of ToF sensors over contact sensors is the absence of mechanically moving parts, which reduces maintenance requirements. In particular for robotics existing LDS (laser distance scanning) sensors are bulky, expensive and have mechanically rotating mirror. ToF imaging sensors are much smaller, more cost effective and do not need mechanically moving parts.

ToF sensors can also be used in darkness and strong sunlight conditions due to their ambient light robustness. Furthermore, these sensors present a wide distance range, being capable of detecting objects up to 10m away. This feature enables their use for indoor and outdoor applications. Finally, the small camera dimensions enable integration into smaller robots such as consumer robot vacuum cleaners or drones.

ToF Sensors Role in Robotics Application

ToF sensors benefit different robotic applications, including:  

• Smart obstacle avoidance

• Navigation and path planning

• 3D mapping and localization (SLAM)

• Wall following

• Hand and finger gesture recognition

• Object or person tracking

• Object picking 

• Cliff detection

Each application has its own complexity and specific focus, and ToF sensors can cover them excellently.

Smart obstacle avoidance

Examples of smart obstacle-avoidance applications include vacuum cleaners, transport/logistic robots, autonomous mobile robots (AMRs), robot forklifts, and warehouse systems. By using ToF sensors, robots can precisely detect the distance to objects in their surroundings. Unlike other technologies, ToF sensors do not require specific training to detect particular obstacles and object borders can be easily extracted from the 3D ToF images. Furthermore, ToF systems with wide field-of-view cameras make it feasible to detect even small obstacles.

Obstacle segmentation using ToF sensor. Image credit: Infineon.

Localization and mapping for robots/drones (SLAM)

ToF sensors are utilized in drone or robot-based 3D mapping solutions to provide precise long-range depth information and high-resolution 3D maps due to their high frame rate. This allows the generating of a 3D point cloud, where each point represents a unique point on the surface ofan object with its own set of Cartesian coordinates.The point cloud is used by SLAM algorithms to generate a precise map of the robot’s environment and localize the robot itself in this map.

Human interaction robots

The depth and amplitude images generated based on ToF data can be used that robots know the human presence, identify persons and determine their location. Depending on its application, the robots can identify intruders or use the 3D point cloud data to understand and respond to human gestures, even detecting subtle movements of the fingers of a person. Moreover, in advanced applications, the robot could identify family members based on biometrics.

Other applications

The depth information provided by ToF measurements improves the success rate of robots used to manipulate, pick up and deliver objects. A ToF sensor provides more robust data than traditional sensors, thus also improving the so called cliff detection of robots avoiding them to fall off stairs or other edges.

ToF Imaging Sensors from Infineon

An attractive option in the market for ToF sensors is the REAL3™ image sensor family. It is developed by Infineon Technologies in collaboration with pmdtechnologies, a premium partner company from Infineon. These sensors use infrared light for ToF-based depth sensing and facilitate a sophisticated ToF pixel with SBI and microlenses for each pixel for a better sensitivity.. The sensors allow a fast readout via MIPI CSI-2. In addition, the power consumption is low, extending the operation time for battery driven devices.

The REAL3 sensors provide high image quality with up to VGA resolution and an enhanced sensing range of up to 10 m, making it possible to build 3D maps in real time. The sensor uses SBI (suppression of background light) technology to boost the ambient light robustness, being capable of working even in bright sunlight conditions. Thus, the features and parameter configuration allow the REAL3 to adapt fast to new operating environments.

Principles of work and usage of REAL3™ 3D image sensors. Image credit: Infineon.

The Infineon portfolio of ToF sensor products is designed to cater to diverse markets. The REAL3 sensor family is commercially available for consumer and automotive applications, with ASIL B/ISO26262 qualification. Each REAL3 model has different resolutions optimized for different applications, covering the requirements of several consumer, automotive, and industrial applications. To make it easier for you to start your project quickly, Infineon has an extensive list of partners specialized in software and hardware worldwide. 

Developers can also benefit from the solutions offered by the partnership between pmdtechnologies and Infineon. The flexx2 family is a development kit for prototyping, research, and mass production, featuring Infineon's latest ToF image sensor for precise depth-sensing. Flexx2 can be connected to Linux or Windows operating systems with USB 3.0. From the industrial perspective, the O3R platform is a starting point for developing robotics solutions, such as autonomous vehicles, for the industrial environment. It can support up to 6 3D ToF cameras with Gigabit Ethernet, USB 3.0 and CAN connections for other sensors.

Conclusion

Precisely detecting the surrounding environment and safely performing tasks have been crucial capabilities to be developed in robots. Sensors using ToF measurements are a very efficient solution for such applications. 

The REAL3™ image sensor family is an excellent example of high-quality ToF-based sensors. They are manufactured with cutting-edge technologies that improve detection capabilities for different conditions and applications. As the use of robots is growing, exploring and taking advantage of these sensors can benefit the development of future products.

References

1. Fatima Khalid, 2022. “ToF 3D image sensors are changing the way we engage with photography and mixed reality”

2. Rutronik, 2022. “VCSELs for ToF Applications - How VCSELs can be best put to use”

3. Tobias Rothermel, Walter Bell, 2022. “Building robots is hard! How to build a smarter robot faster with Time-of-Flight”.

4. Infineon, “ToF 3D image sensors for consumer and industrial”

5. Infineon, 2023. “New i-ToF imager enables smallest 3D camera systems with improved quantum efficiency at optimized cost”