ToF 3D image sensors are changing the way we engage with photography and mixed reality

This is the third article in a 6-part series exploring the technologies enabling the cutting-edge IoT.  

The emergence of video technology brought a revolution in audio-visual communication across the globe. Today, the world heavily relies on photo and video-based mediums for a range of purposes, including business, communication, education, and entertainment. Mechanisms of capturing and presenting video have also seen rapid advancement. The introduction of 3D cameras has allowed users to view things with a greater amount of information than ever before. 

An integral part of this process is the addition of depth information in digital videos which takes them one step closer to reality. This article looks at how Time of Flight (ToF) sensors are used for this purpose and are changing the way we can interact with VR, and other video technologies.

Introducing ToF 3D Image Sensors

Time of Flight or ToF is basically a method to measure the distance to an object based on the flight time of light. ToF sensors or cameras emit a modulated infra-red beam and measure the phase difference between the light received from different surfaces at various distances. This data is translated into distance and gives depth to the illuminated scene. ToF sensors use pixel-by-pixel processing which requires sophisticated algorithms, but its use is becoming increasingly popular in logistics, automotive, and consumer electronics due to its ubiquitous applicability. By scanning in three dimensions, ToF sensors can create 3D images of target objects.

ToF Sensor Working Principle

The operation of a 3D ToF Image sensor is quite straightforward. Detection is based on the principle of Photonic Mixer Devices (PMD). There are two general categories of ToF sensors: direct and indirect sensors. Direct ToF sensors work by emitting light, collecting the reflected signal, and then measuring the delay in the signal [Fig 1]. The time of flight for the light is then converted into distance measurement, which enables the 3D reconstruction of the scene. 

Fig 1. The simple mechanism of direct ToF sensor [Image by Author]

Indirect ToF sensors continuously emit modulated signals and analyse the reflected signals to determine phase shift. Two receiving wells are continuously opened and closed at the modulated light frequency with one receiver operating out of phase with its counterpart. Therefore, each receiver gets a portion of the reflected signal, and based on the amount of light received, the phase shift is determined, as shown in Fig 2. 

Fig 2. Detectors collect light alternately from the object of interest [Image by Author]

Benefits and Applications

ToF provides an optimized solution for 3D enhanced photography by increasing the range without sacrificing the image resolution and at the same time, delivering every digital detail in a low-cost, reliable, and compact package. Latest ToF sensors consume low power, thus making them an attractive choice for mobile devices. Additionally, ToF sensors are less dependent on precise mechanical alignment compared to technologies such as structured light emission. Newer ToF sensors allow the use of higher modulation frequencies which significantly increase the accuracy. 

According to Stratview Research, ToF is expected to see an annual growth of 16.8% from 2021 to 2026 ⁽¹⁾. The applications of ToF cameras are mainly focused on 3D imaging and scanning in both consumer electronics and the logistics industry. It has the potential to revolutionize 3D detection and imaging by providing rich 3D visuals at a low cost.  

ToF sensors can be deployed to aid areas such as facial recognition, security tracking, inventory management, and environmental and architectural mapping. In modern vehicles, ToF sensors enable mapping of the close environment, assist in parking, and many other functions in the cabin such as driver monitoring or gesture control. The most intriguing application, however, lies in consumer electronics and mixed reality. Due to its small form factor and power efficiency, ToF may be the only 3D imaging technology prepared to venture into consumer electronics.

Case Study: Enhancing the AR/MR experience

Incorporation of 3D ToF sensing in consumer electronics is an untapped market with game changing implications. According to Counterpoint Research, incorporation of ToF cameras in smartphones is expected to be a major trend of the future ⁽²⁾. In a recently held webinar, experts from TECNO, Samsung, and DXOMARK Image Labs claimed that features like ToF and Dynamic Vision Sensors (DVS) could enable mobile phone cameras to outperform DSLR cameras ⁽³⁾. Smartphones and gaming devices have utilized 3D ToF cameras in the past for highly realistic augmented/mixed-reality (AR/MR) applications. AR/MR applications blend the real world with virtual elements to give users an enhanced experience of business interactions, 3D design and gaming or entertainment. 

Although AR/MR has not become a mainstream application yet, however, with the rapid progression in the tools behind these technologies, it is a matter of time before it becomes synonymous with everyday smartphone use.

The two main elements required to initiate an MR experience are a smartphone camera and an application that can detect the user’s surroundings and show them on the screen. In this way, the real world can be mapped and virtual images can then be placed into it. To achieve this, the camera of the smartphone must be able to provide an in-depth and detailed image of the scanned area. 

ToF sensors become the main player in this scenario by accurately mapping objects and their surroundings. MR is a promising technological aid in 3D design, for instance, an interior designer can map out a room using their phone and place digitally created furnishings. Similarly, any real-world object can be 3D mapped using a ToF sensor camera and digitally manipulated. A number of such commercial mobile applications already exist but fail to give a realistic mapped result due to the device’s camera ⁽⁴⁾. 

ToF would significantly enhance depth information and in turn, mapped images opening endless possibilities for innovative and quick 3D design. Another application of mixed reality lies in business interactions, whereby professionals can show realistic site/project images for greater understanding. 

MR is also being used to create real-world business backdrops where people can be inserted into a person’s field of view, and a virtual meeting can become a real-life meeting. ToF lies at the heart of this operation as only the depth information provided by 3D imaging can take this experience to a level close to reality. The recently introduced metaverse concept by social networking giants will make use of this precise technology increasing the demand and innovation for ToF sensors.

Gaming is another area that is already utilizing the benefits of ToF in AR/MR domains. Gaming devices and now smartphones are using the gesture recognition capability of ToF sensors for total immersive experiences. When introduced into the front camera of mobile devices, ToF can greatly enhance security measures such as face recognition. With the use of 3D in the rear camera, ToF promises outstanding computational photography and an immersive VR experience. Without extensive post-processing, ToF sensors can effectively render background blur in pictures and videos. 

Ideally, mixed reality is much more advanced compared with virtual and augmented reality technologies. It sets comparatively stringent requirements on processing power and requires highly accurate acquisition of 3D depth data in real time. Moreover, lighting conditions present a major challenge in achieving the desired realism in MR. A robust ToF camera must be able to capture accurate depth data in all lighting conditions ranging from indoor environments like houses and vehicles to outdoors weather-based lighting differences. 

The use of MR in business interactions and day-to-day life may seem a thing of the far future to many. However, the world experienced an exponential increase in the demand for services that blur the boundaries between the virtual and real world in the era of the COVID-19 pandemic leading to an acceleration of non-face-to-face services. It can, therefore, be adequately expected that AR/MR will emerge as essential technologies for work and entertainment in the near future. Needless to say, technologies like ToF, which form the backbone of superior MR performance, will become all the more important.

ToF sensors from Infineon

Infineon introduced the REAL3^TM image sensor family in collaboration with pmdtechnologies, a German company specializing in ToF technology and leading in that area. The sensors use infrared light for Time of Flight sensing and deploy microlenses for each pixel resulting in virtually no loss and higher accuracy. Infineon’s sixth generation of REAL3™ 3D ToF sensors are especially focused on consumer applications, such as smartphone applications specifically AR/MR based photo and videography. The sensor consumes 40% less power than previous generations, which is key to preserving battery life in applications such as gaming that require the ToF camera to be on for long intervals. Keeping in view the strict area constraints of modern smart devices, the ToF sensor is designed to occupy a mere 4.4 × 4.8 mm² footprint, which is a 35% compared to the older generations.

Another distinguishing feature of the REAL3™ imager system is the enhanced sensing range of up to 10 m. A ToF sensor’s depth accuracy increases with higher modulation frequency, however, the sensor’s range suffers. The REAL3™ imager, therefore, utilizes two modulation frequencies and careful post-processing of the collected data to achieve a substantial range without sacrificing depth quality. The sensors are also equipped with pmdtechnologies’ SBI technology (Suppression of Background Illumination) which reduces pixel saturation allowing the camera to operate in various lighting conditions. The sensors can also be dynamically reconfigured via the I²C interface to adapt to new operating environments. With all these features consolidated into a single imaging system, seemingly impossible applications now appear to be a definite reality. Designers must keep a lookout for Infineon’s latest innovations for highly accurate 3D imagers.

Details of ToF Sensors from Infineon for consumer and industrial markets can be accessed here and for automotive from here. 

About the sponsor: Infineon Technologies

Infineon Technologies AG is a world leader in semiconductor solutions that make life easier, safer and greener. Microelectronics from Infineon are the key to a better future. With around 50,280 employees worldwide, Infineon generated revenue of about €11.1 billion in the 2021 fiscal year (ending 30 September) and is one of the ten largest semiconductor companies worldwide. To learn more click here.

References

1.    Stratview Research. Time-of-Flight (ToF) Sensor Market | Market Size, Share & Forecast Analysis | 2021-26 [Internet]. 2022 Jul [cited 2022 Jul 31]. Available from: https://www.stratviewresearch.com/1731/time-of-flight-(ToF)-sensor-market.html

2.    Counterpoint Research. Brighter, steadier, smarter: How smartphone cameras will improve in 2022 [Internet]. BBC Future. [cited 2022 Aug 20]. Available from: https://www.bbc.com/storyworks/future/looking-ahead-with-tecno/brighter-steadier-smarter-how-smartphone-cameras-will-improve-in-2022

3.    Counterpoint Research. Global Mobile Camera Trends 2022: Innovation Talk [Internet]. Counterpoint Research. 2021 [cited 2022 Aug 20]. Available from: https://www.counterpointresearch.com/global-mobile-camera-trends-2022-innovation-talk/

4.    Velichko Y. How AR Fills Your Room with Virtual Decor [Internet]. PostIndustria. 2021 [cited 2022 Jul 31]. Available from: https://postindustria.com/how-ar-fills-your-room-with-virtual-decor-top-home-virtual-furniture-placement-apps-ar-furniture/