As we witness the rise of automated driving, the driver's role is transforming, turning the car into a space for leisure, productivity, and connectivity. The shift from driver to passenger prompts a reimagining of interior concepts. 

Smart In-Cabin Monitoring Systems (ICMS) are required to enable this transformation, offering intuitive interaction, comfort, and new services while always ensuring and even increasing the safety of all occupants. 

In-Cabin Monitoring Systems

In-Cabin Monitoring Systems monitor and analyse the activities and behaviours of occupants within a vehicle's cabin. Their primary goal is to enhance safety, security, and convenience. Easy said, but a quite challenging and complex task that requires the implementation of multiple use-cases with different requirements. 

To enable the highest level of safety, the ICMS checks vital signs and detects driver drowsiness as well as readiness to take back from an autonomous driving mode, like a highway pilot. It monitors who is in the car, aiding in seat belt reminders, optimizing airbag deployment, and preventing children from being left behind. Security is increased by proximity and intrusion detection inside and near the car or by spoofing robust face-authentication to enable secure connectivity to private data and personalized services. Finally, to add convenience and to turn the car into a space for leisure, the ICMS has to make real-time body and finger tracking for intuitive interaction, gesture-controlled rear seat entertainment, and a personalized user experience. 

To fulfill such versatile tasks, the In-Cabin Monitoring System utilizes a combination of sensors, cameras, and artificial intelligence (AI) algorithms. This article focuses on two pivotal technologies that combine to enable cutting-edge ICMSs: radar sensors and Time-of-Flight (ToF) cameras.

Radar sensors

Radar (Radio Detection and Ranging) sensors emit radio waves and measure the time it takes for the waves to bounce off objects and return. This information is used to determine the object's distance, speed, and even direction. When integrated into the cabin environment, radar sensors enable a range of unobstructed in-cabin sensing (1). The ability to measure through most materials and the sensors’ small package size means they are able to be designed seamlessly into the car's design. Furthermore, the low power consumption even allows for operating a radar system in a parked car.

The radar working principle.

Time of Flight (ToF) Cameras

Time-of-Flight (ToF) cameras transmit modulated infrared light onto the scene of interest. The ToF imager captures the reflected light and measures amplitude and phase difference per pixel. The result is a 3D image of the entire scene consisting of the exact distance per pixel plus a gray-scale image. It operates in all lighting conditions: in darkness, strong sunlight or fast-changing lighting conditions. ToF provides the most robust and feature-rich data for vision-based algorithms and the generation of detailed 3D body models of passengers and objects.

Time-of-Flight working principle.

In-Cabin Sensing Use Cases

Radar and Time-of-Flight are, by principle, two completely different technologies. As described, radar sensors utilize radio waves, having the benefit of overcoming occlusions of the car seats, having ultra-low power consumption, and the ability to detect tiny movements like the chest movements of a breathing baby. On the other hand, you do not get any image data of your surroundings in order to detect facial expressions or gaze direction or to do detailed object classification. Meanwhile, ToF cameras provide such details together with the exact distance information while being limited to the field-of-view of the camera lens, the pixel resolution, and occlusions. Both technologies have unique merits and limitations, making them suitable for distinct applications. When having a closer look, they are complementary, and the combination of radar and ToF results in a more robust ICMS that gathers data from two distinct sources to cover all major in-cabin use cases. 

As discussed above, radar sensors and ToF can be combined to create ICMSs that have a broad range of features, utilizing the strengths of each technology resulting in more effective and meticulous monitoring. Below we look at some of the features of a cutting edge ICMS and highlight how each case scenario is delivered using radar and/or ToF sensors.

Child Presence Detection (CPD)

A particularly advantageous feature of an ICMS is the ability to monitor the occupancy of the car’s interior. Occupancy detection is especially useful to sense the presence of unattended children or pets in a vehicle. According to the US Department of Transportation, an average of 38 children under the age of 15 die each year from heat stroke after being left in a vehicle (2). To address the issue, the US Congress issued the Hot Car Act 2021 (3) and the Euro NCAP announced additional award points to vehicle manufacturers who implement the child presence detection (CPD) feature (4). 

A radar sensor is able to scan the area within the vehicle, including seats and footwell areas, and prompt the ICMS to send out a warning if any vital signs are detected while the car is parked and the temperature has reached a certain level. The operation of radars is unobstructed by seating and other non-metal materials, making them able to detect children and pets in back seats, infants placed in rear-facing seats, and even children covered by objects such as blankets. 

Seat Occupancy Detection and Smart Airbags

Seat Occupancy Detection (SOD) localizes the number of vehicle occupants and their specific  locations to optimize seatbelt reminder (SBR) systems and airbag deployment systems. The radar senses when a seat is unoccupied, and then the ICMS communicates to the vehicle’s airbag control module, preventing the airbag from deploying in case of an accident. 

While radar sensors detect occupancy, ToF cameras allow an ICMS to create 3D body models of the driver and passengers. This information can be used to make accurate passenger size and weight estimates and provides the exact positioning of occupants towards the airbag, which can effectively aid in optimized airbag deployment. In particular during autonomous driving modes, when drivers will be allowed to change their seat position, the ICMS can provide the necessary data to adapt airbags and restraints for maximum safety.

Infineon’s Time of Flight (ToF) sensors showing a depth map of in-cabin environment.

Another potential application of SOD lies in seatbelt alerts. In this case, radar can be used to check whether a seat is occupied and coordinate with the seat belt detection systems of the car to check if the passenger is buckled in or not. With the complementary ToF data, the ICMS could even see if the seatbelts are fastened correctly.

As described, having both data from radar and ToF sensors available increases the robustness of the use cases, and the kind of redundant data also supports meeting functional safety requirements for airbag applications.

Intrusion Alert

Radar sensing allows an ICMS to detect when an intruder tries to gain access to a vehicle. This is done by programming the ICMS to detect when an object such as a hand or weapon is inserted into the car through a window, sunroof, or the trunk, and in turn sound an audible alarm or notify the driver on a smart device. This allows the monitoring system to protect the car from theft and secure the interior contents using intrusion alert systems.

Proximity Alert

The internal environment of a vehicle is not the only area that can be monitored using radar sensing. Radars can also be used for constantly monitoring the close proximity of a vehicle and detect if another vehicle is too close or if a person is attempting to damage or break in. Based on this information, an alarm system can be triggered to flash the headlights or honk the horn to ward off potential intruders. Additionally, a notification can also be sent to the driver if they are not near the vehicle. 

Having the right radar system and algorithms, it is possible to enable all above mentioned use cases of CPD, SOD, SBR, and intrusion- and proximity alert with just one radar sensor per car.

Secure Face Authentication and Driver Monitoring

A unique feature of in-cabin ToF cameras is 3D face authentication – securing your vehicle using facial recognition. Incorporating depth-sensing technology, as seen in smartphones and payment terminals, delivers the highest quality of anti-spoofing security. This functionality enables the real connected car, where a driver has seamless access to private data or personalized services like music streaming or any payment functionality – whenever there is the need to authenticate. 

Once a 3D ToF-based driver monitoring camera is installed for 3D face-ID, the same camera can also be used for the basic driver monitoring system (DMS) functionality to ensure drivers are alert to the road: to monitor the driver’s head movement, eye closure and gaze direction and to flag any distraction or signs of drowsiness.

Furthermore, knowing the exact eye position by the 3D ToF data helps to optimize projection features like the optimal view onto a head-up display (HUD).

Personalized Comfort

Once the vehicle has authenticated who the driver is, the system is able to adjust the vehicle's interior to that driver’s personal preferences. A ToF camera recognizes the driver, and the ICMS can then adjust the seat position, climate control, and entertainment settings to create a personalized and comfortable cabin space. 

Enhanced Gesture Control

While gesture control has been around for a while, it has been limited to a very narrow zone near the driver. A ToF sensor with higher resolution enables a wider field of view with enough spatial resolution to properly detect and track fingers enabling passengers to adjust entertainment settings etc, without touching an interface. It even allows interaction with smart surfaces and projections like HUDs and holographic visualizations.

In particular for rear seat entertainment systems, gesture interaction can be very convenient and provides a kind of fun factor and the best entertainment content, like the possibility of gesture-controlled video games. Besides hand- and finger gestures, also body gestures can be used for a most convenient user experience. A ToF camera, for example detects if a passenger is reaching out to a certain area in the car and the interior light is illuminating just this area.

It should be mentioned that depending on the use case the right camera specification and placement have to be considered.

Combination of radar and ToF sensors enable the maximal amount of in-cabin monitoring use-cases.

Infineon’s In-Cabin Solutions

Infineon Technologies has paved the way for a transformative shift in automotive technology by introducing innovative products for in-cabin monitoring. Infineon’s automotive radar sensors and Time-of-Flight solutions are shaping the realm of in-cabin monitoring and detection systems, redefining safety, security, comfort, and user experience within vehicles. 

Infineon's XENSIV™ 60GHz radar transceiver, a powerful yet cost-effective innovation, sets the stage for advanced safety features (5). Operating in the 58 to 62 GHz frequency range, the BGT60ATR24 radar sensor family boasts two transmit, and four receive channels and a low-noise phase-locked loop. The radar is designed with an integrated anti-aliasing loop filter, finite state machine, and RX baseband with ADC ensuring high-performance data precision, which is crucial for safety features like occupancy detection. The BGT60ATR24AIP features Infineon’s antenna in package technology, enabling a very small footprint while reducing efforts at customer side to design antennas. The radar sensor is able to address numerous passive safety applications like left-behind child and pet detection, driver vital monitoring, and occupancy sensing due to its ability to sense micro motions.

To complement the benefits of the radar sensor, Infineon offers its automotive-grade REAL3™ ToF sensors IRS2877A(S) (6). IRS2877A offers a VGA system resolution of 640 x 480 pixels arranged in a way that minimizes crosstalk while maximizing performance and achieving an image circle of only 4mm for the smallest possible lens- and as such, module sizes. Additionally, each pixel is equipped with a patented Suppression of Background Illumination (SBI) circuitry enabling the ToF cameras to operate in all lighting conditions and exhibit sunlight robustness. 

All of this is packaged in a highly minimized plastic package with a footprint of only 9x9 mm, making it fit seamlessly into the interior of a vehicle. This combination of compact size and impressive resolution opens the door to innovative automotive applications, including secure face authentication, driver monitoring and wide-field-of-view in-cabin sensing cameras. IRS2877AS is the ISO 26262 (ASIL-B) compliant variant to address functional safety requirements like smart airbags, the world’s first of its kind.

The flexibility of both sensors allows for discrete integration within the cabin, ensuring their effectiveness without compromising aesthetics. Customer needs, operating conditions, and budgetary limitations determine the number and placement of sensors, optimizing coverage and enabling desired functionalities. Combining data from both sensors can achieve a robust and comprehensive In-Cabin Monitoring System.

Conclusion

The use of radar and ToF sensors in In-Cabin Monitoring Systems represents a significant leap forward in automotive safety and user experience. Integrating features like occupant detection and fine-grained monitoring, ICMSs powered by these sensors have the potential to transform driving into an intelligent, safer, and more personalized experience. As sensing technologies continue to evolve, we can anticipate exciting improvements in the precision, range, and overall capabilities of ToF and radar sensors, paving the way for even more sophisticated applications as we head towards full autonomy. 

References

1.    Soldatos J. Why Radar Sensors are Powerful Enablers for Intelligent IoT Applications [Internet]. [cited 2023 Sep 30]. Available from: https://www.wevolver.com/article/why-radar-sensors-are-powerful-enablers-for-intelligent-iot-applications

2.    Automotive News Europe [Internet]. 2022 [cited 2023 Oct 1]. How to make vehicles safer against hot car deaths. Available from: https://europe.autonews.com/guest-columnist/how-make-vehicles-safer-against-hot-car-deaths

3.    Hot Car Act 2021 [Internet]. 117th Congress (2021-2022) [cited 2023 Dec 15] https://www.congress.gov/bill/117th-congress/house-bill/3164

4.    Euro NCAP. Euro NCAP roadmap 2025 [Internet]. Leuven; 2017 Sep p. 19. Available from: https://cdn.euroncap.com/media/30700/euroncap-roadmap-2025-v4.pdf

5.    Infineon Technologies. 60GHz radar sensors for automotive | Interior sensing - Infineon Technologies [Internet]. [cited 2023 Oct 2]. Available from: https://www.infineon.com/cms/en/product/sensor/radar-sensors/radar-sensors-for-automotive/60ghz-radar/

6.    Infineon Technologies. ToF 3D image sensors for automotive - Infineon Technologies [Internet]. [cited 2023 Oct 2]. Available from: https://www.infineon.com/cms/en/product/sensor/tof-3d-image-sensors/tof-3d-image-sensors-for-automotive/