Real-Time AI Raw Sensor Fusion with IWR6243 mmWave Radar and Camera on NVIDIA's Holoscan Platform

Introduction

Modern robotics, including humanoid robots, autonomous mobile platforms, and industrial automation systems, must perceive their environments with high accuracy, low latency, and consistent reliability across a wide range of operating conditions. Current technology, such as vision sensors alone, often degrades in performance when exposed to fog, glare, dust, rain, or low-light environments, and single-modality depth sensors frequently struggle to deliver stable measurements in cluttered or dynamic scenes.

TI's mmWave radar technology provides a fundamentally different sensing modality that is intrinsically robust to environmental challenges. The IWR6243 mmWave radar sensor delivers precise range, velocity, and angle measurements independent of lighting conditions and with strong resilience to environmental interferences. When raw data is combined with camera-based perception and AI-driven sensor fusion, this capability enables robust and reliable perception of the surrounding environment with much higher fidelity than a radar-only or vision-only approach.

The IWR6243 is a highly integrated, single-chip 4RX and 3TX mmWave radar transceiver operating in the 57 to 64GHz frequency band. This level of integration significantly reduces system complexity, power consumption, and bill-of-materials cost, making it designed for scalable robotics and edge AI deployments. The device supports high-resolution point cloud generation and is designed for real-time sensing applications that require low latency and deterministic performance.

By deploying the IWR6243 alongside NVIDIA's Jetson Thor and the NVIDIA Holoscan Sensor Bridge, developers can seamlessly combine TI's proven mmWave radar sensing with NVIDIA's GPU-accelerated AI and streaming infrastructure. This pairing enables radar data to flow efficiently into AI sensor fusion pipelines while preserving the physical measurement accuracy and reliability that TI radar is known for.

In robotics applications, the IWR6243 enables precise short-range human detection and motion tracking for safe navigation in crowded environments such as warehouses. The direct range and velocity measurement capability of the radar allows the creation of a dynamic safety bubble around the robot, where protective zones can be defined and continuously adjusted based on object distance and relative speed. In humanoid robots, radar sensors further support stable balance control and dynamic obstacle avoidance during walking, running, or manipulation tasks.

System Overview

The system architecture is centered on a low-latency, GPU-centric data path that preserves sensor fidelity while enabling advanced AI processing. The design integrates TI's IWR6243 mmWave radar module, a camera module, NVIDIA's Holoscan Sensor Bridge for high-bandwidth and deterministic sensor data collection, and NVIDIA's Jetson Thor edge compute platform all running on the Holoscan SDK (software development kit) as shown in Figure 2.

The IWR6243 mmWave radar sensor provides raw measurement data through a high-speed CSI-2 interface, enabling direct, low-latency access to raw radar information with minimal transport overhead. This interface allows radar data to be streamed efficiently into the NVIDIA Holoscan Sensor Bridge, where it can be transferred directly into GPU memory using zero-copy mechanisms, all with the scalability and flexibility of Ethernet. This approach maintains the integrity of TI's high-precision radar measurements while minimizing latency and CPU involvement.

Once on the GPU, the Holoscan runtime orchestrates the processing pipeline using a graph-based execution model. Radar signal processing, camera preprocessing, neural network inference, and multimodal fusion are scheduled deterministically, verifying that real-time constraints are met even under high sensor data rates. This architecture allows developers to fully use TI mmWave radar's measurement accuracy while scaling AI workloads efficiently.

AI Sensor Fusion Pipeline

The sensor fusion pipeline begins with synchronized collection of radar and camera data. Radar processing leverages the IWR6243's ability to deliver raw data for range, velocity, and angular measurements, producing point clouds that capture both spatial structure and motion characteristics of the environment. Simultaneously, camera frames are processed using GPU-accelerated AI models that extract semantic information such as object identity, pose, and classification.

The fusion stage aligns TI radar measurements with visual detections in both space and time. Radar-derived depth and velocity information complements vision-based inference that improve robustness and reduce ambiguity. This is particularly valuable in scenarios involving fast-moving objects, partial occlusion, or visually degraded conditions where camera-only perception can fail.

By combining TI's accurate and lighting-independent radar sensing with AI-driven visual semantics, the fused perception output achieves higher confidence, improved tracking stability, and more reliable object classification. The resulting fused data can be used for real-time decision-making, navigation, safety monitoring, or downstream autonomy stacks.

Benefits of TI mmWave Radar in Sensor Fusion

TI's mmWave radar brings unique advantages to sensor fusion systems that are difficult to achieve with other sensing technologies. The ability to directly measure range and velocity enables precise motion understanding that enhances tracking and prediction. The robustness of mmWave sensing allows perception systems to operate reliably in environments where optical sensors degrade, including low light, glare, fog, and dust.

IWR6243's high level of integration reduces system complexity and power consumption while enabling scalable multi-sensor deployments. TI's long-standing expertise in automotive and industrial radar also provides a strong foundation for functional safety, reliability, and long-term availability, which are key critical requirements for robotics and autonomous systems.

When integrated into a Holoscan-based AI pipeline, the IWR6243 acts as the sensor front end that provides reliable sensor data for AI inference to real-world measurements. This combination improves overall system performance and reduces the risk of perception failure in edge deployments.

Getting Started with Hardware

Getting started with a radar-vision sensor fusion system based on Texas Instruments mmWave radar and NVIDIA Holoscan begins with assembling a hardware stack that supports high-bandwidth, low-latency data movement from sensors to a centralized AI compute platform. The reference hardware configuration combines TI's radar evaluation hardware with the NVIDIA Holoscan Sensor Bridge and NVIDIA Jetson Thor.

The IWR6243 cascade EVM serves as the radar sensing front end. This evaluation module integrates two IWR6243 mmWave radar devices along with power management, clocking, and connectivity required for rapid development and validation. The IWR6243 EVM allows developers to configure radar chirps and frame timing while exposing raw radar data suitable for advanced signal processing and AI-based fusion. Using the EVM accelerates bring-up and provides a known-good hardware baseline that reflects the performance characteristics of production radar designs.

To enable efficient data transfer from the IWR6243EVM into the Holoscan ecosystem, a CSI-2 adapter board is used between the radar evaluation module and the NVIDIA Holoscan Sensor Bridge as seen in Figure 4. This adapter board bridges the signals of the radar output interface to the CSI-2 input expected by the sensor bridge.

The Lattice CertusPro-NX Sensor to Ethernet Bridge Board running on low latency, low power, flexible FPGA acts as the central aggregation and transport device for high-bandwidth sensor data. It receives CSI-2 streams from the radar and additional interfaces from other sensors such as cameras. The sensor bridge is responsible for deterministic data movement and for delivering sensor data directly into GPU-accessible memory on the compute platform.

NVIDIA Jetson Thor serves as the centralized compute platform for the system. Jetson Thor provides the GPU performance and AI acceleration required for radar signal processing, camera inference, multimodal fusion, and real-time decision-making. Running the Holoscan SDK on NVIDIA Jetson Thor enables developers to define and execute streaming AI pipelines that process radar and vision data with predictable latency. The centralized compute approach allows the system to scale across different autonomy levels by increasing compute capability without redesigning sensor hardware.

Together, the IWR6243EVM, CSI-2 adapter board, NVIDIA Holoscan Sensor Bridge, a camera module and NVIDIA Jetson Thor form a complete hardware foundation for radar-vision sensor fusion. This configuration allows developers to focus on perception algorithms and AI fusion while relying on a hardware architecture that is optimized for high-bandwidth sensing, low latency, and scalability toward production humanoid and robotics systems. The IWR6243 cascade EVM hardware and the adapter board is available directly from Texas Instruments, and developers should contact their Texas Instruments sales representative for availability, ordering information, and support options.

Getting Started with Software

Getting started with the software stack for radar-vision sensor fusion on NVIDIA Holoscan Bridge involves setting up the Holoscan SDK, enabling the Holoscan Sensor Bridge software components, and deploying the application software that implements radar processing and sensor fusion.

The NVIDIA Holoscan SDK provides the core software framework for building real-time, streaming AI applications. It includes a graph-based execution model, GPU-accelerated operators, deterministic scheduling, and integrated support for sensor inference, visualization and data movement. The SDK runs on NVIDIA Jetson Thor and manages the execution of radar processing and camera inference.

The NVIDIA Holoscan Sensor Bridge software stack complements the SDK by enabling high-bandwidth sensors to stream data efficiently into the Holoscan runtime. It provides the necessary drivers, firmware, and interfaces to move sensor data from the Lattice-based sensor bridge hardware directly into GPU-accessible memory. This software layer abstracts sensor transport details and verifies deterministic timing, allowing application developers to focus on perception algorithms rather than low-level data movement. The tight integration between the Holoscan Sensor Bridge and the Holoscan SDK is critical for maintaining synchronization between radar and camera streams in real-time fusion applications.

On top of the NVIDIA Holoscan platform, the application software implements radar configuration, signal processing, sensor alignment, and fusion logic. In this system, the application software is developed by D3 Embedded and is designed specifically to leverage Texas Instruments mmWave radar, Lattice Semiconductor’s Holoscan Sensor Bridge board and NVIDIA Holoscan infrastructure. The software integrates IWR6243 radar data into a Holoscan-based pipeline, enabling radar point cloud generation, camera alignment, and AI-driven fusion within a centralized processing architecture.

Together, the NVIDIA Holoscan SDK, the Holoscan Sensor Bridge software stack, and the application software from D3 Embedded form a complete software design for real-time radar-vision sensor fusion. This combination enables rapid bring-up, scalable performance, and a clear path from evaluation hardware to production-ready humanoid and robotics systems.

Conclusion

Real-time AI sensor fusion between TI's IWR6243 mmWave radar and camera data on NVIDIA's Holoscan platform delivers a robust, scalable, and production-ready perception design. TI's mmWave radar technology provides accurate, lighting-independent measurement of range and velocity, while NVIDIA Holoscan enables ultra-low-latency GPU processing and AI-driven fusion. Together, these technologies allow developers to build advanced perception systems capable of operating reliably in complex, dynamic, and challenging environments, accelerating the deployment of next-generation robotics and autonomous machines.