Intelligent and Adaptive AR Experiences with TDK

Currently, the AR/VR industry is evolving from large, prototype devices to sleek, all-day wearable systems. This transformation is making immersive experiences increasingly defined by the intelligence and adaptability of the supporting software stack.

TDK has developed Direct Retinal Projection (DRP) technology to enable practical AR experiences, providing unmatched clarity, zero focal strain, and a compact, comfortable form factor suitable for all-day wear. In addition to image display, TDK is actively advancing research and development in areas such as multimodal sensor integration, real-time data fusion, and AI collaboration to support the new user experiences made possible by DRP.

Through these innovations, TDK is building a hardware-optimized AR software platform that interprets spatial context and seamlessly integrates multimodal sensor feedback in real time, paving the way for intelligent and adaptive AR experiences.

Software as the Missing Piece in AR

Direct Retinal Projection (DRP) method, every pixel is controlled by MEMS mirrors and modulated RGB laser light sources, and is directly drawn onto the user’s retina through laser raster scanning. Unlike conventional display devices such as waveguides, microdisplays, or projection systems—which project images onto lenses or glass surfaces—DRP precisely controls light to form images directly on the retina. Technically, the approach of sweeping a focused beam left-to-right (and line-by-line) to build an image bears a conceptual resemblance to the raster-scan technique used in old CRT televisions; both rely on a scanned beam to sequentially form the visible frame. 

However, DRP differs fundamentally in implementation and scale: it uses optical laser beams steered by high-speed MEMS mirrors with nanosecond-level intensity modulation rather than an electron beam on phosphor, and it achieves milliradian angular precision and sub-millisecond timing in a package small and power-efficient enough to be mounted on eyewear for all-day use.

This approach requires control mechanisms fundamentally different from those used in conventional flat-panel displays: the MEMS mirrors’ angular control and the laser sources’ intensity modulation must operate with sub-millisecond latency and angular precision on the order of milliradians.

MEMS mirrors’ angular control and laser sources’ intensity modulation must operate with sub-millisecond latency and milliradian-level angular precision. Even slight jitter, phase delay, or synchronization errors can destabilize the field of view, diminish immersion, or in the worst case, cause visual fatigue. Therefore, in DRP systems, rendering and sensing are inseparable: images must be drawn pixel by pixel, sweep by sweep, in high-precision synchronization with the wearer’s movements and physiological state. For this reason, a highly integrated system combining flexible software correction mechanisms and fast-response control systems is implemented, allowing for precise adjustment to the subtle individual characteristics and unique variations of MEMS mirrors, lasers, and optical paths.＠「

Furthermore, in order to generate AR visuals and provide users with a natural experience, it will likely become essential to integrate high-precision positional data and commands from IMUs (inertial measurement units), eye trackers, master controllers, and the cloud in real time. With information from these sensors and external data sources, the system is expected to instantly respond to the user’s head and gaze movements, as well as changes in the surrounding environment, ensuring that each pixel is rendered at the optimal timing and position. Through this complex coordination, the visuals will dynamically adapt to the user’s actions and environment, creating a highly immersive AR experience that seamlessly blends with the real world. 

Thus, realizing a retinal projection-based AR system using DRP will likely require not only advances in optics, mechanics, and electronic control, but also a sophisticated software platform capable of real-time integration with a wide range of sensors and external data. TDK is advancing technical preparations to realize these capabilities.

A Rendering Engine Tailored to the Retina

At the heart of this platform is TDK’s retinal-aware rendering engine, designed to operate at the high speed and precision required for DRP. TDK’s current AR glasses are already engineered to meet the demands of DRP, incorporating advanced control, computation, and adjustment capabilities.

The DRP renders virtual scenes onto a DRP “canvas: retina” that aligns with the angular sweep of the MEMS scanner. A laser control subsystem then adjusts beam intensity and phase, modulating each color channel accuracy to achieve the desired pixel luminance.

TDK is not resting on this achievement and is developing Active-PIC technology to further enhance performance. Active-PIC enables even higher-quality visuals and resolutions beyond 4K. Additionally, Active-PIC is directly integrated with the laser modulator. Conventional color modulation methods vary the drive current to the laser diode, limiting switching speeds to about 1 GHz and resolution to around 2K. In contrast, TDK’s Active-PIC uses electro-optical phase control, allowing RGB beams to be switched and multiplexed at up to 10 GHz, thereby achieving resolutions above 4K without increasing the form factor. The loop from input event to photon emission is typically completed in under 5 milliseconds, guaranteeing perceptual continuity and eliminating image latency.

TDK’s current AR glasses are already designed to operate at the high speed and precision required for DRP:

• They incorporate advanced control, computation, and adjustment capabilities.
• The software renders virtual scenes onto a DRP “canvas” that aligns with the angular sweep of the MEMS scanner. A laser control subsystem then adjusts beam intensity and phase, modulating each color channel accuracy to achieve the desired pixel luminance.
• However, TDK is not resting on this achievement and is developing Active-PIC technology to further enhance performance.

Retina-Specific Image Processing

Retinal projection offers a focus-free, immersive visual experience that is not affected by the user's eyesight. However, one of the technical challenges that has limited its widespread adoption is the restricted eye-box. In the DRP method, laser beams deliver images to the retina through the pupil, but if the beam misses the pupil, the image is blocked.

TDK addresses this by using optical processing and image recognition technologies to detect the position of the pupil and dynamically adjust the laser projection accordingly. Furthermore, by feeding data from the eye-tracking module back to the rendering core, the system enables foveated rendering, in which only the region around the user’s gaze is rendered in high resolution. This approach maintains visual fidelity while significantly reducing power consumption and computational load.

The platform also offers high flexibility through software, making it easy to accommodate future feature expansions and new algorithms. By controlling the entire DRP imaging pipeline—including MEMS mirrors and software controllers—the system achieves a close integration of physiological information and rendering.

• Retinal projection provides a focus-free and excellent visual experience that is not affected by eyesight.
• However, one technical challenge that has prevented its widespread use is the limited eye-box.
• In the DRP method, laser beams pass through the pupil to project images onto the retina. If the beam does not go through the pupil, it is blocked.
• TDK proposes a method to detect the pupil position using optical processing and image recognition, and to move the laser projection accordingly.
• In addition, data from the eye-tracking module is fed back to the rendering core to maintain visual fidelity.
• TDK achieves all of this while greatly reducing power consumption and computational load.

Realizing Natural Interaction through Sensor Fusion

TDK’s software platform is designed to support multimodal user input by integrating the following sensors and actuators:

• 6-axis IMU for head tracking
• ToF (Time-of-Flight) depth sensor for spatial recognition
• Eye tracking for focus-based control
• Microphone for voice commands and environmental recognition
• Cloud/local AI for advanced recognition and analysis
• Bone conduction speakers for audio output
• Haptic devices for tactile feedback

These data streams are aggregated by the contextual interaction engine, enabling natural UI experiences such as gaze-based overlays, gesture control, voice commands, and context-aware UI repositioning.

Whereas conventional UIs rely on touchscreen input or physical buttons, AR glasses present virtual UI elements in real space, allowing users to interact with floating buttons and menus. While intuitive gesture and gaze-based controls are often depicted in movies, the actual technology and user experience are still evolving.

To operate spatial UIs naturally, it is essential to fuse real-time data from multiple sources—such as hand and finger movement, gaze, and voice—to accurately interpret user intent. TDK’s platform combines sensor fusion with high-speed recognition and processing to deliver a next-generation user experience.

For example, when a user reaches for a virtual button, the ToF sensor detects hand depth, the IMU analyzes gesture trajectory, and gaze confirms intent. Once the button is activated, haptic devices and bone conduction speakers provide feedback faster than human reaction speed.

Real-Time Edge Processing and Cloud Collaboration

To minimize latency and enable standalone operation, TDK is developing a software stack that supports on-device inference for key modules. These functions are implemented in demonstration units or as subsystem components, and are practically usable in individual applications. Gesture recognition and pose estimation are accelerated by lightweight machine learning models running on edge processors within glasses and wearable devices, while heavy processing is offloaded to the cloud as needed.

Furthermore, high-precision recognition and inference are achieved through multimodal sensor fusion technology, which integrates data from multiple sensors such as cameras, IMUs, and microphones. Customizable AI models are also provided, allowing optimization and retraining according to specific use cases and devices, enabling flexible system construction. The platform emphasizes power-efficient design through edge AI and on-device processing, optimizing power consumption and extending battery life.

This hybrid architecture distributes and integrates the load and response time of machine learning processing, enabling real-time DRP rendering that matches human responsiveness—even on battery-constrained devices or during multi-user AR sessions.

For developers, APIs compatible with Unity, etc are provided, along with extensions for DRP-aware rendering and sensor calibration routines. In addition, a comprehensive toolchain and SDK—including simulation environments, debugging tools, and profiling features—are available to support efficient development. Developers can build applications using traditional tools while automatically benefiting from DRP-specific optimizations applied by the runtime.

The Future of Daily Life Enabled by AR Glasses

As AR glasses become widely adopted and the fusion of “A” for Augmentation and “R” for Reality transforms our daily lives, we can look forward to a future where our experiences become richer and more fulfilling than ever before. In this new era, AR will not only overlay information onto our field of vision, but will fundamentally change how we interact with our surroundings, with others, and even with ourselves.

The evolution of AR glasses brings new possibilities to our lives. In the future, all the technologies required for AR glasses—advanced sensors, optical components, environmental recognition, and user-optimized interfaces—will be realized through AR software supported by the cloud and AI. This AR software will not be limited to programmatic applications; it will also include content, systems, and even the transformation of lifestyles. Through AR software platforms that integrate the vast data processing capabilities of the cloud and the intelligent analysis of AI, digital information will seamlessly merge with the real world, enriching everyday experiences. For example, AR will enable people to visualize abstract concepts in real time, navigate complex environments with ease, and connect with others through immersive shared experiences, regardless of physical distance.

In fields such as education, healthcare, and industry, real-time information will be obtained through cloud-based AR software, and AI-driven analysis and remote expert support will be provided through intelligent content and systems. In education, AR will create interactive learning environments where students can explore historical events or scientific phenomena as if they were actually present. In healthcare, AR will assist doctors with real-time data overlays during surgery or patient care, improving accuracy and outcomes. In industry, workers will receive step-by-step guidance and safety alerts directly in their field of view, enhancing productivity and reducing errors.

Even at home, shopping, communication, and entertainment will become more intuitive and convenient thanks to the continuous support of AR software powered by the cloud and AI, which includes not only applications but also immersive content and lifestyle enhancements. For example, when shopping for furniture, you will be able to see how it fits in your room before making a purchase, or communicate with friends and family through lifelike avatars that appear in your living space. AR glasses will not simply be information devices—they will upgrade our lives by integrating with our senses and actions, all made possible by advanced AR software utilizing cloud and AI technologies.

With further advancement of AR glasses, society as a whole will be able to enjoy new experiences and value provided by AR software platforms that leverage the power of the cloud and AI across applications, content, systems, and daily life. These technological innovations will make daily life more comfortable, richer, and freer. There are high expectations for the future lifestyle brought about by comprehensive AR software, the cloud, and AI, as they enable us to transcend the boundaries between the digital and physical worlds, unlocking new forms of creativity, connection, and personal growth.

References

1. Envisioning the Future of AR Glasses Through TDK’s Augmented Reality Solutions _ TDK.pdf
2. 20250203 TDK AR VR.pdf
3. 20241030 English TDK_full-color laser control device.pdf
4. Ring_Controller_for_ARVR.pdf
5. Meta_opticsp_説明資料_v4.pptx