In this article, we will dive deep into the world of simultaneous localization and mapping using Lidar technology. Lidar SLAM has been gaining popularity in recent years, thanks to its versatility and applications across various domains, including autonomous vehicles, mobile robotics, and indoor mapping.

Lidar SLAM: The Ultimate Guide to Simultaneous Localization and Mapping

Lidar and Radar are two popular remote sensing technologies used for detecting and measuring the distance of objects. Lidar works by emitting a laser beam and measuring the time it takes for the reflected light to return to the sensor, while Radar uses radio waves to detect and locate objects. In this article, we’ll focus on the differences between Lidar and Radar. Lidar is more accurate and provides more detailed 3D mapping, while Radar is more versatile and can operate in a wider range of conditions. Both technologies have their own advantages and are used in a variety of applications.

Lidar vs. Radar: Comprehensive Comparison and Analysis

In this episode, we talk about how Toyota and Stanford are collaborating to utilize AI for drifting controls and the efforts behind a team at TUM which has resulted in an autonomous race car performing at the capacity of an amateur F1 driver.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/3793ff13-dbcd-4d01-bee8-121b58085bba?dark=false" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr>This podcast is sponsored by <a href="https://www2.mouser.com/" id="isPasted" rel="nofollow" target="_blank">Mouser Electronics</a>. &nbsp; &nbsp; &nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1662459182875-Mouser-sponsor-1-a.jpg" style="width: 766px;" class="fr-fic fr-dib"><hr><h3 id="isPasted">EPISODE NOTES</h3>(3:55) -&nbsp;<a href="https://www.wevolver.com/article/researchers-program-an-autonomous-vehicle-to-drift-around-obstacles-using-nonlinear-model-predictive-control-nmpc" target="_blank">Drifting Autonomous Vehicle</a>🚗(16:50) -&nbsp;<a href="https://www.wevolver.com/article/formula-1-could-see-driverless-race-cars-as-early-as-2025" target="_blank">Autonomous F1 Races By 2025</a>🏎Episode 86 was brought to you by Mouser Electronics, Farbod &amp; Daniel&rsquo;s favorite electronics distributor. Click <a href="https://resources.mouser.com/automotive/formula-e-racers-help-advance-ev-design" target="_blank">here</a> to read about the Mouser technical resource discussing Formula racing!<hr>As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte" target="_blank">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank">&nbsp;Apple&nbsp;</a>podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank">&nbsp;Spotify</a>, and most of your favorite podcast platforms!

In this episode, we talk about how Toyota and Stanford are collaborating to utilize AI for drifting controls and the efforts behind a team at TUM which has resulted in an autonomous race car performing at the capacity of an amateur F1 driver.

Podcast: The Autonomous Race & Drift Cars Of The Not So Distant Future

Erbium-doped fiber amplifiers (EDFAs) are devices that can provide gain to the optical signal power in optical fibers, often used in long-distance communication fiber optic cables and fiber-based lasers. Invented in the 1980s, EDFAs are arguably one of the most important inventions, and have profoundly impacted our information society enabling signals to be routed across the Atlantic and replacing electrical repeaters.What is interesting about erbium ions in optical communications is that they can amplify light in the 1.55 mm wavelength region, which is where silica-based optical fibers have the lowest transmission loss. The unique electronic intra-4-f shell structure of erbium &ndash; and rare-earth ions in general &ndash; enables long-lived excited states when doped inside host materials such as glass. This provides an ideal gain medium for simultaneous amplification of multiple information-carrying channels, with negligible cross-talk, high temperature stability and low noise figure.Optical amplification is also used in virtually all laser applications, from fiber sensing and frequency metrology, to industrial applications including laser-machining and LiDAR. Today, optical amplifiers based on rare-earth ions have become the workhorse for optical frequency combs (2005 Nobel Prize in Physics), which are used to create the world&rsquo;s most precise atomic clocks.Achieving light amplification with rare-earth ions in photonic integrated circuit can transform integrated photonics. Already in the 1990s, Bell Laboratories were looking into erbium-doped waveguide amplifiers (EDWAs), but ultimately abandoned them because their gain and output power could not match fiber-based amplifiers, while their fabrication doesn&rsquo;t work with contemporary photonic integration manufacturing techniques.Even with the recent rise of integrated photonics, renewed efforts on EDWAs have only been able to achieve less than 1 mW output power, which is not enough for many practical applications. The problem here has been high waveguide background loss, high cooperative upconversion &ndash; a gain-limiting factor at high erbium concentration, or the long-standing challenge in achieving meter-scale waveguide lengths in compact photonic chips.Now, researchers at EPFL, led by Professor Tobias J. Kippenberg, have built an EDWA based on silicon nitride (Si3N4) photonic integrated circuits of a length up to half meter on a millimeter-scale footprint, generating a record output power of more than 145 mW and providing a small-signal net gain above 30 dB, which translates to over 1000-fold amplification in the telecommunication band in continuous operation. This performance matches the commercial, high-end EDFAs, as well as state-of-the-art heterogeneously integrated III-V semiconductor amplifiers in silicon photonics.﻿&ldquo;We overcame the longstanding challenge by applying ion implantation &ndash; a wafer-scale process that benefits from very low cooperative upconversion even at a very high ion concentration &ndash; to the ultralow-loss silicon nitrideintegrated photonic circuits,&rdquo; says Dr Yang Liu, a researcher in Kippenberg&rsquo;s lab, and the study&rsquo;s lead scientist.&ldquo;This approach allows us to achieve low loss, high erbium concentration, and a large mode-ion overlap factor in compact waveguides with meter-scale lengths, which have previously remained unsolved for decades,&rdquo; says Zheru Qiu, a PhD student and co-author of the study.﻿&ldquo;Operating with high output power and high gain is not a mere academic achievement; in fact, it is crucial to the practical operation of any amplifier, as it implies that any input signals can reach the power levels that are sufficient for long-distance high-speed data transmission and shot-noise limited detection; it also signals that high-pulse-energy femtosecond-lasers on a chip can finally become possible using this approach,&rdquo; says Kippenberg.The breakthrough signals a renaissance of rare-earth ions as viable gain media in integrated photonics, as applications of EDWAs are virtually unlimited, from optical communications and LiDAR for autonomous driving, to quantum sensing and memories for large quantum networks. It is expected to trigger follow-up studies that cover even more rare-earth ions, offering optical gain from the visible up to the mid-infrared part of the spectrum and even higher output power.<hr>ReferencesYang Liu, Zheru Qiu, Xinru Ji, Anton Lukashchuk, Jijun He, Johann Riemensberger, Martin Hafermann, Rui Ning Wang, Junqiu Liu, Carsten Ronning, and Tobias J. Kippenberg. A photonic integrated circuit based erbium-doped amplifier. Science, 17 June 2022. DOI:&nbsp;<a href="https://dx.doi.org/10.1126/science.abo2631" rel="noopener" target="_blank">10.1126/science.abo2631</a>

An erbium-doped waveguide amplifier on a photonic integrated chip. Credit: EPFL Laboratory of Photonics and Quantum Measurements/Niels Ackerman

EPFL scientists have built a compact waveguide amplifier by successfully incorporating rare-earth ions into integrated photonic circuits. The device produces record output power compared to commercial fiber amplifiers, a first in the development of integrated photonics over the last decades.

Boosting light power revolutionizes communications and autopilot

Free space optical communication has attracted considerable attention from researchers for a variety of applications. Because of the complexity associated with the existing modulation techniques such as amplitude, phase, and frequency modulation, free-space optical communication systems use intensity modulation with direct detection [1,2]. Several applications such as wide-field lock-in detection for sensitive measurements, mode-locking in lasers, and phase-shift time-of-flight imaging (LiDAR) today require the intensity of the free-space signal to be modulated for a single frequency. However, specific applications typically require a particular frequency at which the signal should be modulated, but for many applications a higher frequency is desirable.&nbsp;Generally, these single-frequency free space intensity modulations should exhibit low optical insertion loss, high modulation efficiency, and a larger acceptance angle. Achieving all of these parameters at megahertz frequency can be challenging. For modulation at gigahertz frequency, the existing research work shows that Bragg cells-based approach has a narrow acceptance angle, electroabsorption modulators (using the quantum-confined Stark effect) have limited modulation efficiency, and Pockel cells (longitudinal and transverse) require bulky crystals. A team of researchers affiliated with the Stanford University, California and Chalmers University of Technology, Gothenburg have carried out work to design a new single-frequency free-space intensity modulator that can achieve a wide acceptance angle, a lower insertion loss, and improved modulation efficiency in the megahertz frequency range [3].&ldquo;Existing lidar systems are big and bulky, but someday, if you want lidar capabilities in millions of autonomous drones or in lightweight robotic vehicles, you&rsquo;re going to want them to be very small, very energy efficient, and offering high performance,&rdquo; explains Okan Atalar, a doctoral candidate in electrical engineering at Stanford and the first author on the new paper in the journal Nature Communications that introduces this compact, energy-efficient device that can be used for lidar.<h2 dir="ltr">Design of a single-frequency free-space intensity modulator</h2>In the paper titled, &ldquo;Longitudinal piezoelectric resonant photoelastic modulator for efficient intensity modulation at megahertz frequencies,&rdquo; the researchers have used a thin Lithium Niobate wafer coated with transparent surface electrodes to design a free space intensity modulator referred to as a &ldquo;longitudinal piezoelectric resonant photoelastic modulator. For intensity modular fabrication, the team used a 50.8 mm diameter, 0.5 mm thick double side polished Y-cut Lithium Niobate wafer. To deal with the trade-off between quality factors and acceptance angles, they choose to use an acoustic resonator. Lithium Niobate as the modulator material has suitable piezoelectric and photoelastic properties to construct an efficient intensity modulator with an acoustic resonator that is able to couple radio frequency into optical communication.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1650216661690-Experimental+characterization+of+the+modulator.png" style="width: 800px;" class="fr-fic fr-dib" alt="Experimental characterization of the modulator">Experimental characterization of the modulator [Image Credit: Research Paper]<div is-sapling-overlay="true" style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 478.75px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'> </div>&ldquo;Critically, lithium niobate is piezoelectric. That is when electricity is introduced through the electrodes, the crystal lattice at the heart of its atomic structure changes shape. It vibrates at very high, very predictable, and very controllable frequencies. And, when it vibrates, lithium niobate strongly modulates light &ndash; with the addition of a couple of polarizers, this new modulator effectively turns light on and off several million times a second,&rdquo; the team explains in the Stanford University blog post. &ldquo;What&rsquo;s more, the geometry of the wafers and the electrodes defines the frequency of light modulation, so we can fine-tune the frequency,&rdquo; Atalar says. &ldquo;Change the geometry and you change the frequency of modulation.&rdquo;In the experimental setup to demonstrate the intensity modulation, they coated the top and bottom surfaces of a double-sided polished LN wafer with tin oxide (ITO) to behave as transparent surface electrodes. To measure and verify the intensity modulation efficiency and that the acoustic mode profile matches the simulation, the coated wafer is placed between the optical polarizers along with a laser beam of 532nm wavelength is passed through the polarizer, wafer electrode region, and polarizer as shown in the figure.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1650216678886-ToF+imaging+using+the+modulator+and+a+standard+camera.png" style="width: 800px;" class="fr-fic fr-dib" alt="ToF imaging using the modulator and a standard camera">ToF imaging using the modulator and a standard camera [Image Credit: Research Paper]<div is-sapling-overlay="true" style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 478.75px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'> </div>The proposed design of the intensity modulator is used for a phase-shift-based ToF imaging of which the setup consists of a standard camera and a modulator. The targets in the scene at different distances reflect back to the receiver with different phases that are related to the distance, space of light in the air, and the intensity modulation frequency. A laser of 635nm wavelength is intensity-modulated at 3.733702 MHz and is used to illuminate the target. The proposed intensity-modulator is located between two optical polarizers. With the ToF imaging setup, the target distances are measured to centimeter-level accuracy.&nbsp;&ldquo;The impact for the proposed modulator is enormous; it has the potential to add the missing 3D dimension to any image sensor, they say. To prove it, the team built a prototype lidar system on a lab bench that used a commercially available digital camera as a receptor. The authors report that their prototype captured megapixel-resolution depth maps while requiring small amounts of power to operate the optical modulator.&rdquo;&nbsp;&ldquo;A future of small-scale lidar with standard image sensors &ndash; and 3D smartphone cameras &ndash; could become a reality,&rdquo; Atalar said noting the future works to reduce energy consumption by at least 10 times the threshold reported in the paper.&nbsp;<h2 dir="ltr">References</h2>[1] Jangjoo, A. and Faghihi, F., &ldquo;Intensity position modulation for free-space laser communication system&rdquo;, in Photonics North 2004: Optical Components and Devices, 2004, vol. 5577, pp. 825&ndash;833. doi:10.1117/12.567474.[2] Xiaoming Zhu and J. M. Kahn, &quot;Free-space optical communication through atmospheric turbulence channels,&quot; in IEEE Transactions on Communications, vol. 50, no. 8, pp. 1293-1300, Aug. 2002, doi: 10.1109/TCOMM.2002.800829.[3] Atalar, O., Van Laer, R., Safavi-Naeini, A.H. et al. Longitudinal piezoelectric resonant photoelastic modulator for efficient intensity modulation at megahertz frequencies. Nat Commun 13, 1526 (2022). https://doi.org/10.1038/s41467-022-29204-9

Lab-based prototype LiDAR system that captured megapixel-resolution depth maps using a commercially available digital camera

A new approach that allows standard image sensors to see light in three dimensions using a novel design for a single-frequency free space intensity modulator.

Acoustic modulator using a thin wafer of Lithium Niobate for free-space optics

As the foundation for reliable decision-making, the perception algorithm has been the backbone for the development of autonomous vehicles. To enable this, autonomous vehicles feed sensor data to perception algorithms that understand the environment wherein sensor fusion with multi-frame tracking is gaining momentum for the detection of 3D objects. The camera-LiDAR fusion has shown significant performance upgrades on workloads associated with a 3D vision&ndash; LiDAR provides an accurate 3D geometry structure, while the camera captures more scene context and semantic information [1]. The fusion of these two sensors has become the fundamental idea to achieve better performance but it is necessary to analyze the attacks on these systems.&nbsp;The security analysis of the perception focused on the image domain with LiDAR-only that introduced spoofing attacks against LiDAR. But a team of researchers identified limitations in the existing security analyses of LiDAR-based perception models that become complex with multi-sensor perception and multi-frame tracking architecture. These approaches required complex models to be deployed with white-box optimizations, restricting the research to no analysis of the black box. Researchers from Duke University and the University of Michigan carried out research to perform an analysis of the camera-LiDAR fusion system under black-box LiDAR spoofing attacks [2]. Additionally, the team defines a novel, context-aware attack, frustum attack, that shows a significant vulnerability to the widely used architecture of LiDAR-only and camera-LiDAR fusion systems.&nbsp;&ldquo;Our goal is to understand the limitations of existing systems so that we can protect against attacks,&rdquo; said Miroslav Pajic, the Dickinson Family Associate Professor of Electrical and Computer Engineering at Duke. &ldquo;This research shows how adding just a few data points in the 3D point cloud ahead or behind of where an object actually, is can confuse these systems into making dangerous decisions.&rdquo;<h2 dir="ltr">Attacks on camera-LiDAR fusion systems</h2>In the paper, &ldquo;Security Analysis of Camera-LiDAR Fusion Against Black-Box Attacks on Autonomous Vehicles&rdquo; the team also introduced a new class of perception attacks for which the attacker only needs to know the location of the true object in the environment. They define five scenarios where these attacks can be used to launch spoofing attacks relative to existing objects in the scene. This work is an extension of the work carried out by Y. Cao, C. Xiao, et. al [3], and J. Sun, Y. Cao, et. al [4] that focused on isolated placement in the 5-8 meter range.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1648675136746-Demonstration+of+Frustum+Attack.png" style="width: 800px;" class="fr-fic fr-dib" alt="Demonstration of Frustum Attack">The attacker launches a malicious frustum attack against a victim AV using a target car [Left]; Physical experiment demonstrates that an attacker can stably spoof longitudinally consistent points in the frustum of a target vehicle. [Right] &nbsp;<div is-sapling-overlay="true" style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 478.75px; height: 63px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'> </div>For consideration that aligns with the existing work, the team defines the attack goals as a false positive outcome, a false-negative outcome as well as translation attack outcome where the detected object&rsquo;s bounding box is translated by some distance. As the team explains in the blogpost on the Duke University website, &ldquo;the new attack strategy works by shooting a laser gun into a car&rsquo;s LIDAR sensor to add false data points to its perception.&rdquo; If these data points are out of range for what car&rsquo;s vision system, then the new research work shows that 3D LiDAR data points placed within an area of the camera&rsquo;s 2D field of view (FOV) can fool the system to produce errors.The above-defined security vulnerability in the camera system is seen as a 3D pyramid with the tip sliced off. The area covered in front of the camera&rsquo;s lens is in the shape of a frustum from where the name comes. Moreover, in the case of the camera placed on top of the camera facing the front, there will be a few data points placed in front of or behind another nearby car that can shift the entire system&rsquo;s perception by several meters&ldquo;This so-called frustum attack can fool adaptive cruise control into thinking a vehicle is slowing down or speeding up,&rdquo; Pajic said. &ldquo;And by the time the system can figure out there&rsquo;s an issue, there will be no way to avoid hitting the car without aggressive maneuvers that could create even more problems.&rdquo;<h2 dir="ltr">Evaluation of the novel&ndash; frustum attack</h2>Researchers evaluated the frustum attack against the state-of-the-art defenses proposed in the existing research work on LiDAR spoofing using various perception algorithms within three distinct LiDAR-only and three distinct camera-LiDAR fusion architectures. These architectures include cascaded-semantic, feature-level, and tracking levels on more than 75 million attack scenarios, making it a rigorous evaluation methodology. The team claims this analysis to be the largest of all on LiDAR spoofing that extensively evaluates multiple architectures of multi-sensor fusion for perception.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1648675246411-Evaluation+of+Frustum+Attack.png" style="width: 800px;" class="fr-fic fr-dib" alt="Evaluation of Frustum Attack">Frustum attack was evaluated on an autonomous vehicle running Baidu Apollo software using perception data from LGSVL simulator. [Image Source: Research Paper]&nbsp;<div is-sapling-overlay="true" style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 478.75px; height: 63px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'> </div>To evaluate the impact of LiDAR attacks on autonomous vehicles equipped with multi-frame tracking, the research work proposed a frustum attack case study using longitudinal sequences of perception data. Start by analyzing the multi-frame fusion and tracking data using representative algorithms and then testing the frustum attack on Baidu Apollo using the LGSVL simulator. &ldquo;The case studies illuminate the high-impact adversarial situations that endanger vehicle and passenger safety that occur under the frustum attack when attacking over multiple time points, effectively deceiving the host vehicle&rsquo;s tracking and control,&rdquo; the researchers note.The results of the paper will be presented on August 10-12 at the 2022 USENIX Security Symposium. More details on the thorough analysis of LiDAR-only and camera-LiDAR perceptions along with the frustum attack case study are available in the research work published on Cornell University&rsquo;s research sharing platform, arXiv under <a href="https://arxiv.org/abs/2106.07098" target="_blank">open access terms</a>.<h2 dir="ltr">References</h2>[1] &nbsp;H. Zhong, H. Wang, Z. Wu, C. Zhang, Y. Zheng, and T. Tang, &ldquo;A survey of LiDAR and camera fusion enhancement,&rdquo; Procedia Computer Science, vol. 183, pp. 579&ndash;588, Jan. 2021, doi:<a href="https://doi.org/10.1016/j.procs.2021.02.100" target="_blank">&nbsp;10.1016/j.procs.2021.02.100</a>.[2] R. Spencer Hallyburton, Yupei Liu, Yulong Cao, Z. Morley Mao, Miroslav Pajic: Security Analysis of Camera-LiDAR Fusion Against Black-Box Attacks on Autonomous Vehicles. DOI arXiv: <a href="https://arxiv.org/abs/2106.07098" target="_blank">2106.07098 [cs.CR]</a>.

The novel class frustum attack leverages that the camera is only a 2D projection of 3D space [Image Source: Research Paper]

A novel class of LiDAR spoofing attacks on autonomous vehicles– the frustum attack, was then later validated using an existing hardware setup.


Security analysis of camera-LiDAR perception for autonomous vehicle system

The key to safe, reliable autonomous vehicles — above even how smart its on-board self-driving system is at its heart — is likely to be found in how efficiently it can process sensor data. Just as with visual acuity tests for human drivers, it’s vital to know that an autonomous vehicle system can spot hazards and react accordingly — no matter how small or distant the problem may be.Traditional two-dimensional camera systems and three-dimensional sensors, like LiDAR (Light Detection and Ranging), may not go far enough for full reliability and safety, leading a team at Google and Alphabet’s autonomous vehicle subsidiary Waymo to look into the fourth dimension: 4D-Net, an approach to object detection which fuses two- and three-dimensional data with the fourth dimension, time, with the claim of significantly improved performance.<h1>Time enough</h1>“This is the first attempt to effectively combine both types of sensors, 3D LiDAR point clouds and onboard camera RGB images, when both are in time,” claim Google Research scientists and paper co-authors AJ Piergiovanni and Anelia Angelova in a joint statement on the work. “We also introduce a dynamic connection learning method, which incorporates 4D information from a scene by performing connection learning across both feature representations.”The 4D-Net approach stems from a simple observation: The majority of modern sensor-equipped vehicles include two- and three-dimensional sensors, typically in the form of multiple camera modules and LiDAR, data from which is gathered across a period of time — but very few efforts have been made to gather everything in a single place and process it as a whole.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1647530490320-4dnet2.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A diagram showing how 4D-Net operates. A block labelled &quot;RGB video feature maps:&quot; takes multiple maps from camera images captured over time and sends them to a connection search syste for projection. A separate box shows point clouds captured over time being sorted into stack pillars, learned features, then a pseudo image processed vi a backbone point cloud network for merging with the projected images and the output of recognised 3D bounding boxes.">The 4D-Net system aims to boost object recognition accuracy at distance, by blending two-dimensional camera imagery with 3D point cloud data — all gathered over to capture motion.4D-Net address this gap, blending three-dimensional point-cloud data with visible-light camera imagery while blending in a time element by processing a series of each captured over a set period. The secret to its success: A novel learning technique which can autonomously find and build connections between the data, fusing it at different levels dynamically in order to boost performance over any of the data feeds alone.“Images in time are […] highly informative, and complementary to both a still image and PCiT [Point Cloud in Time],” the researchers explain of the benefit to the approach. “In fact, for challenging detection cases, motion can be a very powerful clue. While motion can be captured in 3D, a purely PC [Point Cloud]-based method might miss such signals simply because of the sensing sparsity” — the same issue, incidentally, which means distant or small objects can be missed by a LiDAR sensor but picked up on visible-light camera systems or the driver’s naked eye.<h1>Machine learning through time</h1>To handle both types of data, the team turns to a range of pre-processing steps. 3D point cloud data is run through PointPillars, a system for converting the data into a pseudo-image which can be further processed using a convolutional neural network (CNN) designed for two-dimensional data, with a time indicator added to each point to create a denser representation including motion. Conversion into fixed-sized representations, effectively sub-sampling the point cloud, is also used — an approach which densifies the point cloud where data is sparse and sparsifies it where data is dense, thus boosting performance at long ranges.The two-dimensional camera data, meanwhile, is processed into feature maps through Tiny Video Networks, then the data projected to align the 3D points with their corresponding points on the 2D imagery — a process which assumes “calibrated and synchronized sensors.” For point cloud data which lies outside the view of the vehicle’s cameras, a vector of zeroes is applied.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1647530560135-4dnet4.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A diagram showing a multi-feed variant of 4D-Net, in which high-resolution image, medium-resolution image, and video data are linked to the point cloud backbone for generation of 3D boxes.">A variant of the 4D-Net system using multiple resolutions of image and video feed proved ideal, offering additional accuracy gains over the single-feed variant in benchmark testing.The truly smart part of the 4D-Net system, though, comes in the form of its connection architecture search — the means by which it is able to extract the most, and most suitable, information from the fused data. A one-shot lightweight differentiable architecture search finds related information in both 3D and in time and connects it across the two different sensing modalities — and learns the combination of feature representations at various abstraction levels for both sensors.“[This] is very powerful,” the team explains, “as it allows to learn the relations between different levels of feature abstraction and different sources of features.” To further tweak the approach for autonomous vehicles, the team modified the connections to be dynamic, based on the concept of self-attention mechanisms, allowing the network to dynamically select particular visible-light data blocks for information extraction — meaning it can learn how and where to select features based on variable input.<h1>Impressive results</h1>Testing both a single-stream and a multi-stream variant of the system, the latter pulling in additional input streams in the form of still-image and video feeds running at different resolutions, the team claims some impressive gains over rival state-of-the-art approaches.Tested against the Waymo Open Dataset, 4D-Net improved on average precision (AP) across all tested rival approaches. While its performance proved, on average, weaker at shorter distances, its ability to recognize more distant objects — particularly the 50-meter-plus range — is reportedly unsurpassed, particularly when running in multi-stream mode.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1647530549740-4dnet3.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A chart showing the performance of 4D-Net compared to rival approaches StarNet, LaserNet, PointPillars, MVF, Huang et al, PilalrMultiView, and PVRCNN. Overall performance favours 4D-Net; accuracy is lower at 30m detection distances, but noticeably higher at 30-50m and higher still at 50m and more. Runtime, meanwhile, is above LaserNet and PillarMultiView but below PVRCNN.">The team's experimentation shows a significant accuracy gain for 4D-Net over rival approaches at mid- to long-range, though a drop in accuracy for shorter detection distances.“We demonstrate improved state-of-the-art performance and competitive inference run-times,” the team concludes, “despite using 4D sensing and both modalities in time. Without loss of generality, the same approach can be extended to other streams of RGB images, e.g., the side cameras providing critical information for highly occluded objects, or to diverse learnable feature representations for PC [Point Cloud] or images, or to other sensors.”The researchers suggest that the 4D-Net approach can also be used outside the field of autonomous driving, wherever there’ a need to capture different aspects of the same domain by aligning audio, video, text, and image data automatically.The team’s work was presented at the International Conference on Computer Vision (ICCV) 2021, and has been <a href="https://openaccess.thecvf.com/content/ICCV2021/papers/Piergiovanni_4D-Net_for_Learned_Multi-Modal_Alignment_ICCV_2021_paper.pdf" target="_blank">made available under open-access terms</a>. A supporting write-up by AJ Piergiovanni and Anelia Angelova is available <a href="https://ai.googleblog.com/2022/02/4d-net-learning-multi-modal-alignment.html" target="_blank">on the Google AI Blog</a>. The researchers have pledged to make their code available under an open-source license, but it had not been published at the time of writing.<h1>Reference</h1>AJ Piergiovanni, Vincent Casser, Michael S. Ryoo, and Anelia Angelova: <cite>4D-Net for Learned Multi-Modal Alignment, IEEE/CVF ICCV '21</cite>. DOI <a href="https://doi.org/10.1109/ICCV48922.2021.01515" target="_blank">10.1109/ICCV48922.2021.01515</a>.

Designed to find links between 2D imagery and 3D point cloud data captured over time, 4D-Net offers a big boost in long-range object detection.

Designed to address the problem of accurately detecting objects — like other vehicles and pedestrians — at distance, Google and Waymo's 4D-Net offers a novel and generalizable approach to sensor fusion with some impressive results.

4D-Net boosts autonomous driving vision capabilities by fusing point cloud, camera, and time data

This article was discussed in our <a data-saferedirecturl="https://www.google.com/url?q=https://www.wevolver.com/profile/the.next.byte&source=gmail&ust=1662506724055000&usg=AOvVaw3ePqI00MDVgWOHTO-3eNvi" href="https://www.wevolver.com/profile/the.next.byte" id="isPasted" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/3793ff13-dbcd-4d01-bee8-121b58085bba?dark=false" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe>The full article will continue below.<hr><header id="isPasted">A team at the Technical University of Munich (TUM) has developed software which lets race cars compete in motor sports without a driver. The TUM Autonomous Motorsport Team was able to take 1st and 2nd place at the Autonomous Challenges in Indianapolis and most recently at the CES in Las Vegas. Does this technology have the potential to revolutionize racing? Markus Lienkamp, Professor for automotive technology, tells us the answer.</header><section><h2>Prof. Lienkamp, what gave you the idea of entering your team in the two Autonomous Challenges in the USA?</h2>The story began back in 2005, as information scientist Sebastian Thrun from Stanford University won the DARPA Grand Challenge, which was until then highly renowned as the only race for autonomous vehicles, held in the Nevada desert. At the time I was working at Volkswagen and built the car for him. Then came the 2007 Urban Challenge in California, a race for autonomous cars on a paved course under realistic traffic conditions, where once again three Volkswagen vehicles from various universities made the finals. Sebastian Thrun then had the idea of staging speed races similar to Formula 1 on a real race track: That was the birth of the <a href="https://www.indyautonomouschallenge.com/" rel="noreferrer" target="_blank">Autonomous Challenge</a> concept. And it went without saying that I wanted to participate together with a TUM team.<h2>The autonomous vehicles which are currently being tested for street use already contain an enormous amount of AI. What&#39;s the difference between the software you use in the race cars and the systems already in use?</h2>The race track doesn&#39;t have traffic laws or points of reference like marked lanes, traffic lights or traffic signs. Ultimately we&#39;re dealing with &quot;unpredictable&quot; objects, in our case the other race cars. We have to detect them and make predictions about where they&#39;ll move, and that all at speeds of over 250 km/h. We&#39;re able to do this because our software doesn&#39;t concentrate on strictly complying with traffic regulations, as is the case with other vendors. Instead, our software calculates the probable locations of the other objects which it then uses to calculate the optimum solution for our own movements.<h2>How exactly does your software do that?</h2>Our approach uses conventional sensor technologies with lasers, cameras and radar. The software knows the race course and detects the other vehicles. That means it can predict the most probable trajectory, i.e. what location on the course the other vehicle will move to at what point in time, and can then plan its own movements accordingly. Here the computing speed of the software plays an important role. That&#39;s decisive when it comes to safely executing aggressive maneuvers and reacting spontaneously to critical situations.The fact that we made the software generally available as open source code and thus let other teams work on it was enormously helpful as well. That speeds up the development process so that soon the number of people around the world who will be able to work on our code will surpass the development capacities of all the automobile manufacturers put together. Ultimately, though, the software and the car are only as good as the teams behind them. I have an extraordinarily good team whose members worked very hard and well beyond the call of their normal everyday duties to make this success happen. 15 doctoral candidates moved this entire project ahead over the course of four years, and four team leaders made sure the teams succeeded in the individual phases.<h2>Formula 1 is the best-known auto racing format in the world. Do you think it&#39;s possible that Formula 1 will be completely driverless sometime in the future?</h2>That&#39;s both a technical and emotional discussion which affects the spectators more than anything else. Fans always like to bond with particular driver personalities and identify with them in competition. When I see the constantly growing enthusiasm for E-Sports, I think at some point we&#39;ll see both. In technical terms of course we always have to ask when it will be realistic to actually replace the driver. After our experience in Indianapolis, we believe this could be possible by 2025. We can even imagine sending one of our autonomous cars to a real Formula 1 race to compete against human professionals.<h2>How high do you think the chances are that an autonomously driven car could beat a human driver?</h2>In its present form our software has already reached the level of an amateur driver. When we talk about professional-class races, experience shows we&#39;re about half a second behind. So it will probably take a couple more years before our autonomous vehicles are able to beat racing pros. You can compare it to a chess game against the computer: In the beginning the AI was only able to beat hobby players. It took quite a while before it was possible to beat the chess world champion. But it&#39;s definitely possible.<h2>After these great successes in Indianapolis and Las Vegas &ndash; What comes next?</h2>The competition will pretty much continue to CES next year. We&#39;d like to compete once again. We want to put what we&#39;ve learned in the races back on the street &ndash; for example with an autonomous shuttle during the Oktoberfest. And we&#39;ve founded our own software company &ldquo;driveblocks&rdquo; which is industrializing the whole thing and making it ready for series production. That means our expertise will soon be for sale.</section>

Prof. Markus Lienkamp believes in the future of autonomous motorsports. Image: Jacob Kepler

A team at the Technical University of Munich (TUM) has developed software which lets race cars compete in motor sports without a driver. The TUM Autonomous Motorsport Team was able to take 1st and 2nd place at the Autonomous Challenges in Indianapolis and most recently at the CES in Las Vegas.

"Formula 1 could see driverless race cars as early as 2025"

The technology, developed by researchers from the University of Cambridge, the University of Oxford and University College London (UCL), is based on LiDAR (light detection and ranging), and uses LiDAR data to create ultra-high-definition holographic representations of road objects which are beamed directly to the driver’s eyes, instead of 2D windscreen projections used in most head-up displays.While the technology has not yet been tested in a car, early tests, based on data collected from a busy street in central London, showed that the holographic images appear in the driver’s field of view according to their actual position, creating an augmented reality. This could be particularly useful where objects such as road signs are hidden by large trees or trucks, for example, allowing the driver to ‘see through’ visual obstructions. The <a href="https://www.osapublishing.org/oe/fulltext.cfm?uri=oe-29-9-13681&id=450306" target="_blank">results</a> are reported in the journal Optics Express.<iframe src="https://www.youtube.com/embed/pa7o2Gd4ITU?&amp;wmode=opaque&amp;rel=0" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 786px; height: 442px;"></iframe>“Head-up displays are being incorporated into connected vehicles, and usually project information such as speed or fuel levels directly onto the windscreen in front of the driver, who must keep their eyes on the road,” said lead author <a href="http://www.eng.cam.ac.uk/profiles/js2459" target="_blank">Jana Skirnewskaja</a>, a PhD candidate from Cambridge’s Department of Engineering. “However, we wanted to go a step further by representing real objects as panoramic 3D projections.”Skirnewskaja and her colleagues based their system on LiDAR, a remote sensing method that works by sending out a laser pulse to measure the distance between the scanner and an object. LiDAR is commonly used in agriculture, archaeology and geography, but it is also being trialled in autonomous vehicles for obstacle detection.<iframe src="https://www.youtube.com/embed/9kgweRItSFU?&amp;wmode=opaque&amp;rel=0" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 787px; height: 466px;"></iframe>Using LiDAR, the researchers scanned Malet Street, a busy street on the UCL campus in central London. Co-author Phil Wilkes, a geographer who normally uses LiDAR to scan tropical forests, scanned the whole street using a technique called terrestrial laser scanning. Millions of pulses were sent out from multiple positions along Malet Street. The LiDAR data was then combined with point cloud data, building up a 3D model.“This way, we can stitch the scans together, building a whole scene, which doesn’t only capture trees, but cars, trucks, people, signs, and everything else you would see on a typical city street,” said Wilkes. “Although the data we captured was from a stationary platform, it’s similar to the sensors that will be in the next generation of autonomous or semi-autonomous vehicles.”When the 3D model of Malet St was completed, the researchers then transformed various objects on the street into holographic projections. The LiDAR data, in the form of point clouds, was processed by separation algorithms to identify and extract the target objects. Another algorithm was used to convert the target objects into computer-generated diffraction patterns. These data points were implemented into the optical setup to project 3D holographic objects into the driver’s field of view.<div class="fr-img-space-wrap"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1621597362621-Projection+3_CENTRE.jpg" style="width: 788px;" class="fr-fic fr-dib">Image based on LiDAR data (left), converted to a hologram (right).&nbsp;The optical setup is capable of projecting multiple layers of holograms with the help of advanced algorithms. The holographic projection can appear at different sizes and is aligned with the position of the represented real object on the street. For example, a hidden street sign would appear as a holographic projection relative to its actual position behind the obstruction, acting as an alert mechanism.In future, the researchers hope to refine their system by personalising the layout of the head-up displays and have created an algorithm capable of projecting several layers of different objects. These layered holograms can be freely arranged in the driver’s vision space. For example, in the first layer, a traffic sign at a further distance can be projected at a smaller size. In the second layer, a warning sign at a closer distance can be displayed at a larger size.“This layering technique provides an augmented reality experience and alerts the driver in a natural way,” said Skirnewskaja. “Every individual may have different preferences for their display options. For instance, the driver’s vital health signs could be projected in a desired location of the head-up display.“Panoramic holographic projections could be a valuable addition to existing safety measures by showing road objects in real time. Holograms act to alert the driver but are not a distraction.”The researchers are now working to miniaturise the optical components used in their holographic setup so they can fit into a car. Once the setup is complete, vehicle tests on public roads in Cambridge will be carried out.Skirnewskaja is a PhD candidate at the <a href="https://www.ceps-cdt.org/" target="_blank">EPSRC Centre for Doctoral Training (CDT) in Connected Electronic and Photonic Systems</a>, a joint centre with the University of Cambridge and UCL. She also is a fellow of the Foundation of German Business (SDW).<hr>Reference: Jana Skirnewskaja et al. ‘<a href="https://www.osapublishing.org/oe/fulltext.cfm?uri=oe-29-9-13681&id=450306" target="_blank">LiDAR-Derived Digital Holograms for Automotive Head-Up Displays</a>.’ Optics Express (2021). DOI: 10.1364/oe.420740</div>

Image based on LiDAR data (left), converted to a hologram (right).

Researchers have developed the first LiDAR-based augmented reality head-up display for use in vehicles. Tests on a prototype version of the technology suggest that it could improve road safety by ‘seeing through’ objects to alert of potential hazards without distracting the driver.