Whether in murky waters, cramped ship tanks or caves with little light – poor visibility conditions have a huge impact on the environmental perception of autonomous robots. How can they still operate safely and reliably? In the new DeeperSense project, an international consortium led by the German Research Center for Artificial Intelligence (DFKI) is working on technologies that combine the strengths of visual and acoustic sensors with the help of artificial intelligence. The aim is to significantly improve the perception of robotic underwater vehicles in three use cases from the maritime sector. The project is funded by the European Union (EU) with around 3 million euros.&nbsp;The possible applications of robotic systems are numerous. However, in practice, there is often a lack of functional technologies that enable autonomous robots to comprehensively perceive complex environments. Visual sensors such as cameras, which are used in robotics for tasks such as autonomous navigation, manipulation, mapping and object recognition, provide detailed information about the environment. However, their performance is highly dependent on the prevailing light and visibility conditions. In contrast, acoustic sensors for distance measurement are independent of visibility conditions. They also generate image-like data, but with a significantly lower resolution than cameras. In addition, their functionality is severely limited at close range.<h2>Intelligent technologies for practicable robot systems in complex environments</h2>This is where the DeeperSense (Deep-Learning for Multimodal Sensor Fusion) project comes into play, which started on January 1, 2021. The overall objective of the project, which is funded under the EU's Horizon 2020 research framework program, is to significantly improve the environmental perception of service robots, especially in complex and unstructured environments such as underwater. To achieve this, the project pursues an innovative approach: by using artificial intelligence (AI), in particular deep learning as a data-driven machine learning method, it aims to combine the strengths of non-visual and visual sensors to optimize their environment perception capabilities beyond those of the individual sensors. This should not only significantly increase the performance and reliability of autonomous systems, but also open up new functionalities and applications for robotics. <img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjE1NTk4MTM0NDEwLWNzbV9EZWVwZXJTZW5zZV9EaXZlcjJfd2ViX2FkZTM5NmUwNWUuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 788px;" class="fr-fic fr-dib">Mission scenario: AUV monitors diver working under water. © Technisches Hilfswerk (THW)&nbsp; The DeeperSense concept will be demonstrated in three use cases from the particularly demanding underwater sector: diver monitoring in turbid waters, exploration of coral reefs, and seabed mapping. For this purpose, the project consortium brings together leading experts in the fields of maritime robotics, artificial intelligence and underwater sensor technology – the DFKI Robotics Innovation Center, the University of Girona, the University of Haifa and Kraken Robotik GmbH – with end users from the three different application areas – the German Federal Agency for Technical Relief (THW), the Israel Nature and Parks Authority and Tecnoambiente SL.<h2>Intersensory learning for improved environmental perception in underwater applications</h2>DeeperSense is based on the overarching concept of intersensory learning, in which a sensor modality B learns from a sensor modality A. This way, sensor B can provide output that is similar to that of sensor A – in terms of accuracy as well as the type of output and interpretation of the data. For this purpose, sensor A, e.g. a camera, and sensor B, e.g. a sonar, observe the same scene simultaneously. The collected low-resolution sonar data serves as input to an artificial neural network, while the high-resolution camera data serve as output. By gradually adapting the network to the desired output, it learns about the relationships between the input and output data. The result is an algorithm that, once sufficiently trained, is capable of generating a camera-like image based only on the low-resolution sonar data. In the project, this concept will be realized in three core algorithms that meet the specific requirements of the use cases.In the first use case, an autonomous underwater vehicle (AUV) is used to monitor divers at work in turbid waters. Usually, a remote-controlled robot with a camera is used for this purpose, which transmits its data to a control station. However, since optical sensors can only be used to a limited extent underwater, the project partners are also equipping the AUV with a sonar sensor. In DeeperSense, they want to teach this sensor to deliver camera-like images that can be easily interpreted by human personnel at the control station.The second use case deals with the robotic exploration of complex and sensitive underwater structures such as coral reefs. In order to create detailed environmental maps for reliable obstacle detection, the project relies on a combination of visual and acoustic sensors. Here, the challenge will be to reconcile the differing image perspectives. For this purpose, the planned algorithm must be able to recognize objects identified in the data from one sensor in the data of the other. This way, autonomous robots will be able to navigate through a coral reef instead of just going over it. <div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjE1NTk4MjI1NDIxLURlZXBlclNlbnNlX0NvcmFsUmVlZl93ZWIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 788px;" class="fr-fic fr-dib">With DeeperSense AUVs will be able to explore coral reefs at close range. © Israel Nature and Parks Authority (INPA)&nbsp;Finally, in the third use case, the partners want to enable an AUV to generate precise maps of the seafloor. These surveys are typically carried out by ships, which is very expensive. Cameras usually provide supplementary information to verify acoustic data. However, this method is not practical in the darkness of deep waters. In the project, the AUV will use various acoustic sensors to classify sediments on the seafloor. With the help of the DeeperSense concept, the acoustic sensors learn to produce annotated maps of different types of seafloor, just as detailed as it would be possible with cameras under good visibility conditions.</div> <h2>Algorithm development and optimization in the laboratory and under real conditions</h2>During the project, the partners will generate comprehensive training data for the three use cases. The algorithms will be trained using this data and tested under real conditions both in the laboratory and in extensive field tests – in inland waters, in the Mediterranean and the Red Sea. Eventually, the trained algorithms will be optimized for on-board use by underwater vehicles in order to enable real-time execution to support the autonomous robot’s behaviour. The consortium intends to make the results available to the robotics community in online repositories embedded in European research infrastructures.

In the new DeeperSense project, an international consortium led by the German Research Center for Artificial Intelligence (DFKI) is working on technologies that combine the strengths of visual and acoustic sensors with the help of artificial intelligence.

Artificial Intelligence To Optimize The Environmental Perception Of Underwater Robots

In any seemingly quiet pond the still waters actually team with tiny pond dwellers called&nbsp;Euglena gracilis.Unseen to the naked eye, the single-celled organism spirals through the water, pulled along a relatively straight path by a whiplike appendage in search of just the right level of light.But a new paper published Sept. 24 in <a href="http://dx.doi.org/10.1038/s41567-018-0277-7" target="_blank">Nature Physics</a> describes how, under some circumstances, Euglena halts its forward progress and begins tracing out elaborate counterclockwise polygons &mdash; triangles, squares, pentagons &mdash; in a mathematically defined effort to find a better environment.The discovery, led by <a href="https://profiles.stanford.edu/hans-riedel-kruse" target="_blank">Ingmar Riedel-Kruse</a>, assistant professor of bioengineering at Stanford University, could help scientists design tiny swimming robots of the future to be more efficient and effective at maneuvering through the bloodstream, for example, or navigating watery environments.&ldquo;We&rsquo;re trying to understand biological systems in a mathematical way,&rdquo; Riedel-Kruse said. &ldquo;Seemingly simple feedback loops in single cells can actually generate rather complex behaviors in order to accomplish various tasks.&rdquo;<h2>Well-studied organism</h2>Scientists in the 1800s once marveled at finding Euglena &mdash; a greenish oblong with a red eyespot and long, whiplike flagellum for swimming &mdash; under a microscope. Since then, the organism has been observed by countless generations of biology students. With such a history of being watched, it came as a surprise when postdoctoral fellow Alan Tsang first noticed Euglena&rsquo;s novel behavior in a computer model he&rsquo;d developed to study how it moves in relation to light. In his model, when he simulated increased light, the organism began tracing out polygons.Riedel-Kruse remembered being skeptical when Tsang first described what his model predicted.&ldquo;It was hard to believe that it&rsquo;s true,&rdquo; Riedel-Kruse said. &ldquo;I thought there was something wrong with the code.&rdquo; But when the pair checked under the microscope &mdash; increasing light levels as in the simulation &mdash; there were the polygons.The shapes are a result of how Euglena navigates the world. Because the organism normally rolls through the water on its long axis, the eyespot rotates to survey 360 degrees of light. In steady light conditions &mdash; which is normal under a microscope &mdash; it meanders along in a relatively straight path.However, Tsang said, if the eyespot detects increased light intensity, the Euglena makes a hard turn.&ldquo;Then they don&rsquo;t see the light and they swim straight again,&rdquo; Riedel-Kruse said. &ldquo;But since they keep rolling, then after a full cycle they see again the strong light so they make another strong side turn.&rdquo;Enough straight lines followed by sharp turns and a triangle is born.Tsang noticed that over the course of about 30 seconds, Euglena adapted to the stronger light and the turns became less sharp, creating ever-expanding polygons &mdash; squares, then pentagons &mdash; until, finally, the Euglena headed in a relatively straight line.As for why nobody had seen this before, Riedel-Kruse said people rarely alter light levels while observing Euglena under a microscope. But since Tsang was specifically trying to model how the organism moves in relation to light, he did something unusual and the behavior appeared.<h2>A novel behavior</h2>Riedel-Kruse argued that the behavior makes sense for a Euglena swimming along in a pond under a comfortable source of shade. When it suddenly encounters bright sunlight it can turn quickly to seek a patch of shade. By slowly spiraling outward if the first few turns didn&rsquo;t work, the Euglena ups its chances of eventually getting out of the sunlight.Riedel-Kruse&rsquo;s lab studies Euglena in part to better understand how microorganisms navigate their watery worlds. The researchers also integrate what they learn about Euglena into interactive biology setups for education. Euglena is an unusual organism that can both make its own food and eat what it finds in the water. It is related to plants, animals and fungi &mdash; all known as eukaryotes &mdash; but is a separate group with some unique characteristics.&ldquo;Because it is part of an outgroup to most eukaryotic life, you could learn something that is general, and you can also find out how diverse eukaryotic life can be,&rdquo; Riedel-Kruse said. &ldquo;That makes Euglena really interesting to me.&rdquo;What&rsquo;s more, Riedel-Kruse and Tsang said what they learn &mdash; and the mathematical models they developed &mdash; could be useful for microscale robotics.&ldquo;There is an emerging field where people are trying to engineer and program microscopic swarm robotics for things like microsurgery or drug delivery,&rdquo; Tsang said. &ldquo;I definitely see people looking for efficient control mechanisms at the microscale.&rdquo;<iframe src="https://www.youtube.com/embed/oxwP-waWJlQ?&wmode=opaque" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 1115px; height: 610px;"></iframe>

Euglena swim in polygons to avoid light which may help scientists understand biological systems in a mathematical way. | Video by Riedel-Kruse Lab

New insights about how euglena navigate their world could lead to advances in the way miniature robots of the future maneuver through the bloodstream or other watery environments.

Tiny pond dwellers could help scientists design swimming robots

In nature, cockroaches can survive underwater for up to 30 minutes. Now, a robotic cockroach can do even better. <a href="https://wyss.harvard.edu/technology/versatile-ambulatory-microrobots/" rel="noopener noreferrer" target="_blank">Harvard&rsquo;s Ambulatory Microrobot</a>, known as HAMR, can walk on land, swim on the surface of water, and walk underwater for as long as necessary, opening up new environments for this little bot to explore.&nbsp;This next generation HAMR uses multifunctional foot pads that rely on surface tension and surface tension-induced buoyancy when HAMR needs to swim but can also apply a voltage to break the water surface when HAMR needs to sink. This process is called electrowetting, which is the reduction of the contact angle between a material and the water surface under an applied voltage. This change of contact angle makes it easier for objects to break the water surface.Moving on the surface of water allows a microrobot to evade submerged obstacles and reduces drag. Using four pairs of asymmetric flaps and custom designed swimming gaits, HAMR robo-paddles on the water surface to swim. Inspired by the gait of a diving beetle, the researchers exploited the unsteady interaction between the robot&rsquo;s passive flaps and the surrounding water to effectively swim forward and turn.&ldquo;This research demonstrates that microrobotics can leverage small-scale physics &mdash; in this case surface tension &mdash; to perform functions and capabilities that are challenging for larger robots,&rdquo; said Kevin Chen, Ph.D., a postdoctoral fellow at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) and first author of the paper.The most recent research is published in the journal&nbsp;<a href="https://www.nature.com/articles/s41467-018-04855-9" rel="noopener noreferrer" target="_blank">Nature Communications</a>.&ldquo;HAMR&rsquo;s size is key to its performance,&rdquo; said Neel Doshi, graduate student at SEAS and co-author of the paper. &ldquo;If it were much bigger, it would be challenging to be support the robot with surface tension and if it were much smaller, the robot would need a lot more force to break it.&rdquo;HAMR weighs 1.65 grams (about as much as a large paper clip), can carry 1.44 grams of additional payload without sinking and can paddle its legs with a frequency up to 10 Hz. It&rsquo;s coated in Parylene to keep it from shorting under water.<iframe src="https://player.vimeo.com/video/277713323" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 766px; height: 417px;"></iframe>This video shows how the HAMR can transition from land to water, paddle on the surface of water, or sink to the ground to start walking again just as it would on dry land. (Credit: Yufeng Chen, Neel Doshi, and Benjamin Goldberg/Harvard University)&nbsp;Once below the surface of the water, HAMR uses the same gait to walk as it does on dry land and is just as mobile. Getting out of the water, however, is an enormous challenge. The water&rsquo;s surface tension force is twice HAMR&rsquo;s weight and the induced torque causes a dramatic increase of friction on the robot&rsquo;s hind legs. The researchers stiffened the robot&rsquo;s transmission and installed soft pads to the robot&rsquo;s front legs to increase payload capacity and redistribute friction during climbing. Finally, walking up a modest incline, the robot is able break out of the water&rsquo;s hold.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1569985962422-HAMR_no_wire_further_reduce_auto_r17.jpg" style="width: 751px;" class="fr-fic fr-dib">Havard&rsquo;s Ambulatory Microrobot (HAMR) joins the Robobee (right) as a microrobot that can transition between air and water (Credit: Yufeng Chen, Neel Doshi, and Benjamin Goldberg/Harvard University)&nbsp;&ldquo;This robot nicely illustrates some of the challenges and opportunities with small-scale robots,&rdquo; said senior author Robert Wood, Ph.D., Core Faculty member at the Wyss Institute for Biologically Inspired Engineering and the Charles River Professor of Engineering and Applied Sciences at Harvard&rsquo;s Paulson School of Engineering and Applied Sciences. &ldquo;Shrinking brings opportunities for increased mobility &ndash; such as walking on the surface of water &ndash; but also challenges since the forces that we take for granted at larger scales can start to dominate at the size of an insect.&rdquo;

When HAMR needs to sink, its footpads emit a high voltage to break the water surface tension. This process is called electrowetting, which is the reduction of the contact angle between a material and the water surface under an applied voltage. This change of contact angle makes it easier for objects to break the water surface. (Credit: Yufeng Chen, Neel Doshi, and Benjamin Goldberg/Harvard University)

‘HAMR’ can walk on land, swim, and walk under water

Next-generation robotic cockroach can explore under water environments

The next generation of Harvard’s Ambulatory Microrobot (HAMR) can walk on land, swim on the surface of water, and walk underwater, opening up new environments for this little bot to explore.  (Credit: Yufeng Chen, Neel Doshi, and Benjamin Goldberg/Harvard University)

swimming robots

ORGANIZATIONS.

SHAPING THE INDUSTRY.

Deep Trekker

Latest Posts

Artificial Intelligence To Optimize The Environmental Perception Of Underwater Robots

Tiny pond dwellers could help scientists design swimming robots

Next-generation robotic cockroach can explore under water environments

Next-generation robotic cockroach can explore underwater environments