This article was first published on&nbsp;<a href="https://news.mit.edu/2023/helping-robots-handle-fluids-0524" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>. &nbsp;<hr>Imagine you&rsquo;re enjoying a picnic by a riverbank on a windy day. A gust of wind accidentally catches your paper napkin and lands on the water&rsquo;s surface, quickly drifting away from you. You grab a nearby stick and carefully agitate the water to retrieve it, creating a series of small waves. These waves eventually push the napkin back toward the shore, so you grab it. In this scenario, the water acts as a medium for transmitting forces, enabling you to manipulate the position of the napkin without direct contact.Humans regularly engage with various types of fluids in their daily lives, but doing so has been a formidable and elusive goal for current robotic systems. Hand you a latte? A robot can do that. Make it? That&rsquo;s going to require a bit more nuance.&nbsp;<a href="https://openreview.net/forum?id=Cp-io_BoFaE" target="_blank">FluidLab</a>, a new simulation tool from researchers at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), enhances robot learning for complex fluid manipulation tasks like making latte art, ice cream, and even manipulating air. The virtual environment offers a versatile collection of intricate fluid handling challenges, involving both solids and liquids, and multiple fluids simultaneously. FluidLab supports modeling solid, liquid, and gas, including elastic, plastic, rigid objects, Newtonian and non-Newtonian liquids, and smoke and air.&nbsp;At the heart of FluidLab lies FluidEngine, an easy-to-use physics simulator capable of seamlessly calculating and simulating various materials and their interactions, all while harnessing the power of graphics processing units (GPUs) for faster processing. The engine is &ldquo;differential,&rdquo; meaning the simulator can incorporate physics knowledge for a more realistic physical world model, leading to more efficient learning and planning for robotic tasks. In contrast, most existing reinforcement learning methods lack that world model that just depends on trial and error. This enhanced capability, say the researchers, lets users experiment with robot learning algorithms and toy with the boundaries of current robotic manipulation abilities.To set the stage, the researchers tested said robot learning algorithms using FluidLab, discovering and overcoming unique challenges in fluid systems. By developing clever optimization methods, they&#39;ve been able to transfer these learnings from simulations to real-world scenarios effectively.&nbsp;&ldquo;Imagine a future where a household robot effortlessly assists you with daily tasks, like making coffee, preparing breakfast, or cooking dinner. These tasks involve numerous fluid manipulation challenges. Our benchmark is a first step towards enabling robots to master these skills, benefiting households and workplaces alike,&rdquo; says visiting researcher at MIT CSAIL and research scientist at the MIT-IBM Watson AI Lab Chuang Gan, the senior author on a new paper about the research. &ldquo;For instance, these robots could reduce wait times and enhance customer experiences in busy coffee shops. FluidEngine is, to our knowledge, the first-of-its-kind physics engine that supports a wide range of materials and couplings while being fully differentiable. With our standardized fluid manipulation tasks, researchers can evaluate robot learning algorithms and push the boundaries of today&rsquo;s robotic manipulation capabilities.&rdquo;<h2>Fluid fantasia&nbsp;</h2>Over the past few decades, scientists in the robotic manipulation domain have mainly focused on manipulating rigid objects, or on very simplistic fluid manipulation tasks like pouring water. Studying these manipulation tasks involving fluids in the real world can also be an unsafe and costly endeavor.&nbsp;With fluid manipulation, it&rsquo;s not always just about fluids, though. In many tasks, such as creating the perfect ice cream swirl, mixing solids into liquids, or paddling through the water to move objects, it&rsquo;s a dance of interactions between fluids and various other materials. Simulation environments must support &ldquo;coupling,&rdquo; or how two different material properties interact. Fluid manipulation tasks usually require pretty fine-grained precision, with delicate interactions and handling of materials, setting them apart from straightforward tasks like pushing a block or opening a bottle.&nbsp;FluidLab&rsquo;s simulator can quickly calculate how different materials interact with each other.&nbsp;Helping out the GPUs is &ldquo;Taichi,&rdquo; a domain-specific language embedded in Python. The system can compute gradients (rates of change in environment configurations with respect to the robot&#39;s actions) for different material types and their interactions (couplings) with one another. This precise information can be used to fine-tune the robot&rsquo;s movements for better performance. As a result, the simulator allows for faster and more efficient solutions, setting it apart from its counterparts.The 10 tasks the team put forth fell into two categories: using fluids to manipulate hard-to-reach objects, and directly manipulating fluids for specific goals. Examples included separating liquids, guiding floating objects, transporting items with water jets, mixing liquids, creating latte art, shaping ice cream, and controlling air circulation.&nbsp;&ldquo;The simulator works similarly to how humans use their mental models to predict the consequences of their actions and make informed decisions when manipulating fluids. This is a significant advantage of our simulator compared to others,&rdquo; says Carnegie Mellon University PhD student Zhou Xian, another author on the paper. &ldquo;While other simulators primarily support reinforcement learning, ours supports reinforcement learning and allows for more efficient optimization techniques. Utilizing the gradients provided by the simulator supports highly efficient policy search, making it a more versatile and effective tool.&rdquo;<h2>Next steps</h2>FluidLab&rsquo;s future looks bright. The current work attempted to transfer trajectories optimized in simulation to real-world tasks directly in an open-loop manner. For next steps, the team is working to develop a closed-loop policy in simulation that takes as input the state or the visual observations of the environments and performs fluid manipulation tasks in real time, and then transfers the learned policies in real-world scenes.The platform is publicly&nbsp;<a href="https://fluidlab2023.github.io/" target="_blank">publicly available</a>, and researchers hope it will benefit future studies in developing better methods for solving complex fluid manipulation tasks. &ldquo;Humans interact with fluids in everyday tasks, including pouring and mixing liquids (coffee, yogurts, soups, batter), washing and cleaning with water, and more,&rdquo; says University of Maryland computer science professor Ming Lin, who was not involved in the work. &ldquo;For robots to assist humans and serve in similar capacities for day-to-day tasks, novel techniques for interacting and handling various liquids of different properties (e.g. viscosity and density of materials) would be needed and remains a major computational challenge for real-time autonomous systems. This work introduces the first comprehensive physics engine, FluidLab, to enable modeling of diverse, complex fluids and their coupling with other objects and dynamical systems in the environment. The mathematical formulation of &lsquo;differentiable fluids&rsquo; as presented in the paper makes it possible for integrating versatile fluid simulation as a network layer in learning-based algorithms and neural network architectures for intelligent systems to operate in real-world applications.&rdquo;<hr>Gan and Xian wrote the paper alongside Hsiao-Yu Tung a postdoc in the MIT Department of Brain and Cognitive Sciences; Antonio Torralba, an MIT professor of electrical engineering and computer science and CSAIL principal investigator; Dartmouth College Assistant Professor Bo Zhu, Columbia University PhD student Zhenjia Xu, and CMU Assistant Professor Katerina Fragkiadaki. The team&rsquo;s research is supported by the MIT-IBM Watson AI Lab, Sony AI, a DARPA Young Investigator Award, an NSF CAREER award, an AFOSR Young Investigator Award, DARPA Machine Common Sense, and the National Science Foundation.<hr>&quot;Reprinted with permission of&nbsp;<a href="https://news.mit.edu/2023/helping-robots-handle-fluids-0524" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>&quot; &nbsp;

Researchers created "FluidLab," a simulation environment with a diverse set of manipulation tasks involving complex fluid dynamics. Image: Alex Shipps/MIT CSAIL via Midjourney

Researchers create a new simulation tool for robots to manipulate complex fluids in a step toward helping them more effortlessly assist with daily tasks.

Helping robots handle fluids

Image source: Vision, Robotics & Motion, “Robot”, 2023. Accessed via https://vision-robotics.nl/ 

Vision, Robotics & Motion is a highly anticipated event in the manufacturing industry that will take place in the Netherlands on the 7th and 8th of June, 2023. It will celebrate its 20th edition in 2023. 


Young Talent Square at Vision, Robotics & Motion 2023: Start-Ups and Student Teams Discuss Their Technologies

<section id="isPasted">With the rise of extreme weather events, which are becoming more frequent in our warming climate, accurate predictions are becoming more critical for all of us, from farmers to city-dwellers to businesses around the world. To date, climate models have failed to accurately predict precipitation intensity, particularly extremes. While in nature, precipitation can be very varied, with many extremes of precipitation, climate models predict a smaller variance in precipitation with a bias toward light rain.</section><section><h2>Missing piece in current algorithms: cloud organization</h2>Researchers have been working to develop algorithms that will improve prediction accuracy but, as <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Columbia Engineering</a> climate scientists report, there has been a missing piece of information in traditional climate model parameterizations--a way to describe cloud structure and organization that is so fine-scale it is not captured on the computational grid being used. These organization measurements affect predictions of both precipitation intensity and its stochasticity, the variability of random&nbsp;fluctuations in precipitation intensity. Up to now, there has not been an effective, accurate way to measure cloud structure and quantify its impact.</section><section><blockquote>For many years, the scientific community has debated whether to include cloud organization in climate models. Our work provides an answer to the debate and a novel solution for including organization, showing that including this information can significantly improve our prediction of precipitation intensity and variability.<footer>PIERRE GENTINE, DIRECTOR OF THE LEARNING THE EARTH WITH ARTIFICIAL INTELLIGENCE AND PHYSICS CENTER</footer></blockquote></section><section><a href="https://www.pnas.org/doi/10.1073/pnas.2216158120" rel="nofollow" target="_blank">A new study</a> from a team led by <a href="https://www.engineering.columbia.edu/faculty/pierre-gentine" rel="nofollow" target="_blank">Pierre Gentine</a>, director of the <a href="http://news.columbia.edu/news/columbia-ai-based-climate-modeling-center" rel="nofollow" target="_blank">Learning the Earth with Artificial Intelligence and Physics (LEAP) Center</a>, used global storm-resolving simulations and machine learning to create an algorithm that can deal separately with two different scales of cloud organization: those resolved by a climate model, and those that cannot be resolved as they are too small. This new approach addresses the missing piece of information in traditional climate model parameterizations and provides a way to predict precipitation intensity and variability more precisely.&ldquo;Our findings are especially exciting because, for many years, the scientific community has debated whether to include cloud organization in climate models,&rdquo; said Gentine, Maurice Ewing and J. Lamar Worzel Professor of Geophysics in the Departments of <a href="https://www.eee.columbia.edu/" rel="nofollow" target="_blank">Earth and Environmental Engineering</a> and Earth Environmental Sciences and a member of the<a href="https://datascience.columbia.edu/" rel="nofollow" target="_blank">&nbsp;Data Science Institute</a>. &ldquo;Our work provides an answer to the debate and a novel solution for including organization, showing that including this information can significantly improve our prediction of precipitation intensity and variability.&rdquo;</section><section><h2>Using AI to design neural network algorithm</h2>Sarah Shamekh, a PhD student working with Gentine, developed a neural network algorithm that learns the relevant information about the role of fine-scale cloud organization (unresolved scales) on precipitation. Because Shamekh did not define a metric or formula in advance, the model learns implicitly--on its own--how to measure the clustering of clouds, a metric of organization, and then uses this metric to improve the prediction of precipitation. Shamekh trained the algorithm on a high-resolution moisture field, encoding the degree of small-scale organization.&nbsp;&ldquo;We discovered that our organization metric explains precipitation variability almost entirely and could replace a stochastic parameterization in climate models,&rdquo; said Shamekh, lead author of the study, published May 8, 2023, by PNAS. &ldquo;Including this information significantly improved precipitation prediction at the scale relevant to climate models, accurately predicting precipitation extremes and spatial variability.&rdquo;</section><section><h2>Machine-learning algorithm will improve future projections</h2>The researchers are now using their machine-learning approach, which implicitly learns the sub-grid cloud organization metric, in climate models. This should significantly improve the prediction of precipitation intensity and variability, including extreme precipitation events, and enable scientists to better project future changes in the water cycle and extreme weather patterns in a warming climate.&nbsp;</section><section><h2>Future work</h2>This research also opens up new avenues for investigation, such as exploring the possibility of precipitation creating memory, where the atmosphere retains information about recent weather conditions, which in turn influences atmospheric conditions later on, in the climate system. This new approach could have wide-ranging applications beyond just precipitation modeling, including better modeling of the ice sheet and ocean surface.</section>

Credit: "Rain Storm Colorado Springs Colorado" by Brokentaco/Flickr is licensed under CC BY 2.0

Columbia Engineers develop machine-learning algorithm that will help researchers to better understand and mitigate the impact of extreme weather events, which are becoming more frequent in our warming climate.

New Algorithm Predicts Precipitation, Especially Extreme Events, More Accurately

This article was first published on&nbsp;<a href="https://news.mit.edu/2023/researchers-identify-similar-materials-images-0523" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>. &nbsp;<hr>A robot manipulating objects while, say, working in a kitchen, will benefit from understanding which items are composed of the same materials. With this knowledge, the robot would know to exert a similar amount of force whether it picks up a small pat of butter from a shadowy corner of the counter or an entire stick from inside the brightly lit fridge.Identifying objects in a scene that are composed of the same material, known as material selection, is an especially challenging problem for machines because a material&rsquo;s appearance can vary drastically based on the shape of the object or lighting conditions.Scientists at MIT and Adobe Research have taken a step toward solving this challenge. They developed a technique that can identify all pixels in an image representing a given material, which is shown in a pixel selected by the user.The method is accurate even when objects have varying shapes and sizes, and the machine-learning model they developed isn&rsquo;t tricked by shadows or lighting conditions that can make the same material appear different.Although they trained their model using only &ldquo;synthetic&rdquo; data, which are created by a computer that modifies 3D scenes to produce many varying images, the system works effectively on real indoor and outdoor scenes it has never seen before. The approach can also be used for videos; once the user identifies a pixel in the first frame, the model can identify objects made from the same material throughout the rest of the video.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1684915256774-ezgif-3-743d977334.gif" style="width: 766px;" class="fr-fic fr-dib">The researchers&#39; technique can also be used to select similar materials in a video. The user identifies a pixel in the first frame (red dot in the far-left image on the yellow fabric) and the system automatically identifies objects made from the same material throughout the rest of the video. Image: Courtesy of the researchersIn addition to applications in scene understanding for robotics, this method could be used for image editing or incorporated into computational systems that deduce the parameters of materials in images. It could also be utilized for material-based web recommendation systems. (Perhaps a shopper is searching for clothing made from a particular type of fabric, for example.)&ldquo;Knowing what material you are interacting with is often quite important. Although two objects may look similar, they can have different material properties. Our method can facilitate the selection of all the other pixels in an image that are made from the same material,&rdquo; says Prafull Sharma, an electrical engineering and computer science graduate student and lead author of a <a data-lf-fd-inspected-kn9eq4roawkarlvp="true" href="https://arxiv.org/pdf/2305.13291.pdf" target="_blank">paper</a> on this technique.Sharma&rsquo;s co-authors include Julien Philip and Michael Gharbi, research scientists at Adobe Research; and senior authors William T. Freeman, the Thomas and Gerd Perkins Professor of Electrical Engineering and Computer Science and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); Fr&eacute;do Durand, a professor of electrical engineering and computer science and a member of CSAIL; and Valentin Deschaintre, a research scientist at Adobe Research. The research will be presented at the SIGGRAPH 2023 conference.<h2>A new approach</h2>Existing methods for material selection struggle to accurately identify all pixels representing the same material. For instance, some methods focus on entire objects, but one object can be composed of multiple materials, like a chair with wooden arms and a leather seat. Other methods may utilize a predetermined set of materials, but these often have broad labels like &ldquo;wood,&rdquo; despite the fact that there are thousands of varieties of wood.Instead, Sharma and his collaborators developed a machine-learning approach that dynamically evaluates all pixels in an image to determine the material similarities between a pixel the user selects and all other regions of the image. If an image contains a table and two chairs, and the chair legs and tabletop are made of the same type of wood, their model could accurately identify those similar regions.Before the researchers could develop an AI method to learn how to select similar materials, they had to overcome a few hurdles. First, no existing dataset contained materials that were labeled finely enough to train their machine-learning model. The researchers rendered their own synthetic dataset of indoor scenes, which included 50,000 images and more than 16,000 materials randomly applied to each object.&ldquo;We wanted a dataset where each individual type of material is marked independently,&rdquo; Sharma says.Synthetic dataset in hand, they trained a machine-learning model for the task of identifying similar materials in real images &mdash; but it failed. The researchers realized distribution shift was to blame. This occurs when a model is trained on synthetic data, but it fails when tested on real-world data that can be very different from the training set.To solve this problem, they built their model on top of a pretrained computer vision model, which has seen millions of real images. They utilized the prior knowledge of that model by leveraging the visual features it had already learned.&ldquo;In machine learning, when you are using a neural network, usually it is learning the representation and the process of solving the task together. We have disentangled this. The pretrained model gives us the representation, then our neural network just focuses on solving the task,&rdquo; he says.<h2>Solving for similarity</h2>The researchers&rsquo; model transforms the generic, pretrained visual features into material-specific features, and it does this in a way that is robust to object shapes or varied lighting conditions.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1684915401459-ezgif-3-c9363940ca.gif" style="width: 766px;" class="fr-fic fr-dib">The system the researchers developed to identify similar materials is robust to changes in lighting conditions, as seen in this example of match heads burning. Image: Courtesy of the researchersThe model can then compute a material similarity score for every pixel in the image. When a user clicks a pixel, the model figures out how close in appearance every other pixel is to the query. It produces a map where each pixel is ranked on a scale from 0 to 1 for similarity.&ldquo;The user just clicks one pixel and then the model will automatically select all regions that have the same material,&rdquo; he says.Since the model is outputting a similarity score for each pixel, the user can fine-tune the results by setting a threshold, such as 90 percent similarity, and receive a map of the image with those regions highlighted. The method also works for cross-image selection &mdash; the user can select a pixel in one image and find the same material in a separate image.During experiments, the researchers found that their model could predict regions of an image that contained the same material more accurately than other methods. When they measured how well the prediction compared to ground truth, meaning the actual areas of the image that are comprised of the same material, their model matched up with about 92 percent accuracy.In the future, they want to enhance the model so it can better capture fine details of the objects in an image, which would boost the accuracy of their approach.&ldquo;Rich materials contribute to the functionality and beauty of the world we live in. But computer vision algorithms typically overlook materials, focusing heavily on objects instead. This paper makes an important contribution in recognizing materials in images and video across a broad range of challenging conditions,&rdquo; says Kavita Bala, Dean of the Cornell Bowers College of Computing and Information Science and Professor of Computer Science, who was not involved with this work. &ldquo;This technology can be very useful to end consumers and designers alike. For example, a home owner can envision how expensive choices like reupholstering a couch, or changing the carpeting in a room, might appear, and can be more confident in their design choices based on these visualizations.&rdquo;<hr>&quot;Reprinted with permission of&nbsp;<a href="https://news.mit.edu/2023/researchers-identify-similar-materials-images-0523" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>&quot; &nbsp;

A machine-learning model can identify all the pixels in an image that represent a given material. Developed at MIT, the technique could someday be used in computer vision systems that help robots interact with objects in the real world. Pictured is an artist’s interpretation of the new system. Image: MIT News with iStock

This machine-learning method could assist with robotic scene understanding, image editing, or online recommendation systems.

Researchers use AI to identify similar materials in images

ETH student Alexander Bayer was inspired to upgrade the traditional white cane by a blind classmate in high school. Together with three other ETH students, he is working in the Student Project House on the &ldquo;NextGuide&rdquo; project, a smart cane for the blind and visually impaired equipped with an integrated camera and the ability to provide haptic feedback. The cane not only indicates the direction in which an individual should walk to avoid obstacles but also communicates via vibrations whether they are standing in front of a door, pedestrian crossing or staircase.&nbsp;

ETH students have developed a smart cane featuring an integrated camera that detects its surroundings and a tactile pointer that helps blind and visually impaired individuals navigate their way safely.

The smart cane for the blind and visually impaired

This article was first published on&nbsp;<a href="https://news.mit.edu/2023/new-machine-learning-model-ocean-currents-0517" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>. &nbsp;<hr>To study ocean currents, scientists release GPS-tagged buoys in the ocean and record their velocities to reconstruct the currents that transport them. These buoy data are also used to identify &ldquo;divergences,&rdquo; which are areas where water rises up from below the surface or sinks beneath it.By accurately predicting currents and pinpointing divergences, scientists can more precisely forecast the weather, approximate how oil will spread after a spill, or measure energy transfer in the ocean. A new model that incorporates machine learning makes more accurate predictions than conventional models do, a <a data-lf-fd-inspected-kn9eq4roawkarlvp="true" href="https://arxiv.org/pdf/2302.10364.pdf" target="_blank">new study</a> reports.A multidisciplinary research team including computer scientists at MIT and oceanographers has found that a standard statistical model typically used on buoy data can struggle to accurately reconstruct currents or identify divergences because it makes unrealistic assumptions about the behavior of water.The researchers developed a new model that incorporates knowledge from fluid dynamics to better reflect the physics at work in ocean currents. They show that their method, which only requires a small amount of additional computational expense, is more accurate at predicting currents and identifying divergences than the traditional model.This new model could help oceanographers make more accurate estimates from buoy data, which would enable them to more effectively monitor the transportation of biomass (such as Sargassum seaweed), carbon, plastics, oil, and nutrients in the ocean. This information is also important for understanding and tracking climate change.&ldquo;Our method captures the physical assumptions more appropriately and more accurately. In this case, we know a lot of the physics already. We are giving the model a little bit of that information so it can focus on learning the things that are important to us, like what are the currents away from the buoys, or what is this divergence and where is it happening?&rdquo; says senior author Tamara Broderick, an associate professor in MIT&rsquo;s Department of Electrical Engineering and Computer Science (EECS) and a member of the Laboratory for Information and Decision Systems and the Institute for Data, Systems, and Society.Broderick&rsquo;s co-authors include lead author Renato Berlinghieri, an electrical engineering and computer science graduate student; Brian L. Trippe, a postdoc at Columbia University; David R. Burt and Ryan Giordano, MIT postdocs; Kaushik Srinivasan, an assistant researcher in atmospheric and ocean sciences at the University of California at Los Angeles; Tamay &Ouml;zg&ouml;kmen, professor in the Department of Ocean Sciences at the University of Miami; and Junfei Xia, a graduate student at the University of Miami. The research will be presented at the International Conference on Machine Learning.<h2>Diving into the data</h2>Oceanographers use data on buoy velocity to predict ocean currents and identify &ldquo;divergences&rdquo; where water rises to the surface or sinks deeper.To estimate currents and find divergences, oceanographers have used a machine-learning technique known as a Gaussian process, which can make predictions even when data are sparse. To work well in this case, the Gaussian process must make assumptions about the data to generate a prediction.A standard way of applying a Gaussian process to oceans data assumes the latitude and longitude components of the current are unrelated. But this assumption isn&rsquo;t physically accurate. For instance, this existing model implies that a current&rsquo;s divergence and its vorticity (a whirling motion of fluid) operate on the same magnitude and length scales. Ocean scientists know this is not true, Broderick says. The previous model also assumes the frame of reference matters, which means fluid would behave differently in the latitude versus the longitude direction.&ldquo;We were thinking we could address these problems with a model that incorporates the physics,&rdquo; she says.They built a new model that uses what is known as a Helmholtz decomposition to accurately represent the principles of fluid dynamics. This method models an ocean current by breaking it down into a vorticity component (which captures the whirling motion) and a divergence component (which captures water rising or sinking).In this way, they give the model some basic physics knowledge that it uses to make more accurate predictions.This new model utilizes the same data as the old model. And while their method can be more computationally intensive, the researchers show that the additional cost is relatively small.<h2>Buoyant performance</h2>They evaluated the new model using synthetic and real ocean buoy data. Because the synthetic data were fabricated by the researchers, they could compare the model&rsquo;s predictions to ground-truth currents and divergences. But simulation involves assumptions that may not reflect real life, so the researchers also tested their model using data captured by real buoys released in the Gulf of Mexico.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1684368694423-ezgif-5-7519252a77.gif" style="width: 766px;" class="fr-fic fr-dib">This shows the trajectories of approximately 300 buoys released during the Grand LAgrangian Deployment (GLAD) in the Gulf of Mexico in the summer of 2013, to learn about ocean surface currents around the Deepwater Horizon oil spill site. The small, regular clockwise rotations are due to Earth&#39;s rotation. Credit: Consortium of Advanced Research for Transport of Hydrocarbons in the EnvironmentIn each case, their method demonstrated superior performance for both tasks, predicting currents and identifying divergences, when compared to the standard Gaussian process and another machine-learning approach that used a neural network. For example, in one simulation that included a vortex adjacent to an ocean current, the new method correctly predicted no divergence while the previous Gaussian process method and the neural network method both predicted a divergence with very high confidence.The technique is also good at identifying vortices from a small set of buoys, Broderick adds.Now that they have demonstrated the effectiveness of using a Helmholtz decomposition, the researchers want to incorporate a time element into their model, since currents can vary over time as well as space. In addition, they want to better capture how noise impacts the data, such as winds that sometimes affect buoy velocity. Separating that noise from the data could make their approach more accurate.&ldquo;Our hope is to take this noisily observed field of velocities from the buoys, and then say what is the actual divergence and actual vorticity, and predict away from those buoys, and we think that our new technique will be helpful for this,&rdquo; she says.&ldquo;The authors cleverly integrate known behaviors from fluid dynamics to model ocean currents in a flexible model,&rdquo; says Massimiliano Russo, an associate biostatistician at Brigham and Women&rsquo;s Hospital and instructor at Harvard Medical School, who was not involved with this work. &ldquo;The resulting approach retains the flexibility to model the nonlinearity in the currents but can also characterize phenomena such as vortices and connected currents that would only be noticed if the fluid dynamic structure is integrated into the model. This is an excellent example of where a flexible model can be substantially improved with a well thought and scientifically sound specification.&rdquo;This research is supported, in part, by the Office of Naval Research, a National Science Foundation (NSF) CAREER Award, and the Rosenstiel School of Marine, Atmospheric, and Earth Science at the University of Miami.<hr>&quot;Reprinted with permission of&nbsp;<a href="https://news.mit.edu/2023/new-machine-learning-model-ocean-currents-0517" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>&quot; &nbsp;

Computer scientists at MIT joined forces with oceanographers to develop a machine-learning model that incorporates knowledge from fluid dynamics to generate more accurate predictions about the velocities of ocean currents. This figure shows drifting buoy trajectories in the Gulf of Mexico superimposed on surface currents. The red dots mark the buoys’ positions on March 9, 2016, and the tails are 14 days long. Image: Edward Ryan and Tamay Özgökmen from the University of Miami.

A new machine-learning model makes more accurate predictions about ocean currents, which could help with tracking plastic pollution and oil spills, and aid in search and rescue.

A better way to study ocean currents

Grasping objects of different sizes, shapes and textures is a problem that is easy for a human, but challenging for a robot. Researchers from the University of Cambridge designed a soft, 3D printed robotic hand that cannot independently move its fingers but can still carry out a range of complex movements.&nbsp;The robot hand was trained to grasp different objects and was able to predict whether it would drop them by using the information provided from sensors placed on its &lsquo;skin&rsquo;.&nbsp;This type of passive movement makes the robot far easier to control and far more energy-efficient than robots with fully motorised fingers. The researchers say their adaptable design could be used in the development of low-cost robots that are capable of more natural movement and can learn to grasp a wide range of objects. The <a href="https://onlinelibrary.wiley.com/doi/full/10.1002/aisy.202200390" target="_blank">results</a> are reported in the journal Advanced Intelligent Systems.&nbsp;<iframe width="640" height="360" src="https://www.youtube.com/embed/haWziO6aOqc?&wmode=opaque&rel=0&enablejsapi=1&origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>In the natural world, movement results from the interplay between the brain and the body: this enables people and animals to move in complex ways without expending unnecessary amounts of energy. Over the past several years, soft components have begun to be integrated into robotics design thanks to advances in 3D printing techniques, which have allowed researchers to add complexity to simple, energy-efficient systems.&nbsp;The human hand is highly complex, and recreating all of its dexterity and adaptability in a robot is a massive research challenge. Most of today&rsquo;s advanced robots are not capable of manipulation tasks that small children can perform with ease. For example, humans instinctively know how much force to use when picking up an egg, but for a robot this is a challenge: too much force, and the egg could shatter; too little, and the robot could drop it. In addition, a fully actuated robot hand, with motors for each joint in each finger, requires a significant amount of energy.&nbsp;In <a href="http://www.eng.cam.ac.uk/profiles/fi224" target="_blank">Professor Fumiya&nbsp;</a><a href="http://www.eng.cam.ac.uk/profiles/fi224" target="_blank">Iida</a>&rsquo;s <a href="https://birlab.org/" target="_blank">Bio-Inspired Robotics Laboratory</a> in Cambridge&rsquo;s Department of Engineering, researchers have been developing potential solutions to both problems: a robot hand than can grasp a variety of objects with the correct amount of pressure while using a minimal amount of energy.&nbsp;&ldquo;In <a href="http://www.eng.cam.ac.uk/news/3d-printed-robot-hand-plays-piano" target="_blank">earlier experiments</a>, our lab has shown that it&rsquo;s possible to get a significant range of motion in a robot hand just by moving the wrist,&rdquo; said co-author Dr Thomas George-Thuruthel, who is now based at University College London (UCL) <a href="https://www.ucl.ac.uk/ucl-east/study-and-research/robotics-and-autonomous-systems" target="_blank">East</a>. &ldquo;We wanted to see whether a robot hand based on passive movement could not only grasp objects, but would be able to predict whether it was going to drop the objects or not, and adapt accordingly.&rdquo;&nbsp;The researchers used a 3D-printed anthropomorphic hand implanted with tactile sensors, so that the hand could sense what it was touching. The hand was only capable of passive, wrist-based movement.&nbsp;The team carried out more than 1,200 tests with the robot hand, observing its ability to grasp small objects without dropping them. The robot was initially trained using small 3D printed plastic balls, and grasped them using a pre-defined action obtained through human demonstrations.&nbsp;&ldquo;This kind of hand has a bit of springiness to it: it can pick things up by itself without any actuation of the fingers,&rdquo; said first author Dr Kieran Gilday, who is now based at EPFL in Lausanne, Switzerland. &ldquo;The tactile sensors give the robot a sense of how well the grip is going, so it knows when it&rsquo;s starting to slip. This helps it to predict when things will fail.&rdquo;&nbsp;The robot used trial and error to learn what kind of grip would be successful. After finishing the training with the balls, it then attempted to grasp different objects including a peach, a computer mouse and a roll of bubble wrap. In these tests, the hand was able to successfully grasp 11 of 14 objects.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1684333513339-CENTRE_RobotHand_CREDIT+Bio-Inspired+Robotics+Laboratory%2C+University+of+Cambridge.jpg" style="width: 766px;" class="fr-fic fr-dib">Credit: Bio-Inspired Robotics Laboratory, University of Cambridge.&nbsp;&ldquo;The sensors, which are sort of like the robot&rsquo;s skin, measure the pressure being applied to the object,&rdquo; said Dr George-Thuruthel. &ldquo;We can&rsquo;t say exactly what information the robot is getting, but it can theoretically estimate where the object has been grasped and with how much force.&rdquo;&nbsp;&ldquo;The robot learns that a combination of a particular motion and a particular set of sensor data will lead to failure, which makes it a customisable solution,&rdquo; said Dr Gilday. &ldquo;The hand is very simple, but it can pick up a lot of objects with the same strategy.&rdquo;&nbsp;&ldquo;The big advantage of this design is the range of motion we can get without using any actuators,&rdquo; said Professor Iida. &ldquo;We want to simplify the hand as much as possible. We can get lots of good information and a high degree of control without any actuators, so that when we do add them, we&rsquo;ll get more complex behaviour in a more efficient package.&rdquo;&nbsp;A fully actuated robotic hand, in addition to the amount of energy it requires, is also a complex control problem. The passive design of the Cambridge-designed hand, using a small number of sensors, is easier to control, provides a wide range of motion, and streamlines the learning process.&nbsp;In future, the system could be expanded in several ways, such as by adding computer vision capabilities, or teaching the robot to exploit its environment, which would enable it to grasp a wider range of objects.&nbsp;<hr>Reference: K. Gilday, Dr. T. George-Thuruthel, Prof. F. Iida. &lsquo;<a href="https://onlinelibrary.wiley.com/doi/full/10.1002/aisy.202200390" target="_blank">Predictive Learning of Error Recovery with a Sensorised Passivity-based Soft Anthropomorphic Hand</a>.&rsquo; Advanced Intelligent Systems (2023). 10.1002/aisy.202200390&nbsp;

Researchers have designed a low-cost, energy-efficient robotic hand that can grasp a range of objects – and not drop them – using just the movement of its wrist and the feeling in its ‘skin’.  

It's all in the wrist: energy-efficient robot hand learns how not to drop the ball

Proteins are essential to the biological functions of most living organisms. They have evolved to interact with other proteins, nucleic acids, lipids etc., and all of those interactions form large, &ldquo;supra-molecular&rdquo; complexes. This means that understanding protein interactions is crucial for understanding many cellular processes.In a big step forward, scientists in the group of Matteo Dal Peraro at EPFL have developed a new tool called PeSTo (short for Protein Structure Transformer) that can predict the specific regions on the surface of a protein that can interact with other proteins, but also nucleic acids, lipids, ions, and small molecules. These interfaces are crucial for the formation of supramolecular complexes and function modulation.<blockquote>...this method effectively captures the complex interactions within protein structures to enable an accurate prediction of protein binding interfaces&rdquo;.<footer>Lucien Krapp, the main developer of PeSTo</footer></blockquote>PeSTo is built on a neural network based on transformer technology. In the context of machine learning, a transformer is a type of neural network designed to process sequential data, such as natural language by using self-attention mechanisms to weigh the importance of different parts of the input sequence and make predictions. The technology was introduced by Google Brain in 2017 and are now at the core of many modern AI tools.<h2>How does PeSTo work?</h2>&ldquo;The model evaluates the chemical and physical context of each atom by examining all nearby atoms,&rdquo; says Lucien Krapp, the main developer of PeSTo. &ldquo;Using the self-attention mechanism, it focuses on significant atoms and interactions within the protein structure. It means that this method effectively captures the complex interactions within protein structures to enable an accurate prediction of protein binding interfaces&rdquo;.Because PeSTo&rsquo;s predictions are based solely on the position in space and the type of atoms, it can make predictions without needing to describe the physics and chemistry of the protein interface using additional external methods. This eliminates the &lsquo;overhead&rsquo; of pre-computing molecular surfaces and additional properties, making it much faster, robust and more general than current methods.It also means that PeSTo can run fast enough to process large volumes of protein structure data, e.g. ensembles from molecular dynamics simulations or entire foldomes. Ultimately, this enables faster discovery of interfaces that go unseen in conventional static structures resolved experimentally.PeSTo outperforms other methods for predicting protein interaction interfaces and can predict interactions with nucleic acids, lipids, ligands, ions, and small molecules with high confidence. The model&#39;s low computational cost makes it a valuable tool for the scientific community.<h2>PeSTo applied to the human foldome</h2>The researchers unleashed PeSTo on the human foldome, a growing database of predicted protein structures. They analyzed the interactions that human proteins have with other molecules, and produced detailed information about the human &quot;interfaceome&quot; &ndash; the sum total of all protein interacting interfaces in the human body. To do this, the researchers used the&nbsp;<a href="http://alphafold.ebi.ac.uk/" rel="noopener" target="_blank">AlphaFold European Bioinformatics Institute (AF-EBI)</a> database.The researchers have made PeSTo available in&nbsp;<a href="http://pesto.epfl.ch/" target="_blank">a user-friendly web server</a>, free of charge and prior registration. The server can take any protein structure in PDB format. The predicted interfaces can be visualized directly in the browser with additional information on the confidence of the prediction on a per-residue basis.Publishing in Nature Communications, the scientists highlight numerous advantages of PeSTo over older methods, particularly that it can work with all kinds of molecules without needing to know all the details about their chemistry and physics. This makes PeSTo a more flexible, powerful and general tool for studying molecular systems and their interactions.<hr>ReferencesLucien F. Krapp, Luciano A. Abriata, Fabio Cort&eacute;s Rodriguez, Matteo Dal Peraro. PeSTo: parameter-free geometric deep learning for accurate prediction of protein binding interfaces. Nature Communications 18 April 2023. DOI:&nbsp;<a href="http://www.nature.com/articles/s41467-023-37701-8" rel="noopener" target="_blank">10.1038/s41467-023-37701-8</a>

The geometric deep-learning method (PeSTo) used to predict protein binding interfaces. The amino acids involved in the protein binding interface are highlighted in red. Credit: Lucien Krapp (EPFL)

Scientists at EPFL have developed PeSTo, an AI model that can predict, with high confidence, the binding interfaces of proteins when they bind other proteins, nucleic acids, lipids, ions, and small molecules. The model's low computational cost enables the processing of large amounts of structural data, opening up opportunities for new biological discoveries.

PeSTo: a new AI tool for predicting protein interactions

This article was first published on&nbsp;<a href="https://news.mit.edu/2023/using-reflections-shiny-objects-camera-0510" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>. &nbsp;<hr>As a car travels along a narrow city street, reflections off the glossy paint or side mirrors of parked vehicles can help the driver glimpse things that would otherwise be hidden from view, like a child playing on the sidewalk behind the parked cars.Drawing on this idea, researchers from MIT and Rice University have created a computer vision technique that leverages reflections to image the world. Their method uses reflections to turn glossy objects into &ldquo;cameras,&rdquo; enabling a user to see the world as if they were looking through the &ldquo;lenses&rdquo; of everyday objects like a ceramic coffee mug or a metallic paper weight. &nbsp;&nbsp;Using images of an object taken from different angles, the technique converts the surface of that object into a virtual sensor which captures reflections. The AI system maps these reflections in a way that enables it to estimate depth in the scene and capture novel views that would only be visible from the object&rsquo;s perspective. One could use this technique to see around corners or beyond objects that block the observer&rsquo;s view.This method could be especially useful in autonomous vehicles. For instance, it could enable a self-driving car to use reflections from objects it passes, like lamp posts or buildings, to see around a parked truck.&ldquo;We have shown that any surface can be converted into a sensor with this formulation that converts objects into virtual pixels and virtual sensors. This can be applied in many different areas,&rdquo; says Kushagra Tiwary, a graduate student in the Camera Culture Group at the Media Lab and co-lead author of a <a data-lf-fd-inspected-kn9eq4roawkarlvp="true" href="https://arxiv.org/pdf/2212.04531.pdf" target="_blank">paper</a> on this research.Tiwary is joined on the paper by co-lead author Akshat Dave, a graduate student at Rice University; Nikhil Behari, an MIT research support associate; Tzofi Klinghoffer, an MIT graduate student; Ashok Veeraraghavan, professor of electrical and computer engineering at Rice University; and senior author Ramesh Raskar, associate professor of media arts and sciences and leader of the Camera Culture Group at MIT. The research will be presented at the Conference on Computer Vision and Pattern Recognition.<h2>Reflecting on reflections</h2>The heroes in crime television shows often &ldquo;zoom and enhance&rdquo; surveillance footage to capture reflections &mdash; perhaps those caught in a suspect&rsquo;s sunglasses &mdash; that help them solve a crime.&nbsp;&ldquo;In real life, exploiting these reflections is not as easy as just pushing an enhance button. Getting useful information out of these reflections is pretty hard because reflections give us a distorted view of the world,&rdquo; says Dave.This distortion depends on the shape of the object and the world that object is reflecting, both of which researchers may have incomplete information about. In addition, the glossy object may have its own color and texture that mixes with reflections. Plus, reflections are two-dimensional projections of a three-dimensional world, which makes it hard to judge depth in reflected scenes.The researchers found a way to overcome these challenges. Their technique, known as ORCa (which stands for Objects as Radiance-Field Cameras), works in three steps. First, they take pictures of an object from many vantage points, capturing multiple reflections on the glossy object.Then, for each image from the real camera, ORCa uses machine learning to convert the surface of the object into a virtual sensor that captures light and reflections that strike each virtual pixel on the object&rsquo;s surface. Finally, the system uses virtual pixels on the object&rsquo;s surface to model the 3D environment from the point of view of the object.<h2>Catching rays</h2>Imaging the object from many angles enables ORCa to capture multiview reflections, which the system uses to estimate depth between the glossy object and other objects in the scene, in addition to estimating the shape of the glossy object. ORCa models the scene as a 5D radiance field, which captures additional information about the intensity and direction of light rays that emanate from and strike each point in the scene.The additional information contained in this 5D radiance field also helps ORCa accurately estimate depth. And because the scene is represented as a 5D radiance field, rather than a 2D image, the user can see hidden features that would otherwise be blocked by corners or obstructions.In fact, once ORCa has captured this 5D radiance field, the user can put a virtual camera anywhere in the scene and synthesize what that camera would see, Dave explains. The user could also insert virtual objects into the environment or change the appearance of an object, such as from ceramic to metallic.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1683771128106-ezgif-1-b97d11cf83.gif" style="width: 766px;" class="fr-fic fr-dib">The additional information that is captured in the 5D radiance field that ORCa learns enables a user to change the appearance of objects in the scene, in this case, by rendering the glossy sphere and mug as metallic objects instead. Credit: Courtesy of the researchers&ldquo;It was especially challenging to go from a 2D image to a 5D environment. You have to make sure that mapping works and is physically accurate, so it is based on how light travels in space and how light interacts with the environment. We spent a lot of time thinking about how we can model a surface,&rdquo; Tiwary says.<h2>Accurate estimations</h2>The researchers evaluated their technique by comparing it with other methods that model reflections, which is a slightly different task than ORCa performs. Their method performed well at separating out the true color of an object from the reflections, and it outperformed the baselines by extracting more accurate object geometry and textures.They compared the system&rsquo;s depth estimations with simulated ground truth data on the actual distance between objects in the scene and found ORCa&rsquo;s predictions to be reliable.&nbsp; &nbsp;&ldquo;Consistently, with ORCa, it not only estimates the environment accurately as a 5D image, but to achieve that, in the intermediate steps, it also does a good job estimating the shape of the object and separating the reflections from the object texture,&rdquo; Dave says.Building off of this proof-of-concept, the researchers want to apply this technique to drone imaging. ORCa could use faint reflections from objects a drone flies over to reconstruct a scene from the ground. They also want to enhance ORCa so it can utilize other cues, such as shadows, to reconstruct hidden information, or combine reflections from two objects to image new parts of a scene.&ldquo;Estimating specular reflections is really important for seeing around corners, and this is the next natural step to see around corners using faint reflections in the scene,&rdquo; says Raskar.&ldquo;Ordinarily, shiny objects are difficult for vision systems to handle. This paper is very creative because it turns the longstanding weakness of object shininess into an advantage. By exploiting environment reflections off a shiny object, the paper is not only able to see hidden parts of the scene, but also understand how the scene is lit. This enables applications in 3D perception that include, but are not limited to, an ability to composite virtual objects into real scenes in ways that appear seamless, even in challenging lighting conditions,&rdquo; says Achuta Kadambi, assistant professor of electrical engineering and computer science at the University of California at Los Angeles, who was not involved with this work. &ldquo;One reason that others have not been able to use shiny objects in this fashion is that most prior works require surfaces with known geometry or texture. The authors have derived an intriguing, new formulation that does not require such knowledge.&rdquo;<hr>&quot;Reprinted with permission of&nbsp;<a href="https://news.mit.edu/2023/using-reflections-shiny-objects-camera-0510" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>&quot; &nbsp;

Researchers from MIT and Rice University have created a computer vision technique that leverages reflections to image the world by using them to turn glossy objects into “cameras,” enabling a user to see the world as if they were looking through the “lenses” of everyday objects like a ceramic coffee mug or a metallic paper weight. Credit: Courtesy of the researchers