<html><head></head><body>The results build on evidence that providing richer information during artificial intelligence (AI) training can make autonomous robots more adaptive to new situations, improving their safety and effectiveness.Adding descriptions of a tool’s form and function to the training process for the robot improved the robot’s ability to manipulate newly encountered tools that were not in the original training set. A team of mechanical engineers and computer scientists presented the new method, <a href="https://openreview.net/pdf?id=nPw7jaGBrCG" target="_blank">Accelerated Learning of Tool Manipulation with LAnguage</a>, or ATLA, at the Conference on Robot Learning on Dec. 14.<iframe src="https://player.vimeo.com/video/781953242" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" style="width: 764px; height: 492px;"></iframe>Robotic arms have great potential to help with repetitive or challenging tasks, but training robots to manipulate tools effectively is difficult: Tools have a wide variety of shapes, and a robot’s dexterity and vision are no match for a human’s.“Extra information in the form of language can help a robot learn to use the tools more quickly,” said study coauthor <a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of <a href="https://mae.princeton.edu/" target="_blank">mechanical and aerospace engineering</a> at Princeton who leads the <a href="https://irom-lab.princeton.edu/majumdar" target="_blank">Intelligent Robot Motion Lab</a>.The team obtained tool descriptions by querying GPT-3, a large language model released by OpenAI in 2020 that uses a form of AI called deep learning to generate text in response to a prompt. After experimenting with various prompts, they settled on using “Describe the [feature] of [tool] in a detailed and scientific response,” where the feature was the shape or purpose of the tool.“Because these language models have been trained on the internet, in some sense you can think of this as a different way of retrieving that information,” more efficiently and comprehensively than using crowdsourcing or scraping specific websites for tool descriptions, said <a href="https://engineering.princeton.edu/faculty/karthik-narasimhan" target="_blank">Karthik Narasimhan</a>, an assistant professor of <a href="https://www.cs.princeton.edu/people/profile/karthikn" target="_blank">computer science</a> and coauthor of the study. Narasimhan is a lead faculty member in Princeton’s <a href="https://princeton-nlp.github.io/" target="_blank">natural language processing (NLP) group</a>, and contributed to the original GPT language model as a visiting research scientist at OpenAI.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1671785093867-pushing_tasks.gif" style="width: 300px;" class="fr-fic fr-dib">Princeton researchers have found that human-language descriptions of tools can accelerate the learning of a simulated robotic arm lifting and using a variety of tools. Animation by the researchers; GIF by Neil Adelantar&nbsp;This work is the first collaboration between Narasimhan’s and Majumdar’s research groups. Majumdar focuses on developing AI-based policies to help robots — <a href="https://engineering.princeton.edu/news/2020/11/17/machine-learning-guarantees-robots-performance-unknown-territory" target="_blank">including flying and walking robots</a> — generalize their functions to new settings, and he was curious about the potential of recent “massive progress in natural language processing” to benefit robot learning, he said.For their simulated robot learning experiments, the team selected a training set of 27 tools, ranging from an axe to a squeegee. They gave the robotic arm four different tasks: push the tool, lift the tool, use it to sweep a cylinder along a table, or hammer a peg into a hole. The researchers developed a suite of policies using machine learning training approaches with and without language information, and then compared the policies’ performance on a separate test set of nine tools with paired descriptions.This approach is known as meta-learning, since the robot improves its ability to learn with each successive task. It’s not only learning to use each tool, but also “trying to learn to understand the descriptions of each of these hundred different tools, so when it sees the 101st tool it’s faster in learning to use the new tool,” said Narasimhan. “We’re doing two things: We’re teaching the robot how to use the tools, but we’re also teaching it English.”The researchers measured the success of the robot in pushing, lifting, sweeping and hammering with the nine test tools, comparing the results achieved with the policies that used language in the machine learning process to those that did not use language information. In most cases, the language information offered significant advantages for the robot’s ability to use new tools.One task that showed notable differences between the policies was using a crowbar to sweep a cylinder, or bottle, along a table, said <a href="https://zhiyir.mycpanel.princeton.edu/" target="_blank">Allen Z. Ren</a>, a Ph.D. student in Majumdar’s group and lead author of the research paper.“With the language training, it learns to grasp at the long end of the crowbar and use the curved surface to better constrain the movement of the bottle,” said Ren. “Without the language, it grasped the crowbar close to the curved surface and it was harder to control.”The research was supported in part by the Toyota Research Institute (TRI), and is part of a larger TRI-funded project in Majumdar’s research group aimed at improving robots’ ability to function in novel situations that differ from their training environments.“The broad goal is to get robotic systems — specifically, ones that are trained using machine learning — to generalize to new environments,” said Majumdar. Other TRI-supported work by his group has addressed <a href="https://arxiv.org/abs/2202.05894" target="_blank">failure prediction</a> for vision-based robot control, and used an <a href="https://arxiv.org/abs/2107.06353" target="_blank">“adversarial environment generation”</a> approach to help robot policies function better in conditions outside their initial training.The article, Leveraging language for accelerated learning of tool manipulation, was presented Dec. 14 at the Conference on Robot Learning. Besides Majumdar, Narasimhan and Ren, coauthors include Bharat Govil, Princeton Class of 2022, and Tsung-Yen Yang, who completed a Ph.D. in electrical engineering at Princeton this year and is now a machine learning scientist at Meta Platforms Inc.In addition to TRI, support for the research was provided by the U.S. National Science Foundation, the Office of Naval Research, and the School of Engineering and Applied Science at Princeton University through the generosity of William Addy ’82.</body></html>

Princeton researchers have found that human-language descriptions of tools can accelerate the learning of a simulated robotic arm lifting and using a variety of tools. Animation by the researchers; GIF by Neil Adelantar

Exploring a new way to teach robots, Princeton researchers have found that human-language descriptions of tools can accelerate the learning of a simulated robotic arm lifting and using a variety of tools.

Words prove their worth as teaching tools for robots

robotic arms

Nathalie de Leon. Photo by Sameer A. Khan/Fotobuddy

A perfect diamond may be a jeweler’s dream, but for quantum scientists, beauty lies in a stone’s tiny defects. These flaws hold the power to trap and control single electrons, opening possibilities for both quantum computing and mapping molecular structures.

Defects in diamond could sense structures of single molecules

Using specially designed diamonds with nitrogen-vacancy centers, researchers at Princeton University and the University of Wisconsin-Madison have developed a technique to measure noise in a material by studying correlations, and they can use this information to learn the spatial structure and time-varying nature of the noise. In this image, a diamond with near-surface nitrogen-vacancy centers is illuminated by green laser light from a microscope objective lens. Photo by David Kelly Crow

Electromagnetic noise poses a major problem for communications, prompting wireless carriers to invest heavily in technologies to overcome it. But for a team of scientists exploring the atomic realm, measuring tiny fluctuations in noise could hold the key to discovery.

New technique reveals changing shapes of magnetic noise in space and time

Researchers at Princeton University and the University of Washington have developed an ultracompact camera the size of a coarse grain of salt. The system relies on a technology called a metasurface, which is studded with 1.6 million cylindrical posts and can be produced much like a computer chip. Image courtesy of the researchers

Micro-sized cameras have great potential to spot problems in the human body and enable sensing for super-small robots, but past approaches captured fuzzy, distorted images with limited fields of view.

Researchers shrink camera to the size of a salt grain

If you’ve been listening to more podcasts while stuck at home this year, you may have noticed a side effect of the uptick in virtual conversations: a decline in audio quality. Interviews conducted by phone or video chat often include background noise, reverberation and distortion.

More by Molly Sharlach

Defects in diamond could sense structures of single molecules

New technique reveals changing shapes of magnetic noise in space and time

Researchers shrink camera to the size of a salt grain