In this episode, we talk about how a fashion company spun-off from MIT has developed the world’s first truly one-size-fits-all garment.

Podcast: Novel Fabrics + Robots -- Sustainable One-size-fits-all Fashion Revolution

A pick and place robot in an industrial setup

Understanding the inception, evolution, composition, and types of pick and place robots and their impact on various industrial sectors 

Pick and Place Robots: An In-Depth Guide to Their Functionality and Applications

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/7a6ed573-f46a-40bb-b396-d38d1207aa2e?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Until recently, bespoke tailoring — clothing made to a customer’s individual specifications — was the only way to have garments that provided the perfect fit for your physique. For most people, the cost of custom tailoring is prohibitive. But the invention of active fibers and innovative knitting processes is changing the textile industry.“We all wear clothes and shoes,” says Sasha MicKinlay MArch ’23, a recent graduate of the MIT Department of Architecture. “It’s a human need. But there’s also the human need to express oneself. I like the idea of customizing clothes in a sustainable way. This dress promises to be more sustainable than traditional fashion to both the consumer and the producer.”McKinlay is a textile designer and researcher at the Self-Assembly Lab who designed the <a href="https://selfassemblylab.mit.edu/4d-knit-dress" target="_blank">4D Knit Dress</a> with Ministry of Supply, a fashion company specializing in high-tech apparel. The dress combines several technologies to create personalized fit and style. Heat-activated yarns, computerized knitting, and robotic activation around each garment generates the sculpted fit. A team at Ministry of Supply led the decisions on the stable yarns, color, original size, and overall design.“Everyone’s body is different,” says Skylar Tibbits, associate professor in the Department of Architecture and founder of the Self-Assembly Lab. “Even if you wear the same size as another person, you're not actually the same.”<iframe width="640" height="360" src="https://player.vimeo.com/video/893959788" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;&nbsp;</iframe>4D Knit Dress: Transforming Style. Video: Self-Assembly Lab<h2 id="isPasted">Active textiles</h2>Students in the Self-Assembly Lab have been working with dynamic textiles for several years. The yarns they create can change shape, change property, change insulation, or become breathable. Previous applications to tailor garments include <a href="https://selfassemblylab.mit.edu/active-textile-tailoring" target="_blank">making sweaters</a> and <a href="https://news.mit.edu/2022/form-fitting-face-mask-lavender-tessmer-0718" target="_blank">face masks</a>. Tibbits says the 4D Knit Dress is a culmination of everything the students have learned from working with active textiles.McKinlay helped produce the active yarns, created the concept design, developed the knitting technique, and programmed the lab’s industrial knitting machine. Once the garment design is programmed into the machine, it can quickly produce multiple dresses. Where the active yarns are placed in the design allows for the dress to take on a variety of styles such as pintucks, pleats, an empire waist, or a cinched waist.“The styling is important,” McKinlay says. “Most people focus on the size, but I think styling is what sets clothes apart. We’re all evolving as people, and I think our style evolves as well. After fit, people focus on personal expression.”<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEwMTM1NjY2MDgwLU1pbmlzdHJ5X0ZJTkFMX2RyZXNzZXMuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Designers explored this series of possible styles for the robotic heat activation of the 4D Knit Dress. Image courtesy of the Self-Assembly Lab.Danny Griffin MArch ’22, a current graduate student in architectural design, doesn’t have a background in garment making or the fashion industry. Tibbits asked Griffin to join the team due to his experience with robotics projects in construction. Griffin translated the heat activation process into a programmable robotic procedure that would precisely control its application.“When we apply heat, the fibers shorten, causing the textile to bunch up in a specific zone, effectively tightening the shape as if we’re tailoring the garment,” says Griffin. “There was a lot of trial and error to figure out how to orient the robot and the heat gun. The heat needs to be applied in precise locations to activate the fibers on each garment. Another challenge was setting the temperature and the timing for the heat to be applied.”It took a while to determine how the robot could&nbsp;reach all areas of the dress.“We couldn’t use a commercial heat gun — which is like a handheld hair dryer — because they’re too large,” says Griffin. “We needed a more compact design. Once we figured it out, it was a lot of fun to write the script for the robot to follow.”A dress can begin with one design — pintucks across the chest, for example — and be worn for months before having heat re-applied to alter its look. Subsequent applications of heat can tailor the dress further.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEwMTM1NjgyOTQyLVJvYm90aWNfYXJtX2F0X3dvcmsuanBlZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Robotic heat activation can be seen during a demonstration at Ministry of Supply in Boston. Photo: Olivia Mintz<h2>Beyond fit and fashion</h2>Efficiently producing garments is a “big challenge” in the fashion industry, according to Gihan Amarasiriwardena ’11, the co-founder and president of Ministry of Supply.“A lot of times you'll be guessing what a season's style is,” he says. “Sometimes the style doesn't do well, or some sizes don’t sell out. They may get discounted very heavily or eventually they end up going to a landfill.”“Fast fashion” is a term that describes clothes that are inexpensive, trendy, and easily disposed of by the consumer. They are designed and produced quickly to keep pace with current trends. The 4D Knit Dress, says Tibbits, is the opposite of fast fashion. Unlike the traditional “cut-and-sew” process in the fashion industry, the 4D Knit Dress is made entirely in one piece, which virtually eliminates waste.“From a global standpoint, you don’t have tons of excess inventory because the dress is customized to your size,” says Tibbits.McKinlay says she hopes use of this new technology will reduce the amount of waste in inventory that retailers usually have at the end of each season.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzEwMTM1NzEwMDIxLU1pbmlzdHJ5X0ZJTkFMX0RyYWZ0MDUgY29weS5qcGVnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">A complete 4D Knit Dress comes out of a Stoll industrial knitting machine after being produced in one piece, without any cutting or sewing. Photo courtesy of the Self-Assembly Lab.“The dress could be tailored in order to adapt to these changes in styles and tastes,” she says. “It may also be able to absorb some of the size variations that retailers need to stock. Instead of extra-small, small, medium, large, and extra-large sizes, retailers may be able to have one dress for the smaller sizes and one for the larger sizes. Of course, these are the same sustainability points that would benefit the consumer.”The Self-Assembly Lab has collaborated with Ministry of Supply on projects with active textiles for several years. Late last year, the team debuted the <a href="https://selfassemblylab.mit.edu/4d-knit-dress" target="_blank">4D Knit Dress</a> at the company’s flagship store in Boston, complete with a robotic arm working its way around a dress as customers watched. For Amarasiriwardena, it was an opportunity to gauge interest and receive feedback from customers interested in trying the dress on.“If the demand is there, this is something we can create quickly” unlike the usual design and manufacturing process, which can take years, says Amarasiriwardena.Griffin and McKinlay were on hand for the demonstration and pleased with the results. For Griffin, with the “technical barriers” overcome, he sees many different avenues for the project.“This experience leaves me wanting to try more,” he says.McKinlay too would love to work on more styles.“I&nbsp;hope this research project helps people rethink or reevaluate their relationship with clothes,” says McKinlay. “Right now when people purchase a piece of clothing it has only one ‘look.’ But, how exciting would it be to purchase one garment and reinvent it to change and evolve as you change or as the seasons or styles change? I'm hoping that's the takeaway that people will have.”

Caption:In late 2023 the MIT Self Assembly Lab and the high-tech fashion company Ministry of Supply debuted their 4D Knit Dress at the latter's store Boston, complete with a robotic arm working its way around a dress as customers watched. Credits:Image courtesy of MIT Self Assembly Lab

Developed by the Self-Assembly Lab, the 4D Knit Dress uses several technologies to create a custom design and a custom fit, while addressing sustainability concerns.

Is this the future of fashion?

In this episode, we delve into the cognitive strategies employed by researchers at EPFL to augment the human body with an additional robotic arm and learn about the profound impact of cognitive enhancements on the integration of advanced robotic limbs

Podcast: Why Robotic Limbs Are Critical To The Human Race

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/85f51311-32bd-4a6e-953c-0d6777ee40fc?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Neuroengineer Silvestro Micera develops advanced technological solutions to help people regain sensory and motor functions that have been lost due to traumatic events or neurological disorders. Until now, he had never before worked on enhancing the human body and cognition with the help of technology.<iframe width="640" height="360" src="https://www.youtube.com/embed/w78rRbaH7Wg?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-gtm-yt-inspected-8="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="720627293" title="Cognitive strategies for augmenting the body with a wearable, robotic arm">&nbsp;</iframe>Now in a study published in Science Robotics, Micera and his team report on how diaphragm movement can be monitored for successful control of an extra arm, essentially augmenting a healthy individual with a third – robotic – arm.“This study opens up new and exciting opportunities, showing that extra arms can be extensively controlled and that simultaneous control with both natural arms is possible,” says Micera, Bertarelli Foundation Chair in Translational Neuroengineering at EPFL, and professor of Bioelectronics at Scuola Superiore Sant’Anna.The study is part of the <a href="https://nccr-robotics.ch/research/wearable-robotics/third-arm/" target="_blank" rel="noopener">Third-Arm project</a>, previously funded by the Swiss National Science Foundation (NCCR Robotics), that aims to provide a wearable robotic arm to assist in daily tasks or to help in search and rescue. Micera believes that exploring the cognitive limitations of third-arm control may actually provide gateways towards better understanding of the human brain.Micera continues, “The main motivation of this third arm control is to understand the nervous system. If you challenge the brain to do something that is completely new, you can learn if the brain has the capacity to do it and if it’s possible to facilitate this learning. We can then transfer this knowledge to develop, for example, assistive devices for people with disabilities, or rehabilitation protocols after stroke.”“We want to understand if our brains are hardwired to control what nature has given us, and we’ve shown that the human brain can adapt to coordinate new limbs in tandem with our biological ones,” explains Solaiman Shokur, co-PI of the study and EPFL Senior Scientist at the Neuro-X Institute. “It’s about acquiring new motor functions, enhancement beyond the existing functions of a given user, be it a healthy individual or a disabled one. From a nervous system perspective, it’s a continuum between rehabilitation and augmentation.”To explore the cognitive constraints of augmentation, the researchers first built a virtual environment to test a healthy user’s capacity to control a virtual arm using movement of his or her diaphragm. They found that diaphragm control does not interfere with actions like controlling one’s physiological arms, one’s speech or gaze.In this virtual reality setup, the user is equipped with a belt that measures diaphragm movement. Wearing a virtual reality headset, the user sees three arms: the right arm and hand, the left arm and hand, and a third arm between the two with a symmetric, six-fingered hand.“We made this hand symmetric to avoid any bias towards either the left or the right hand,” explains Giulia Dominijanni, PhD student at EPFL’s Neuro-X Institute.In the virtual environment, the user is then prompted to reach out with either the left hand, the right hand, or in the middle with the symmetric hand. In the real environment, the user holds onto an exoskeleton with both arms, which allows for control of the virtual left and right arms. Movement detected by the belt around the diaphragm is used for controlling the virtual middle, symmetric arm. The setup was tested on 61 healthy subjects in over 150 sessions.“Diaphragm control of the third arm is actually very intuitive, with participants learning to control the extra limb very quickly,” explains Dominijanni. “Moreover, our control strategy is inherently independent from the biological limbs and we show that diaphragm control does not impact a user’s ability to speak coherently.”The researchers also successfully tested diaphragm control with an actual robotic arm, a simplified one that consists of a rod that can be extended out, and back in. When the user contracts the diaphragm, the rod is extended out. In an experiment similar to the VR environment, the user is asked to reach and hover over target circles with her left or right hand, or with the robotic rod.Besides the diaphragm, but not reported in the study, vestigial ear muscles have also been tested for feasibility in performing new tasks. In this approach, a user is equipped with ear sensors and trained to use fine ear muscle movement to control the displacement of a computer mouse.“Users could potentially use these ear muscles to control an extra limb,” says Shokur, emphasizing that these alternative control strategies may help one day for the development of rehabilitation protocols for people with motor deficiencies.Part of the third arm project, previous studies regarding the <a href="https://actu.epfl.ch/news/a-smart-artificial-hand-for-amputees-merges-user-a/" target="_blank">control of robotic arms</a> have been focused on helping amputees. The latest Science Robotics study is a step beyond repairing the human body towards augmentation.“Our next step is to explore the use of more complex robotic devices using our various control strategies, to perform real-life tasks, both inside and outside of the laboratory. Only then will we be able to grasp the real potential of this approach,” concludes Micera.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTY4NDIyNzU0LTgxOTJ4NTQ2NC5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Martina Gini controls a simplified robotic arm with breathing. © 2023 EPFL / Alain Herzog, CC-BY-SA<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTY4NTQ4Mjg5LWV6Z2lmLTUtNDVjNDI4M2Y2Zi5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">Silvestro Micera (left) and Leonardo Pollina (right) prepare for the third arm virtual reality experiment. © 2023 EPFL / Alain Herzog, CC-BY-SA

EPFL scientists show that breathing may be used to control a wearable extra robotic arm in healthy individuals, without hindering control of other parts of the body.

Cognitive strategies to augment the body with an extra robotic arm

Imagine that dinner could be prepared by a robot in your own home with just the push of a button. It sounds like something straight out of a far-flung future. Now, researchers at CMU are laying the groundwork for this technology with artificial intelligence.&nbsp;Alison Bartch, Atharva Dikshit, and Abraham George, under the guidance of <a href="https://engineering.cmu.edu/directory/bios/barati-farimani-amir.html" target="_blank">Amir Barati Farimani</a>, assistant professor of <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineeringOpens in new window</a>, trained a robot arm on different fruit slices and shapes to help improve its accuracy with cutting and moving real pieces of food on a cutting board.&nbsp;“So I’m cooking and have a cluttered cutting board. I’m reasoning about how to chop and how to select the objects that I want to cut, without cutting things that I don’t want,” said Bartch, a Ph.D. candidate in mechanical engineering. “It’s something that humans are pretty good at. And it’s a surprisingly difficult problem for robots.”<blockquote>It’s something that humans are pretty good at. And it’s a surprisingly difficult problem for robots.<cite>Alison Bartch<cite id="isPasted">,&nbsp;Ph.D. student<cite id="isPasted">, Mechanical Engineering</cite></cite> </cite></blockquote>The researchers combined two vision foundational models—models trained on large visual data sets—to help the robot arm recognize the shape and the type of fruit and vegetable slices. The AI would then determine what motion to use on these slices to produce the correct results with 70% accuracy.One of the models was only trained on whole fruits and vegetables, so the researchers needed to fine-tune the AI to more accurately recognize slices with a relatively small amount of newly collected data. According to Bartch, their fine-tuned framework can potentially be used for other related research; the combined foundational models, however, can be used for a variety of other AI projects in the future.“That’s just one of the things in a lot of robotics and machine learning—a lot of the data sets are already collected,” said George, also a Ph.D. candidate in mechanical engineering.&nbsp;Bartch said that the research focuses more on the “practicalities” of cutting food, such as producing the correct number of slices and moving pieces out of the way, making this research more applicable to a real kitchen than other research.“For this work, we focused more on the pipeline surrounding it, as opposed to the actual chop action,” Bartch said. “You want to pick which object to cut. You want to clear the scene before you cut. After you cut, detecting where the objects are then and continuing to plan. A lot of existing work focused on cutting assumed the position of the vegetable and that there is nothing else in the environment.”&nbsp;However, it will still be a while before AI will be chopping vegetables in homes. According to Bartch, there is still a lot of research and fine-tuning needed, and their robot still needs some human input. It’s difficult to replicate human reasoning in machines, even with tasks many humans find simple.“I think there is definitely a really long way to go before it’s in a person’s house.” Bartch said. “Because reasoning about cooking is really difficult. When you want to go into more practical situations, you need the system to be able to reason when the food is done.Atharva Dikshit, a master’s student in mechanical engineering, also contributed to this research. This research was submitted to the Journal of Intelligent and Robotic Systems.

Researchers at CMU combined two vision foundational models—models trained on large visual data sets—to help a robot arm recognize the shape and the type of fruit and vegetable slices.

Chop, chop: Improving food prep with the power of AI

Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know. The technique involves quantifying the fuzziness of human language and using that measurement to tell robots when to ask for further directions. Telling a robot to pick up a bowl from a table with only one bowl is fairly clear. But telling a robot to pick up a bowl when there are five bowls on the table generates a much higher degree of uncertainty — and triggers the robot to ask for clarification.Because tasks are typically more complex than a simple “pick up a bowl” command, the engineers use large language models (LLMs) — the technology behind tools such as ChatGPT — to gauge uncertainty in complex environments. LLMs are bringing robots powerful capabilities to follow human language, but LLM outputs are still frequently unreliable, said <a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of <a href="https://mae.princeton.edu/" target="_blank">mechanical and aerospace engineering</a> at Princeton and the senior author of a <a href="https://openreview.net/forum?id=4ZK8ODNyFXx" target="_blank">study</a> outlining the new method.“Blindly following plans generated by an LLM could cause robots to act in an unsafe or untrustworthy manner, and so we need our LLM-based robots to know when they don’t know,” said Majumdar.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1701898003603-knowno-ambiguities_WEB-2048x871.jpg" style="width: 766px;" class="fr-fic fr-dib">The researchers tested their method on a simulated robotic arm and on two types of robot hardware. The most complex experiments (left) involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. Another set of experiments used a tabletop robotic arm (right) tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm (center) added an additional layer of ambiguity. Images by the researchersThe system also allows a robot’s user to set a target degree of success, which is tied to a particular uncertainty threshold that will lead a robot to ask for help. For example, a user would set a surgical robot to have a much lower error tolerance than a robot that’s cleaning up a living room.“We want the robot to ask for enough help such that we reach the level of success that the user wants. But meanwhile, we want to minimize the overall amount of help that the robot needs,” said&nbsp;<a href="https://allenzren.github.io/" target="_blank">Allen Ren</a>, a graduate student in mechanical and aerospace engineering at Princeton and the study’s lead author. Ren received a&nbsp;<a href="https://www.corl2023.org/awards" target="_blank">best student paper award</a> for his Nov. 8 presentation at the Conference on Robot Learning in Atlanta. The new method produces high accuracy while reducing the amount of help required by a robot compared to other methods of tackling this issue.The researchers tested their method on a simulated robotic arm and on two types of robots at Google facilities in New York City and Mountain View, California, where Ren was working as a student research intern. One set of hardware experiments used a tabletop robotic arm tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm added an additional layer of ambiguity.The most complex experiments involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. In one example, a human asks the robot to “place the bowl in the microwave,” but there are two bowls on the counter — a metal one and a plastic one.The robot’s LLM-based planner generates four possible actions to carry out based on this instruction, like multiple-choice answers, and each option is assigned a probability. Using a statistical approach called conformal prediction and a user-specified guaranteed success rate, the researchers designed their algorithm to trigger a request for human help when the options meet a certain probability threshold. In this case, the top two options — place the plastic bowl in the microwave or place the metal bowl in the microwave — meet this threshold, and the robot asks the human which bowl to place in the microwave.In another example, a person tells the robot, “There is an apple and a dirty sponge … It is rotten. Can you dispose of it?” This does not trigger a question from the robot, since the action “put the apple in the compost” has a sufficiently higher probability of being correct than any other option.“Using the technique of conformal prediction, which quantifies the language model’s uncertainty in a more rigorous way than prior methods, allows us to get to a higher level of success” while minimizing the frequency of triggering help, said the study’s senior author&nbsp;<a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of mechanical and aerospace engineering at Princeton.Robots’ physical limitations often give designers insights not readily available from abstract systems. Large language models “might talk their way out of a conversation, but they can’t skip gravity,” said coauthor&nbsp;<a href="https://andyzeng.github.io/" target="_blank">Andy Zeng</a>, a research scientist at Google DeepMind. “I’m always keen on seeing what we can do on robots first, because it often sheds light on the core challenges behind building generally intelligent machines.”Ren and Majumdar began collaborating with Zeng after he gave a talk as part of the&nbsp;<a href="https://robo.princeton.edu/seminar" target="_blank">Princeton Robotics Seminar</a> series, said Majumdar. Zeng, who earned a computer science Ph.D. from Princeton in 2019, outlined Google’s efforts in using LLMs for robotics, and brought up some open challenges. Ren’s enthusiasm for the problem of calibrating the level of help a robot should ask for led to his internship and the creation of the new method.“We enjoyed being able to leverage the scale that Google has” in terms of access to large language models and different hardware platforms, said Majumdar.Ren is now extending this work to problems of active perception for robots: For instance, a robot may need to use predictions to determine the location of a television, table or chair within a house, when the robot itself is in a different part of the house. This requires a planner based on a model that combines vision and language information, bringing up a new set of challenges in estimating uncertainty and determining when to trigger help, said Ren.The paper, “<a href="https://openreview.net/forum?id=4ZK8ODNyFXx" target="_blank">Robots That Ask for Help: Uncertainty Alignment for Large Language Model Planners</a>,” was presented Nov. 8 at the Conference on Robot Learning. In addition to Ren, Zeng and Majumdar, coauthors include Anushri Dixit and Alexandra Bodrova of Princeton; and Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu and Dorsa Sadigh of Google DeepMind. The research was supported in part by the U.S. National Science Foundation and the Office of Naval Research.

Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know and ask for clarification from a human. Photo by the researchers

Modern robots know how to sense their environment and respond to language, but what they don’t know is often more important than what they do know. Teaching robots to ask for help is key to making them safer and more efficient.

How do you make a robot smarter? Program it to know what it doesn't know.

Imagine you’re visiting a friend abroad, and you look inside their fridge to see what would make for a great breakfast. Many of the items initially appear foreign to you, with each one encased in unfamiliar packaging and containers. Despite these visual distinctions, you begin to understand what each one is used for and pick them up as needed.Inspired by humans' ability to handle unfamiliar objects, a group from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) designed Feature Fields for Robotic Manipulation (<a href="https://arxiv.org/abs/2308.07931" target="_blank">F3RM</a>), a system that blends 2D images with foundation model features into 3D scenes to help robots identify and grasp nearby items. F3RM can interpret open-ended language prompts from humans, making the method helpful in real-world environments that contain thousands of objects, like warehouses and households.F3RM offers robots the ability to interpret open-ended text prompts using natural language, helping the machines manipulate objects. As a result, the machines can understand less-specific requests from humans and still complete the desired task. For example, if a user asks the robot to “pick up a tall mug,” the robot can locate and grab the item that best fits that description.“Making robots that can actually generalize in the real world is incredibly hard,” says Ge Yang, postdoc at the National Science Foundation AI Institute for Artificial Intelligence and Fundamental Interactions and MIT CSAIL. “We really want to figure out how to do that, so with this project, we try to push for an aggressive level of generalization, from just three or four objects to anything we find in MIT’s Stata Center. We wanted to learn how to make robots as flexible as ourselves, since we can grasp and place objects even though we've never seen them before.”<h2>Learning “what’s where by looking”</h2>The method could assist robots with picking items in large fulfillment centers with inevitable clutter and unpredictability. In these warehouses, robots are often given a description of the inventory that they're required to identify. The robots must match the text provided to an object, regardless of variations in packaging, so that customers’ orders are shipped correctly. For example, the fulfillment centers of major online retailers can contain millions of items, many of which a robot will have never encountered before. To operate at such a scale, robots need to understand the geometry and semantics of different items, with some being in tight spaces. With F3RM’s advanced spatial and semantic perception abilities, a robot could become more effective at locating an object, placing it in a bin, and then sending it along for packaging. Ultimately, this would help factory workers ship customers’ orders more efficiently.“One thing that often surprises people with F3RM is that the same system also works on a room and building scale, and can be used to build simulation environments for robot learning and large maps,” says Yang. “But before we scale up this work further, we want to first make this system work really fast. This way, we can use this type of representation for more dynamic robotic control tasks, hopefully in real-time, so that robots that handle more dynamic tasks can use it for perception.”The MIT team notes that F3RM’s ability to understand different scenes could make it useful in urban and household environments. For example, the approach could help personalized robots identify and pick up specific items. The system aids robots in grasping their surroundings — both physically and perceptively.“Visual perception was defined by David Marr as the problem of knowing ‘what is where by looking,’” says senior author Phillip Isola, MIT associate professor of electrical engineering and computer science and CSAIL principal investigator. “Recent foundation models have gotten really good at knowing what they are looking at; they can recognize thousands of object categories and provide detailed text descriptions of images. At the same time, radiance fields have gotten really good at representing where stuff is in a scene. The combination of these two approaches can create a representation of what is where in 3D, and what our work shows is that this combination is especially useful for robotic tasks, which require manipulating objects in 3D.”<h2>Creating a “digital twin”</h2>F3RM begins to understand its surroundings by taking pictures on a selfie stick. The mounted camera snaps 50 images at different poses, enabling it to build a <a href="https://www.matthewtancik.com/nerf" target="_blank">neural radiance field</a> (NeRF), a deep learning method that takes 2D images to construct a 3D scene. This collage of RGB photos creates a “digital twin” of its surroundings in the form of a 360-degree representation of what’s nearby.In addition to a highly detailed neural radiance field, F3RM also builds a feature field to augment geometry with semantic information. The system uses&nbsp;<a href="https://openai.com/research/clip" target="_blank">CLIP</a>, a vision foundation model trained on hundreds of millions of images to efficiently learn visual concepts. By reconstructing the 2D CLIP features for the images taken by the selfie stick, F3RM effectively lifts the 2D features into a 3D representation.<h2>Keeping things open-ended</h2>After receiving a few demonstrations, the robot applies what it knows about geometry and semantics to grasp objects it has never encountered before. Once a user submits a text query, the robot searches through the space of possible grasps to identify those most likely to succeed in picking up the object requested by the user. Each potential option is scored based on its relevance to the prompt, similarity to the demonstrations the robot has been trained on, and if it causes any collisions. The highest-scored grasp is then chosen and executed. To demonstrate the system’s ability to interpret open-ended requests from humans, the researchers prompted the robot to pick up Baymax, a character from Disney’s “Big Hero 6.” While F3RM had never been directly trained to pick up a toy of the cartoon superhero, the robot used its spatial awareness and vision-language features from the foundation models to decide which object to grasp and how to pick it up. F3RM also enables users to specify which object they want the robot to handle at different levels of linguistic detail. For example, if there is a metal mug and a glass mug, the user can ask the robot for the “glass mug.” If the bot sees two glass mugs and one of them is filled with coffee and the other with juice, the user can ask for the “glass mug with coffee.” The foundation model features embedded within the feature field enable this level of open-ended understanding.“If I showed a person how to pick up a mug by the lip, they could easily transfer that knowledge to pick up objects with similar geometries such as bowls, measuring beakers, or even rolls of tape. For robots, achieving this level of adaptability has been quite challenging,” says MIT PhD student, CSAIL affiliate, and co-lead author William Shen. “F3RM combines geometric understanding with semantics from foundation models trained on internet-scale data to enable this level of aggressive generalization from just a small number of demonstrations.” Shen and Yang wrote the paper under the supervision of Isola, with MIT professor and CSAIL principal investigator Leslie Pack Kaelbling and undergraduate students Alan Yu and Jansen Wong as co-authors. The team was supported, in part, by Amazon.com Services, the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research’s Multidisciplinary University Initiative, the Army Research Office, the MIT-IBM Watson AI Lab, and the MIT Quest for Intelligence. Their work will be presented at the 2023 Conference on Robot Learning.

Feature Fields for Robotic Manipulation (F3RM) enables robots to interpret open-ended text prompts using natural language, helping the machines manipulate unfamiliar objects. The system’s 3D feature fields could be helpful in environments that contain thousands of objects, such as warehouses. Images courtesy of the researchers.

By blending 2D images with foundation models to build 3D feature fields, a new MIT method helps robots understand and manipulate nearby objects with open-ended language prompts.

Using language to give robots a better grasp of an open-ended world

&nbsp; Achieving precision is&nbsp;a must&nbsp;in ultrasonic inspections, but&nbsp;it's&nbsp;not always straightforward due to&nbsp;some&nbsp;challenges like interference and noise. These issues can arise from various sources, including surface rust and environmental dust. To address these challenges,&nbsp;there's&nbsp;an exciting development on the horizon: the introduction of ultrasonic testing (UT) UAVs, often referred to as UT drones or UT flying robots. These innovative robots come equipped with&nbsp;state-of-the-art&nbsp;technology and, importantly, enhance safety by&nbsp;eliminating&nbsp;the need for human inspectors to enter high-risk areas.&nbsp;What's more, they have the capability to reach elevated inspection sites, effectively reducing the risks associated with working at heights while ensuring highly accurate results.&nbsp; &nbsp; &nbsp;One of the standout players in this field is SKYRON,&nbsp;the&nbsp;intelligent tethered&nbsp;flying&nbsp;robot equipped with a customized 38DL PLUS ultrasonic thickness gauge from Olympus. This technology not only enhances the safety of inspections but also elevates precision. The 38DL PLUS gauge is designed to tackle challenging conditions, whether that involves a rusty surface, a dusty environment,&nbsp;high temperatures, or access to difficult spaces. SKYRON can conduct meticulous inspections even in challenging locations.&nbsp;&nbsp; &nbsp; <img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1699488071311-1699488071311.png" class="fr-fic fr-dii">&nbsp;Moreover, what sets SKYRON apart is its ingenious built-in brush system, integrated into its rotary arm. This system is engineered to smooth rough surfaces and remove debris before an inspection, ensuring clean and&nbsp;accurate&nbsp;readings.&nbsp;It's&nbsp;a critical feature that ensures the reliability of the collected data. Furthermore,&nbsp;there's&nbsp;no need to interrupt the inspection process to change the brush head since&nbsp;it's&nbsp;seamlessly integrated into SKYRON's rotary arm. Additionally, before each inspection, SKYRON&nbsp;applies&nbsp;couplant&nbsp;gel, further streamlining the process.&nbsp;SKYRON brings even more improvements to the table. It boasts a continuous power supply, allowing it to fly for an unlimited duration, which is particularly&nbsp;advantageous&nbsp;for projects&nbsp;where&nbsp;relying on batteries would be inconvenient. The constant power supply also enables a higher payload capacity,&nbsp;facilitating&nbsp;the integration of various sensors, including advanced NDT,&nbsp;DFT,&nbsp;PAUT,&nbsp;and an array of robotic features.&nbsp;SKYRON is not just a UAV;&nbsp;it's&nbsp;a versatile platform that can seamlessly incorporate various sensors and features. It ensures safe and secure operations, minimizing the risk of unexpected issues during flights. Moreover, it guarantees&nbsp;accurate&nbsp;contact-based inspections of critical structures. SKYRON&nbsp;can fly,&nbsp;attach&nbsp;and measure multiple assets, capturing precise thickness measurements and conducting corrosion checks. What truly sets it apart is its adaptability, making it highly effective in challenging environments, and demanding conditions where&nbsp;maintaining&nbsp;a clear line of sight can be a challenge. In summary, SKYRON stands as a valuable&nbsp;tool&nbsp;for addressing a variety of inspection challenges and spearheading a new era of ultrasonic precision.&nbsp;In conclusion, SKYRON, equipped with its remarkable 38DL PLUS ultrasonic thickness gauge and ingenious built-in brush system,&nbsp;embodies&nbsp;the future of ultrasonic inspections. The seamless integration of these advanced features not only enhances the safety of inspections but also ensures the highest level of precision, regardless of the conditions. SKYRON&nbsp;doesn't&nbsp;just&nbsp;prepares&nbsp;a new era of ultrasonic precision; it serves as a shining example of what's possible when technology and innovation join forces. With its adaptability and versatility,&nbsp;it&nbsp;is ready&nbsp;to tackle a wide array of inspection challenges and&nbsp;help to&nbsp;redefine the standards of excellence in the field. As the skies&nbsp;open up&nbsp;to these remarkable flying robots.&nbsp;&nbsp;

Achieving precision is a must in ultrasonic inspections, but it's not always straightforward

Advancing Ultrasonic Inspections

At EPFL's&nbsp;<a href="https://www.epfl.ch/labs/create/" target="_blank">CREATE lab</a>, under the guidance of Josie Hughes, a breakthrough has been made in the realm of soft robotics. Drawing inspiration from the versatile movement of elephant trunks and octopus tentacles, the team introduced the trimmed helicoid — a novel robotic structure that promises greater compliance and control in robotic designs. With a blend of keen biological observation and computational modeling, the researchers have now unveiled a soft robot arm capable of intricate tasks, ensuring safer human-robot interactions. The findings, detailing both the structure and methodology, are a collaboration with the Department of Cognitive Robotics at TU Delft and were published in Nature's new journal, npj Robotics.Professor Hughes highlighted the importance of this development: "Through the invention of a new architectured structure, the trimmed helicoid, we've designed a robot arm that excels in control, range of motion, and safety. When the novel architecture is combined with distributed actuation— where multiple actuators are placed throughout a structure or device—this robot arm has a vast range of motion, high precision, and is inherently safe for human interaction."Whereas traditional robots are rigid, often making them unsuitable for delicate tasks or close human interactions, CREATE's soft robot arm is designed for safer interactions with humans and adaptability to a wider range of tasks. With an unprecedented combination of flexibility and precision, the soft and compliant nature of the arm reduces potential risks during human-robot interactions. This opens doors for its application in healthcare, elderly care, and more. Unlike their rigid counterparts, the soft robot arm can adapt to different shapes and surfaces, making it an ideal tool for intricate tasks like picking fruits or handling fragile items. In industry, it might become the go-to solution for delicate assembly lines, working alongside humans, augmenting their capacity instead of replacing them. The agricultural industry could also benefit from its gentle touch in handling crops, accompanying workers to lessen their workload during intense harvesting periods.The crux of the research lies in the robotic arm’s novel architecture. The researchers have creatively modified a spring-like spiral, which they call a 'helicoid', by trimming parts of it to give it diverse functionalities. This seemingly simple act has allowed them to precisely control how flexible or stiff the spiral becomes in different directions. By adjusting its shape, they can make the inner part resistant to being squashed and the outer part flexible enough to bend easily. With this special design, they've created a soft robot that can move and act in ways previously unseen, showing the kind of dexterity and soft touch found in nature, like in an elephant's trunk or an octopus's tentacle."By observing these animals and developing a novel architectured structure, we aim to mimic this range of motion and control found in Nature," Josie Hughesnoted. To do so, the team employed advanced computer modeling to turn observations into tangible results. Using these models, they iteratively tested their innovative spiral designs — into a final trimmed helicoid shape. Qinghua Guan and Francesco Stella, who spearheaded the actual creation of the robot, gave insights into the design and optimization process: "We introduce a specific surface into the computer model, then trim and adjust. Computational methods guide us, helping assess the optimal geometric structure for maximum workspace and compliance." The result? A robotic creation, drawing from nature, but refined with precise human ingenuity and computer modeling. “In the end, our computer models were accurate to the point where we only had to build one version of the arm.”The advancements at EPFL's CREATE lab symbolize a pivotal shift in the robotics field. Traditional robotic applications, dominated by rigid mechanics, could see a shift towards this softer, more human-friendly counterpart. A patent has been filed for this first commercial soft manipulator, and a EPFL and TU Delft joint start-up has been launched under the name Helix Robotics. As Professor Hughes aptly summarized, "Merging keen observations from nature with precise computational modeling has unveiled the potential of soft robotics for future commercial applications. As we move forward, our aim is to bring robots closer to humans, not just in proximity but in understanding and collaboration. We hope that this soft robotic arm exemplifies a future where machines assist, complement, and understand human needs more deeply than ever before."<hr>Professor Josie Hughes will be showing the Helix robot along with several other of her creations at the&nbsp;<a href="https://swissroboticsday.ch/" target="_blank" rel="noopener">Swiss Robotics Day</a> at ETHZ on November 3rd. Many other EPFL professors will be showcasing their work. The day is dedicated to the best of Swiss robotics across industry, startups, universities, research labs, nonprofits, and government. The event features high-profile speakers (Boston Dynamics, Disney) and panelists, along with startup pitches and 50+ exhibition booths with live demos.ReferencesGuan, Q., Stella, F., Della Santina, C. et al. Trimmed helicoids: an architectured soft structure yielding soft robots with high precision, large workspace, and compliant interactions. npj Robot 1, 4 (2023). https://doi.org/10.1038/s44182-023-00004-7

EPFL researchers have designed a bio-inspired robot with a novel trimmed helicoid structure that allows for a wide range of motion and safe interaction with humans.

Soft, elephant trunk-like robot for close interaction with humans

<h3 id="isPasted">Let’s explore some methods on how to achieve this.</h3>First and foremost, there exists the “Straightforward Programming” approach. This means: getting a skilled person to meticulously describe a task, to such an extent that one can write either a simple script or an intricate algorithm that can guide the robot to do the task. This method is great and highly effective when the task is well-defined and specific, something often done in industrial automation. However, the complexity can escalate rapidly, making it challenging. Thus, if the approach is successful, that’s excellent – but if not, we must ponder – what alternative strategies can we employ?Secondly, an alternative approach to consider is “Programming by Demonstration“. This method involves guiding the robot in performing the task. By equipping the robot with advanced force sensing capabilities and vision algorithms, this technique can prove to be quite effective. However, it’s crucial to note that its effectiveness is reliant on the robot’s abilities. Usually, robots are equipped with rudimentary grippers rather than adept, dexterous hands, which can limit the complexity of tasks they can perform.&nbsp;Okay, so how can we effectively record human actions for a robot to mimic?Ideally, we’d guide a human through the task, document it in detail, and then replay it to the robot. We might even consider feeding this information into a machine-learning system, enabling it to learn from our human-guided demonstration. Alternatively, at the very least, we can analyze these human activities, deconstruct them into simpler parts, and use this information to develop more efficient algorithms in the way that a song is deconstructed in a recording studio.&nbsp;OK, so how about, filming the person doing the task and using “machine vision” to track what they are doing?It’s certainly a good starting point. This approach provides valuable insights into the nuances of human behaviour – such as hand-switching moments, object-holding techniques, and tool usage – But our experience is that it’s really hard to track human movement&nbsp;using cameras. You need clear lines of sight, and you’ll have lots of trouble tracking the fine movements of the fingers and hands.&nbsp;Even the most advanced image-based tracking systems, while capable of monitoring hands with minimal occlusion, fail to consistently track all joint movements. Consequently, the resultant data tends to be somewhat inconsistent and noisy.&nbsp;Alright, what about using motion capture?Motion capture systems, commonly employed in film production and sports science, operate by attaching easily visible markers to the individual, such as passive elements like white dots or active elements like pulsed LEDs. This method certainly helps with the accuracy of tracking. However, it doesn’t entirely overcome the previously mentioned issues related to occlusion. Additionally, calibrating the optical systems to an error margin of less than a few millimetres is quite challenging – which means that if the task involves intricate movements, such as rolling a pen between fingers, it’s likely it’ll miss a lot.&nbsp;&nbsp;Okay. I need it to be accurate, what’s my best option then?For precise human movement tracking, your best bet is to use a tight-fitting movement-tracking&nbsp;<a href="https://www.shadowrobot.com/shadow-hand-glove/" target="_blank">glove</a>. This glove can incorporate a multitude of techniques to achieve such a feat, some even providing additional features to assist the operator (and some that make it more difficult for the operator). For instance, some gloves can relay a sense of weight, touch, or stiffness, all functionalities grouped under the term ‘haptics.’ The realm of haptics is broad, including diverse technologies from ultrasound devices transmitting signals to fingers, electrical skin stimulation, vibrating piezoelectric elements, to inflating air balloons, miniature motors, or linear actuators, among many others.Gloves that measure human hand movement can do it by utilising:<ul><li dir="ltr">Sensors that fit around the joints and track the specific movement of the joint</li><li dir="ltr">Mechanical linkages that go from one end of the finger to the other and track the whole movement of the finger</li><li dir="ltr">Inertial sensors (that clever combination of accelerometer, gyroscope and possibly magnetometer) - worn at various points on the hand to follow the movement</li><li dir="ltr">Wireless measuring sensors - detecting the location of the fingers relative to a source worn on the hand somewhere.</li><li dir="ltr">Cameras that look away from the hand and track the movement of the world. (I’m not sure I’ve seen this one used in anger but I’m sure it’s possible!)</li></ul> Even then several variations of these technologies exist. For instance, some sensors measure the flex of a glove, correlating it to joint movement. Other types of sensors stretch and track their own deformation to gauge distance. Occasionally, mechanical linkages are placed over individual joints for precise motion tracking.&nbsp;Each of these methods presents a unique approach to capturing and interpreting human movement<h3 id="isPasted">Our experience:</h3>From our extensive experience, we have found that accurately measuring the mechanical properties of the human hand can be challenging, as it often leads to significant inaccuracies. The placement of the measurement sensor is crucial, and unfortunately, the potential for slippage is high – gloves and similar apparatus tend to shift excessively, complicating the data collection. Relating the measurements to the actual rotation of a finger joint’s axis, especially the thumb, is another daunting task!<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NTM2Nzk3ODE3LWxvbmcgLSBQb2xoZW11cyAmIFNoYWRvdyAoMTkyMCB4IDEwODAgcHgpLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 766px;" class="fr-fic fr-dib">Mechanical linkages, although valuable, can quickly become cumbersome. There is also a risk that they could discomfort the operator to the point where task completion becomes impossible (too heavy, too rigid etc). Inertial sensors present a viable option; however, (once again) preventing them from drifting is generally difficult, and the precision level they offer often falls short of what we needWe have found that wireless measurement sensors, particularly those from&nbsp;<a href="https://www.shadowrobot.com/shadow-robot-polhemus/" target="_blank">Polhemus</a>, give us excellent results. These sensors entail a transmitting source worn on the palm and receivers located at the fingertips, providing us with high-quality data for hand movement tracking. However, there remains the challenge of locating the hand within the wider three-dimensional space. For this, we have found that the HTC Vive is an excellent solution and so you will often see the Vive tracker featured in our demonstrations.Now – what we want is to accurately know where to put the robot hand – which isn’t quite the same thing as knowing accurately where the human hand is. We have to take the data from the human hand and map it to the robot hand. This is a kinematic challenge – the movements of the human and the movements of the robot don’t agree entirely, and we need to convert one to the other. The best method for mapping human data to a robot hand will depend on the specific application. For example, if you need to control a robot’s hand in real-time, then direct mapping may be the best option. If you are only interested in controlling a robot’s hand for offline analysis, then inverse kinematics or hybrid mapping may be more feasible options. Needless to say, the more complex the robot the more challenging this becomes. Lucky for us complex robots are a speciality at Shadow!&nbsp;So, the human wears a set of gloves, and they control the robot to do the task. After completing a few rudimentary training tasks the operator gets the hang of the process and they are doing it pretty well. What do we do to learn from this?We can precisely monitor the instructions sent to the robot and reproduce them. This is ideal when operating in a controlled environment where all variables remain consistent – a scenario commonly found in an automation setting within a quality assurance laboratory. In the past, we employed this technique to build a functional robotic kitchen. While it allows you to execute repetitive tasks, it lacks the ability to cope with variation.We can capture and analyze data from numerous tests, assessing them for consistency. If the results are largely similar, they can be directly applied. However, if they vary a lot, we can delve into our algorithm toolbox to try and understand these discrepancies. This is where it helps to have a <a href="https://www.shadowrobot.com/artificial-intelligence-machine-learning/" target="_blank">wide range of data</a> from <a href="https://www.shadowrobot.com/dexterous-hand-series/" target="_blank">the robot</a>, from force, <a href="https://www.shadowrobot.com/shadow-robot-11-lessons-about-tactile-sensors/" target="_blank">touch</a>, and joint sensors. The question arises: Can we employ data from vision, such as cameras aimed at the work area, to discern what’s different? To be effective, this vision data needs to be synchronized with the robot sensor data, necessitating a shared framework for all data streams.We can gather data from multiple tests and use it by “Feeding it to the AI” – meaning use it to instruct a machine learning system on how to replicate similar movements from similar data.This can be done in several ways.We might train a traditional learning model on the datasets so that similar inputs generate similar outputs. Alternatively, if we have enough data, we could use a transformer model to generate movements. We can even break down the data into smaller segments and train a network to assemble these fragments into the correct sequences.Given the impressive capabilities of modern machine learning systems in generating text, images, video, and even code, this approach appears highly promising – and it’s only the beginning!So, does this sound like it could help with a problem that you face? If it does, then let’s have&nbsp;<a href="https://www.shadowrobot.com/blog/data-humans-to-robots/hand@shadowrobot.com" target="_blank">a chat to</a> see if the Shadow Robot team can help!

Humans excel at manual tasks, and ideally, robots should surpass us in these functions. We have the advantage due to our superior dexterity, and we can perform tasks that robots should, in theory, be capable of.
As we advance robotics our goal is to effectively transfer our human skills to robots.

Getting Data from Humans to Robots

This article originally appeared in <a href="https://news.itmo.ru/en/science/life_science/news/13440/" target="_blank" rel="noopener noreferrer">ITMO News</a> and has been republished with permission.<hr>A family of medical devices named implants are manufactured to replace a missing organ or its parts. Dental implants are implanted into the jaw to support replacement teeth. Apart from dentistry, implants are also used to resurface a knee, repair skull bones, as well as for other purposes.&nbsp;All medical devices are customized to each patient's unique characteristics. In the case of dental implants, doctors must be aware of a person's skull bones, gums, and roots to determine the appropriate form and size of the future implant.&nbsp;To make an implant last longer, as well as heal faster and easier, its surface is enhanced with functional features, like antibacterial properties and biocompatibility. The latter is usually accomplished by roughening up the implant's surface with sandblasting, followed by acid and anodizing to make the surface antimicrobial. So far, there is no one-stop complex to execute both procedures – except for the one currently under development at ITMO University.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3MTk5NTcwODA3LXAxMzQ0MC5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 766px;" class="fr-fic fr-dib">The research team (from left to right): Yulia Karlagina, Galina Romanova, Andrey Lisichnikov, and Fedor Inochkin. Photo courtesy of Evgeniy Shamshin<h2 dir="ltr" id="isPasted">The laser-based complex</h2>Researchers from ITMO’s Institute of Laser Technologies and their colleagues from Laser Center designed a multiuse robotic complex for surface treatment of diverse medical materials. According to the developers, the technology is equipped with a fiber laser system and possesses several advantages:&nbsp;Versatility. The technology is well-suited for use with medical devices such as scalpels and surgical scissors, as well as a variety of implants, including teeth, skull, hip, and knee replacements.&nbsp;Turnkey treatment in one system. The team created and merged several laser treatment options that are used consecutively:<ul><li dir="ltr">The technology for antibacterial surface properties prevents bacterial complications, which lead to implant rejection. Laser heating makes oxide layers grow on the surface of a titanium implant, which demonstrates bactericidal properties when exposed to a UV source.</li><li dir="ltr">The technology for biocompatible surface property. The laser is applied to generate a relief with unique properties on the surface, followed by the laser ablation process, which creates a biomimetic micro- and nanorelief that promotes excellent adhesion of proteins in the early stages of osseointegration, cell proliferation, and differentiation, ensuring implant success.&nbsp;</li><li dir="ltr">The technology for applying non-toxic identification labels on medical devices. Legally binding product labeling makes it possible to track the life cycle of a product, from its early production to the end user.&nbsp;</li></ul><blockquote>“We came up with a versatile device that makes it possible to functionalize a wide spectrum of medical equipment without any additional consumables or tools. The single system can boast several treatment options, related to the biocompatible and antibacterial properties of products, their roughness, as well as labeling,” states&nbsp;Yulia Karlagina, a junior researcher at ITMO’s Institute of Laser Technologies.</blockquote><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1697199551134-ezgif-2-348e4b0195.gif" style="width: 766px;" class="fr-fic fr-dib">The robot laser complex in action. Video courtesy of Evgeniy ShamshinCompatibility with custom devices. All it takes is to load a 3D model of the implant into the program and indicate which elements of the 3D electronic geometric model need to be processed. The software was developed by the team at ITMO.&nbsp;The complex consists of a six-axis robotic manipulator and a laser scanning system mounted on the robot's arm, which ensures high accuracy of laser positioning, achieved by optical methods, and universality of robotic systems. Its ability to focus a laser beam on a product’s surface in all six spatial coordinates makes it possible to process products of complex shapes in one technological cycle.The team incorporated a computer vision system to make it easier for the operator to determine the position of the product in reality. In addition to its position, the system reports on its orientation and geometrical parameters to ensure precision laser processing.<blockquote>“The existing robotic solutions are aimed at mass-production items. Once adjusted for a specific device, they perform a series of programmed actions again and again. This approach is not suitable for small-scale or one-of-a-kind equipment, whereas our technology can be easily customized for a new product,” explains&nbsp;Fedor Inochkin, a researcher at ITMO’s Institute of Laser Technologies<img data-fr-image-pasted="true" alt="" src="https://news.itmo.ru/en/science/life_science/news/13440/" class="fr-fic fr-dii"></blockquote><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3MTk5Mzg4MjEyLTEzNDQwMDEuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 766px;" class="fr-fic fr-dib">An operator positions a medical device and loads a 3D model into the system, selects the areas to be processed, and starts the program. The robot manipulator travels from one area to another with a laser beam. The procedure time varies depending on the chosen technology and item size. Photo courtesy of Evgeniy ShamshinA compact automated laser system for dental implants and their components is another device created by the team to serve exclusively dental purposes.&nbsp;<blockquote>“The systems have different functions, which affects their price and audience. If a customer needs to process a certain implant, like a dental one, they should purchase the second device, whereas the first one will be more beneficial with a wide range of items, including custom ones. For instance, a person who suffered from head injury can receive an individually designed skull implant. The multiuse system allows specialists to incorporate a desired set of properties into personalized implants in accordance to their unique shape,” notes Yulia Karlagina.&nbsp;</blockquote><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3MTk5NDEwOTc4LTEzNDQwMDIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 766px;" class="fr-fic fr-dib">An operator positions an implant, closes the complex’s cover, chooses the type of device under study and processing mode in the program, and, finally, starts the program. The implant is ready in 10-15 minutes. Photo courtesy of Evgeniy Shamshin<h2 dir="ltr" id="isPasted">Prospects of the technology</h2>The ITMO researchers in collaboration with their co-authors from Samara State University have carried out preliminary testing of both complexes, in particular preclinical trials of the processed medical devices. All samples were successfully tested in vitro and in vivo and are recommended for clinical trials.&nbsp;In the future, the developers plan to run acceptance tests to ensure the complexes are ready for operation and mass production. The team also intends to attract new industrial partners. In the long run, the systems are expected to be used in the production of medical equipment and implants.&nbsp;<blockquote>“We already have surface functionalization technologies for implants and soon we, with our colleagues from Laser Center, will start to look for new partners who will integrate them into their productions. If requested, we can also modify our products to the client’s needs – for instance, we can add a second robot manipulator that will upload new implants into the system instead of an operator,” notes Galina Romanova, a senior researcher at ITMO’s Institute of Laser Technologies.&nbsp;</blockquote><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3MTk5NDQxMTQzLTEzNDQwMDMuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 766px;" class="fr-fic fr-dib">Galina Romanova. Photo courtesy of Evgeniy ShamshinThe project is implemented as part of the Government Decree No. 218 on request by Laser Center. Additionally, the developers continue to run research within ITMO’s 2030 Development Strategy.&nbsp;The research team has been working on the processing of medical titanium alloys for several years now. Together with their partners from Lenmiriot, they <a href="https://news.itmo.ru/en/science/new_materials/news/9655/" target="_blank">developed</a> a design of an implant surface that helps implants heal faster without inflammation. In 2022, Lenmiriot <a href="https://news.itmo.ru/en/startups_and_business/partnership/news/12313/" target="_blank">announced</a> the market launch of the dental implants made with the new laser technology.

The robot laser complex. Photo courtesy of Evgeniy Shamshin

Researchers from ITMO University have created a multipurpose robot complex for laser treatment of medical device surfaces, like those of dental and skull implants. The designed technology can be utilized to imbue metal implants with antibacterial and biocompatible properties, as well as mark medical items. All one needs to do is load a 3D model of an implant into a program, set a processing trajectory, and pick a surface attribute of choice.

A Multiuse Robot for Medical Applications Designed at ITMO

Effective time management is essential when conducting inspections on container ships, as it directly influences the cost-effectiveness and competitiveness of maritime cargo transportation. Inspection plays a&nbsp;crucial&nbsp;role in ensuring container ship safety, averting cargo damage,&nbsp;maintaining&nbsp;compliance with environmental rules, and preserving the vessel's value.&nbsp;Ensuring the safety and dependability of container ships involves employing fundamental methods such as contact-based Ultrasonic Thickness (UT) inspections and visual assessments.&nbsp;These inspections proactively&nbsp;identify&nbsp;potential issues before they escalate into significant concerns, thereby aiding in the prevention of maritime mishaps.&nbsp;&nbsp; Regularly inspecting cargo vessels becomes a complex task due to their significant size,&nbsp;necessitating&nbsp;careful maneuvering to access various compartments, structures, and cargo holds.&nbsp;Traditional approaches, like rope access and scaffolding, come with inherent risks and time consumption. Addressing these hurdles, innovative flying robots have&nbsp;emerged&nbsp;as&nbsp;an optimal&nbsp;solution for ship inspections.&nbsp;Avestec's&nbsp;intelligent airborne robot, SKYRON, effectively addresses the difficulties&nbsp;encountered&nbsp;in ship inspections.&nbsp;Watch the video above to see how SKYRON performed UT and visual inspections on a container ship in Vancouver, Canada. It focused on checking the back wall of the cargo hold, the guides for the columns, and the ship's side facing the water. This inspection happened while the ship was parked at the port to unload cargo, highlighting the importance of saving time. SKYRON is a highly advanced flying robot designed for various inspections, reducing risks for&nbsp;humans&nbsp;and cutting down on inspection time and costs.&nbsp;

Watch How Flying Robots Like SKYRON Are Redefining Safety Checks at Sea 

From Robo-Flyers to Smart Scans, Unveiling the Future of Vessel Safety and Performance

<img alt="A blue line drawing of a person wearing a hard hatDescription automatically generated" src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1695324692147-1695324692147.png" id="isPasted" class="fr-fic fr-dii">For the management of hydropower conveyance systems within pulp and paper facilities, surge tanks stand as critical components, serving as key players for efficient operations. A robust maintenance strategy and a secure work environment are paramount for these systems to thrive. Non-destructive testing (NDT) inspections emerge as a vital practice in maintaining surge tank performance. However, circumstances can be challenging, as showcased in a recent case undertaken by Avestec.&nbsp;The situation presented a dual challenge. Initially, accessing the surge tanks proved intricate due to obstructive towering trees on one front. Subsequently, a haunting specter of potential asbestos contamination&nbsp;emerged&nbsp;on the opposing&nbsp;side, adding&nbsp;an element of danger that&nbsp;couldn&#39;t&nbsp;be ignored. The confluence of these obstacles demanded an innovative approach, one that safeguarded&nbsp;workers&rsquo;&nbsp;well-being while ensuring thorough assessments of the tanks.&nbsp;<img alt="A group of men in safety gearDescription automatically generated with medium confidence" src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1695324672404-1695324672404.jpeg" class="fr-fic fr-dii">&nbsp;In response to these complex challenges,&nbsp;Avestec&nbsp;utilized&nbsp;the capabilities of SKYRON, an intelligent flying robot.&nbsp;SKYRON&nbsp;emerged&nbsp;as the perfect&nbsp;inspection&nbsp;tool, smoothly navigating the intricate&nbsp;environment&nbsp;and flawlessly conducting ultrasonic thickness tests on the tank shell.&nbsp;By doing so, it enabled inspection without exposing human personnel to the hazards.&nbsp;Operating SKYRON was a precise coordination of human&nbsp;expertise&nbsp;and&nbsp;cutting-edge&nbsp;technology.&nbsp;A skilled pilot and inspector collaborated to guide the flying robot with precision. While the robot soared over the dense canopy of trees, the team&nbsp;maintained&nbsp;a safe distance, evading the potential asbestos threat. A&nbsp;first-person&nbsp;view&nbsp;camera&nbsp;system&nbsp;interface&nbsp;facilitated&nbsp;real-time monitoring as SKYRON ventured beyond the limits of direct line of sight.&nbsp;The&nbsp;results&nbsp;of&nbsp;Avestec&#39;s&nbsp;inspection were&nbsp;truly&nbsp;remarkable.&nbsp;Within&nbsp;a single day, SKYRON&nbsp;conducted thorough&nbsp;inspections,&nbsp;collecting&nbsp;577 ultrasonic thickness readings in&nbsp;60 minutes&nbsp;flight time.&nbsp; <img alt="A large metal tower with a ladderDescription automatically generated" src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1695324672418-1695324672418.jpeg" class="fr-fic fr-dii">&nbsp;This achievement marked a transformative change in the methods of inspecting surge tanks. Through overcoming challenging conditions and executing seamless inspections, SKYRON proves its advantages over conventional approaches, triumphing even in the most demanding environments.&nbsp;

SKYRON's Cutting-Edge Solution Overcomes Hazardous Obstacles and Delivers Rapid Results

Revolutionizing Tank Inspections: How Intelligent Flying Robots Ensure Safety and Efficiency

Offshore platform inspections have always been a demanding challenge. The extreme conditions, towering structures, and inaccessible spots make it a true test of human endurance and safety.&nbsp;Conventional methods involved ropes and scaffolding, costing time, money, and, most importantly, risking the lives of inspectors.&nbsp;That&#39;s&nbsp;where the&nbsp;game-changer&nbsp;comes in, offering a revolutionary solution that puts human safety at the forefront.&nbsp;Flying robots use&nbsp;cutting-edge&nbsp;technology to navigate the complexities of offshore platforms effortlessly. This transformational tool redefines the inspection landscape, introducing a more efficient, cost-effective,&nbsp;and, above all, safer&nbsp;way of working.&nbsp;The challenges of elevated heat make inspections hard. But now, the use of flying robots like SKYRON has ushered in a notable improvement in inspection efficiency, simultaneously mitigating inherent risks. &nbsp;These robotic systems guarantee precise measurements without compromising the well-being and safety of inspectors. This transformative approach empowers the inspection workforce.&nbsp;Watch the video above to witness&nbsp;SKYRON&nbsp;in action.&nbsp;It&#39;s&nbsp;not just about technology;&nbsp;it&#39;s&nbsp;about rewriting the rules of offshore platform inspections. As we step into this new era, safety takes center stage, making sure that inspectors no longer face unnecessary risks.&nbsp;It&#39;s&nbsp;an exciting shift towards a safer and more efficient future, where innovation meets the challenges head-on.&nbsp;

SKYRON marked history as the pioneer in contact-based NDT inspection on a massive offshore platform, setting a new standard for safety and efficiency. 

robotic arms

Profiles