In this episode, we discuss research coming out of Columbia University to better connect humans with robots by having them detect, process, and respond to facial expressions!<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/f153920a-aca0-4937-863a-2cc925d11434?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">EPISODE NOTES</h3><a href="https://www.wevolver.com/article/robot-can-you-say-cheese" target="_blank">https://www.wevolver.com/article/robot-can-you-say-cheese</a>Become a founding reader of our newsletter: read.thenextbyte.com

In this episode, we discuss research coming out of Columbia University to better connect humans with robots by having them detect, process, and respond to facial expressions!

Podcast: Finally! A Humanoid Robot You Can Trust

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/f153920a-aca0-4937-863a-2cc925d11434?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Columbia engineers build Emo, a silicon-clad robotic face that makes eye contact and uses two AI models to anticipate and replicate a person’s smile before the person actually smiles -- a major advance in robots predicting human facial expressions accurately, improving interactions, and building trust between humans and robots.<section id="isPasted">What would you do if you walked up to a robot with a human-like head and it smiled at you first? You’d likely smile back and perhaps feel the two of you were genuinely interacting. But how does a robot know how to do this? Or a better question, how does it know to get you to smile back?While we’re getting accustomed to robots that are adept at verbal communication, thanks in part to advancements in large language models like ChatGPT, their nonverbal communication skills, especially facial expressions, have lagged far behind. Designing a robot that can not only make a wide range of facial expressions but also know when to use them has been a daunting task.&nbsp;</section><section><h2>Tackling the challenge</h2>The <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Creative Machines Lab</a> at <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Columbia Engineering</a> has been working on this challenge for more than five years. In a new study published today in <a href="https://www.science.org/doi/10.1126/scirobotics.adi4724" rel="nofollow" target="_blank">Science Robotics</a>, the group unveils Emo, a robot that anticipates facial expressions and executes them simultaneously with a human. It has even learned to predict a forthcoming smile about 840 milliseconds before the person smiles, and to co-express the smile simultaneously with the person.The team, led by <a href="https://www.engineering.columbia.edu/faculty/hod-lipson" rel="nofollow" target="_blank">Hod Lipson</a>, a leading researcher in the fields of artificial intelligence (AI) and robotics, faced two challenges: how to mechanically design an expressively versatile robotic face which involves complex hardware and actuation mechanisms, and knowing which expression to generate so that they appear natural, timely, and genuine. The team proposed training a robot to anticipate future facial expressions in humans and execute them simultaneously with a person. The timing of these expressions was critical -- delayed facial mimicry looks disingenuous, but facial co-expression feels more genuine since it requires correctly inferring the human's emotional state for timely execution.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzExNTg1NTM0ODUxLXl1aGFuZy1odWEtb2YtY3JlYXRpdmUtbWFjaGluZXMtbGFiLWZhY2UtdG8tZmFjZS13aXRoLWVtby5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Yuhang Hu of Creative Machines Lab face-to-face with Emo. Image: Courtesy of Creative Machines Lab<h2 id="isPasted">How Emo connects with you</h2>Emo is a human-like head with a face that is equipped with 26 actuators that enable a broad range of nuanced facial expressions. The head is covered with a soft silicone skin with a magnetic attachment system, allowing for easy customization and quick maintenance. For more lifelike interactions, the researchers integrated high-resolution cameras within the pupil of each eye, enabling Emo to make eye contact, crucial for nonverbal communication.The team developed two AI models: one that predicts human facial expressions by analyzing subtle changes in the target face and another that generates motor commands using the corresponding facial expressions.To train the robot how to make facial expressions, the researchers put Emo in front of the camera and let it do random movements. After a few hours, the robot learned the relationship between their facial expressions and the motor commands -- much the way humans practice facial expressions by looking in the mirror. This is what the team calls “self modeling” – similar to our human ability to imagine what we look like when we make certain expressions.Then the team ran videos of human facial expressions for Emo to observe them frame by frame. After training, which lasts a few hours, Emo could predict people’s facial expressions by observing tiny changes in their faces as they begin to form an intent to smile.“I think predicting human facial expressions accurately is a revolution in HRI. Traditionally, robots have not been designed to consider humans' expressions during interactions. Now, the robot can integrate human facial expressions as feedback,” said the study’s lead author Yuhang Hu, who is a PhD student at Columbia Engineering in Lipson’s lab. “When a robot makes co-expressions with people in real-time, it not only improves the interaction quality but also helps in building trust between humans and robots. In the future, when interacting with a robot, it will observe and interpret your facial expressions, just like a real person.”<iframe width="640" height="360" src="https://www.youtube.com/embed/pWTTzR_wXuQ?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="556492795" title="Human-Robot Facial Co-expression">&nbsp;</iframe>Go inside the Creative Machines Lab to watch Emo's facial co-expression. Video Credit: Yuhang Hu, Columbia Engineering<h2 id="isPasted">What’s next</h2>The researchers are now working to integrate verbal communication, using a large language model like ChatGPT into Emo. As robots become more capable of behaving like humans, Lipson is well aware of the ethical considerations associated with this new technology.“Although this capability heralds a plethora of positive applications, ranging from home assistants to educational aids, it is incumbent upon developers and users to exercise prudence and ethical considerations,” says Lipson, James and Sally Scapa Professor of Innovation in the Department of <a href="https://www.me.columbia.edu/" rel="nofollow" target="_blank">Mechanical Engineering</a> at Columbia Engineering, co-director of the <a href="https://make.columbia.edu/" rel="nofollow" target="_blank">Makerspace</a> at Columbia, and a member of the <a href="https://datascience.columbia.edu/" rel="nofollow" target="_blank">Data Science Institute</a>. “But it’s also very exciting -- by advancing robots that can interpret and mimic human expressions accurately, we're moving closer to a future where robots can seamlessly integrate into our daily lives, offering companionship, assistance, and even empathy. Imagine a world where interacting with a robot feels as natural and comfortable as talking to a friend.”&nbsp;</section>

The study's lead author, Yuhang Hu, a PhD student at Columbia Engineering in Hod Lipson’s lab. Credit: John Abbott/Columbia Engineering

Columbia engineers build Emo, a silicon-clad robotic face that makes eye contact and uses two AI models to anticipate and replicate a person’s smile before the person actually smiles 

Robot, can you say 'Cheese'?

In this compelling podcast, host Raymond Yin, explores the rapidly advancing realm of smart home connectivity, offering valuable insights for tech enthusiasts, homeowners, and those intrigued by the potential of interconnected living environments.<h2 dir="ltr">The Essence of Interoperability in Smart Homes</h2>A central theme of the episode is the critical importance of interoperability in smart homes. Yin and LaPré delve into how devices from various manufacturers can seamlessly communicate and collaborate, creating an integrated and efficient home environment. This aspect of smart home technology is not just a convenience but a necessity in our increasingly interconnected world.<a href="https://www.wevolver.com/article/matter-the-future-of-smart-home-connectivity-and-its-impact-on-product-development" target="_blank" rel="noopener noreferrer">Read more about the Matter protocol here. </a><h3 dir="ltr">Benefits of a Unified Ecosystem</h3>The podcast highlights the numerous advantages of a unified smart home ecosystem. The duo discusses how such integration leads to heightened convenience, bolstered security, and improved energy efficiency. These benefits underscore the transformative impact of smart home technologies on our daily lives.The conversation also acknowledges the challenges and complexities inherent in the smart home technology sector. As the field evolves rapidly, staying abreast of these changes and addressing the emerging issues is essential for both developers and consumers.<h3 dir="ltr">Insights from an Industry Expert</h3>Bringing a wealth of knowledge to the table, Chris LaPré shares his expertise on the current landscape of smart home connectivity. He sheds light on recent advancements in wireless technologies and the significance of open standards in promoting interoperability and shaping the future of connected homes.<h3 dir="ltr">Visioning the Future of Connected Homes</h3>The episode provides a glimpse into the future of smart homes, emphasizing the role of interoperability in its evolution. This forward-looking perspective is particularly engaging for listeners keen to understand where smart home technology is headed.In conclusion, "The Tech Between Us" episode with Raymond Yin and Chris LaPré offers a comprehensive and insightful exploration of the present and future of smart home connectivity. It's a must-listen for anyone interested in the intersection of technology and everyday living, providing a clear view of the technological landscape that lies between us and our homes. The podcast also encourages its audience to subscribe and stay updated on future episodes, ensuring they remain at the forefront of technological advancements in the field.

In a recent episode of the podcast "The Tech Between Us," host Raymond Yin engages in a compelling discussion with Chris LaPré, the Chief Technology Officer at the Connectivity Standards Alliance. 

The Tech Between Us Podcast: Exploring the Matter Protocol for a Smarter Home

Just a year ago, on November 29, 2022, Enchanted Tools announced its entrance into the world of robotics with the introduction of its humanoid robot, Mirokaï. With the goal of producing 100,000 robots by 2032, Enchanted Tools' engineers are working tirelessly to bring their dream to life. Miroki has traveled from Paris to Las Vegas to Tokyo, gaining popularity and acclaim along the way. Laurent Vasse, Enchanted Tools' Industrial Director, shared the resources and technological support provided by maxon teams to help meet these challenges.<h2 dir="ltr">Enchanted Tools: useful robots to re-enchant the world.</h2>Enchanted Tools is a start-up with the goal of transforming the field of robotics. To make this possible, they have developed robot-characters called Mirokaï, which weigh 28kg and stand at 1.23m tall. These robots are designed to be both endearing and useful, with a unique appearance that combines animal and manga characteristics. The Mirokaï serve three main purposes.<ul><li dir="ltr">Move autonomously, thanks to an orbital globe on which they are perched.</li><li dir="ltr">Grip objects, using a universal handle that gives them a 97% success rate.</li><li dir="ltr">Interact with humans, in a timely manner, by being able to understand and execute voice commands.&nbsp;</li></ul><blockquote><blockquote>"The Mirokaï have been designed with simplification in mind: simplifying input, simplifying navigation and simplifying interaction, to help those who help on a daily basis, while providing ethical answers to today's robotics and AI challenges." says Jérôme Monceaux, CEO of Enchanted Tools.</blockquote></blockquote>Mirokaï robots are capable of carrying loads up to 3kg, which makes them suitable for various sectors such as hospitals, hotels, airports, and restaurants. These robots can relieve staff from performing the most tedious and physically demanding logistical tasks. What sets these robots apart from others is their animation-inspired design, and they are programmed to interact with people emotionally to enhance their adaptability.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTAzNTczODU4LVNlzIFxdWVuY2UgMTEuMDBfMjhfMTNfMjcucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">The Mirokaï robot character weighs 28kg and is 1.23m tall.<blockquote>“Since the first prototype was presented to the press, we have successfully developed a dozen demonstrators, which have enabled us to validate the robot’s impact, particularly with the help of healthcare professionals.” explains Laurent Vasse, Industrial Director at Enchanted Tools.</blockquote>Enchanted Tools' 70 engineers are now moving on to industrialization after a phase of concurrent engineering to strike the perfect balance between time-to-market, performance, and price.Open urban factory project to revolutionize robotic industrializationIn 2024, the Mirokaï will become a reality as the first pilot line is implemented. This will allow the production of approximately 50 robots by the end of the year. Previously, the R&amp;D engineers had designed and assembled demonstrators in the R&amp;D offices. However, with the introduction of the pilot line, the robot assembly process can be designed under conditions that closely mimic the industrial model. Laurent Vasse provided this information.Enchanted Tools is planning to open a factory in the heart of Paris. The company wants to show that robotics can re-enchant the world instead of dehumanizing it. The factory will not only produce robots but also introduce the world of robotics, mechatronics, and AI to people. According to Laurent Vasse, the company's goal is to change people's beliefs about this industry and democratize it. They hope to inspire people to pursue careers in this field as well.Within the factory Enchanted Tools there will be a broad range of activities and opportunities for engagement. Including:&nbsp;<ul><li dir="ltr">Industrial area,&nbsp;where assembly operations can be mastered.</li><li dir="ltr">Maintenance and assistance, where Enchanted Tools customers can benefit from the support they need once the robots are on the market.</li><li dir="ltr">R&amp;D&nbsp;Centre&nbsp;where Enchanted Tools engineers further develop the company's know-how.</li></ul>The factory is scheduled to open its doors in the summer of 2024, when all media attention will be focused on Paris, for the Olympic Games."We imagined this concept of an urban factory as one that could be replicated in other parts of the world, where we encounter the same issues of development, brand awareness, and robot maintenance. We therefore plan to duplicate this concept in North America, China, or Japan, to support our commercial development with this type of entity, which will combine all the skills we need to deploy in these regions" further explains Laurent Vasse.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTAzNzkwNTM4LW1heG9uIEVuY2hhbnRlZCBUb29scyBCdXJlYXUgZXR1ZGVzIDIucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">maxon Engineers worked closely with the Enchanted Tools team to solve technical challenges.<h2 dir="ltr" id="isPasted">100,000 robots in 10 years</h2>The pilot-line aims to produce around 50 robots in 2024, then 200 - 300 units in 2025. From 2026 onwards, Enchanted Tools is set to use a larger factory to produce a thousand robots a year, then ramp up production to 100,000 robots by 2032. "From the start, our goal has been to produce robots in large quantities, on a massive scale. This means having working methods, in terms of development and industrialization, that are revolutionary for the world of robotics”, explains Laurent Vasse. For the teams at Enchanted Tools, the challenge is to find the right balance between improving control and retaining a form of agility. In other words, Enchanted Tools strives to retain the flexibility that is the strength of start-ups, while applying robust methodologies worthy of more mature industries, to successfully initiate the industrialization phase.&nbsp;<blockquote>"To succeed in this challenge, we needed partners like maxon, capable of following us in this way, with excellent technological mastery and the ability to adapt and buildup in industrialization,", says Laurent Vasse, Industrial Director of Enchanted Tools.</blockquote><h2 dir="ltr">maxon x Enchanted Tools: strong collaboration brings the project to life.</h2>From the start of the project, Enchanted Tools' teams worked hand in hand with precision mechatronic system company, maxon France. Maxon assisted Enchanted Tool's to select the perfect drive system, size the products correctly, and customize certain elements according to the current and future needs of their project. "Right from the start, we were thinking in terms of industrialization and volume", says Kévin Schwartz, sales engineer for maxon France. He continues: "There's a huge gap between current production volumes and projections for the coming years. Thanks to the maxon group's industrial volumes and our production facilities, we were able to be extremely reactive, since these products were already in our catalog, and we were able to make the necessary customizations.”<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzAyNTA0MDE3MTA5LU1heG9uIEZyYW5jZSAtIDI3IEp1aW4gMjAyMy00OTIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib">The prototypes are tested against a variety of constraints.&nbsp;The maxon team then helped Enchanted Tools with product sizing and the choice of motorization ranges, taking into account the technical constraints and very tight deadlines required to present this new robot. EC-i 40 motors with planetary gearboxes were chosen for the Ball Bot, while ECX brushless motors with a diameter of 22mm are used for the orbital axes.To control industrialization, the maxon and Enchanted Tools project teams worked on different categories of prototypes (A, B and C). Each prototype is tested against functional, design, service life, and normative constraints, moving toward the final product and production, this approach enables the teams to anticipate the product's maturity.&nbsp;<blockquote>"The advantage of this approach is that it's robust when it comes to mass production. We go through all the design validation stages, to eliminate as many risks as possible, and then be able to calmly launch large volumes for industrialization", adds Kévin Schwartz.</blockquote>Thanks to the technological support provided by <a href="https://www.maxongroup.fr/maxon/view/content/index" target="_blank" rel="noopener noreferrer">maxon France</a>, the Enchanted Tools teams were able to move ahead quickly with industrialization. This partnership is one crucial step towards the realization of the Mirokaï project, which is set to influence the robotics landscape.&nbsp;

A year ago, French company Enchanted Tools unveiled its prototype humanoid robot, Mirokaï. The ambitious team plans to produce 100,000 robots by 2032. 



Building a humanoid robot in under a year

If you've ever been annoyed about a supermarket purchase that cost more at the checkout than the shelf price tag suggested, you've probably thought, irritated, that the retailer was trying to trick you. In reality, store price mismatches are often caused by errors that inevitably appear at large stores. Prices can change as often as several times per day, and with thousands of price tags that must be updated manually, the stores simply don’t have enough employees to update prices immediately.Adapta Robotics, a Bucharest-based robotics startup, offers an almost automated solution to this problem. They created ERIS (Effective Retail Intelligent Scanner), called MARCEL in Carrefour stores, a robot that automatically scans all the labels on a shelf, reads and compares the prices with the correct ones, and prints updated labels. At the same time, the robot also analyzes how fully the store shelves are stocked and, in the future, could even check the accuracy of the placement of the products themselves. ERIS was officially launched on November 3rd, 2022, and the company has already partnered with one of the leading European retail giants, Carrefour, to use the robot in their stores.ERIS is not the first successful product for the Bucharest-based company. Adapta Robotics has previously launched MATT, a robot that tests physical inputs for device screens in many industries, from smart washing machines to instrument panels on airplanes, on the international market.Founded by the members of a robotics team created by students from the Polytechnic University of Bucharest, with awards at dozens of professional competitions, Adapta Robotics got its start as a division of the software development company <a href="http://rinf.tech/" target="_blank">rinf.tech</a>, and later turned into an independent company. I spoke with the three founders of the company – Diana Baicu, Mihai Crăciunescu, and Cristian Dobre – about how they turned their passion for robotics into a successful business, how the robots they invented work and about the training and development process of using them.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284618626-Adapta_Founders-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">Co-founders of Adapta Robotics, from left to right: Cristian Dobre, Diana Baicu, and Mihai Crăciunescu. PHOTO: Andrei Cojan/Adapta Robotics<h3 id="isPasted">Mindcraft Stories: How did Adapta Robotics get started?</h3>Mihai Crăciunescu:&nbsp;Our team started many years ago at the Bucharest Polytechnic when we were building robots to participate in various competitions. After a while, when we won almost all the competitions we went to, national or international, we decided to take the step to create a company.Cristian Dobre:&nbsp;Robotics is a very expensive hobby, and for this reason, we had to work hard. After a few summer jobs, I ended up at <a href="http://rinf.tech/" target="_blank">rinf.tech</a>. They wanted to build a robot to test mobile devices, but they didn't have the know-how. I joined their team, and they helped us apply for a European funding project to develop such a robot, which we called MATT.<h3>M.S: So MATT was the first project you had as a robotics business?</h3>Diana Baicu:&nbsp;MATT was the first prototype we made when we were a division within <a href="http://rinf.tech/" target="_blank">rinf.tech</a>, this was around 2015. Since then, and especially in the last two years, we started to separate from them and create an identity of our own as a new robotics startup. In the meantime, we also started working on developing ERIS.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284638871-Adapta_Lab_Detail-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">Andrei Cojan/Adapta Robotics<h3 id="isPasted">M.S: Where did you get the idea of a robot to scan the labels on the shelves?</h3>M.C.:&nbsp;The shorter answer is that we talked with a retailer from Romania, who presented us with some problems they faced in the stores, and we saw it as a kind of technical challenge.Retailers can pay serious fines if the prices on the shelf are not in line with the actual ones. And the mislabeling part isn't bad faith on the part of a retailer, it's an error. To stay competitive, retailers issue daily price changes or temporary promotions on certain products. Logistically, at a store with 10,000 labels, making sure every label is updated every time is a problem. Human error will still occur somewhere.As engineers who have been developing robots for so many years, so obviously, our solution was a robot.D.B.:&nbsp;From discussions with various actors in the retail environment, we realized that most solutions in this field propose autonomous platforms. Which, of course, from a technical point of view sounds very good, but it is not necessarily the most efficient solution. This is because the problems identified by that robot must still be solved by humans.With ERIS, we decided from the beginning that it must be operated by a human. Thus, we make the pipeline more efficient, as the operator can solve the problems that arise on the spot.For example, when ERIS starts scanning after being placed in front of a shelf, there is a distance it must maintain from it. All of this information is specified by the robot, which tells the operator how close or far away to stand, as well as the speed at which it can scan the tags.We started with one prototype, of course, which we tested in a pilot project in stores. Now we've done a redesign based on everything I've learned in about a year and a half since ERIS has been running.<h3>M.S: How exactly does ERIS work?</h3>C.D.:&nbsp;ERIS is a tool. You can think of it as a mega scanner. Instead of looking at every item on a shelf and every barcode to check every price, as happens when you check manually, it scans the entire surface, and the operator just has to push it past the shelf.ERIS is equipped with sixteen cameras. Eight of these are distance-measuring cameras — that is, instead of recording color, the camera will return a black-and-white image; white means that the product is very close, and black means it is very far.After taking in the data, the robot analyzes each image captured with conventional cameras and uses a software detection model to locate the regions where the price tags are located.M.C.:&nbsp;The robot takes images from all the cameras simultaneously and builds a single large vertical image, almost two meters high, as long as the shelf. From this, it identifies price label locations.Once it has discovered the areas with price tag, they are cropped out of the big picture and sent to two neural networks: one that identifies the barcode and another that reads the price. The latter is developed with a machine learning model created by us. Once it has all of this information, it can query the current product price against the retailer's updated database.There is also a separate module that identifies shelf occupancy to alert the retailer if the shelf is empty or out of stock.The accuracy of the entire system is over 98%, in some cases even 100%, because the entire flow has been optimized to work as well as possible in the real store environment.D.B.:&nbsp;If the operator identifies a wrong label, he can print it on the spot and replace it because there are two printers on the robot. Alternatively, it also has the option to print several corrected labels at once after it has finished scanning. In this case, the robot has a location finder feature that can show the operator where the wrong label is.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284660931-ezgif-2-e22b91ac4c.png" style="width: 300px;" class="fr-fic fr-dib">ERIS1 in Store. Adapta Robotics<h3 id="isPasted">M.S: How did you train ERIS to detect labels and shelf occupancy in a store?</h3>M.C.:&nbsp;The biggest challenge we had was the datasets, that is, gathering enough relevant information that is labeled correctly so that you can train a neural network with the accuracy I mentioned earlier. In the beginning, we took many pictures of store shelves, which we had to manually annotate or use online annotation services.The volume of data required was enormous, and the quality of the results returned by online services was very poor. That is, out of 1,000 pictures, maybe 200 were properly annotated.We invested a lot of time and resources in a simulator, where we simulated the store with products, with real data from the store, through which we were able to train our networks with high accuracy.D.B.:&nbsp;We made a kind of game using Unreal Engine, a virtual store, to generate our data so that we could get a large amount of data with minimal annotation effort. At the same time, this also allowed us to more easily configure the models in case of later changes. If we change the retailer, for example, the solution works, by and large, but small adjustments still need to be made.C.D.:&nbsp;The colleague who worked on this area made a very cool algorithm that generates the shelves programmatically. Each time you started the simulator, their arrangement in the store remained the same, but the labels and shelf occupancy differed.A very big advantage is that in a game environment, you can also acquire data about the distance to objects and test that part. Such data can be obtained either if you go with the camera that measures the distance and scan the whole store or, much easier, if you load the algorithms into such a simulator. With Unreal Engine, it's very simple to calculate the distance from the camera to a product on the shelf.M.C.:&nbsp;Each retailer has labels with a different format and maybe even a different font. And we need to make sure that ERIS can recognize the price regardless of these variables. They sometimes exist even within the same store. For example, they are yellow or electronic tags, with a promotion or without a promotion - you can get up to ten different kinds. The solution must be robust to such changes.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284691878-ERIS_two_versions_MATT-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">ERIS (left) and two versions of the MATT robot. Andrei Cojan/Adapta Robotics<h3 id="isPasted">M.S: How efficient is ERIS compared to a human operator?</h3>D.B.:&nbsp;We are at the point where we have already seen that the robot is more efficient than a human operator. By our calculations, comparing how long it would take an experienced operator to scan a shelf of around 200-300 tags and how long it would take ERIS to do the same, it came out to an 85% improvement. Taking into account the label change part, the efficiency percentage is 45-50%. Our plan is to bring this percentage somewhere to 65-70%.<h3>M.S: What other plans to improve the ERIS robot do you have in the future?</h3>D.B.:&nbsp;We already have the second version, which we have already developed and are planning to introduce on the market. We optimized both the hardware side and costs, improved performance, and made it more reliable and easier to use. This will also increase the number of imaging cameras to ten.C.D:&nbsp;ERIS is already very efficient, and most data processing is done directly on the robot. To give you an idea, in one minute, around 60 GB of data are transmitted and processed on ERIS. We envision, in the future, that this data will be aggregated and transmitted to the retail and logistics systems so that it can be used there as well.D.B:&nbsp;That would mean integration with other store systems to know, for example, the historical inventory stock patterns. We have a lot of plans on that side, including an analytics platform based on all the information that ERIS gets to actually help the retailer understand everything that's going on in the store.M.C.:&nbsp;Another functionality we have on the radar is a planogram, through which you can check if a product is positioned where it should be on the shelf. For example, product manufacturers pay a premium to have their products on the top shelf, third shelf, etc., usually to be more visible, and the retailer has to ensure that those products are there. That's where you get into a detection area of the product itself, which is more complicated than the tag, but that's a feature we're discussing for the future.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284710441-MATT_testing_tablet_touchscreen-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">MATT testing the tablet touchscreen. Adapta Robotics<h3 id="isPasted">M.S: As for MATT, how did you come up with the idea for a robot that tests inputs for various devices?</h3>M.C.:&nbsp;MATT was designed as a robot to automate testing on phones, around 2015. Now you can automate pretty much everything that happens on a phone with emulation software, so MATT is not that relevant for smartphones anymore.It's just that other industries have emerged that require testing for touchscreens, such as automotive. There are other situations where you need to integrate a phone with a device and make sure that the Bluetooth connection works or that if you press a button on the phone, something happens on the other device, like in the case of appliances that you control through the phone.I saw that MATT is perfect for this, so I put it on input testing, and it became kind of a benchmark in this industry. Most of our customers for MATT are in the United States but also in the Netherlands, Germany, France, Spain, and even Lithuania.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1702284743662-Adapta_Team_Photo-1536x1024.jpg" style="width: 300px;" class="fr-fic fr-dib">Andrei Cojan/Adapta Robotics<h3 id="isPasted">M.S: What expansion plans do you have for the future?</h3>M.C:&nbsp;We are actively scaling the use of ERIS, primarily in Romania, because we are a Romanian startup. But our targets are abroad because that's where the impact will be much more relevant. In Romania, labor in stores is also cheaper. And as there is always the question of return on investment for the retailer, in countries like Sweden, Norway, Great Britain, Germany, or France, the investment will be much more attractive for the retailer.The vast majority of our competitors are well-known or are credibly well-funded companies. We have to focus on quality, that is, that robot does what we promise it does.Many of the initiatives in this field fail because they have a lot of nice words in the presentation, but when they get to launch a robot, it does not perform the basic functionalities or performs them very poorly. To get a "piece of the pie", we focus on quality: the AI and the robot must work at very good percentages so that we can deliver a reliable solution that actually improves store operations.<hr>This Q&amp;A interview session with Adapta Robotics happened at&nbsp;<a href="http://gotech.world/" target="_blank">gotech.world</a> 2022 Expo, where the company introduced the second iteration of ERIS, the retail robot.

ERIS, a robot specialized in scanning shelf labels created by Adapta Robotics, will soon be seen in supermarkets all across Romania

The Robot That Solves the Wrong Price Label Problem on Store Shelves

The recent advances in Large Language Models (LLM) have made a lot of noise recently – everybody is using ChatGPT. Robotics is no exception; there has been a lot of talk about how to integrate them in various kind of robotic use cases; in this article, we will talk about their applications to social robots like NAO and Pepper.&nbsp;<h2>Social Robots? &nbsp;</h2>Social robots are designed to be used in a places, interacting with people, and as such are a natural fit for conversation-oriented language models. But they have been used for years before that, and the field of human-robot interaction is well-developed.&nbsp;Compared to a GUI chat interface like ChatGPT, social robots have to deal with extra uncertainty linked to speech recognition (the input the langauge model received might not be what the user intended to say), and handling several users speaking, some of which might not even intend to speak to the robot.&nbsp;<h2>Improving what the robot understands&nbsp;</h2>One advantage LLMs have is in understanding open-ended instructions. There are several challenges in understanding voice commands:&nbsp;<ol><li>Speech recognition: turning the raw sounds into text; how effective this is is going to depend of how the user speaks (accents, slurred speech, volume, clarity of enunciation), the characteristics of the microphone, surrounding sound, etc.&nbsp;</li><li>Variety of expressions: In practice, people have a huge variety of ways of expressing even simple things (For example: "I'm looking for an ATM", "Would you show me the nearest ATM?", "Is there an ATM around here?", "Where's the nearest ATM?","Do you know where I can withdraw some money?", "Where can I withdraw cash around here?", "Is there an ATM in this building?" etc.); and handling all of those properly usually requires some bespoke customisation via a dialogue engine like dialogflow or qichat.&nbsp;</li><li>Context: In addition to the explicit information contains in a question like the ATM example above, some information comes from what was previously discussed ("How much does one cost?" "How can I get there?"), or from gestures such as pointing ("Is it that way?" "What's that?"); keeping track of that information usually requires more bespoke logic related to keeping track of users.&nbsp;</li></ol>These are all challenges that could be handled previously with some work; but LLMs introduced a great increase in quality: they can be very good at handling a wide variety of expressions out of the box. They are also very good at taking into account any context included in the conversation log (typically sent alongside the prompt); this usually includes previous conversation but (with a bit more work) can include any information that can be infered from the sensors (for example, by using an object recognition model to turn what the robot sees into a text description).&nbsp;One more surprising improvement when using LLMs is on speech recognitions – more specifically, even if there are mistakes in the speech recognition (that can be misheard words, but also homophones like their/there/they're), the LLM will often use context and common sense to handle the input appropriately anyway, giving the impression of a robot that hears better.&nbsp;<h2>Improving how the robot answers&nbsp;</h2>One obvious output is Speech, and plugged into a speech-to-text engine, a LLM can easily create a conversational robot.&nbsp;However, this brings a few new issues:&nbsp;<ul><li>One notorious ones is hallucinations, a problem that's far from solved for LLMs. The robot can understand a wider variety of inputs, his answers to those might be confident but highly inaccurate – if there's a word he doesn't understand, he might decide to invent a fanciful product description; if an unfamiliar name is given, he might make someone up. This can be mitigated, but at the risk of making the robot repetitive or boring – it's a fine balance, and it might depend on the use case.&nbsp;</li><li>A less obvious one is speech that's too verbose. If ChatGPT talks too much, the user can just skim the paragraphs; but a speaking robot takes time and can be much more annoying. This can be improved with better prompting, with pretraining etc. as well as UX tweaks (e.g. a way of interrupting the robot) but requires careful attention. More generally, the LLM's output might sometimes not be as good quality as hand-crafted one.&nbsp;</li></ul>However, a robot is more than a smart speaker – it might have arms, wheels, etc. and the LLM can be made to control those, either directly or from a library of presets (e.g. navigation waypoints) - in effect, the output of the LLM is not just speech but actions.&nbsp;<h2>How to deal with it the best&nbsp;</h2>How to deal with hallucinations and other problems may depend of the use case:&nbsp;<ul><li>For entertainment purposes, the hallucinations may not be a problem, and can even be part of the fun, as long as that is well-presented to the user.&nbsp;</li><li>For information purposes, a choice needs to be made between using LLM output and risking hallucinations, or using fixed, well-prepared outputs.&nbsp;</li><li>One should also keep in mind other non-speech actions and see how many of those can be controlled by the LLM.&nbsp;</li></ul>Apart from the technical performance on input and output, the largest strengths of LLMs is flexibility – adapting the robot to a new use case, or to a specific context for a new use case, can be as simple as editing a prompt, and produces a behavior of decent quality.&nbsp;This allows for much quicker iterations. &nbsp;<h2>Further directions&nbsp;</h2>In robotics (social or not) and elsewhere, engineers, researchers and tinkerers all over the world are experimenting with interesting ways to use LLMs. We've covered a few we have experience with here, but there are many more directions that could be applicable, such as:&nbsp;<ul><li>integrating vision models to plug it with what the robot sees, to make it even more aware of the context&nbsp;</li><li>making the robot proactively decide what to do (not just respond to what he heard, but decide to e.g. walk up to such-and-such a person and engage conversation)&nbsp;</li><li>giving the robot a long-time memory&nbsp;</li></ul>LLMs offer great possibilities for social robots if used well, and it will be interesting to observe how the field evolves in the coming years – in the coming months even.&nbsp;

A Pepper robot showing a ChatGPt-type dialogue

Lessons learnt from integrating ChatGPT in Pepper and NAO

Bringing the LLM revolution to social robots

In this episode, we discuss a novel new approach for increasing the field of view of autonomous systems by leveraging reflections of blindspots on reflective bodies.

Podcast: Blind Spot Begone! Cars and Robots Can Now See Around Corners

This article was first published on <a href="https://news.mit.edu/2023/using-robotics-supercharge-health-care-0123" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>. &nbsp;&nbsp;<hr>Since its founding in 1998, Vecna Technologies has developed a number of ways to help hospitals care for patients. The company has produced intake systems to respond to Covid-19 patient surges, prediction systems to manage health complications in maternity wards, and telepresence robots that have allowed sick people to stay connected with friends and loved ones.The differences among those products have also led to a number of transformations and spinoffs, including material handling company Vecna Robotics and the health care nonprofit VecnaCares. Vecna Technologies co-founders Deborah Noel Theobald &rsquo;95 and Daniel Theobald &rsquo;95, SM &rsquo;98 say each of those pivots has been driven by a desire to build a robotics company that makes a positive impact on the world.&ldquo;We knew we wanted to do robotics and do something good in the world,&rdquo; Deborah says of the team&rsquo;s mindset. &ldquo;We founded Vecna thinking, &lsquo;How can these new web technologies influence and improve health care?&rsquo; That&rsquo;s the arc MIT set me on and something I&rsquo;ve been excited to pursue ever since.&rdquo;<h2>&ldquo;A fun ride&rdquo;</h2>As a child, Deborah Theobald wanted to be an astronaut. The desire led her to MIT, which had one of the few aerospace engineering programs for undergraduates. She got interested in the health care industry while studying the health effects of long-term space exploration with Professor Dava Newman, the Apollo Program Professor of Astronautics at MIT who is now also the director of the Media Lab.Deborah also met Daniel Theobald at MIT. Daniel had been building robots since he was a child and was majoring in mechanical engineering.The two began thinking about starting a company, and Daniel even applied to the MIT $100K Entrepreneurship Competition (then the $10K) with a rough idea for a robotics company.For their master&rsquo;s degrees, Deborah went to the University of Maryland to continue studying health effects in space, while Daniel stayed at MIT, working on several robotics projects. When Daniel graduated in 1998, Vecna was born.From day one, the company had a policy of paying employees to spend 10 percent of their work week doing community service.&ldquo;We found that our focus on giving back benefited the business in so many ways that it was absolutely, unambiguously the right thing to do,&rdquo; Daniel says. &ldquo;For one, it was a self-filtering mechanism. People joined Vecna who believed in giving back and wanted to be part of something that was socially responsible. And we found those are also the people that make amazing employees.&rdquo;The founders got their first big break with a government contract to build a health care portal that allowed patients, managers, and providers to communicate and share documents. The contract also provided flexibility for the founders to explore other avenues for the business.The pair went on to earn a number of government grants for one-off projects, some of which blossomed into successful commercial products. Another grant tasked them with building models to help hospitals predict and manage hospital acquired infections (HAIs), which kill tens of thousands of people in the U.S. each year. The resulting tool ended up being deployed in about 100 hospitals.&ldquo;At the time, people were using spreadsheets to pull in data from different systems &hellip; and trying to comprehend what kind of infection it was,&rdquo; Deborah says, noting that doctors usually start infected patients on general antibiotics before they can classify the disease. &ldquo;Our tool allowed them to pull that information together faster, reducing their stay in hospitals &mdash; and all the trauma and pain that goes with that &mdash; by weeks.&rdquo;The company&rsquo;s next product was a patient registration system that used kiosks to streamline patient intake at hospitals. During the Covid-19 pandemic, Vecna turned the platform into a text-based check-in service for clinics. The service is being used by thousands of hospitals today.Subsequent mobile versions of that system have been used to deliver medication, allow doctors to hold virtual consultations, and even help immunocompromised students to attend school virtually and avoid isolation.Vecna&rsquo;s emphasis on community service led the team to explore ways to apply the company&rsquo;s technologies in low-resource settings, leading to the creation of the company&rsquo;s nonprofit arm, VecnaCares.In 2014, VecnaCares brought their VGo mobile robot to Liberia and Sierra Leone to help with the Ebola response, allowing doctors to see patients without going through a time-consuming decontamination process. The company&rsquo;s patient intake software was also used to register and manage patients with Ebola and other diseases.VecnaCares has since partnered with groups including the International Rescue Committee, the International Committee of the Red Cross, International Medical Corp, and the Special Olympics for a variety of projects. It&rsquo;s also honed its algorithms to help low-resource hospitals manage staff shortages in maternity wards, helping nurses focus their attention on the babies and mothers most at risk of complications.&ldquo;One of the places we&rsquo;re deployed has 10,000 births a year, so at any one time there may be 40 women laboring in that hospital, which has one operating room for all C-sections,&rdquo; Deborah explains. &ldquo;Our tool can intake women, do an assessment, and notify clinicians if someone&rsquo;s high risk and needs checking-in on. It leads to better outcomes and helps manage some of the complications that have led to a high rate of infant and maternal mortality in these areas.&rdquo;After years of robot development and commercialization, the founders decided their robots may be better suited for warehouses than health care. In 2017,&nbsp;Daniel&nbsp;spun out Vecna Robotics to focus exclusively on robotics for industrial settings like manufacturing, logistics, and order fulfillment.&ldquo;We&rsquo;ve sort of done four different growths and exits,&rdquo; Deborah explains. &ldquo;It&rsquo;s been a fun ride.&rdquo;<h2>Continuing to innovate</h2>As it nears the 25th anniversary of its founding, Vecna Technologies is far from finished. Its leaders believe the firm&rsquo;s products and expertise can play a significant role in the burgeoning home health care and extended care industries, helping patients stay out of hospitals while remaining safe.&ldquo;As we look at the aging population, that burden of care is really going to fall on family members as well as [health care organizations],&rdquo; Deborah says. &ldquo;I&rsquo;d love to be able to provide better tools for them to care for loved ones, which is often unpaid and unrecognized.&rdquo;Later this year, the company will release an inexpensive home care robot that can move autonomously or by remote control to help care for people struggling with diseases like Alzheimer&rsquo;s. The robots will be part of Vecna&rsquo;s &ldquo;Be There Network&rdquo; that health care providers can use to provide care for large numbers of patients despite staff shortages.&ldquo;Now you can see and hear and feel like you&rsquo;re actually there to more seamlessly interact with the environment,&rdquo; Deborah says. &ldquo;We see that as the wave of the future now that people have begun to embrace telepresence. There are so many uses for this robot. People keep coming up with more ideas as they catch the vision.&rdquo;No matter what the future holds for Vecna &mdash; whose motto is &ldquo;Better technology, better world&rdquo; &mdash; the founders say the company will continue exploring new applications where its technologies could make a real difference in people&rsquo;s lives.<hr>&quot;Reprinted with permission of&nbsp;<a href="https://news.mit.edu/2023/using-robotics-supercharge-health-care-0123" id="isPasted" rel="noopener noreferrer" target="_blank">MIT News</a>&quot; &nbsp;

Vecna Technologies, founded by MIT alumni Deborah Theobald and Daniel Theobald, has applied robotic and digital technologies to a number of applications in the health care space with an eye toward impact. Image: Courtesy of Vecna

Vecna Technologies, founded by a pair of MIT alumni, has followed a long and winding path to help people in health care settings.

Using robotics to supercharge health care

The warm feeling of being hugged is known to have a number of positive effects &mdash; and humans are far from the only creatures to enjoy social touch. Robots, meanwhile, are traditionally cold, sterile, and not entirely cuddly &mdash; at least, outside the realm of children&rsquo;s entertainment.But what if a robot could deliver a satisfying hug? That&rsquo;s the question asked by Alexis E. Block and colleagues in the HuggieBot project, with results from a third-generation study having just been accepted for publication. Such a machine could provide comfort to the lonely, have potential medical benefits in care settings, or act as a surrogate when people simply can&rsquo;t be together in person, as has been so common during the ongoing pandemic.Enter HuggieBot 3.0, claimed by its creators to be &ldquo;the first fully autonomous human-sized hugging robot that recognizes and responds to the user&rsquo;s intra-hug gestures.&rdquo;<h1>The birth of HuggieBot</h1>Work on the original HuggieBot began in 2016, when Block was studying for a Master&rsquo;s degree in robotics. The idea: Providing the ability to send a hug anywhere in the world, but in a manner which feels natural and enjoyable &mdash; following what Block and colleagues would come to call the &ldquo;six commandments&rdquo; of robotic hugging, a series of design guidelines which included the need for such a robot to be soft, warm, human-sized, and able to autonomously enter into and exit from an embrace.HuggieBot 2.0, put into testing in 2020, took the concept a stage further by using visual and haptic perception to deliver adaptive hugging in closed-loop system, but it wasn&rsquo;t until HuggieBot 3.0 that the machine would gain a key feature: The ability to autonomously detect, and reciprocate, &ldquo;intra-hugging gestures&rdquo; through a probabilistic behavior algorithm &mdash; the creation of which contributed an additional five new design guidelines, on top of a revision to one of the original six.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjUxMDA4ODg5NTI0LWh1Z2dpZTQuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="An image of the HuggieBot robot embracing a researcher; its arms, covered in soft fabric, are seen holding the researcher as the researcher leans in to the hug. A screen on the head shows a smiling cartoon face; a skirt is seen to the bottom of the picture, hiding a framework which gives the robot its human-like height.">Built using off-the-shelf robot arms with added soft padding, an inflatable and warmed chest area, and given a welcoming face, HuggieBot is an undeniably huggable robot.In its tested incarnation, HuggieBot 3.0 incorporates a custom sensing system dubbed HuggieChest which uses two inflated chambers of polyvinyl chloride to provide a soft surface. As a hug takes place a barometric pressure sensor and microphone serve to detect human touch, transmitting data via an Arduino Mega microcontroller board to a Robot Operating System (ROS)-based computer located in the robot&rsquo;s 3D-printed head &mdash; behind a display which provides a friendly and welcoming animated visage.The robot&rsquo;s own hugs, meanwhile, are delivered using a pair of Kinova JACO arms, mounted to a custom metal frame and each with six degrees of freedom. &ldquo;These arms were selected for being anthropomorphic, quiet, and safe,&rdquo; the researchers explain. &ldquo;Their movement speed is limited in firmware, and they can be mechanically overpowered by the user if necessary.&rdquo;<h1>Hug autonomy</h1>Initially, the team&rsquo;s testing took place as a &ldquo;Wizard of Oz&rdquo; set-up &mdash; where a remote operator, not seen by participants, controlled the robot&rsquo;s motions. Once data had been gathered from 512 human trials across 32 participants, it was used to train a machine learning system capable of detecting and classifying a range of gestures carried out by humans during a hug and which the team classified into four key intra-hug gestures: Remaining still, moving vertically, tapping or patting on the back, and tightening hold in a squeeze.&ldquo;Rather than maximizing user acceptance for each robot gesture, which would result in the robot only squeezing the user,&rdquo; the researchers note, &ldquo;our behavior algorithm balances exploration and exploitation to create a natural, spontaneous robot that provides comforting hugs.&rdquo;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjUxMDA4OTA1MjM5LWh1Z2dpZTIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="Diagrams of the four &quot;intra-hug gestures&quot; defined by the research team: Remaining still, moving vertically, tapping or patting on the back, and tightening the hold in a squeeze.">The researchers developed a system for detecting, classifying, and responding to four key gestures using a pressure sensor and a microphone in the robot&#39;s torso.Further testing on 16 participants certainly seems to bear that claim out: Twelve of the users were found to hug the robot for longer, subjects declared that they felt &ldquo;significantly more understood&rdquo; by the robot, and stated that the revised robot was &ldquo;significantly nicer to hug&rdquo; in its new interactive form.Perhaps the biggest outcome of the work, however, is in its extension of the earlier &quot;commandments&quot;: The team revised a guideline on hug initiation, creating a new framework for inviting a potential participant and ensuring consent, while adding new guidelines on ensuring a hugging robot is capable of automatically adapting to the user&rsquo;s height in order to avoid some below-the-belt contact which occurred during HuggieBot 3.0&rsquo;s testing, adding a requirement for gesture detection and perception, improving the speed and variety of robot responses to said gestures, and having the robot proactively provide gestures of its own.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjUxMDA4OTE0MDUyLWh1Z2dpZTMuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="A flow chart for the HuggieBot 3.0, showing how the robot starts by waiting for a person then initiates a hug verbally, opens its arms, begins the hug, then responds to or initiates a range of intra-hug gestures with varying probability before releasing the user at the end.">HuggieBot 3.0&#39;s gesture classification system allows it to respond to hug participants in a number of ways, providing variety and offering a more realistic experience.There&rsquo;s more to the team&rsquo;s work than simply building a better hugging robot: The researchers point to the body of work showing the benefits to physical interaction between humans, and how it may be possible to deliver at least some of the those benefits using autonomous or remotely-controlled robotic surrogate &mdash; or, as part of future work on the platform, potentially generalize HuggieBot&rsquo;s capabilities to a robot which could provide companionship and assistance in care homes, helping to address concerns regarding an ageing population and a dearth of care workers.The team&rsquo;s work has been accepted in to the journal <cite><a href="https://dl.acm.org/doi/10.1145/3526110" target="_blank">ACM Transactions on Human-Robot Interaction</a></cite>, and has been made available in preprint form under free access terms.<h1>References</h1>Alexis E. Block, Hasti Seifi, Otmar Hilliges, Roger Gassert, and Katherine J. Kuchenbecker: <cite>In the Arms of a Robot: Designing Autonomous Hugging Robots with Intra-Hug Gestures, ACM Transactions on Human-Robot Interaction, Accepted February 2022</cite>. DOI <a href="https://doi.org/10.1145/3526110" target="_blank">10.1145/3526110</a>.Alexis E. Block, Sammy Christen, Roger Gassert, Otmar Hilliges, and Katherine J. Kuchenbecker: <cite>The Six Hug Commandments: Design and Evaluation of a Human-Sized Hugging Robot with Visual and Haptic Perception, Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction</cite>. DOI <a href="https://doi.org/10.1145/3434073.3444656" target="_blank">10.1145/3434073.3444656</a>.

HuggieBot, with its soft pressure-sensing torso, 3D-printed animated head, and six-degree-of-freedom arms, is a research platform into high-quality autonomous hugging.

While the concept of a hugging robot may sound bizarre, the researchers behind the project — now in its third generation — believe that such a device could have a major impact on everything from social telepresence to elder care.

Intra-hug gestures prove key to making HuggieBot 3.0 the most cuddly robot around

In this episode, we talk about how TUM researchers are trying to model the effects of climate change on forest fires using a neural network, why robots assisting with getting dressed is more challenging than it seems, and how drones are being used to safely evacuate the elderly in nursing homes during emergencies. As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/ea6cb464-230a-493a-a54e-88cddd95368e?dark=false" class="fr-draggable"></iframe>&nbsp;<h3 id="isPasted">EPISODE NOTES</h3>(0:45) -&nbsp;<a href="https://www.wevolver.com/article/how-climate-change-and-fires-are-shaping-the-forests-of-the-future" target="_blank">Modeling How Climate Change Impacts Forest Fires</a>:&nbsp;Wildfires have been wreaking havoc on the nation’s forests and it is now more important than ever to understand how our changing climate is impacting our most treasured ecosystems. Computer simulations can provide great insight but when considering the number of variables that contribute to forest fires it becomes clear that traditional approaches would be too computationally intensive to be feasible. That’s where the researchers at the Technical University of Munich come in: they’ve built a neural network capable of recognizing patterns between the key parameters that result in forest fires which uses only a fraction of the computational power required via the traditional approach.&nbsp;(6:35) -&nbsp;<a href="https://www.wevolver.com/article/getting-dressed-with-help-from-robots" target="_blank">Getting Dressed With Help From Robots</a>:&nbsp;Do you find yourself thinking “I wish a robot could do this” when dressing yourself in the morning? Well, soon you might be able to get one of those exact robots in your own home! MIT researchers have been developing a robot to assist people when getting dressed and although they’ve made some impressive progress, there are some bumps they’ve hit on their path all centered around one object: people. People’s varying preferences regarding getting clothed and safety in general has caused their algorithm to become overwhelmed. Regardless, the team is making great progress in the human-robot-interaction front and paving the way for similar future projects.(15:15) -&nbsp;<a href="https://www.wevolver.com/article/blue-jay-unveils-drone-that-evacuates-elderly-in-case-of-fire-in-nursing-homes" target="_blank">Drone To Evacuate Nursing Homes In Case Of Emergency</a>:&nbsp;A student group - named the ‘Blue Jays’ - from Technical University of Eindhoven have developed a drone to calmly guide the elderly out of nursing homes during emergencies. The drone is powered by a neural network to analyze the facial features of people and determine their emotions which it can respond to. Their hope is that this aerial companion can help provide a sense of comfort to those who are distressed while the first responders operating the drone can instruct those in the building on how to safely exit.&nbsp;--About the podcast:Every day, some of the most innovative universities, companies, and individual technology developers share their knowledge on Wevolver. To ensure we can also provide this knowledge for the growing group of podcast listeners, we started a collaboration with two young engineers, Daniel Scott Mitchell &amp; Farbod Moghaddam who discuss the most interesting content in this podcast series.&nbsp;To learn more about this show, please visit the&nbsp;<a href="https://www.wevolver.com/profile/the.next.byte" target="_blank">shows page</a>. By following the page, you will get automatic updates by email when a new show is published.Be sure to give us a follow and review on&nbsp;<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" rel="nofollow" target="_blank">Apple</a> podcasts,&nbsp;<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" rel="nofollow" target="_blank">Spotify</a>, and most of your favorite podcast platforms!

In this episode, we talk about how TUM researchers are trying to model the effects of climate change on forest fires using a neural network, why robots assisting with getting dressed is more challenging than it seems, how drones are being used to evacuate elderly in nursing homes during emergencies.

Podcast: Climate Change AI, Robots Getting You Dressed, Drones Evacuating Nursing Homes

In this episode, we talk about a graduate student’s thesis which resulted in a robot that’ll allow you to hug loved ones that are far away, how artificial photosynthesis might hold the key to our energy crisis, and a novel technique to conduct spinal stimulation therapy via non-invasive surgery. As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b376e804-dca2-4c3a-9012-cfbbdf936706?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h3 id="isPasted">EPISODE NOTES</h3>Take a few seconds to leave us a review. It really helps!&nbsp;<a href="https://apple.co/2RIsbZ2" target="_blank">https://apple.co/2RIsbZ2</a> if you do it and send us proof, we’ll give you a shoutout on the show.(0:48) - Hugging robots:&nbsp;Have you ever missed someone so much that you wish you could just send them a hug? Well, now you can! Alexis E. Block and her team at ETH Zurich have been developing various iterations of HuggieBot - a robot whose goal is to give the best hugs possible - to minimize the strain of distance between loved ones, especially during times of crisis.&nbsp;(6:45) - Artificial Leaf:&nbsp;A team at EPFL is looking at the world’s best engineer (nature) to address the global energy crisis by developing an artificial leaf capable of photosynthesis: converting water and sunlight into oxygen and chemical energy.&nbsp;(13:50) - Non-invasive Spinal Implants:&nbsp;1 in 12 Americans suffer from severe back pain and spinal stimulation surgery isn’t widely used to address this problem because it is either extensive and invasive or non-invasive and not very effective. Researchers at the University of Cambridge have tackled this problem with an approach that uses microfluidic channels, soft robotic materials, and flexible electronics to be extensive and non-invasive.--About the podcast:Every day, some of the most innovative universities, companies, and individual technology developers share their knowledge on Wevolver. To ensure we can also provide this knowledge for the growing group of podcast listeners, we started a collaboration with two young engineers, Daniel Scott Mitchell &amp; Farbod Moghaddam who discuss the most interesting content in this podcast series.&nbsp;To learn more about this show, please visit the&nbsp;<a href="https://www.wevolver.com/profile/the.next.byte" target="_blank">shows page</a>. By following the page, you will get automatic updates by email when a new show is published.Be sure to give us a follow and review on&nbsp;<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" rel="nofollow" target="_blank">Apple</a> podcasts,&nbsp;<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" rel="nofollow" target="_blank">Spotify</a>, and most of your favorite podcast platforms!

In this episode, we talk about a graduate student’s thesis which resulted in a robot that’ll allow you to hug loved ones that are far away, how artificial photosynthesis might hold the key to our energy crisis, and a novel technique to conduct spinal stimulation therapy via non-invasive surgery.

Podcast: Hugging Robots, Scalable Solar Fuel, Non-Invasive Spinal Implants

In this episode, we talk about a research from Penn State that has proven the concept of printing tissue during surgery, a joint effort between MIT &amp; Harvard to develop a hair brushing robot, and a breakthrough in anode free and sodium based batteries coming out of Washington University in St. Louis. &nbsp;As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/9c426031-52da-4f56-801d-0370ae454607?dark=false" class="fr-draggable"></iframe>&nbsp;<h3>EPISODE NOTES</h3>(0:50) -&nbsp;<a href="https://www.wevolver.com/article/skin-and-bones-repaired-by-bioprinting-during-surgery" target="_blank">Bioprinting During Surgery</a>:&nbsp;Trauma to the head or face can be difficult to address and oftentimes requires bone grafts from the patient or cadaverous. Professor Ibrahim Ozbolat’s team at Penn State has addressed this by developing bio inks for hard and soft tissue enabling in situ bioprinting during surgery.&nbsp;(7:40) -&nbsp;<a href="https://www.wevolver.com/article/untangle-your-hair-with-help-from-robots" target="_blank">Hairbrush Robot</a><a href="https://www.wevolver.com/article/organic-material-may-help-design-energy-efficient-heating-and-cooling-systems" target="_blank">:</a>Nurses spend 18-40% of their time on direct patient care which involves hygienic maintenance like brushing hair. A joint effort from MIT &amp; Harvard aims to alleviate so of their workload by automating the hair brushing process using robots.(13:10) -&nbsp;<a href="https://www.wevolver.com/article/bai-lab-develops-stable-efficient-anode-free-sodium-battery" target="_blank">Anode Free Batteries</a>:Lithium ion batteries power almost all major electronic appliances that we use today; however the main ingredient is scarce and therefore ~25% of the cost we pay is just for the battery. Professor Pen Bai from Washington University wants to address this issue using their new anode free sodium based battery cells that were developed.--About the podcast:Every day, some of the most innovative universities, companies, and individual technology developers share their knowledge on Wevolver. To ensure we can also provide this knowledge for the growing group of podcast listeners, we started a collaboration with two young engineers, Daniel Scott Mitchell &amp; Farbod Moghaddam who discuss the most interesting content in this podcast series.&nbsp;To learn more about this show, please visit the&nbsp;<a href="https://www.wevolver.com/profile/the.next.byte" target="_blank">shows page</a>. By following the page, you will get automatic updates by email when a new show is published.Be sure to give us a follow and review on <a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" rel="nofollow" target="_blank">Apple</a> podcasts, <a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" rel="nofollow" target="_blank">Spotify</a>, and most of your favorite podcast platforms!

In this episode, we talk about a research from Penn State that has proven the concept of printing tissue during surgery, a joint effort between MIT & Harvard to develop a hair brushing robot, and a breakthrough in anode free and sodium based batteries coming out of Washington University in St. Louis

Podcast: Tissue Printing During Surgery, Hair Brushing Robots, Smaller & Cheaper Batteries

After the introduction of digital tools in design and creative disciplines almost three decades ago, there has been an ongoing interest in connecting the physical and the digital worlds. Most of these hybridizations have been directly influenced by the industrial/engineering origins of some of these digital technologies. For instance, using CNC/Robotic arms as tools for fabricating digitally designed artifacts or seeking time/cost-efficient production methods are a few examples.&nbsp;Although there is a tremendous value in this "tool-oriented" perspective, there seems to be a missed opportunity to use these digital/physical platforms, not only as "tools"--for translating one platform to another, but to combine them into a synthesized mixed design/experience "medium"; a mixed medium where it benefits from both digital and physical to curate a new design process, artifact or experience.&nbsp;In one of their latest experimental research projects Mixed Robotic Interface (MRI Γ), researchers at the Robotically Augmented Design (RAD) Lab at the College of Architecture and Environmental Design (CAED) at Kent State University, explored potentials of robotics and augmented reality technologies as possible mediums for hybrid cyberphysical storytelling and atmosphere making. Using the natural power of robotics and AR as platforms that co-exist in both digital and physical environments, MRI Γ explores possibilities of creating/curating a hybridized experience arrangement, to narrate a story with actual and virtual actors and setups.&nbsp;Developed as part of the MRI Γ research investigation led by Ebrahim Poustinchi assistant professor of architecture and the founder/director of the RAD Lab, graduate students Bradly Bowman and Trever Swanson formed a performance setup for storytelling, using a combination of physical images/posters, AR content as the performance set/stage, and an industrial robotic arm as the performer.&nbsp;<iframe src="https://player.vimeo.com/video/447412833" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 778px; height: 565px;"></iframe> The story—called “Child Robot!,” is based on a para-fictional scenario speculating on a future child room where the robot arm plays the role of the child, illustrating a futuristic/post-human ambiance. As a cyberphysical storytelling arrangement that simultaneously uses element from the actual and virtual to narrate, precision, and coordination between the digital and the physical is crucial. The setup is a product of a precise calibration of the physical robot arm—KR6 R900 Sixx KUKA robot, the physical and virtual room, and the virtually animated toys in the scene (child-robot room) coordinated in a responsive time-based manner. &nbsp;Using augmented reality procedures in conjunction with the physical setup, “Child Robot!” enables the users to experience a blend of reality, fantasy, actual and virtual through a custom-made software application, curated motion, and a physical scene.&nbsp;Powered using Unity 3D game engine, custom animation based robotic motion-control platform—Oriole, and image tracking techniques, “Child Robot!” facilitates its audience to experience the story—and its hybridized digital/physical content, from multiple points of view and perspectives. Liberating the experience from both completely immersive digital experience—VR experience, and a physical single perspective performance—regular physical experience, MRI blurs the boundaries between the audience, the performer, and the performance setup.&nbsp;To synchronize and coordinate the robot's movement and the digital game events in the AR platform, two sets of data have been used. A physical image-tracker works as an anchor for the digital—AR, content, and establishes a coordinate system for the scene's virtual overlay. This tracker enables the coordination between the robot's static position and the static framing of the digital setup. However, for the robot—from the physical world, to interact with the digital game objects, there is a need for realtime communication. Throughout the experience/performance, the robot continually sends its realtime position/orientation (TCP Plane data) to Unity 3D game engine. This information enables interaction between the game objects—toys, and the robot.To overlay the digital renderings—AR material, over the real/physical footage of the setup without overwriting the robot itself, RAD Lab researchers used greenscreen techniques to separate the robot from its physical context. Later in the process by using a custom-made “Child Robot!” AR application, and employing Chroma Key image processing techniques, the green background has been replaced with a coordinated digital content in realtime.&nbsp;While holding a device—e.g., cellphone, equipped with the custom-made “Child Robot!” AR application, the audience can walk through the MRI Γ setup, physically interact with the robot—the performer, and watch the playful interaction of the physical robot and the digital room.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1597308421054-003.jpg">The medium for experiencing the “Child-Robot! ”is a heterogeneous mixture of reality and its augmentation through the lens of a custom-made cell-phone app. The robot-arm as the performer/narrator of the story remains connected to the world of physicality while the digital window augments its surroundings.Although MRI Γ research—and “Child Robot!” as one of its examples, are at the early stages of experimentation and development, one of the main aims of this research is to move beyond immersion—both complete digital immersion (VR) or physical. It seeks a hybrid medium—a mixed interface, where the physicality of the sound, presence, and breeze of the robot’s motion, combines with the digitality of rendered animated content of AR to narrate one consistent story and create/curate one cohesive cyberphysical experience.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1597308435023-002.jpg">Robot arm interacting with virtual objects through synchronized digital/physical realtime animation.

Using Chroma Key image processing techniques in Unity 3D, the green screen is being replaced by the animated digital scene.

Using AR and robotics as Storytelling Mediums

Mixed Robotic Interface (MRI Gamma)

If you&rsquo;ve ever been caught in the dreaded sidewalk tango, failing to gracefully maneuver around someone directly in your path, you have some idea of the challenges ahead for a Stanford University robot, JackRabbot 2. With a squat body, friendly eyes and a two-fingered arm, JackRabbot 2 is tasked with learning all it can about moving with, around and between humans.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1571628349145-JackRabbot-3709-555x833.jpg" style="width: 766px;" class="fr-fic fr-dib">JackRabbot 2 has an arm and expressive eyes, which can help communicate the robot&rsquo;s next move to people around it.&nbsp;(Image credit: Amanda Law)&nbsp;The new robot &ndash; an update of the stylish&nbsp;<a href="https://news.stanford.edu/2016/06/01/stanfords-social-robot-jackrabbot-seeks-understand-pedestrian-behavior/" target="_blank">JackRabbot</a> &ndash; is part of a field of research interested in getting robots closer to humans so they can work as generalized personal helpers: delivering packages, cleaning house and grabbing a snack from the fridge.&ldquo;The JackRabbot project is developing a robot that doesn&rsquo;t just navigate an environment by following the behavior of a traditional robotic system, such as going from Point A to Point B and avoiding bumping into obstacles,&rdquo; said&nbsp;<a href="https://profiles.stanford.edu/silvio-savarese" target="_blank">Silvio Savarese</a>, associate professor of computer science, who leads the JackRabbot project. &ldquo;We want a robot that is also aware of the surroundings and the social aspects of human-robotics interactions, so it can move among humans in a more natural way.&rdquo;Right now, the researchers are testing the robot&rsquo;s new features &ndash; its predecessor had neither an arm nor a face &ndash; and continuing to gather a massive amount of detailed data for the algorithm that will inform JackRabbot 2&rsquo;s socially aware autonomous navigation. The&nbsp;<a href="http://cvgl.stanford.edu/projects/jackrabbot/" target="_blank">Stanford Vision and Learning Lab,</a> home to JackRabbot 2, officially unveiled the new robot at an on-campus event Sept. 20.<h2>No, you first</h2>Robotic arms are usually used to interact with physical objects, such as opening a door or picking up a container. JackRabbot 2&rsquo;s arm will be capable of these actions but will also convey intention &ndash; signaling a person to go ahead or stop with hand gestures. Facial expressions and sounds will also help JackRabbot communicate with the people around it.JackRabbot 2 is packed with sensors to help it navigate the world. These include multiple depth-sensing and stereo cameras, GPS and three LIDAR sensors, which send and detect pulsed laser light to define the surrounding environment in three dimensions.The researchers estimate they will need at least 24 hours of data in various environments &ndash; indoors and outdoors during different times of day, in crowds of varying density and configuration &ndash; to teach the artificially intelligent algorithm that will allow JackRabbot 2 to navigate autonomously with humanlike etiquette. The algorithm is designed so that it could work with other, similar robots as well.&ldquo;There are many behaviors that we humans&nbsp;subconsciously follow &ndash; when I&rsquo;m walking through crowds, I maintain personal distance or, if I&rsquo;m talking with you, someone wouldn&rsquo;t go between us and interrupt,&rdquo; said Ashwini Pokle, a graduate student in the Stanford Vision and Learning Lab. &ldquo;We&rsquo;re working on these deep learning algorithms so that the robot can adapt these behaviors and be more polite to people.&rdquo;The data collection process includes training the algorithm on pre-existing videos featuring all kinds of modes of transportation: walkers, skateboarders, bicyclists and people on scooters. The group will also crowdsource data from a video game where people maneuver a simulation of JackRabbot 2 through online environments. And for some real-world exposure, the researchers take JackRabbot 2 out for walks, navigating it manually in the way they want it to someday move itself.&ldquo;Every time we go around campus or outside, we want to collect all of the data that the robot perceives through its sensors,&rdquo; said Roberto Mart&iacute;n-Mart&iacute;n, a postdoctoral fellow in the Stanford Vision and Learning Lab. &ldquo;We train the robot to understand these data and infer things like &lsquo;Where are people in these images?&rsquo; or &lsquo;How do I move if I receive this sensor data?&rsquo;&rdquo;Both JackRabbots are intentionally cute &ndash; and named for Stanford&rsquo;s resident jackrabbits. Since these robots are designed to move among us without supervision, the researchers wanted to make people comfortable around them. They did such a good job of making them approachable, however, that the robots have also had to learn what an in-coming hug looks like and how to pause for selfies.<h2>The last-mile problem</h2>Although the JackRabbot project may be a step toward personal robots, the most immediate application is the &ldquo;last-mile problem.&rdquo; This is the attempt to bridge the gap between large-scale delivery hubs and individual businesses or homes &ndash; an environment that is drastically different and less defined by rules and boundaries than roads where autonomous cars roam.One day, JackRabbot descendants may be a regular sight, grabbing packages from an autonomous delivery truck and bringing them to your front door. At Stanford, this may be an imminent reality as the researchers are hoping to have JackRabbot 2 try out its delivery-bot skills on campus.

The JackRabbot project team with JackRabbot 2 (from left to right): Patrick Goebel, Noriaki Hirose, Tin Tin Wisniewski, Amir Sadeghian, Alan Federman, Silvio Savarese, Roberto Martín-Martín, Pin Pin Tea-mangkornpan and Ashwini Pokle (Image credit: Amanda Law)

Like its predecessor, JackRabbot 2 is learning how to navigate safely through spaces occupied by people, following the rules of human etiquette. What it learns could help it move comfortably among us in the future.

Stanford's JackRabbot 2: The polite pedestrian robot

Shimon, a robot in the Center of Music Technology and School of Music.

social robotics

Profiles