CUDA Cores and Tensor Cores are specialized units within NVIDIA GPUs; the former are designed for a wide range of general GPU tasks, while the latter are specifically optimized to accelerate AI and deep learning through efficient matrix operations.

Tensor Cores vs CUDA Cores: The Powerhouses of GPU Computing from Nvidia

Neural networks have made a seismic impact on how engineers design controllers for robots, catalyzing more adaptive and efficient machines. Still, these brain-like machine-learning systems are a double-edged sword: Their complexity makes them powerful, but it also makes it difficult to guarantee that a robot powered by a neural network will safely accomplish its task. The traditional way to verify safety and stability is through techniques called Lyapunov functions. If you can find a Lyapunov function whose value consistently decreases, then you can know that unsafe or unstable situations associated with higher values will never happen. For robots controlled by neural networks, though, prior approaches for verifying Lyapunov conditions didn’t scale well to complex machines.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and elsewhere have now developed new techniques that rigorously certify Lyapunov calculations in more elaborate systems. Their algorithm efficiently searches for and verifies a Lyapunov function, providing a stability guarantee for the system. This approach could potentially enable safer deployment of robots and autonomous vehicles, including aircraft and spacecraft.To outperform previous algorithms, the researchers found a frugal shortcut to the training and verification process. They generated cheaper counterexamples — for example, adversarial data from sensors that could’ve thrown off the controller — and then optimized the robotic system to account for them. Understanding these edge cases helped machines learn how to handle challenging circumstances, which enabled them to operate safely in a wider range of conditions than previously possible. Then, they developed a novel verification formulation that enables the use of a scalable neural network verifier, α,β-CROWN, to provide rigorous worst-case scenario guarantees beyond the counterexamples.“We’ve seen some impressive empirical performances in AI-controlled machines like humanoids and robotic dogs, but these AI controllers lack the formal guarantees that are crucial for safety-critical systems,” says Lujie Yang, MIT electrical engineering and computer science (EECS) PhD student and CSAIL affiliate who is a co-lead author of a new paper on the project alongside Toyota Research Institute researcher Hongkai Dai SM ’12, PhD ’16. “Our work bridges the gap between that level of performance from neural network controllers and the safety guarantees needed to deploy more complex neural network controllers in the real world,” notes Yang.For a digital demonstration, the team simulated how a quadrotor drone with lidar sensors would stabilize in a two-dimensional environment. Their algorithm successfully guided the drone to a stable hover position, using only the limited environmental information provided by the lidar sensors. In two other experiments, their approach enabled the stable operation of two simulated robotic systems over a wider range of conditions: an inverted pendulum and a path-tracking vehicle. These experiments, though modest, are relatively more complex than what the neural network verification community could have done before, especially because they included sensor models. “Unlike common machine learning problems, the rigorous use of neural networks as Lyapunov functions requires solving hard global optimization problems, and thus scalability is the key bottleneck,” says Sicun Gao, associate professor of computer science and engineering at the University of California at San Diego, who wasn’t involved in this work. “The current work makes an important contribution by developing algorithmic approaches that are much better tailored to the particular use of neural networks as Lyapunov functions in control problems. It achieves impressive improvement in scalability and the quality of solutions over existing approaches. The work opens up exciting directions for further development of optimization algorithms for neural Lyapunov methods and the rigorous use of deep learning in control and robotics in general.”Yang and her colleagues’ stability approach has potential wide-ranging applications where guaranteeing safety is crucial. It could help ensure a smoother ride for autonomous vehicles, like aircraft and spacecraft. Likewise, a drone delivering items or mapping out different terrains could benefit from such safety guarantees.The techniques developed here are very general and aren’t just specific to robotics; the same techniques could potentially assist with other applications, such as biomedicine and industrial processing, in the future. While the technique is an upgrade from prior works in terms of scalability, the researchers are exploring how it can perform better in systems with higher dimensions. They’d also like to account for data beyond lidar readings, like images and point clouds.As a future research direction, the team would like to provide the same stability guarantees for systems that are in uncertain environments and subject to disturbances. For instance, if a drone faces a strong gust of wind, Yang and her colleagues want to ensure it’ll still fly steadily and complete the desired task.&nbsp; Also, they intend to apply their method to optimization problems, where the goal would be to minimize the time and distance a robot needs to complete a task while remaining steady. They plan to extend their technique to humanoids and other real-world machines, where a robot needs to stay stable while making contact with its surroundings. Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at TRI, and CSAIL member, is a senior author of this research. The paper also credits University of California at Los Angeles PhD student Zhouxing Shi and associate professor Cho-Jui Hsieh, as well as University of Illinois Urbana-Champaign assistant professor Huan Zhang. Their work was supported, in part, by Amazon, the National Science Foundation, the Office of Naval Research, and the AI2050 program at Schmidt Sciences. The researchers’ paper will be presented at the 2024 International Conference on Machine Learning.

MIT CSAIL researchers helped design a new technique that can guarantee the stability of robots controlled by neural networks. This development could eventually lead to safer autonomous vehicles and industrial robots. Image: Alex Shipps/MIT CSAIL

Neural network controllers provide complex robots with stability guarantees, paving the way for the safer deployment of autonomous vehicles and industrial machines.

Creating and verifying stable AI-controlled systems in a rigorous and flexible way

The world of artificial intelligence (AI) has recently seen significant advancements in generative models, a type of machine-learning algorithms that “learn” patterns from set of data in order to generate new, similar sets of data. Generative models are often used for things like drawing images and natural language generation – a famous example are the models used to develop chatGPT.Generative models have had remarkable success in various applications, from image and video generation to composing music and to language modeling. The problem is that we are lacking in theory, when it comes to the capabilities and limitations of generative models; understandably, this gap can seriously affect how we develop and use them down the line.One of the main challenges has been the ability to effectively pick samples from complicated data patterns, especially given the limitations of traditional methods when dealing with the kind of high-dimensional and complex data commonly encountered in modern AI applications.Now, a team of scientists led by Florent Krzakala and Lenka Zdeborová at EPFL has investigated the efficiency of modern neural network-based generative models. The study, now published in&nbsp;PNAS, compares these contemporary methods against traditional sampling techniques, focusing on a specific class of probability distributions related to spin glasses and statistical inference problems.The researchers analyzed generative models that use neural networks in unique ways to learn data distributions and generate new data instances that mimic the original data.The team looked at flow-based generative models, which learn from a relatively simple distribution of data and “flow” to a more complex one; diffusion-based models, which remove noise from data; and generative autoregressive neural networks, which generate sequential data by predicting each new piece based on the previously generated ones.The researchers employed a theoretical framework to analyze the performance of the models in sampling from known probability distributions. This involved mapping the sampling process of these neural network methods to a Bayes optimal denoising problem – essentially, they compared how each model generates data by likening it to a problem of removing noise from information.The scientists drew inspiration from the complex world of spin glasses, materials with intriguing magnetic behavior, to analyze modern data generation techniques. This allowed them to explore how neural network-based generative models navigate the intricate landscapes of data.The approach allowed them to study the nuanced capabilities and limitations of the generative models against more traditional algorithms like Monte Carlo Markov Chains (algorithms used to generate samples from complex probability distributions) and Langevin Dynamics (a technique for sampling from complex distributions by simulating the motion of particles under thermal fluctuations).The study revealed that modern diffusion-based methods may face challenges in sampling due to a first-order phase transition in the algorithm’s denoising path. What this means is that they can run into problems because of sudden change in how they remove noise from the data they're working with. Despite identifying regions where traditional methods outperform, the research also highlighted scenarios where neural network-based models exhibit superior efficiency.This nuanced understanding offers a balanced perspective on the strengths and limitations of both traditional and contemporary sampling methods. The research is a guide to more robust and efficient generative models in AI; by providing a clearer theoretical foundation, it can help develop next-generation neural networks capable of handling complex data generation tasks with unprecedented efficiency and accuracy.<hr>ReferencesDavide Ghio, Yatin Dandi, Florent Krzakala, Lenka Zdeborovà. Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective. PNAS 24 June 2024. DOI:&nbsp;<a href="https://doi.org/10.1073/pnas.2311810121" target="_blank" rel="noopener">10.1073/pnas.2311810121</a>

Researchers at EPFL have made a breakthrough in understanding how neural network-based generative models perform against traditional data sampling techniques in complex systems, unveiling both challenges and opportunities for AI's future in data generation.

Navigating the labyrinth: How AI tackles complex data sampling

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies. However, because it is being introduced to production sites in the manufacturing industry, machine vision generally refers to digital image technologies utilized in automatic inspection and process automation in factories. Machine vision consists of the following:・Imaging section (cameras) ・Interface section (board/general-purpose interface) ・Processing engine section (computer) ・Processing control section (software)etc. The image data obtained by the imaging section is loaded into the processing engine via the interface section. In the processing engine section, the processing control section performs binarization*1, edge detection*2, and other forms of image processing. Next, it outputs the image data that has completed image processing to robots and industrial equipment.*1: Processing that converts the pixel values (threshold values) that are preset for the image to the two values of black and white. *2: Processing that detects the bright and dark areas in an image.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327121894-img-0001-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: Machine vision configurationMachine vision is used in automatic inspection, automatic recognition, and automatic measurement, etc. For example, automatic inspection and automatic recognition are very fast compared to manual inspection. Moreover, machine vision used in automatic measurement can accurately identify small value differences that humans cannot measure. In addition, linking machine vision to AI functions is expected to produce more intelligent uses going forward.<h2>Differences Between Machine Vision, Computer Vision, and Robot Vision</h2>Computer vision and robot vision are technologies that are similar to machine vision, but they each play different roles. Here, we explain how their roles differ.<h3>Differences Between Machine Vision and Computer Vision</h3>"Computer vision" is a technology that adds equipment control functions to machine vision functions. Computer vision can control various types of equipment based on data output by machine vision. For example, this would make it possible to combine the image data output by machine vision with CAD data generated by the computer to control processors or to adjust the conveyor transport speed based on image data that captured the progress of line work. In addition, computer vision has a broad scope of utilization that naturally includes smart factories as well as referee assistance systems and virtual replays in sports and projection mapping, etc. in entertainment.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327141371-img-0002-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: Example of computer vision utilization<h3 id="isPasted">Differences Between Machine Vision and Robot Vision</h3>"Robot vision" is a technology that adds machine vision to robot visual functions. Machine vision attaches cameras to various types of equipment to obtain image data, but robot vision attaches cameras to the end of robot arms to obtain image data. This enables the robot to operate based on visual information obtained by itself. For example, a robot that does not use robot vision can only execute certain operations programmed in advance. However, when a robot equipped with robot vision moves its robot arm closer to an object, it then sequentially identifies the target based on image data from the camera to execute operations. Because this type of operational control is possible, a robot equipped with robot vision is capable of detailed, precise, and flexible operations that cannot be specified with a program.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327158790-img-0003-en.png" style="width: 100%px;" class="fr-fic fr-dib">Figure 3: Example of robot vision utilization<h2 id="isPasted">Example of Machine Vision Utilization in a Smart Factory</h2>Machine vision is used in various types of work in a smart factory. Here, we introduce an example of machine vision utilization.<h3>Reading Characters at Ultrahigh-speed – OCR (Optical Character Recognition/Reader)</h3>OCR is a technology that scans handwritten or printed characters with an image scanner or digital camera. Machine vision scans the 2D code, serial number, and date of manufacture, etc. printed on products. The scanned data is utilized in inspections of the print quality or whether the correct characters are printed as well as the screening of the next process, etc. In many cases, the products being scanned by OCR pass through the line at high speeds. Therefore, machine vision uses cameras that can capture images at high speed and image processing systems that are capable of high-speed processing.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327179168-img-0004+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 4: OCR on a manufacturing line<h3 id="isPasted">Scanning Shapes in 2D and 3D – Visual Inspections</h3>The utilization of machine vision is considered to be the most effective means of visual inspection on a manufacturing line. Because machine vision-based visual inspection is able to inspect the accuracy of products after cutting and molding and the mounting condition of components, etc. with high speed and accuracy, it is essential for improving production efficiency and maintaining quality. The surfaces of target objects can vary widely in terms of their glossiness and light absorption, etc. The lighting used in machine vision enables products with various surface conditions to be inspected by changing the method of illumination and the color of the light rays. In addition, 3D cameras equipped with a stereophonic*3, time-of-flight*4, and other measurement systems as well as structured light*5 and other imaging systems can also measure the depth.*3: Method that uses two cameras to perform triangulation and measure the distance to an object. *4: Method that measures the distance to an object according to the time it takes for light emitted by a light source to reflect off of the object and return to a light detector. *5: Lighting method that emits light with a specific shading pattern.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327197939-img-0005.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 5: Visual inspection with a 3D camera<h3 id="isPasted">Grabbing Specific Components – Picking</h3>"Picking" on a manufacturing line refers to the task of collecting (pickup) the necessary components and products. Utilizing machine vision makes it possible to identify and pick components with the specified color and shape even if components and products with various colors and shapes are mixed together in one case.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327216910-img-0006.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 6: Picking by a robot<h3 id="isPasted">Providing Visual Functions to Robots – Robot Operational Control</h3>In recent years, industrial robots have also been used in precision work due to their high operating precision. For example, they play an active role in installing, soldering, and welding microcomponents. However, controlling such work with programming requires high-precision positioning devices. With robot vision utilizing machine vision, high-precision positioning devices are not required for the robot to perform precise positioning with the image data from the camera mounted on the end of the robot arm.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327233332-img-0007.jpg" style="width: 100%px;" class="fr-fic fr-dib">Figure 7: Installing microcomponents with a robot<h2 id="isPasted">Future of Machine Vision – AI Vision</h2>As various types of industrial equipment shift to using AI, one form of machine vision evolution that shows promise is "AI vision."<h3>What is AI Vision?</h3>AI vision is a system that equips lightweight AI chips in smartphones, etc. with cameras and camera functions to make inferences about captured image data. AI vision resembles computer vision in that it uses AI (artificial intelligence), but computer vision mainly makes inferences with a cloud AI service (Figure 8). On the other hand, AI vision equips an AI module in an edge device to make inferences, and the inference results are stored on the edge server. The edge server communicates with the cloud AI only when necessary. (Figure 9).<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1721327267580-1721327267580.png" class="fr-fic fr-dib">Therefore, AI vision has the following advantages over computer vision: ・Able to decrease the volume of communication with the cloud AI service ・Able to prevent external leaks of data collected by the sensors ・Superior responsiveness Furthermore, the cameras and smartphones used in AI vision are called "edge AI devices" while the method that makes inferences on the edge device is called "edge AI."<h3>What Technologies Are Required to Realize AI Vision?</h3>Although AI vision has the advantages described above, AI modules equipped on edge devices require the following types of functions. ・Superior energy conservation due to restrictions on battery capacity ・Ability to perform high-speed processing due to the responsiveness requirement ・Superior heat dissipation due to installation in a limited space More than anything, it is important that the AI modules be compact.<h2>Summary</h2>This article explained the topic of machine vision and the similar phrases of computer vision and robot vision as well as AI vision. Going forward, it is expected that machine vision and AI will evolve in a close relationship with the evolution in deep learning for image data. In addition, new AI systems and products are expected to be developed through the proper use of cloud AI and edge AI. These trends will likely become essential technologies for smart factory robots, automatic inspection, automatic recognition, and automatic measurement equipment that require visual information that is closer to the human eye and decision-making capabilities that exceed that of humans.

In a broad sense, "machine vision" refers to the utilization of visual information and optical information using digital technologies.

The Eyes of Smart Factories - What is Machine Vision?

Proteins have evolved to excel at everything from contracting muscles to digesting food to recognizing viruses. To engineer better proteins, including antibodies, scientists often iteratively mutate the amino acids – the units that are arranged in a sequence to make up proteins – in different positions until the resulting protein has an improved function, like eliciting a stronger immune response or capturing carbon dioxide from the atmosphere more efficiently.But there are more possible amino acid sequences than there are grains of sand in the world. And finding the best protein and, therefore, the best potential drug, is often expensive or impossible.Stanford scientists have developed a new machine learning-based method to more quickly and accurately predict the molecular changes that will lead to better antibody drugs. Published in Science&nbsp;on July 4, the approach combines the 3D structure of the protein backbone with large language models based on amino acid sequence, and allows researchers to find, in minutes, rare and desirable mutations that would otherwise only be found with exhaustive experiments.Led by <a href="https://profiles.stanford.edu/peterkim" data-extlink="" target="_blank">Peter S. Kim</a>, professor of biochemistry and institute scholar at <a href="https://chemh.stanford.edu/" data-extlink="" target="_blank">Sarafan ChEM-H</a>, and <a href="https://profiles.stanford.edu/brian-hie" data-extlink="" target="_blank">Brian Hie</a>, assistant professor of chemical engineering, the team showed that they could improve a once FDA-approved SARS-CoV-2 antibody that had been discontinued due to its ineffectiveness against a new strain in November 2022. Their approach resulted in a 25-fold improvement against the virus.“A lot of effort in AI and drug development is centered around amassing tons of data about how well a certain molecule performs a certain task so that a computer can learn enough to design a better version,” said Kim. “What’s remarkable is that we’ve shown that structure can be used in lieu of a lot of that data, and the computer will still learn.”“Now, more antibodies actually get a shot at being optimized,” said Hie, who is also an innovation investigator at the<a href="https://arcinstitute.org/" data-extlink="" target="_blank">&nbsp;Arc Institute</a>.<h2>Bent into shape</h2>When faced with the challenge of finding the best amino acid sequence, scientists will often make millions and test them in miniaturized, simplified versions of biological systems. They hope that the best drug in a dish will also be the best drug in humans.“It’s a lot of guess and check,” said Hie. “The goal of a lot of intelligent algorithms is to remove the guesswork from this.”To speed up the process, scientists have developed ChatGPT-like machine learning algorithms that are trained on the amino acid sequences of millions of proteins to predict desirable mutations.These models, however, often point scientists toward sequences that, once produced in the lab, are unstable or worse than where they started.This is partially because protein function depends not only on the sequence of amino acids but also on the 3D structure of that sequence. For example, to trigger an immune response, antibodies must be the right shape to bind to molecules that sit atop the surface of viruses.The key, the team thought, to developing a better prediction algorithm was structure. So, they constrained the long list of possible beneficial mutations – as determined by the sequence-based large language model – to only those that would preserve the 3D shape of the starting protein.&nbsp;<h2>Testing ground</h2>In December 2022, the team put it to the test on a recently discontinued SARS-CoV-2 antibody therapy.“The prevailing theory was that trying to improve this antibody would fail,” said<a href="https://profiles.stanford.edu/varun-shanker" data-extlink="" target="_blank">&nbsp;Varun Shanker</a>, a medical student, graduate student in biophysics, and lead author on the study. “The virus was too smart. It evolved as it spread through millions of people to know exactly how to mutate to avoid these antibodies.”Using purely sequence-based models to optimize the protein resulted in a modest twofold increase in effectiveness. But with their structure-guided approach, the team saw a 25-fold increase.“We were finally catching up to the virus,” said Shanker, who is also a fellow in the Chemistry/Biology Interface Training Program at Sarafan ChEM-H.&nbsp;<h2>Teaching an old model new tricks</h2>Most efforts in using AI to build better drugs rely on “training” or “supervising” the model, which involves generating huge amounts of data about the function and performance of unique protein sequences. This approach takes a lot of time, and results in a model tailored to specific protein performing a specific task.This model does not require any input about what the protein does, how well it does it, or any lab experiments. Because structure is so closely tied to function, the protein’s coordinates become a proxy for performance. For the COVID antibody work, they constrained the structure not just to the antibody itself, but to the antibody when it is bound to the virus. From there, their model “learned” some rules of antibody binding without ever needing to be taught.Early experiments show that the approach is generalizable to other kinds of proteins, like enzymes, which help catalyze chemical reactions in our bodies. So far, the researchers have found that the model points scientists to tens of proteins, and, on average, half are better than the starting point.This tool could be useful to quickly respond to emerging or evolving diseases. It also lowers the barrier to making more effective medicines. Stronger medicines mean lower doses are necessary, which means that a given quantity could benefit more patients. For infectious diseases like HIV, where studies have shown that large but infrequent doses of an antibody can protect patients from infection, this could be transformational.The team is making their model and code freely available to anyone.“This is an exciting example of the power of deep learning to democratize the process of building better proteins,” said Shanker. “This not only allows people to develop new medicines, but also opens up new areas of scientific exploration that had been inaccessible.”Theodora Bruun, a&nbsp;<a href="https://knight-hennessy.stanford.edu/" data-extlink="" target="_blank">Knight-Hennessy Scholar</a>, is a co-author on the study. Kim is a member of&nbsp;<a href="http://biox.stanford.edu/" data-extlink="" target="_blank">Bio-X</a>, the&nbsp;<a href="https://humanperformance.stanford.edu/" data-extlink="" target="_blank">Wu Tsai Human Performance Alliance</a>, the&nbsp;<a href="http://chri.stanford.edu/" data-extlink="" target="_blank">Maternal &amp; Child Health Research Institute</a>, and the&nbsp;<a href="https://neuroinstitute.stanford.edu/" data-extlink="" target="_blank">Wu Tsai Neurosciences Institute</a>. Hie is a member of&nbsp;<a href="http://biox.stanford.edu/" data-extlink="" target="_blank">Bio-X</a>, the Dieter Schwarz Foundation SDS Faculty Fellow at&nbsp;<a href="https://datascience.stanford.edu/" data-extlink="" target="_blank">Stanford Data Science</a>, and an Innovation Investigator at the&nbsp;<a href="https://arcinstitute.org/" data-extlink="" target="_blank">Arc Institute</a>. Shanker is part of the Stanford&nbsp;<a href="https://med.stanford.edu/mstp.html" data-extlink="" target="_blank">Medical Scientist Training Program</a>.&nbsp;

Biochemistry Professor Peter S. Kim co-led research on a new machine learning-based method for designing antibody drugs. The method was shown to improve a once FDA-approved SARS-CoV-2 antibody that had been discontinued due to ineffectiveness. | Steve Fisch

The method, which combines a ChatGPT-like large language model with information about a protein’s 3D shape, could make it easier and faster to develop better medicines for infectious diseases, cancer, and other conditions.

New AI approach optimizes development of antibody drugs

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/d97ba401-cff2-4367-adb7-043c01d4a88a?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>When Haoguang Cai and Khem Agarwal decided they wanted to work on a project to present at the annual IEEE Rising Stars conference, they knew they wanted to build something that would be of interest and use to the Institute for Electrical and Electronics Engineers audience of students and young technical professionals.The <a href="https://www.sv.cmu.edu/index.html" rel="noopener" target="_blank">Silicon ValleyOpens in new window</a> graduate students, who care about making AI that benefits people, came up with an idea that achieved both goals. They knew that IEEE Rising Star attendees are focused on career opportunities so they set to work devising a system that could help job seekers improve their performance and become more confident in job interviews.Cai and Agarwal also knew that many of their fellow students practiced interviewing with friends or colleagues. But when mock interview options were exhausted or not available, they still wanted more practice and helpful feedback.The system that the pair developed and later demonstrated at the Las Vegas conference analyzes a user’s facial and speech expression to detect what emotions they are conveying and tracks their gaze or eye motion to detect where they are focusing their attention. Feedback in both real time and after the interview session gives users insight as to how they are likely being perceived during an interview.“When I showed the live demo on stage, everyone wanted to try it. They really saw its potential and lined up after the workshop to try it and learn how we did it,” said Cai.<blockquote>When I showed the live demo on stage, everyone wanted to try it. They really saw its potential and lined up after the workshop to try it and learn how we did it.<cite>Haoguang Cai, MSSM Student, SV - III</cite></blockquote>The project originated in their Software Engineering Management class. Their instructor, <a href="https://engineering.cmu.edu/directory/bios/fang-catherine.html" target="_blank">Catherine Fang</a>, was scheduled to speak at the January 2024 conference. The distinguished service professor is an entrepreneur who has co-founded two companies, holds multiple U.S. patents, has worked for LinkedIn, Yahoo, and Sun Microsystems, and serves as an associate editor for the IEEE reliability society.“I’m so proud of all of our students, but this team really stood out. They have a very solid technical and product management background. Their presentation was a big success,” said Fang.<blockquote>I’m so proud of all of our students, but this team really stood out. They have a very solid technical and product management background. Their presentation was a big success.<cite>Catherine Fang, Distinguished Service Professor, SV - III</cite></blockquote>She added that emotion detection technology is an emerging trend with many useful consumer applications that she has seen demonstrated at other big conferences.Cai and Agarwal, who will both earn their Master of Science in Software Management (MSSM) from the Integrated Innovation Institute in December 2024, knew how to program the machine learning model to identify and measure facial expressions and eye tracking but they needed a data set with a large number of labelled images that represented a diverse set of people including those of varying color, age, and gender.They contacted Hume AI, a research lab and technology company that uses artificial intelligence to measure emotional expression, which was able to provide access to their extensive data set through use of their Application Programming Interface (API). APIs are commonly used in software development to enable integration between different systems or services for applications such as accessing data from a remote server.Jeremy Hadfield, who is a technical product manager at Hume, has supported the pair through the development of their emotion detection system. He was recently a guest at a campus event.Cai, who hopes to complete his transition from a machine learning engineer to product manager in the AI field after graduation, will intern with Hume AI this summer. Agarwal is planning for a career that will enable him to use AI to create virtual reality learning tools.They both agree that the challenge and extra work to build and demonstrate their emotion detection system was a worthwhile endeavor and that it only furthered their desire to use AI to build products that will help people.

Face expression analysis that users can view in real time while answering interview questions indicate the emotion they are conveying.

Silicon Valley students use AI to help job seekers improve performance and build confidence.

Emotion detection system puts a smile on their face

Hidden occupational health risks impact both employees and businesses across sectors such as manufacturing, logistics, retail, and construction. By addressing these risks early and promoting safer working conditions, companies can achieve a 46% increase in efficiency!&nbsp;&nbsp;To discuss the critical role of safe work environments, Xsens is hosting a crash course webinar series that consists of two chapters. The first chapter,&nbsp;<a href="https://www.movella.com/resources/live-stream/workplace-ergonomics-maximising-workplace-health-and-efficiency" target="_blank" rel="noreferrer noopener">Workplace Ergonomics: Maximize Employee Well-being and Productivity</a>, was held on April 30, and you can&nbsp;now&nbsp;<a href="https://bit.ly/3W7ePEn" target="_blank" rel="noopener noreferrer">register for the second session</a>, which takes place on July 9.&nbsp;&nbsp;<iframe width="640" height="360" src="https://www.youtube.com/embed/NxUrHnd9xmQ?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;</iframe>&nbsp;&nbsp;Here are the topics we discussed in the first chapter: &nbsp;&nbsp;<ol start="1"><li data-leveltext="%1." data-font="Aptos" data-listid="1" data-list-defn-props='{"335552541":0,"335559685":720,"335559991":360,"469769242":[65533,0],"469777803":"left","469777804":"%1.","469777815":"hybridMultilevel"}' data-aria-posinset="1" data-aria-level="1">What is workplace ergonomics, and how does it impact occupational health and safety?&nbsp;&nbsp;</li></ol>Workplace ergonomics is a critical aspect of occupational health &amp; safety. It involves tailoring workspaces and processes in line with human physical and cognitive abilities. It aims to reduce the risk of musculoskeletal disorders (MSDs) and enhance a facility's safety and productivity. Poor ergonomics can lead to significant financial impacts, including high injury costs and operational disruptions. On average, workplace injuries cost businesses over $20,000 per incident and result in about 20 days (about 3 weeks) of lost work time.&nbsp;&nbsp;<ol start="2"><li data-leveltext="%1." data-font="Aptos" data-listid="1" data-list-defn-props='{"335552541":0,"335559685":720,"335559991":360,"469769242":[65533,0],"469777803":"left","469777804":"%1.","469777815":"hybridMultilevel"}' data-aria-posinset="2" data-aria-level="1">Why is workplace ergonomics so important for organizations? &nbsp;&nbsp;</li></ol>Poor ergonomics in workplaces can result in high injury costs and challenges in maintaining operational continuity. Furthermore, companies need to abide by international standards, such as ISO, OSHA, and EN norms, that serve as objective tools for ergonomic assessments and imply legal consequences for non-compliant facilities.&nbsp;&nbsp;<ol start="3"><li data-leveltext="%1." data-font="Aptos" data-listid="1" data-list-defn-props='{"335552541":0,"335559685":720,"335559991":360,"469769242":[65533,0],"469777803":"left","469777804":"%1.","469777815":"hybridMultilevel"}' data-aria-posinset="3" data-aria-level="1">What are some of the different ergonomic assessment methods and tools often used by organizations?&nbsp;&nbsp;</li></ol>There are several options for conducting ergonomic assessment today:&nbsp;<ul><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props='{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}' data-aria-posinset="1" data-aria-level="1">Traditional ergonomic assessments rely on manual methods such as checklists and video recordings to observe worker postures and movements based on international standards such as ISO and OSHA.&nbsp;&nbsp;</li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props='{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}' data-aria-posinset="2" data-aria-level="1">Advanced software tools, including Siemens Process Simulate and CATIA, offer digital workspace design and ergonomic assessment capabilities.&nbsp;&nbsp;</li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props='{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}' data-aria-posinset="3" data-aria-level="1">AI-powered video analysis tools automate assessments by tracking joint angles and movements but may not fully capture dynamic work environments.&nbsp;&nbsp;</li><li data-leveltext="" data-font="Symbol" data-listid="2" data-list-defn-props='{"335552541":1,"335559685":720,"335559991":360,"469769226":"Symbol","469769242":[8226],"469777803":"left","469777804":"","469777815":"hybridMultilevel"}' data-aria-posinset="4" data-aria-level="1">Motion capture, allowing for rapid data collection and&nbsp;objective&nbsp;analysis, offers a quick return on investment by reducing injury rates and associated costs.&nbsp;&nbsp;</li></ul>Prioritizing improved ergonomics can profoundly impact worker well-being and satisfaction. Businesses can lower costs and boost productivity by reducing injuries and creating safer work environments. &nbsp;&nbsp;&nbsp;Do not miss the second chapter of our webinar series, where we will talk about the role of motion capture in promoting occupational health &amp; safety, real-life use cases, and ROI! Register today:&nbsp;<a href="https://bit.ly/3W7ePEn" target="_blank" rel="noopener noreferrer">https://bit.ly/3W7ePEn</a>&nbsp;&nbsp;

A woman wearing Xsens Awinda suit sorting our fruits in the supermarket environment. The pose analytics are visualized using Scalefit

Improved occupational health and safety can enhance productivity by 46% and boost employee well-being. To address these opportunities, Xsens is launching a crash course on topics like injury risk reduction, regulation compliance, and the positive influence of safer workspaces.

A crash course on occupational health & safety: safer workplaces for improved productivity

<table style="width: 100%;"><tbody><tr><td style="width: 100%;">SAE will host a two-day training course, "Machine Learning for Safety Experts," with their partner Fraunhofer IKS on October 22-23, 2024, in Munich. The course aims to equip safety engineers with the knowledge to integrate Machine Learning into safety-critical applications while adhering to standards like ISO 26262 and ISO 21448. Using established methods, participants will learn to develop safety lifecycles, derive safety requirements, and address Machine Learning insufficiencies. The training is essential for engineers looking to enhance their skills in ensuring the safety of Machine Learning based functions. <a href="https://www.sae.org/learn/content/c2208/?utm_source=https://www.sae.org/learn/content/c2208/&utm_medium=facebook,_email,_instagram,_linkedin_etc&utm_campaign=sae_ml_training" target="_blank" class="button-main-action">Read more or register for the event here</a>.</td></tr></tbody></table>Machine Learning (ML) is increasingly pivotal across various engineering disciplines, transforming problem-solving and innovation capabilities. In the automotive sector, Machine Learning drives advancements in autonomous driving systems, enhancing vehicle safety and functionality. Healthcare benefits from Machine Learning through improved diagnostic accuracy and patient care personalization, while industrial automation optimizes efficiency and predictive maintenance, reducing downtime and costs.However, as Machine Learning becomes integral to engineering solutions, the imperative for engineers to continuously advance their understanding of Machine Learning safety grows. Integrating Machine Learning into safety-critical applications presents a unique challenge, especially in adhering to stringent safety standards.The ISO 26262 and ISO 21448 standards are particularly pertinent, focusing on automotive safety and the safety of intended functionality, respectively. These standards guide engineers in developing, validating, and certifying Machine Learning based systems to ensure they meet rigorous safety requirements.&nbsp;The upcoming "Machine Learning for Safety Experts" training by SAE and Fraunhofer IKS in Munich is a timely initiative. It will address this crucial skill gap and ensure engineers are well-equipped to handle the intricacies of safe Machine Learning applications.<h2 dir="ltr">Ensuring Safety in Machine Learning Applications</h2>Integrating Machine Learning with established safety standards, particularly in sectors like automotive, where safety is paramount, presents significant challenges. The complexity of Machine Learning systems, which often behave unpredictably, necessitates adherence to robust safety protocols to mitigate risks not traditionally accounted for in older standards.<h3 dir="ltr">Overview of Relevant Safety Standards</h3>Both ISO 26262 and ISO 21448 are critical in guiding the integration of Machine Learning into safety-critical applications. They provide a structured framework for evaluating and managing the risks associated with Machine Learning applications in automotive and other engineering fields. This ensures that vehicles and other engineered systems become more autonomous and remain safe for end-users under all conditions.The interaction between these standards and Machine Learning development is a growing area of focus as engineers seek to align cutting-edge technological advancements with rigorous safety requirements. Ensuring the safety of Machine Learning applications requires a balance between innovation and stringent compliance with these evolving safety standards.<h3 dir="ltr">Challenges of Integration with Safety Standards</h3>One of the principal standards, ISO 26262, focuses on functional safety for all road vehicles, highlighting the need for risk assessment and mitigation strategies throughout the product lifecycle. This standard has been crucial in addressing risks associated with electrical and electronic systems in vehicles. However, the integration of Machine Learning brings new dimensions to these challenges because Machine Learning systems can fail in ways that are hard to predict with traditional safety analysis techniques.ISO 21448, also known as Safety of the Intended Functionality (SOTIF), complements ISO 26262 by addressing safety risks in vehicle operations that occur without system failures. This includes scenarios where vehicles perform as designed but fail to handle unexpected real-world conditions, such as unusual weather or road obstacles. SOTIF guides how to design, verify, and validate Machine Learning systems to ensure they behave safely under various operating conditions.<h2 dir="ltr">SAE Course</h2>The "Machine Learning for Safety Experts" course by SAE and Fraunhofer IKS is designed to train safety engineers in integrating machine learning into safety-critical applications.&nbsp;Scheduled for October 22-23, 2024, in Munich, this training focuses on applying fundamental safety principles of Machine Learning across various applications. Participants will learn to develop Machine Learning safety-related functions, derive safety requirements, and address Machine Learning insufficiencies through a comprehensive toolkit of methods. This course is crucial for engineers aiming to enhance the safety of Machine Learning functions in automotive and other critical sectors.&nbsp;For more details, <a href="https://www.sae.org/learn/content/c2208/?utm_source=https://www.sae.org/learn/content/c2208/&utm_medium=facebook,_email,_instagram,_linkedin_etc&utm_campaign=sae_ml_training" target="_blank">Check the course details here.</a><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE5NDk0MjIwMTMxLXJvYWQgY3Jvc3MuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Course participants will learn to develop Machine Learning-based safety functions, establish safety requirements, and address Machine Learning deficiencies using a comprehensive set of methods. Image credit: City of Bellevue, the UW and Microsoft.<h3 dir="ltr" id="isPasted">Key Learning Objectives of the Training</h3>The Machine Learning for Safety Experts training is designed to provide safety engineers with the essential knowledge and skills to integrate Machine Learning (ML) into safety-critical applications, ensuring compliance with standards like ISO 26262 and ISO 21448.<h4 dir="ltr">Understanding the Impact of Machine Learning on Functional Safety&nbsp;</h4>One of the primary learning outcomes is recognizing the profound impact of Machine Learning on functional safety. Participants will gain insights into how Machine Learning can influence the safety lifecycle of systems, especially in the context of automotive safety. This includes understanding the potential risks and the necessity of robust safety arguments to mitigate these risks effectively.<h4 dir="ltr">Applying Machine LearningSpecific Safety Measures&nbsp;</h4>The course emphasizes the application of Machine Learningspecific safety measures. This includes learning about the methodologies and tools tailored to address the unique challenges of Machine Learning algorithms. Participants will be trained on runtime monitoring, robustness testing, and other techniques to ensure that Machine Learning systems perform reliably under diverse conditions, aligning with the stringent requirements of ISO 26262 and SOTIF.<h4 dir="ltr">Developing Safety Requirements for Machine LearningBased Systems</h4>A crucial part of the training is the development of safety requirements specific to Machine Learningbased systems. Engineers will learn to derive and implement safety requirements from a system-level perspective, ensuring that Machine Learning components meet the necessary safety standards. This involves comprehensive training on validating and verifying Machine Learning systems to ensure they adhere to the safety protocols outlined in ISO 26262 and ISO 21448.This training equips engineers with the expertise to navigate the complexities of integrating Machine Learning into safety-critical systems, making it an essential program for those looking to stay ahead in the rapidly evolving field of safety engineering.<h2 dir="ltr">Practical Benefits of the Training</h2>The "Machine Learning for Safety Experts" training is invaluable for engineers addressing prevalent Machine Learning safety challenges in real-world applications. This training equips participants with the necessary skills and knowledge to effectively integrate Machine Learning into safety-critical systems, ensuring adherence to stringent safety standards like ISO 26262 and ISO 21448. The course will discuss Machine Learning in three core application areas of the Automotive Industry, Healthcare, and Manufacturing.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE5NDk0MjQ2MjkzLWNyb3NzIDMuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Image credit:&nbsp;<a href="https://link.springer.com/chapter/10.1007/978-981-15-6728-5_2#auth-Zhanxiang-Chai" target="_blank" id="isPasted">Zhanxiang Chai</a> et al.&nbsp;<h2 dir="ltr" id="isPasted">Who should attend? &nbsp;</h2>The "Machine Learning for Safety Experts" training course is tailored for automotive safety engineers at all experience levels who must assess and ensure the safety of functions and systems incorporating Machine Learning (ML). This course is especially relevant for professionals working with safety-critical applications in the automotive industry, where compliance with standards such as ISO 26262 and ISO 21448 is crucial.<h3 dir="ltr">Course prerequisites:</h3><ul><li dir="ltr">Pre-existing Safety Knowledge: Participants should have a solid understanding of general safety principles and practices.</li><li dir="ltr">Experience with Safety Standards: Familiarity with applying ISO 26262 (functional safety for road vehicles) and ISO 21448 (safety of the intended functionality) is essential.</li><li dir="ltr">Basic Understanding of Machine Learning Techniques: A fundamental grasp of Machine Learning concepts and techniques is required to benefit fully from the training.</li></ul><h2 dir="ltr">Expert Instructors and Their Credentials</h2>The training is led by two distinguished experts from Fraunhofer IKS:<h3 dir="ltr">Karsten Roscher</h3><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE5NDk0MDg4MTMzLWthcnN0ZW4uanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">With over a decade of research experience in connected mobility, advanced driver assistance systems, and Machine Learning for computer vision and communication systems, Karsten is a leading advocate for safe intelligence. He heads the Dependable Perception &amp; Imaging department at Fraunhofer IKS and is actively involved in German and European standardization bodies. His extensive work developing reliable perception systems across automotive, rail, industrial automation, and medical domains highlights his deep expertise in integrating Machine Learning with safety standards.<h3 dir="ltr">Philipp Schleiss</h3><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzE5NDk0MTEzMzM4LXBoaWxpcHAtc2NobGVpc3MuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Philipp has been a pivotal figure at Fraunhofer since 2013, focusing on increasing the dependability of embedded and autonomous systems. He leads the department in systems safety engineering and specializes in tool-based design automation, continuous safety assurance, and real-time guarantees for automated driving. Philipp's contributions to advancing safety standards and his role in developing methods for the safe implementation of Machine Learning in autonomous systems make him an invaluable resource for course participants.These instructors bring a wealth of knowledge and practical experience, ensuring participants gain theoretical insights and actionable skills to apply Machine Learning safety measures in their professional practices effectively.&nbsp; <a href="https://www.sae.org/learn/content/c2208/?utm_source=https://www.sae.org/learn/content/c2208/&utm_medium=facebook,_email,_instagram,_linkedin_etc&utm_campaign=sae_ml_training" target="_blank" class="button-main-action">Find out more information here.&nbsp;</a><h2 dir="ltr">Conclusion</h2>Staying current in Machine Learning safety practices is crucial in today's rapidly evolving technological landscape. The "Machine Learning for Safety Experts" training by Fraunhofer IKS provides engineers with the tools and knowledge needed to integrate Machine Learning into safety-critical applications effectively. Continuous learning and professional training are essential to keep pace with technological advancements and ensure the safe deployment of Machine Learning systems in various industries. This course equips participants to lead their teams confidently, ensuring that safety remains a top priority as they leverage the power of advanced Machine Learning techniques.<h2 dir="ltr">References</h2><ol><li dir="ltr"><a href="https://www.iks.fraunhofer.de/en/events/machine-learning-for-safety-experts1.html" target="_blank">https://www.iks.fraunhofer.de/en/events/machine-learning-for-safety-experts1.html</a></li><li dir="ltr"><a href="https://www.sae.org/learn/content/c2208/" target="_blank">https://www.sae.org/learn/content/c2208/</a></li><li dir="ltr"><a href="https://saemobilus.sae.org/courses/machine-learning-safety-experts-c2208" target="_blank">https://saemobilus.sae.org/courses/machine-learning-safety-experts-c2208</a></li><li dir="ltr"><a href="https://www.iks.fraunhofer.de/en/services/ai-act-high-risk-systems.html" target="_blank">https://www.iks.fraunhofer.de/en/services/ai-act-high-risk-systems.html</a></li><li dir="ltr"><a href="https://www.automate.org/ai/industry-insights/ai-in-the-real-world-4-case-studies-of-success-in-industrial-manufacturing" target="_blank">https://www.automate.org/ai/industry-insights/ai-in-the-real-world-4-case-studies-of-success-in-industrial-manufacturing</a></li><li dir="ltr"><a href="https://integranxt.com/blog/success-stories-how-ml-solutions-transformed-businesses-worldwide/" target="_blank">https://integranxt.com/blog/success-stories-how-Machine Learningsolutions-transformed-businesses-worldwide/</a></li><li dir="ltr"><a href="https://www.omdena.com/blog/machine-learning-examples" target="_blank">https://www.omdena.com/blog/machine-learning-examples</a></li><li dir="ltr"><a href="https://learntube.ai/blog/data-science/machine-learning/real-world-applications-of-machine-learning-case-studies-and-success-stories/" target="_blank">https://learntube.ai/blog/data-science/machine-learning/real-world-applications-of-machine-learning-case-studies-and-success-stories/</a></li><li dir="ltr"><a href="https://pg-p.ctme.caltech.edu/blog/ai-ml/machine-learning-in-healthcare-applications-use-cases-careers" target="_blank">https://pg-p.ctme.caltech.edu/blog/ai-ml/machine-learning-in-healthcare-applications-use-cases-careers</a></li><li dir="ltr"><a href="https://ar5iv.org/pdf/1908.04674" target="_blank">https://ar5iv.org/pdf/1908.04674</a></li><li dir="ltr"><a href="https://www.frontiersin.org/articles/10.3389/fcomp.2023.1132580/full" target="_blank">https://www.frontiersin.org/articles/10.3389/fcomp.2023.1132580/full</a></li><li dir="ltr"><a href="https://www.ptc.com/en/blogs/alm/iso-26262-vs-sotif-iso-pas-21448-whats-the-difference" target="_blank">https://www.ptc.com/en/blogs/alm/iso-26262-vs-sotif-iso-pas-21448-whats-the-difference</a></li></ol>

The upcoming "Machine Learning for Safety Experts" training by SAE and Fraunhofer IKS in Munich on October 22-23, 2024 is a timely initiative. It will address this crucial skill gap and ensure engineers are well-equipped to handle the intricacies of safe Machine Learning applications.

Mastering Machine Learning for Critical Applications

For an AI agent, this is easier said than done. Current approaches often utilize multiple hand-crafted machine-learning models to tackle different parts of the task, which require a great deal of human effort and expertise to build. These methods, which use visual representations to directly make navigation decisions, demand massive amounts of visual data for training, which are often hard to come by.To overcome these challenges, researchers from MIT and the MIT-IBM Watson AI Lab devised a navigation method that converts visual representations into pieces of language, which are then fed into one large language model that achieves all parts of the multistep navigation task.Rather than encoding visual features from images of a robot’s surroundings as visual representations, which is computationally intensive, their method creates text captions that describe the robot’s point-of-view. A large language model uses the captions to predict the actions a robot should take to fulfill a user’s language-based instructions.Because their method utilizes purely language-based representations, they can use a large language model to efficiently generate a huge amount of synthetic training data.While this approach does not outperform techniques that use visual features, it performs well in situations that lack enough visual data for training. The researchers found that combining their language-based inputs with visual signals leads to better navigation performance.“By purely using language as the perceptual representation, ours is a more straightforward approach. Since all the inputs can be encoded as language, we can generate a human-understandable trajectory,” says Bowen Pan, an electrical engineering and computer science (EECS) graduate student and lead author of a&nbsp;<a href="https://arxiv.org/pdf/2310.07889" target="_blank">paper on this approach</a>.Pan’s co-authors include his advisor, Aude Oliva, director of strategic industry engagement at the MIT Schwarzman College of Computing, MIT director of the MIT-IBM Watson AI Lab, and a senior research scientist in the Computer Science and Artificial Intelligence Laboratory (CSAIL); Philip Isola, an associate professor of EECS and a member of CSAIL; senior author Yoon Kim, an assistant professor of EECS and a member of CSAIL; and others at the MIT-IBM Watson AI Lab and Dartmouth College. The research will be presented at the Conference of the North American Chapter of the Association for Computational Linguistics.<h2>Solving a vision problem with language</h2>Since large language models are the most powerful machine-learning models available, the researchers sought to incorporate them into the complex task known as vision-and-language navigation, Pan says.But such models take text-based inputs and can’t process visual data from a robot’s camera. So, the team needed to find a way to use language instead.Their technique utilizes a simple captioning model to obtain text descriptions of a robot’s visual observations. These captions are combined with language-based instructions and fed into a large language model, which decides what navigation step the robot should take next.The large language model outputs a caption of the scene the robot should see after completing that step. This is used to update the trajectory history so the robot can keep track of where it has been.The model repeats these processes to generate a trajectory that guides the robot to its goal, one step at a time.To streamline the process, the researchers designed templates so observation information is presented to the model in a standard form — as a series of choices the robot can make based on its surroundings.For instance, a caption might say “to your 30-degree left is a door with a potted plant beside it, to your back is a small office with a desk and a computer,” etc. The model chooses whether the robot should move toward the door or the office.“One of the biggest challenges was figuring out how to encode this kind of information into language in a proper way to make the agent understand what the task is and how they should respond,” Pan says.<h2>Advantages of language</h2>When they tested this approach, while it could not outperform vision-based techniques, they found that it offered several advantages.First, because text requires fewer computational resources to synthesize than complex image data, their method can be used to rapidly generate synthetic training data. In one test, they generated 10,000 synthetic trajectories based on 10 real-world, visual trajectories.The technique can also bridge the gap that can prevent an agent trained with a simulated environment from performing well in the real world. This gap often occurs because computer-generated images can appear quite different from real-world scenes due to elements like lighting or color. But language that describes a synthetic versus a real image would be much harder to tell apart, Pan says.&nbsp;Also, the representations their model uses are easier for a human to understand because they are written in natural language.“If the agent fails to reach its goal, we can more easily determine where it failed and why it failed. Maybe the history information is not clear enough or the observation ignores some important details,” Pan says.In addition, their method could be applied more easily to varied tasks and environments because it uses only one type of input. As long as data can be encoded as language, they can use the same model without making any modifications.But one disadvantage is that their method naturally loses some information that would be captured by vision-based models, such as depth information.However, the researchers were surprised to see that combining language-based representations with vision-based methods improves an agent’s ability to navigate.“Maybe this means that language can capture some higher-level information than cannot be captured with pure vision features,” he says.This is one area the researchers want to continue exploring. They also want to develop a navigation-oriented captioner that could boost the method’s performance. In addition, they want to probe the ability of large language models to exhibit spatial awareness and see how this could aid language-based navigation.

Someday, you may want your home robot to carry a load of dirty clothes downstairs and deposit them in the washing machine in the far-left corner of the basement. The robot will need to combine your instructions with its visual observations to determine the steps it should take to complete this task.

Researchers use large language models to help robots navigate

In this episode, we discuss research coming out of the Karlsruhe Institute of Technology to detect the emotions of tennis players better than humans can!

Podcast: Catching Feelings with 69 percent Accuracy: AI Reads Emotions in Sports

Boosting the performance of solar cells, transistors, LEDs, and batteries will require better electronic materials, made from novel compositions that have yet to be discovered.To speed up the search for advanced functional materials, scientists are using AI tools to identify promising materials from hundreds of millions of chemical formulations. In tandem, engineers are building machines that can print hundreds of material samples at a time based on chemical compositions tagged by AI search algorithms.But to date, there’s been no similarly speedy way to confirm that these printed materials actually perform as expected. This last step of material characterization has been a major bottleneck in the pipeline of advanced materials screening.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1719275773623-MIT_Accelerating-Materials-Vickmark-02-PRESS.jpg" style="width: 100%px;" class="fr-fic fr-dib">To speed up the search for advanced functional materials, scientists are using a new computer vision approach to identify new solar cell materials from hundreds of millions of chemical formulations. Credit: Bryce VickmarkNow, a new computer vision technique developed by MIT engineers significantly speeds up the characterization of newly synthesized electronic materials. The technique automatically analyzes images of printed semiconducting samples and quickly estimates two key electronic properties for each sample: band gap (a measure of electron activation energy) and stability (a measure of longevity).The new technique accurately characterizes electronic materials 85 times faster compared to the standard benchmark approach.The researchers intend to use the technique to speed up the search for promising solar cell materials. They also plan to incorporate the technique into a fully automated materials screening system.“Ultimately, we envision fitting this technique into an autonomous lab of the future,” says MIT graduate student Eunice Aissi. “The whole system would allow us to give a computer a materials problem, have it predict potential compounds, and then run 24-7 making and characterizing those predicted materials until it arrives at the desired solution.”“The application space for these techniques ranges from improving solar energy to transparent electronics and transistors,” adds MIT graduate student Alexander (Aleks) Siemenn. “It really spans the full gamut of where semiconductor materials can benefit society.”Aissi and Siemenn detail the new technique in a <a href="https://www.nature.com/articles/s41467-024-48768-2" target="_blank">study appearing today</a> in Nature Communications. Their MIT co-authors include graduate student Fang Sheng, postdoc Basita Das, and professor of mechanical engineering Tonio Buonassisi, along with former visiting professor Hamide Kavak of Cukurova University and visiting postdoc Armi Tiihonen of Aalto University.<h2>Power in optics</h2>Once a new electronic material is synthesized, the characterization of its properties is typically handled by a “domain expert” who examines one sample at a time using a benchtop tool called a UV-Vis, which scans through different colors of light to determine where the semiconductor begins to absorb more strongly. This manual process is precise but also time-consuming: A domain expert typically characterizes about 20 material samples per hour — a snail’s pace compared to some printing tools that can lay down 10,000 different material combinations per hour.“The manual characterization process is very slow,” Buonassisi says. “They give you a high amount of confidence in the measurement, but they’re not matched to the speed at which you can put matter down on a substrate nowadays.”To speed up the characterization process and clear one of the largest bottlenecks in materials screening, Buonassisi and his colleagues looked to computer vision — a field that applies computer algorithms to quickly and automatically analyze optical features in an&nbsp;image.“There’s power in optical characterization methods,” Buonassisi notes. “You can obtain information very quickly. There is richness in images, over many pixels and wavelengths, that a human just can’t process but a computer machine-learning program can.”The team realized that certain electronic properties — namely, band gap and stability — could be estimated based on visual information alone, if that information were captured with enough detail and interpreted correctly.With that goal in mind, the researchers developed two new computer vision algorithms to automatically interpret images of electronic materials: one to estimate band gap and the other to determine stability.The first algorithm is designed to process visual data from highly detailed, hyperspectral images.“Instead of a standard camera image with three channels — red, green, and blue (RBG) — the hyperspectral image has 300 channels,” Siemenn explains. “The algorithm takes that data, transforms it, and computes a band gap. We run that process extremely fast.”The second algorithm analyzes standard RGB images and assesses a material’s stability based on visual changes in the material’s color over time.“We found that color change can be a good proxy for degradation rate in the material system we are studying,” Aissi says.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1719275806756-MIT_Accelerating-Materials-Vickmark-03-PRESS.jpg" style="width: 100%px;" class="fr-fic fr-dib">MIT professor of mechanical engineering Tonio Buonassisi (left) is a co-author on the paper outlining the new technique for identifying advanced materials. Credit: Bryce Vickmark<h2>Material compositions</h2>The team applied the two new algorithms to characterize the band gap and stability for about 70 printed semiconducting samples. They used a robotic printer to deposit samples on a single slide, like cookies on a baking sheet. Each deposit was made with a slightly different combination of semiconducting materials. In this case, the team printed different ratios of perovskites — a type of material that is expected to be a promising solar cell candidate though is also known to quickly degrade.“People are trying to change the composition — add a little bit of this, a little bit of that — to try to make [perovskites] more stable and high-performance,” Buonassisi says.Once they printed 70 different compositions of perovskite samples on a single slide, the team scanned the slide with a hyperspectral camera. Then they applied an algorithm that visually “segments” the image, automatically isolating the samples from the background. They ran the new band gap algorithm on the isolated samples and automatically computed the band gap for every sample. The entire band gap extraction process process took about six minutes.“It would normally take a domain expert several days to manually characterize the same number of samples,” Siemenn says.To test for stability, the team placed the same slide in a chamber in which they varied the environmental conditions, such as humidity, temperature, and light exposure. They used a standard RGB camera to take an image of the samples every 30 seconds over two hours. They then applied the second algorithm to the images of each sample over time to estimate the degree to which each droplet changed color, or degraded under various environmental conditions. In the end, the algorithm produced a “stability index,” or a measure of each sample’s durability.&nbsp;As a check, the team compared their results with manual measurements of the same droplets, taken by a domain expert. Compared to the expert’s benchmark estimates, the team’s band gap and stability results were 98.5 percent and 96.9 percent as accurate, respectively, and 85 times faster.“We were constantly shocked by how these algorithms were able to not just increase the speed of characterization, but also to get accurate results,” Siemenn says. &nbsp;“We do envision this slotting into the current automated materials pipeline we’re developing in the lab, so we can run it in a fully automated fashion, using machine learning to guide where we want to discover these new materials, printing them, and then actually characterizing them, all with very fast processing.”This work was supported, in part, by First Solar.&nbsp;

MIT graduate students Eunice Aissi, left, and Alexander Siemenn, have developed a technique that automatically analyzes visual features in printed samples (pictured) to quickly determine key properties of new and promising semiconducting materials. Credit: Bryce Vickmark

The technique characterizes a material’s electronic properties 85 times faster than conventional methods.

New computer vision method helps speed up screening of electronic materials

Much has been discussed about the promise and limitations of large-language models in industries such as education, healthcare and even manufacturing. But what about energy? Could large-language models (LLMs), like those that power ChatGPT, help run and maintain the energy grid?&nbsp;New research, co-authored by&nbsp;<a href="https://seas.harvard.edu/person/na-li" target="_blank">Na Li</a>, Winokur Family Professor of Electrical Engineering and Applied Mathematics at the&nbsp;<a href="https://seas.harvard.edu/" target="_blank">Harvard John A. Paulson School of Engineering and Applied Sciences&nbsp;</a>(SEAS) suggests that LLMs could play an important role in co-managing some aspects of the grid, including emergency and outage response, crew assignments and wildfire preparedness and prevention. But security and safety concerns need to be addressed before LLMs can be deployed in the field.&nbsp;“There is so much hype with large-language models, it’s important for us to ask what LLMs can do well and, perhaps more importantly, what they can’t do well, at least not yet, in the power sector,” said&nbsp;<a href="https://engineering.tamu.edu/electrical/profiles/lxie.html" target="_blank">Le Xie</a>, Professor of Electrical &amp; Computer Engineering at Texas A&amp;M University and corresponding author of the study. &nbsp;“The best way to describe the potential of LLMs in this sector is as a co-pilot. It’s not a pilot yet — but it can provide advice, a second opinion, and very timely responses with very few training data samples, which is really beneficial to human decision making.”The research is published in&nbsp;<a href="https://www.cell.com/joule/abstract/S2542-4351(24)00237-X?_returnURL=https%3A%2F%2Flinkinghub.elsevier.com%2Fretrieve%2Fpii%2FS254243512400237X%3Fshowall%3Dtrue" target="_blank">Joule.</a>&nbsp;The research team, which included engineers from Houston-based energy-provider CenterPoint Energy and grid-operator Midcontinent Independent System Operator, used GPT models to explore the capabilities of LLMs in the energy sector — and identified both strengths and weaknesses.&nbsp;The strengths of LLMs — their ability to generate logical responses from prompts, to learn based on limited data, to delegate tasks to embedded tools and to work with non-text data such as pictures — could be leveraged to perform tasks such as detecting broken equipment, real-time electricity load forecasting, and analyzing wildfire patterns for risk assessments.But there are significant challenges to implementing LLMs in the energy sector — not the least of which is the lack of grid-specific data to train the models. For obvious security reasons, crucial data about the U.S. power system is not publicly available and cannot be made public. Another issue is the lack of safety guardrails. The power grid, like autonomous vehicles, needs to prioritize safety and incorporate large safety margin when making real-time decisions. LLMs also need to get better about providing reliable solutions and transparency around their uncertainties, said Li.&nbsp;“We want foundational LLMs to be able to say ‘I don’t know’ or ‘I only have 50% certainty about this response’, rather than give us an answer that might be wrong,” said Li. “We need to be able to count on these models to provide us with reliable solutions that meet specified standards for safety and resiliency.”All of these challenges give engineers a roadmap for future work.&nbsp;“As engineers, we want to highlight these limitations because we want to see how we can improve them,” said Li. “Power system engineers can help improve security and safety guarantees by either fine tuning the foundational LLM or developing our own foundational model for the power systems. One exciting part of this research is that it is a snapshot in time. Next year or even sooner, we can go back and revisit all these challenges and see if there has been any improvement.”&nbsp;This research was co-authored by Subir Majumder, Lin Dong, Fatemeh Doudi, Yuting Cai, Chao Tian, Dileep Kalathil of Texas A&amp;M University; Kevin Ding of CenterPoint Energy; and Anupam A. Thatte of Midcontinent Independent System Operator.

The promise and limitations of large-language models in the energy sector

Bringing GPT to the grid

The Vision, Robotics &amp; Motion trade event, held annually in Brabanthallen, 's-Hertogenbosch, Netherlands, successfully concluded its 2024 edition on June 5th and 6th. This event offered a comprehensive overview of solutions for smart automation and digitalization opportunities in the manufacturing sector.This year's event aimed to provide insights into robotics, vision, and motion implementation for engineers and technical specialists in the high-tech and manufacturing industries. It featured over 130 exhibitors, including manufacturers, system integrators, suppliers, and advisors.Wevolver attended the event and spoke to some of the exhibitors. Below is a list of some of the brilliant companies Wevolver spoke to.&nbsp;<h2 dir="ltr">Key Highlights from the event</h2><h3 dir="ltr"><a href="https://www.youtube.com/watch?v=LE2Kz4ruee8" target="_blank">Wevolver at Vision, Motion and Robotics 2024</a></h3><h2 dir="ltr"><a href="http://beckhoff.com" target="_blank" rel="noopener noreferrer">Beckhoff</a></h2>Bechhoff, a German-based company, showcased its machine automation products. Of particular note was their MX system, a standout development in cabinetless machine control. This system integrates IPC, power supply, motion control, and I/O components all on a single plate that can be directly mounted on the machine. This eliminates the need for traditional control cabinets, offering significant time and space savings in industrial settings.<h2 dir="ltr"><a href="http://onlogic.com" target="_blank" rel="noopener noreferrer">Onlogic</a></h2>OnLogic, headquartered in the Netherlands, is a global provider of industrial hardware solutions. Their products are notable for being fanless, ensuring that no dust infiltrates the system. In dusty, industrial environments, fanless equipment is essential to ensure longevity and durability.&nbsp;<h2 dir="ltr"><a href="https://nl.mitsubishielectric.com/nl/" target="_blank">Mitsubishi Electric</a></h2>Mitsubishi Electric showcased a large range of products, from cobots to standard robots and new server motion products to software solutions. They also introduced AI-driven analysis solutions, 3D simulation and inspection software, highlighting their integrated approach to automation and data management.&nbsp;<h2 dir="ltr"><a href="http://vs-t.eu" target="_blank" rel="noopener noreferrer">VS Technology</a></h2>VS Technology is a Japanese lens lighting and filter manufacturing company. They displayed a variety of optical solutions including telecentric, fixed focal length, and line scan lenses. They highlighted their ability to offer comprehensive optical solutions across industries, including specialized applications requiring high precision. The company has developed a solution for telecentric lenses with a working distance of up to 220 mm.<h2 dir="ltr"><a href="http://staubli.com" target="_blank" rel="noopener noreferrer">Staubli Robotics</a></h2>Staubli group has mainly three divisions: robotics, connectors, and textiles. At MVR, they displayed their precision robots, emphasizing their applications in sensitive environments such as food and pharmaceuticals. Their robots are known for high speed, precision, repeatability, and easy maintenance.<h2 dir="ltr"><a href="http://fanuc.eu" target="_blank" rel="noopener noreferrer">Fanuc Benelux BV</a></h2>Fanuc presented their cobot, a collaborative robot that stops when you touch it. It is absolutely safe to be in the neighbourhood of this collaborative robot, which instantly stops upon touch and restarts when touched twice. The robot is equipped with integrated vision, vision system, camera, and software, all from Fanuc.&nbsp;Next, they presented their G and C-code-based Power Motion. The code language and G-program enables it to control more than 20 axes simultaneously. This makes it easy for CNC operators to program the motion control themselves.Then, there was IPC, a new feature for Fanuc that allows for quick actions and fast data control. IPC makes it easy to have a 3D image of the parts inside the box. In that specific case, there were three Bin Pickers installed, and cameras, two on top and one on the side. The two on top had a 3D image of the parts inside the box. The third one was installed for quality control of the product.<h2 dir="ltr"><a href="http://cobotx.nl" target="_blank" rel="noopener noreferrer">Cobotx</a></h2>Cobotx, a leading system integrator of cobots and robots, showcased a pick-and-place cobot. This cobot is designed to operate safely close to humans, eliminating the need for fences or physical barriers. During a demonstration, the cobot exhibited its ability to slow down to a secure operating speed when a human approached, ensuring safety. Once the person moved out of potential danger, the cobot automatically resumed high-efficiency mode, optimizing productivity.<h2 dir="ltr"><a href="http://vselectronics.in" target="_blank" rel="noopener noreferrer">VS Electronics</a></h2>VS Electronics showcased their cutting-edge electronic components and modules that are integral to robotic systems. They introduced a new range of sensors and controllers that enhance the operational capabilities of industrial robots, focusing on increasing automation reliability.<h2 dir="ltr"><a href="http://robotize.nl" target="_blank" rel="noopener noreferrer">Robotize</a></h2>Robotize exhibited their latest autonomous mobile robots (AMRs) designed for logistics and warehouse operations. Their robots feature advanced navigation and mapping technologies, allowing for efficient materials handling without human intervention.<h2 dir="ltr"><a href="http://dormac-cobots.nl" target="_blank" rel="noopener noreferrer">Dormac cobots</a>, distributors for Doosan Robotics</h2>Dormac Cobots showcased their collaboration with Doosan Robotics, presenting a range of collaborative robots (cobots) that are easy to integrate into existing production lines. These cobots are particularly noted for their user-friendly interfaces and versatile applications.<h2 dir="ltr"><a href="http://isotron.eu" target="_blank" rel="noopener noreferrer">Isotron</a></h2>Isotron presented their latest industrial automation solutions, including PLCs and interface modules that enhance the efficiency and scalability of manufacturing processes. Their products are known for robust performance in demanding industrial environments.<h2 dir="ltr"><a href="http://multirotorresearch.com" target="_blank" rel="noopener noreferrer">Multi Rotor Research</a> (MRR Drones)</h2>MRR Drones unveiled their new drone models designed for industrial inspections and aerial surveillance. These drones are equipped with advanced imaging technologies and are capable of operating in challenging environments.<h2 dir="ltr"><a href="https://mitutoyo.eu/" target="_blank" rel="noopener noreferrer">Mitutoyo</a></h2>Mitutoyo introduced their latest precision measurement instruments, including digital calipers and micrometers. Their products provide high accuracy in quality control processes across various industries. This accuracy is usually required in industrial and scientific research.&nbsp;<h2 dir="ltr"><a href="http://cadmes.com" target="_blank" rel="noopener noreferrer">Cadmes</a></h2>Cadmes displayed their comprehensive CAD/CAM solutions that streamline the design and manufacturing processes. They highlighted software enhancements that improve collaboration and efficiency in engineering teams.&nbsp;<h2 dir="ltr"><a href="http://automation.be" target="_blank" rel="noopener noreferrer">Automation</a>, Schneider Electric partner</h2>As a partner of Schneider Electric, Automation showcased its integrated solutions for energy management and automation. Their technologies are designed to enhance efficiency, sustainability and operational control in smart manufacturing environments.&nbsp;<h2 dir="ltr"><a href="http://mandersautomation.nl" target="_blank" rel="noopener noreferrer">Manders Automation</a></h2>Manders Automation featured their custom automation systems that cater to complex industrial tasks. They emphasized solutions that combine robotics, vision systems, and advanced control technologies to optimize production workflows.<h3 dir="ltr">A New Focus on Digital Solutions</h3>&nbsp;This year's trade show was expanded to include a new pillar: Digital Innovation. This new focus centered on exploring digitization opportunities and solutions for companies in the manufacturing industry. Companies like Mitsubishi Electric and Fanuc presented AI-driven analysis solutions, 3D simulation, and inspection software, and Industrial PC (IPC) technology, highlighting a data-driven approach to automation.Further demonstrating the importance of digitization, <a href="https://www.cadmes.com/" target="_blank">Cadmes</a> displayed software improvements for better collaboration and efficiency in engineering teams and Automation, a Schneider Electric partner, showcased integrated solutions for energy management and automation, promoting sustainable and efficient operations.These digital solutions promise significant benefits for manufacturers. Real-time data analysis allows for predictive maintenance, reduced downtime, and optimized production processes. Improved communication between machines and software facilitates a more connected and efficient factory floor.<h3 dir="ltr">Industry Impact of the VRM show</h3>The event addressed current manufacturing challenges by showcasing advancements in automation, digitalization, and human-robot collaboration. These advancements can help manufacturers:Many of the solutions showcased offer enhancements in production efficiency. For example, the integration of Fanuc’s G-code-based motion control allows for precise operations over multiple axes, significantly speeding up production without sacrificing quality.The event highlighted how advanced robotics, smart automation, and precision engineering are critical in meeting customer demands for higher-quality products delivered in shorter timescales. Companies like VS Electronics and Isotron displayed components that support high-performance robotics systems capable of operating under demanding industrial conditions.<h3 dir="ltr">Conclusion</h3>Vision, Robotics &amp; Motion 2024 was a resounding success. It brought together industry leaders and innovators to showcase the future of manufacturing and automation.&nbsp;Feedback from attendees and exhibitors was overwhelmingly positive, with many praising the breadth and quality of the solutions presented.Looking forward, the next edition of the trade show is expected to further expand on the themes of digitalization and automation with even more innovative solutions. As technology evolves, so too will the capabilities and applications of robotics and motion control in industrial settings.<h3 dir="ltr">Additional Information</h3>Visit the official website for more details on this year's highlights or to stay updated about the next Vision, Robotics &amp; Motion event. This resource offers comprehensive information and updates, helping industry professionals stay ahead in a rapidly evolving market.Are you curious about Vision, Robotics &amp; Motion 2025? Don’t miss out:&nbsp; 📅 &nbsp; &nbsp;11th and 12th June, 2025📍 &nbsp; &nbsp;Brabanthallen 's-Hertogenbosch, Netherlands&nbsp;

The Vision, Robotics & Motion trade event, held annually in Brabanthallen, 's-Hertogenbosch, Netherlands, successfully concluded its 2024 edition on June 5th and 6th. 

Wrap-Up of the Vision, Robotics & Motion Show 2024

Scientists from ITMO have come up with a three-step algorithm for the development of manipulators and walking or wearable robots. Compared to the usual methods, the new approach results in lighter, more efficient, affordable, and compact mechanisms. Walking robots that can move without having a full map of their surroundings are just one example of the designs produced by the new method. Another is the jumping robot made by the researchers to demonstrate their new algorithm.<h1 dir="ltr" id="isPasted">Why it’s hard to combine movement and energy efficiency in robots</h1>Industrial robots need to move with high speed and precision. These are achieved with powerful electric drives that are very demanding in terms of the energy they consume. This means that such robots often need to be connected to a power outlet, making them stationary.Mobile robots are a whole other matter, however. They have to be designed in a way that allows them to perform all their tasks with just the energy they can get from a battery. And it can’t be the case of just increasing the battery’s capacity either – here, “more powerful” also means “heavier.” If a robot wastes half the energy gained from a heavier battery just to carry this battery around, it’s not a very efficient robot. That’s why researchers engineer robotic systems in a way that will make use of the mechanism’s inherent dynamics, thus requiring less energy.<h1 dir="ltr">Overcoming the challenge</h1>In robotics, such mechanisms are designed with morphology in mind. This means that most of the dynamics needed for the robot’s movements are provided by its body (the spread of its weight, its elasticity, or mechanical links). The control systems for such robots don’t need to program their every move, instead focusing on supporting its existing dynamics.&nbsp;When designing such robots, engineers must follow a formalized process that takes into account not only control algorithms, but also the design of the robot’s mechanics. According to Ivan Borisov, an engineer at ITMO’s Research Center “Strong AI in Industry,” for a long time robotics specialists concentrated on developing new algorithms while still using the same hardware and principles as in industrial robots.&nbsp;However, the success of Cassie the robot <a href="https://www.researchgate.net/publication/331449776_Walk_this_way_To_be_useful_around_people_robots_need_to_learn_how_to_move_like_we_do" target="_blank">has shown</a> that in order to achieve high speed, robustness, and capacity to navigate unstructured environments, robots need to be designed differently. First and foremost, they need to be adaptive and energy efficient.<iframe width="640" height="360" src="https://www.youtube.com/embed/a_YGPbWJO5g?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h1 dir="ltr" id="isPasted">The method from ITMO</h1>ITMO scientists used the principles of morphological design to suggest their own method of designing mechanisms for manipulator, locomotion, and wearable robots. The method is based on a three-step algorithm.First step: creating an open kinetic chain. Here, the algorithm determines which parts should make up the mechanism and the trajectories they need to follow in order to produce the desired movements.Second step: closing the chain of parts that make up the mechanism (the kinetic chain is closed when each link in the chain has over two connections). Here, the algorithm determines which links in the chain need to be connected to the core mechanism so that a drive can be transported to a different joint. At this point, the program may also decide to remove or add certain connections in order to lower inertia. This helps preserve energy and speed up the robot.&nbsp;Third step: placing electric drives with maximal efficiency. At this stage, physical and virtual elastic elements are also integrated into the mechanism. With these, the robot can adapt to the environment, cushion itself against rough and uneven surfaces, and conserve energy.By targeting performance and maximum functionality at every step, this approach produces more efficient devices.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1718581571013-1323200.jpg" style="width: 100%px;" class="fr-fic fr-dib">Ivan Borisov. Photo by Dmitry Grigoryev / ITMO.NEWS<blockquote id="isPasted">“Using our method, we can develop any leverage mechanism, such as a leg mechanism for a walking robot, a finger mechanism for a grasping device, or parts of an exoskeleton. What’s even more, we can make robots that are able to function in unstructured or unknown environments. For instance, a walking robot that will be able to map out their surroundings even in bad lighting or an anthropomorphic wrist that can grasp objects it can’t see,” explains Ivan Borisov.</blockquote>To demonstrate the algorithm’s capacity, the researchers developed a jumping robot that can move on uneven terrain. The robot moves thanks to self-sustained vibrations of its drive and springs; it can change the speed and direction of its movements. Thanks to the new method used in its design, the robot requires a drive four times less powerful than a similar robot designed with the classical approach.&nbsp;According to Ivan Borisov, the method squeezes the most out of each part of the algorithm, maximizing the drives’ efficiency and using elastic parts to restore some of the used energy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1718581765992-ezgif-4-4d6d52b5a1.gif" style="width: 100%px;" class="fr-fic fr-dib">Flexible jumping robot. Video by Dmitry Grigoryev / ITMO.NEWS<h1 dir="ltr" id="isPasted">What’s next&nbsp;</h1>The new method will be presented at the prestigious IEEE International Conference on Robotics and Automation (SJR 1.92), where ITMO will be this year’s only Russian participant.Additionally, the researchers are developing software that will use the algorithm to generate designs of walking robots and trajectories of their electric drives. It will also test the suggested solutions in a simulated environment.A similar&nbsp;<a href="https://github.com/aimclub/rostok" target="_blank">program</a> for grasping devices has already been developed at ITMO. This one can design a robot based on the parameters of an object it needs to grasp. For instance, if you need a device for grasping apples, you need to simply indicate the fruit’s characteristics – then, the algorithm will generate the mechanism’s structure with optimal parameters.According to the researchers, as the method involves randomization, it can produce several solutions for the same problem, allowing the human operating it to choose the best one.&nbsp;Both algorithms are meant to assist designers in making mechatronic and robotic systems, including industrial, collaborative, or manipulator robots.

Flexible jumping robot. Photo by Dmitry Grigoryev / ITMO.NEWS

Scientists from ITMO have come up with a three-step algorithm for the development of manipulators and walking or wearable robots. Compared to the usual methods, the new approach results in lighter, more efficient, affordable, and compact mechanisms. 

Scientists Suggest New Method to Make More Adaptive and Autonomous Robots

Recent advances in robotics and artificial intelligence allow robots to carry out diverse tasks in manufacturing, warehouse operations, delivery services and more, benefitting humans who would otherwise undertake these physically demanding – and potentially dangerous – jobs.However, robots remain passive to their environment as they are not yet able to interact intelligently with or directly manipulate it to respond to events that may prevent mission completion. This shortfall becomes particularly pronounced in unpredictable environments where unexpected physical barriers or gaps can thwart robots’ progress toward mission objectives. In these situations, current autonomy algorithms are left with no recourse but to declare mission failure.&nbsp;<a href="https://engineering.wustl.edu/faculty/Yiannis-Kantaros.html" target="_blank">Yiannis Kantaros</a>, assistant professor in the Preston M. Green Department of Electrical &amp; Systems Engineering, has received a five-year, $591,457 Faculty Early Career Development (CAREER) Award from the National Science Foundation (NSF) to address these limitations. With the NSF’s support, Kantaros aims to advance robot intelligence and enable teams of robots to interact collaboratively with their perceived environment to overcome unanticipated obstacles and events.CAREER awards support junior faculty who model the role of teacher-scholar through outstanding research, excellence in education and the integration of education and research within the context of the mission of their organization.&nbsp;<a href="https://engineering.wustl.edu/faculty/research/index.html#nsf_career#nsf_career" target="_blank">One-third of current McKelvey Engineering faculty have received the award</a>.&nbsp;“Safety-critical applications such as disaster relief and search-and-rescue missions often occur in unstructured environments that are challenging for humans to navigate, requiring teams of robots to be deployed,” Kantaros said. “One major challenge is the unpredictable nature of these environments, which can hinder robots from completing tasks due to unexpected obstacles, debris, structural changes or physical gaps that block access to critical areas.“Unlike humans, robots lack the ability to interact intelligently with their surroundings to overcome these issues,” Kantaros continued. “My research will address this challenge by enabling teams of robots to interact with their environment and overcome challenges by intelligently reconfiguring it, for example, by pushing obstacles out of the way or using objects to bridge gaps.”By marrying artificial perception with decision-making capabilities, Kantaros will equip robots with the ability to not only perceive their surroundings but also to take a more active approach and modify surroundings to achieve their goals. A central component of his project is developing perception-based planning capabilities for robot teams operating in unknown and unstructured environments.“The proposed research will result in fundamental robot capabilities that will substantially enhance multi-robot planning methods in unstructured environments,” Kantaros said. “The developed planning algorithms will enable robot teams to interact with the perceived environment, accommodating imperfect AI-enabled perception and effectively responding to unforeseen events that could hinder task completion.”Imagine a fleet of robots working in tandem, dynamically reshaping their environment to navigate through treacherous terrain or deliver aid to remote locations with unprecedented efficiency and adaptability.&nbsp;“These autonomy algorithms are expected to offer unprecedented flexibility in various humanitarian and scientific applications in unknown environments, leading to enhancements in mission performance and robot safety, ultimately saving lives and resources,” Kantaros noted. “Important applications include search and rescue operations, disaster relief efforts, and environmental monitoring and surveillance.”

Yiannis Kantaros studies teams of autonomous robots working collaboratively in the Washington University Miniature City (WashU Mini-City), constructed in McKelvey Hall. (Photo: Yiannis Kantaros)

Yiannis Kantaros will enable teams of robots to interact collaboratively, perceive and respond to their environment with a CAREER Award from the National Science Foundation

Advancing robot autonomy in unpredictable environments

Matroid-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid

Plainsight-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Mon Oct 30 2023 14:34:39 GMT+0200 (Eastern European Standard Time)

Edge Impulse-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Xsens-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:53 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:58 GMT+0300 (Eastern European Summer Time)

Movella-Mon Oct 30 2023 14:34:24 GMT+0200 (Eastern European Standard Time)

Movella-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Photoneo-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Photoneo-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Photoneo-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Photoneo-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Photoneo-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Photoneo-Wed Sep 20 2023 08:53:41 GMT-0400 (Eastern Daylight Time)

Photoneo-Wed Sep 20 2023 08:54:57 GMT-0400 (Eastern Daylight Time)

Photoneo-Wed Sep 20 2023 08:55:20 GMT-0400 (Eastern Daylight Time)

Photoneo-Wed Sep 20 2023 09:08:47 GMT-0400 (Eastern Daylight Time)

Photoneo-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Photoneo-Mon Oct 30 2023 08:42:19 GMT-0400 (Eastern Daylight Time)

Georgia Institute of Technology-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Georgia Institute of Technology-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Georgia Institute of Technology-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Georgia Institute of Technology-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Georgia Institute of Technology-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Computer Vision

An AI-generated depiction of a modern day chipset

Gaining a comprehensive understanding of the distinct capabilities and applications of TPUs and GPUs is crucial for developers and researchers aiming to navigate the complex and rapidly changing terrain of artificial intelligence.

TPU vs GPU in AI: A Comprehensive Guide to Their Roles and Impact on Artificial Intelligence

Clustering - A Common Unsupervised Learning Technique

Understanding What is Unsupervised Learning, the Mechanisms, Types, and Applications of Various Algorithms and Challenges it presents in Machine Learning

What is Unsupervised Learning: A Comprehensive Guide

Exploring the key concepts related to Unsupervised vs Supervised Learning, understanding the fundamental principles, major algorithms and their real-world applications, and practical distinctions between supervised and unsupervised learning.

Unsupervised vs Supervised Learning: A Comprehensive Comparison

Artificial intelligence (AI) is a wide-ranging tool that enables people to rethink how we integrate information, analyze data, and use the resulting insights to improve decision making


Computer Vision

Profiles