Health-monitoring apps can help people manage chronic diseases or stay on track with fitness goals, using nothing more than a smartphone. However, these apps can be slow and energy-inefficient because the vast machine-learning models that power them must be shuttled between a smartphone and a central memory server.Engineers often speed things up using hardware that reduces the need to move so much data back and forth. While these machine-learning accelerators can streamline computation, they are susceptible to attackers who can steal secret information.To reduce this vulnerability, researchers from MIT and the MIT-IBM Watson AI Lab created a machine-learning accelerator that is resistant to the two most common types of attacks. Their chip can keep a user’s health records, financial information, or other sensitive data private while still enabling huge AI models to run efficiently on devices.The team developed several optimizations that enable strong security while only slightly slowing the device. Moreover, the added security does not impact the accuracy of computations. This machine-learning accelerator could be particularly beneficial for demanding AI applications like augmented and virtual reality or autonomous driving.While implementing the chip would make a device slightly more expensive and less energy-efficient, that is sometimes a worthwhile price to pay for security, says lead author Maitreyi Ashok, an electrical engineering and computer science (EECS) graduate student at MIT.“It is important to design with security in mind from the ground up. If you are trying to add even a minimal amount of security after a system has been designed, it is prohibitively expensive. We were able to effectively balance a lot of these tradeoffs during the design phase,” says Ashok.Her co-authors include Saurav Maji, an EECS graduate student; Xin Zhang and John Cohn of the MIT-IBM Watson AI Lab; and senior author Anantha Chandrakasan, MIT’s chief innovation and strategy officer, dean of the School of Engineering, and the Vannevar Bush Professor of EECS. The research will be presented at the IEEE Custom Integrated Circuits Conference.<h2>Side-channel susceptibility</h2>The researchers targeted a type of machine-learning accelerator called digital in-memory compute. A digital IMC chip performs computations inside a device’s memory, where pieces of a machine-learning model are stored after being moved over from a central server.The entire model is too big to store on the device, but by breaking it into pieces and reusing those pieces as much as possible, IMC chips reduce the amount of data that must be moved back and forth.But IMC chips can be susceptible to hackers. In a side-channel attack, a hacker monitors the chip’s power consumption and uses statistical techniques to reverse-engineer data as the chip computes. In a bus-probing attack, the hacker can steal bits of the model and dataset by probing the communication between the accelerator and the off-chip memory.Digital IMC speeds computation by performing millions of operations at once, but this complexity makes it tough to prevent attacks using traditional security measures, Ashok says.She and her collaborators took a three-pronged approach to blocking side-channel and bus-probing attacks.First, they employed a security measure where data in the IMC are split into random pieces. For instance, a bit zero might be split into three bits that still equal zero after a logical operation. The IMC never computes with all pieces in the same operation, so a side-channel attack could never reconstruct the real information.But for this technique to work, random bits must be added to split the data. Because digital IMC performs millions of operations at once, generating so many random bits would involve too much computing. For their chip, the researchers found a way to simplify computations, making it easier to effectively split data while eliminating the need for random bits.Second, they prevented bus-probing attacks using a lightweight cipher that encrypts the model stored in off-chip memory. This lightweight cipher only requires simple computations. In addition, they only decrypted the pieces of the model stored on the chip when necessary.Third, to improve security, they generated the key that decrypts the cipher directly on the chip, rather than moving it back and forth with the model. They generated this unique key from random variations in the chip that are introduced during manufacturing, using what is known as a physically unclonable function.“Maybe one wire is going to be a little bit thicker than another. We can use these variations to get zeros and ones out of a circuit. For every chip, we can get a random key that should be consistent because these random properties shouldn’t change significantly over time,” Ashok explains.They reused the memory cells on the chip, leveraging the imperfections in these cells to generate the key. This requires less computation than generating a key from scratch.“As security has become a critical issue in the design of edge devices, there is a need to develop a complete system stack focusing on secure operation. This work focuses on security for machine-learning workloads and describes a digital processor that uses cross-cutting optimization. It incorporates encrypted data access between memory and processor, approaches to preventing side-channel attacks using randomization, and exploiting variability to generate unique codes. Such designs are going to be critical in future mobile devices,” says Chandrakasan.<h2>Safety testing</h2>To test their chip, the researchers took on the role of hackers and tried to steal secret information using side-channel and bus-probing attacks.Even after making millions of attempts, they couldn’t reconstruct any real information or extract pieces of the model or dataset. The cipher also remained unbreakable. By contrast, it took only about 5,000 samples to steal information from an unprotected chip.The addition of security did reduce the energy efficiency of the accelerator, and it also required a larger chip area, which would make it more expensive to fabricate.The team is planning to explore methods that could reduce the energy consumption and size of their chip in the future, which would make it easier to implement at scale.“As it becomes too expensive, it becomes harder to convince someone that security is critical. Future work could explore these tradeoffs. Maybe we could make it a little less secure but easier to implement and less expensive,” Ashok says.

A new chip can efficiently accelerate machine-learning workloads on edge devices like smartphones while protecting sensitive user data from two common types of attacks — side-channel attacks and bus-probing attacks. Image: Chip figure courtesy of the researchers; MIT News; iStock.

Researchers have developed a security solution for power-hungry AI models that offers protection against two common attacks.

This tiny chip can safeguard user data while enabling efficient computing on a smartphone

In this episode, we discuss research coming out of Columbia University to better connect humans with robots by having them detect, process, and respond to facial expressions!

Podcast: Finally! A Humanoid Robot You Can Trust

Low-cost, wearable sensors could increase access to care for patients with Parkinson’s disease. New machine-learning approaches and a baseline of data from healthy older adults improve the accuracy of the results from such sensors, University of Illinois Urbana-Champaign researchers and clinical collaborators found in a new study.&nbsp;“This study demonstrates that the expansion of datasets with healthy older adult motion data and integration with deep learning approaches can help improve the accuracy of detecting differences in motor impairment in persons with Parkinson's disease for use in future telemedicine sessions,” said study leader <a href="https://experts.illinois.edu/en/persons/manuel-enrique-hernandez" target="_blank">Manuel Enrique Hernandez</a>, a professor of <a href="https://medicine.illinois.edu/academics/departments/biomedical-and-translational-sciences" target="_blank">biomedical and translational sciences</a> within the <a href="https://medicine.illinois.edu/" target="_blank">Carle Illinois College of Medicine</a> at the U. of I.To have their symptoms monitored, patients with Parkinson’s disease must periodically undergo assessment — typically a time-consuming, in-person visit to a specialist with limited availability, Hernandez said.&nbsp;Evaluation by telemedicine would improve access to care for patients but is made challenging by the lack of quantifiable measurements. For example, the physician can watch the patient move, but cannot assess rigidity or muscle tone. While some progress has been made in using wearable sensors to assess motor skills, the cost limits their use.&nbsp;To address these challenges, the Illinois team focused on ways to improve assessment using low-cost wearable sensors.&nbsp;“Ideally, we’d have something that’s completely passive, gathering data as a person goes through their everyday environment and using that information to provide guidance on their overall motor function and progression of neurological symptoms. But then you face the big challenge of how to parse all that information and make it useful to a clinician,” Hernandez said. “That’s what led to our strategy of improving the machine learning, honing in on activities useful for assessment and looking at healthy individuals as a baseline.”&nbsp;The researchers adapted the gold standard of clinical assessment, the Movement Disorders Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale. The MDS-UPDRS outlines specific tasks that a patient would perform and qualitative observations a physician would make during an exam and organizes them into categories for scoring.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1713912642131-225490.jpg" style="width: 100%px;" class="fr-fic fr-dib">Subjects completed tasks normally done during in-person clinical assessment with sensors on their arms, hands and feet to gather data. Image courtesy of Manuel Enrique HernandezFor their study, the researchers had patients perform tasks and muscle movements while wearing sensors to provide the data in the categories scored by MDS-UPDRS. They used data from both patients with Parkinson’s disease and from healthy older adults to train their machine learning model.&nbsp;“Improving machine learning models requires a lot of data. We hypothesized that healthy individuals can serve as the basis for what to expect with potential age-related changes. Then we could build a comparative model to get a sense of when there might be a marked change in terms of motor function or motor impairment,” said Hernandez, who also is affiliated with the departments of <a href="https://ahs.illinois.edu/kch-home" target="_blank">kinesiology and community health</a> and <a href="https://bioengineering.illinois.edu/" target="_blank">bioengineering</a>, as well as the <a href="https://beckman.illinois.edu/" target="_blank">Beckman Institute for Advanced Science and Technology</a> at Illinois.&nbsp;The researchers also used deep learning techniques known as pretraining to make the model more robust, better able to identify important data points and filter out irrelevant ones. The pretraining and the addition of healthy older adult data improved the accuracy of identifying motor impairments in hand movement tasks by more than 12% over the current standard model. Additionally, including data from healthy older adults improved the accuracy of assessing every task for the upper and lower body, except for toe tapping.&nbsp;The <a href="https://www.mdpi.com/1424-8220/23/21/9004" target="_blank">results</a> were reported in the journal Sensors.Next, the researchers hope to collaborate further with neurologists and clinicians to expand their model. They aim to identify a small number of additional tasks that could quantitatively measure more information about the classical symptoms of Parkinson’s disease, such as tremor, while maintaining low cost and ease of use for the sensors.&nbsp;“There’s a very large need for us to better understand and better quantify changes that are ongoing with Parkinson’s disease,” Hernandez said. “The ability to use wearable sensors while performing activities that are part of the course of clinical evaluations in a telehealth setting might open up the doors for more objective, more quantifiable information that can help direct treatment for individuals with Parkinson’s disease and hopefully improve quality of life moving forward.”<a href="https://ise.illinois.edu/directory/profile/r-sowers" target="_blank">Richard Sowers</a>, Illinois professor of <a href="https://ise.illinois.edu/" target="_blank">industrial and enterprise systems engineering</a> and <a href="https://math.illinois.edu/" target="_blank">mathematics</a>, and James R. Brasic, professor and neurologist at Johns Hopkins School of Medicine, New York University and NYC Health + Hospitals, were coauthors of the paper. &nbsp;&nbsp;<hr>Editor’s note: &nbsp;&nbsp;To reach Manuel Enrique Hernandez, email <a href="mailto:mhernand@illinois.edu" target="_blank">mhernand@illinois.edu</a>.&nbsp;The paper, “A deep learning approach for automatic and objective grading of the motor impairment severity in Parkinson’s disease for use in tele-assessments,” is available <a href="https://www.mdpi.com/1424-8220/23/21/9004" target="_blank">online</a>. DOI: 10.3390/s23219004This research had no external funding.&nbsp;

Illinois professors Richard Sowers, left, and Manuel Hernandez used machine learning techniques to improve data analysis from wearable sensors to detect symptoms and monitor the progression of Parkinson’s disease. Photo by Fred Zwicky

Low-cost, wearable sensors could increase access to care for patients with Parkinson’s disease. New machine-learning approaches and a baseline of data from healthy older adults improve the accuracy of the results from such sensors

Wearable sensors for Parkinson's can improve with machine learning, data from healthy adults

To build AI systems that can collaborate effectively with humans, it helps to have a good model of human behavior to start with. But humans tend to behave suboptimally when making decisions.This irrationality, which is especially difficult to model, often boils down to computational constraints. A human can’t spend decades thinking about the ideal solution to a single problem.Researchers at MIT and the University of Washington developed a way to model the behavior of an agent, whether human or machine, that accounts for the unknown computational constraints that may hamper the agent’s problem-solving abilities.Their model can automatically infer an agent’s computational constraints by seeing just a few traces of their previous actions. The result, an agent’s so-called “inference budget,” can be used to predict that agent’s future behavior.In a new paper, the researchers demonstrate how their method can be used to infer someone’s navigation goals from prior routes and to predict players’ subsequent moves in chess matches. Their technique matches or outperforms another popular method for modeling this type of decision-making.Ultimately, this work could help scientists teach AI systems how humans behave, which could enable these systems to respond better to their human collaborators. Being able to understand a human’s behavior, and then to infer their goals from that behavior, could make an AI assistant much more useful, says Athul Paul Jacob, an electrical engineering and computer science (EECS) graduate student and lead author of a&nbsp;<a href="https://openreview.net/pdf?id=W3VsHuga3j" target="_blank">paper on this technique</a>.“If we know that a human is about to make a mistake, having seen how they have behaved before, the AI agent could step in and offer a better way to do it. Or the agent could adapt to the weaknesses that its human collaborators have. Being able to model human behavior is an important step toward building an AI agent that can actually help that human,” he says.Jacob wrote the paper with Abhishek Gupta, assistant professor at the University of Washington, and senior author Jacob Andreas, associate professor in EECS and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). The research will be presented at the International Conference on Learning Representations.<h2>Modeling behavior</h2>Researchers have been building computational models of human behavior for decades. Many prior approaches try to account for suboptimal decision-making by adding noise to the model. Instead of the agent always choosing the correct option, the model might have that agent make the correct choice 95 percent of the time.However, these methods can fail to capture the fact that humans do not always&nbsp;behave suboptimally in the same way.Others at MIT have also&nbsp;<a href="https://news.mit.edu/2020/building-machines-better-understand-human-goals-1214" target="_blank">studied more effective ways</a> to plan and infer goals in the face of suboptimal decision-making.To build their model, Jacob and his collaborators drew inspiration from prior studies of chess players. They noticed that players took less time to think before acting when making simple moves and that stronger players tended to spend more time planning than weaker ones in challenging matches.“At the end of the day, we saw that the depth of the planning, or how long someone thinks about the problem, is a really good proxy of how humans behave,” Jacob says.They built a framework that could infer an agent’s depth of planning from prior actions and use that information to model the agent’s decision-making process.The first step in their method involves running an algorithm for a set amount of time to solve the problem being studied. For instance, if they are studying a chess match, they might let the chess-playing algorithm run for a certain number of steps. At the end, the researchers can see the decisions the algorithm made at each step.Their model compares these decisions to the behaviors of an agent solving the same problem. It will align the agent’s decisions with the algorithm’s decisions and identify the step where the agent stopped planning.From this, the model can determine the agent’s inference budget, or how long that agent will plan for this problem. It can use the inference budget to predict how that agent would react when solving a similar problem.<h2>An interpretable solution</h2>This method can be very efficient because the researchers can access the full set of decisions made by the problem-solving algorithm without doing any extra work. This framework could also be applied to any problem that can be solved with a particular class of algorithms.“For me, the most striking thing was the fact that this inference budget is very interpretable. It is saying tougher problems require more planning or being a strong player means planning for longer. When we first set out to do this, we didn’t think that our algorithm would be able to pick up on those behaviors naturally,” Jacob says.The researchers tested their approach in three different modeling tasks: inferring navigation goals from previous routes, guessing someone’s communicative intent from their verbal cues, and predicting subsequent moves in human-human chess matches.Their method either matched or outperformed a popular alternative in each experiment. Moreover, the researchers saw that their model of human behavior matched up well with measures of player skill (in chess matches) and task difficulty.Moving forward, the researchers want to use this approach to model the planning process in other domains, such as reinforcement learning (a trial-and-error method commonly used in robotics). In the long run, they intend to keep building on this work toward the larger goal of developing more effective AI collaborators.This work was supported, in part, by the MIT Schwarzman College of Computing Artificial Intelligence for Augmentation and Productivity program and the National Science Foundation.

MIT and other researchers developed a framework that models irrational or suboptimal behavior of a human or AI agent, based on their computational constraints. Their technique can help predict an agent’s future actions, for instance, in chess matches. Image: iStock

A new technique can be used to predict the actions of human or AI agents who behave suboptimally while working toward unknown goals.

To build a better AI helper, start by modeling the irrational behavior of humans

This story by Landon Hall was first published in the&nbsp;<a href="https://viterbischool.usc.edu/news/2024/04/teaching-robots-to-walk-on-the-moon-and-maybe-rescue-one-another/" target="_blank">USC Viterbi School of Engineering newsroom</a>.Georgia Tech alumna&nbsp;Feifei Qian&nbsp;(M.S. PHYS 2011, Ph.D. ECE 2015), an assistant professor of electrical and computer engineering at the USC Viterbi School of Engineering and School of Advanced Computing, leads the NASA LASSIE project alongside co-investigator&nbsp;Frances Rivera-Hernández, an assistant professor in the School of Earth and Atmospheric Sciences at Georgia Tech.&nbsp;Sharissa Thompson, a graduate student at Georgia Tech, is a student intern on the NASA Curiosity Rover project.The Palmer Glacier on Oregon’s Mount Hood isn’t the moon, but it’s a good place to practice.Some 6,000 feet up the snow-capped mountain, located about 70 miles east of Portland, a multidisciplinary team from the University of Southern California, Texas A&amp;M University, Georgia Institute of Technology, Oregon State University, Temple University, the University of Pennsylvania, and NASA gathered to turn loose a four-legged robot named Spirit into the wild.The team that included engineers, cognitive scientists, geoscientists, and planetary scientists field-tested Spirit as part of the LASSIE Project: Legged Autonomous Surface Science in Analog Environments. Spirit covered a variety of challenging terrains, using his spindly metal legs to amble over, across, and over around shifting dirt, slushy snow, and boulders during five days of testing in summer 2023. Sometimes he expertly traversed the hillside, while at other moments he teetered and fell over — all part of the process to better understand the substrate properties and learn to better walk on these extreme terrains. The practice time Spirit logged produced data that will be used to train future robots for use on intergalactic surfaces, like Earth’s moon and perhaps planets in our solar system.“A legged robot needs to be able to detect what is happening when it interacts with the ground underneath, and rapidly adjust its locomotion strategies accordingly,” says Feifei Qian, an assistant professor of electrical and computer engineering at the USC Viterbi School of Engineering and School of Advanced Computing, which is leading the project funded by NASA. “When the robot leg slips on ice or sinks into soft snow, it inspires us to look for new principles and strategies that can push the boundary of human knowledge and enable new technology. We learn and improve from the observed failures.”Watch this&nbsp;<a href="https://www.youtube.com/watch?v=wBTyelFFE1A" target="_blank">5-minute video</a>&nbsp;produced for the team by documentary filmmaker Sean Grasso.<iframe width="640" height="360" src="https://www.youtube.com/embed/wBTyelFFE1A?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> Spirit learns from every step.“Similar to the way that when we walk on uneven surfaces as humans, we can sort of detect how the ground is shifting beneath our feet, a legged robot is capable of the exact same thing,” says Cristina Wilson, a cognitive scientist at Oregon State University.<h2>The more machines the merrier</h2>Qian’s group doesn’t intend to stop at just one robot, wandering the wilderness alone. She and her former colleagues at Penn, Cynthia Sung, Mark Yim, Daniel Koditschek, and Douglas Jerolmack, received a two-year, $2 million grant from NASA they’re calling the TRUSSES Project: Temporarily, Robots Unite to Surmount Sandy Entrapments, Then Separate. They want to help the space agency put teams of robots on the Moon and have them work together on tasks. They would take the knowledge they came in with, and the data they collect on the mission, and communicate those details to each other.“They would sense how the ground conditions are,” Qian says, “and then exchange that information with one another, and collectively form a map of locomotion risk estimation. The team of robots can then use this traversal risk map to inform their planetary explorations: ‘There is an extremely soft sand patch that might be high-risk for wheeled rovers. Come over here, this might be a safer area.’ ”The robots in mind for this kind of work would be more than just Spirit: There would be a wheeled rover (great for payload and long distances), a Hexapedal robot (intermediate payload but better mobility than the wheeled), and dog-like ones like the rugged version of Spirit (highest mobility, shorter distances). And here’s the coolest part of that research, the part that sounds like something the Transformers would do. Or at least a team of castaways on Survivor: If one got in a jam, made immovable by loose dirt or a rock or a ravine, his bot-mates would arrive and link together and form a bridge, or a pyramid, to hoist their pal to safety. And then back to work.“When they plan for the strategy to pull the robot up, they’ll decide what force to exert and what position the robot should go to, while also compiling the terrain information,” Qian says. “That’s the key idea of how to use these capabilities: to both prevent and recover from locomotion failures in extreme terrain.”<h2>Back to Mount Hood</h2>Spirit gets around a variety of natural environments, to learn how to better move on challenging terrains. Qian has let him off his leash on Southern California beaches, and the multi-university team has field-tested him in the soft granules of White Sands National Park in New Mexico. But <a href="https://youtu.be/wBTyelFFE1A" target="_blank">the video</a> shot at Mount Hood shows just how otherworldly that landscape can be in these planetary-analog environments. This provides Spirit with plenty of opportunities to learn on earth, before potentially exploring other planets.“You look around us, it would be very hard to drive up this,” Ryan Ewing, a geologist from NASA Johnson Space Center, <a href="https://youtu.be/wBTyelFFE1A?feature=shared" target="_blank">shares</a>. “But as a legged being, as humans, we can step around it easily. A dog could walk around it easily. So this project is the proving ground that we can enable new science and new mobility on environments that are like other planets.”In fact, a dog is indeed frisking about: Howard, Wilson’s German shepherd, wandered about, with the kind of agility Spirit could only dream of.“We are going to observe how Howard moves in different types of snow and ice conditions,” Qian <a href="https://youtu.be/wBTyelFFE1A?feature=shared" target="_blank">says</a>. “What exactly, out of those combined motions, allows him to succeed on challenging terrain?”The LASSIE Project calls for two more trips for Spirit: to Mount Hood this summer, and to White Sands next year. The TRUSSES team, from USC and Penn, also plans to visit White Sands next year with Spirit and the other, new, multitasking robots. Imagine WALL-E with friends.<hr>The NASA PSTAR (Planetary Science and Technology Through Analog Research) number for this project is 80NSSC22K1313.

Researchers at Georgia Tech have teamed up with NASA and five peer institutions to teach dog-like robots to navigate craters of the moon and other challenging planetary surfaces.

Good Dog: LASSIE Spirit Learns to Walk on the Moon

NVIDIA Jetson AGX Orin Module and Developer Kit

Nvidia Jetson and Edge Impulse facilitate Edge AI development, offering computational power and model training platforms for real-world applications.

Building High-Performance Edge AI Solutions with Nvidia Jetson and Edge Impulse

Machine vision represents a fascinating fusion of hardware and software designed to empower machines to 'see' and interpret the world around them. This technology, critical in manufacturing, automates tasks that require visual understanding, enhancing efficiency, accuracy, and safety on production lines. From guiding robots to inspecting products for defects, machine vision systems translate visual inputs into actionable information, fundamentally transforming industrial operations.The evolution of machine vision is closely linked with advancements in artificial intelligence (AI). As AI technologies have progressed, they have significantly enhanced the capabilities of machine vision systems, enabling more complex and nuanced interpretation of visual data. This integration has been pivotal in driving Industry 4.0, the ongoing automation of traditional manufacturing and industrial practices using smart technology.&nbsp;Augmented with AI, machine vision is an enabler of automation and a key player in creating intelligent, adaptable, and efficient manufacturing environments. The convergence of machine vision and AI pushes the boundaries of what's possible in industrial automation, leading to unprecedented productivity and innovation.<h2 dir="ltr">The Core Components of Machine Vision Systems</h2>Machine vision systems are intricate setups that combine hardware and software to enable machines to 'see' and make decisions based on visual inputs. These systems play a pivotal role in various applications, including automated inspection, process control, and guiding robots, significantly contributing to industrial automation and robotics advancements.<h3 dir="ltr">Hardware: The Observational Components</h3>The foundational hardware components of machine vision systems include cameras and sensors, which serve as the 'eyes' of the system. Cameras equipped with image sensors capture visual data from the environment. These cameras range from simple, compact smart cameras for specific applications to more complex multi-camera systems offering greater versatility and data collection capabilities.&nbsp;Sensors are crucial for determining the system's specifications, such as resolution, directly impacting the amount of data collected and the system's ability to discern environmental changes. The resolution, defined by the total number of pixels in rows and columns, affects not just image clarity but also processing speed, with monochrome cameras often facilitating faster processing than colour counterparts due to simpler data.<h3 dir="ltr">Software: The Analytical Brain</h3>On the software side, machine vision systems are powered by image processing algorithms and, increasingly, by deep learning and AI technologies. These software components are tasked with analysing and interpreting the captured visual data. The traditional approach to image processing involves predefined rules for analysing images, which can be rigid and limited in scope.<h3 dir="ltr">The Synergy of Hardware and Software</h3>The interplay between the hardware's data collection capabilities and the software's processing power defines the effectiveness of a machine vision system. Adequate lighting and lens quality are also essential, ensuring the captured images are clear and detailed enough for accurate analysis. For example, the choice between backlighting and front-lighting setups can significantly affect the system's ability to capture usable data in real-world applications.<h2 dir="ltr">Implementing Machine Vision in Manufacturing</h2>Implementing machine vision in manufacturing significantly enhances quality control, defect detection, and production efficiency. This technology integration facilitates a transition from manual to intelligent manufacturing, providing high detection accuracy, speed, and improved coordination.Machine vision technology is pivotal in collecting information, detecting quality, positioning, automatic sorting, and intelligent monitoring within manufacturing. It addresses the challenges of poor quality, low precision, low efficiency, and high labour intensity traditionally associated with manual operations. Adopting machine vision in manufacturing accelerates production, elevates detection accuracy, and minimises the risk of accidents.<h3 dir="ltr">Enabling Algorithms for Object Detection, Identification, and Classification</h3>The backbone of machine vision systems in industrial settings is a suite of sophisticated algorithms that enable object detection, identification, and classification. These algorithms leverage optical tools and computer technology to collect and process data, simulating human visual functions.&nbsp;With the advancement of AI, machine learning, and deep learning technologies, machine vision systems have achieved higher accuracy and broader application scopes. These technologies encourage traditional manufacturing processes to become more intelligent, requiring expertise across various fields, including optical imaging, signal processing, image processing, and deep learning.<h2 dir="ltr">Current Trends and Future Directions</h2>The evolution of machine vision technology is rapidly transforming the landscape of industrial automation, robotics, and beyond. At the forefront of this transformation is integrating 3D imaging and point cloud data, offering enhanced spatial analysis and enabling more complex applications. Let's explore the cutting-edge research and predictions for future advancements in this exciting field.<h3 dir="ltr">Integrating 3D Imaging and Point Cloud Data</h3>In recent years, significant advancements have been made in deep learning techniques for processing 3D point cloud data. This has revolutionised applications in fields such as robotics and autonomous vehicles by allowing for detailed spatial analysis of environments. Techniques exist for various tasks, including 3D object classification, tracking, pose estimation, segmentation, and point cloud completion. Transformative methodologies like Pointr and SnowflakeNet have emerged, highlighting the potential for these technologies to evolve further and impact various domains.<h3 dir="ltr">Predictions for Future Advancements</h3><ul><li dir="ltr">Augmented Reality (AR) Applications: AR's immersive experience can benefit greatly from advancements in machine vision by providing more accurate and real-time interaction with the physical world. This could revolutionise medical imaging, retail, and education by offering more intuitive and interactive ways to engage with information and environments.</li><li dir="ltr">Further Integration with Robotic Process Automation (RPA): Machine vision technology allows robots to perform more complex tasks more efficiently and accurately. Combining machine vision and RPA could lead to significant productivity boosts from manufacturing to logistics by automating tasks that require understanding and interpreting visual data.</li><li dir="ltr">Ease of Use and Accessibility: There's a trend towards making machine vision systems more accessible and easier to use. Efforts are being made to develop "ease-of-use" and "no-code" solutions to democratise the technology, making it more accessible to non-experts.</li><li dir="ltr">Growth in 3D Imaging for Vision Guided Robotics: 3D imaging is no longer emerging but becoming a mature automated imaging offering. From bin picking to product palletizing, these solutions promise to expand the use of VGR by making it more user-friendly and applicable to a broader range of use cases.</li><li dir="ltr">AI and Deep Learning as Automated Inspection Tools: Deep learning has become a valuable tool for segmentation and classification in automated inspection. It enables the identification of images or features within images more intuitively, akin to human perception.</li></ul><h2 dir="ltr">Practical Applications of Machine Vision in Industry</h2>Machine vision technology transforms operations across various sectors, particularly in the automotive and electronics industries, by enhancing product quality and operational efficiency.<h3 dir="ltr">Automotive Industry</h3>Machine vision is essential in inspections and robotic guidance in the automotive sector. Companies like Ford and General Motors (GM) use machine vision to ensure the safety and efficiency of manufacturing processes. Machine vision systems are employed for various applications, including flexible automation, model verification, and quality improvement. The technology offers significant advantages such as adaptable tooling, model verification on flexible lines, and in-station process control for immediate feedback. This enables manufacturers to uphold stringent quality standards and eliminate waste, ultimately contributing to producing a quality product.<h3 dir="ltr">Oil and Gas Industry</h3>Machine vision is critical in inspections and robotic guidance in the oil and gas sector. Companies like Wipro are leveraging machine vision in several cutting-edge ways, including visual inspections of oil rigs, automated quality inspections, and remote monitoring of safety-critical assets. For instance, machine vision enables magnetic crawlers with cameras to inspect difficult-to-reach areas, such as great heights and remote locations within industrial plants. Robots equipped with machine vision can proactively detect internal cracks in gas pipelines, potentially saving millions of dollars by preventing failures. Moreover, these robots can grade the corrosion of pipes based on drone-captured images and videos, facilitating more preventive and cost-effective operations.<h3 dir="ltr">Manufacturing Industry</h3>Machine vision systems are employed for various applications, including flexible automation, model verification, and quality improvement. The technology offers significant advantages such as adaptable tooling, model verification on flexible lines, and in-station process control for immediate feedback. This enables manufacturers to uphold stringent quality standards and eliminate waste, ultimately contributing to producing a quality product.<h3 dir="ltr">Electronics Industry</h3>While the specifics on electronics were not directly covered in the sources, it's evident that the benefits and applications of machine vision in the automotive industry, such as inspection, quality control, and robot guidance, are also pertinent to the electronics sector. High precision and accuracy are essential in electronics manufacturing, where machine vision can identify defects, verify component placement, and ensure product integrity, much like in automotive manufacturing.<h3 dir="ltr">Key Benefits</h3><ul><li dir="ltr">Increased Efficiency: Automation of inspection processes and real-time decision-making capabilities speed up production lines and reduce manual intervention.&nbsp;</li><li dir="ltr">Reduced Error Rates: High-resolution cameras and sophisticated software in modern machine vision systems enable precise defect detection, minimising the risk of errors.</li><li dir="ltr">Enhanced Safety: In the automotive industry, ensuring the proper assembly of over 30,000 parts per vehicle is crucial for driver safety. Machine vision facilitates the flawless fabrication of each component.</li><li dir="ltr">Flexibility and Adaptability:&nbsp;Companies highlight the demand for more flexible automation as a driver for increased use of machine vision. The technology's adaptability allows easy integration into existing production lines, supporting model verification and quality improvements without needing hard-tooled custom fixtures.</li><li dir="ltr">Improved Quality: The ability of machine vision systems to provide feedback for in-station process control plays a vital role in quality assurance. By enabling the detection of defects at the source, these systems help maintain high-quality standards and reduce the cost and complexity of end-of-line repairs.</li></ul>Machine vision technology continues to evolve, offering scalable and versatile solutions that cater to the growing demands of industrial automation.<h2 dir="ltr">Challenges and Considerations</h2>Adopting machine vision technologies has several challenges and considerations that organisations must navigate to leverage their full potential effectively. These challenges include high initial costs, integration complexities, and the need for specialised expertise. By addressing these factors, organisations can better position themselves to take advantage of the numerous benefits that machine vision technologies offer.<h3 dir="ltr">High Initial Costs</h3>Implementing machine vision technologies often requires a significant initial investment. This includes the cost of hardware such as cameras and sensors, software for image processing and analysis, and the infrastructure needed to support these technologies. While the costs of components and engineering have decreased, making the technology more accessible, the initial setup still represents a substantial financial commitment.<h3 dir="ltr">Integration Complexities</h3>Integration involves ensuring compatibility between new and existing systems, configuring machine vision technologies to meet specific operational needs, and optimising performance within the current workflow. This complexity is compounded by the rapid pace of technological advancement, which introduces new standards and capabilities that systems must adapt to.<h3 dir="ltr">Need for Specialized Expertise</h3>Organisations often need professionals skilled in both the technical aspects of machine vision systems and the specific applications within their industry. This expertise is necessary for designing effective solutions, troubleshooting issues, and updating systems to incorporate the latest advancements in the field. The challenge here is finding and retaining talent with the necessary skill set.By proactively addressing these challenges, organisations can harness the power of machine vision technologies to drive innovation and competitive advantage in their operations.<h2 dir="ltr">Conclusion</h2>The transformative impact of machine vision on manufacturing is undeniable. Machine vision has become a cornerstone of modern industrial automation through its ability to automate complex inspection tasks, enhance quality control, and drive efficiencies. Its application spans various sectors, providing unparalleled accuracy and speed in tasks once deemed challenging for automated systems. Machine vision systems minimise downtime, reduce waste, and ensure product quality by enabling more reliable and faster inspections, significantly improving manufacturing.The journey of machine vision from a supportive technology to a pivotal component of industrial automation underscores its transformative impact on manufacturing. As we look to the future, continued innovation, coupled with strong partnerships between technology providers and manufacturers, will be key to unlocking the full potential of machine vision. Embracing these advancements will ensure that industries keep pace with modern production demands and set new standards for efficiency, quality, and innovation.<h2 dir="ltr">References</h2><ol><li dir="ltr"><a href="https://www.automate.org/vision/industry-insights/trends-and-lessons-from-the-2021-machine-vision-market" target="_blank">https://www.automate.org/vision/industry-insights/trends-and-lessons-from-the-2021-machine-vision-market</a></li><li dir="ltr"><a href="https://www.frontiersin.org/articles/10.3389/frobt.2021.627280/full" target="_blank">https://www.frontiersin.org/articles/10.3389/frobt.2021.627280/full</a></li><li dir="ltr"><a href="https://www.cognex.com/what-is/deep-learning/challenges-to-machine-vision" target="_blank">https://www.cognex.com/what-is/deep-learning/challenges-to-machine-vision</a></li><li dir="ltr"><a href="https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry" target="_blank">https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry</a></li><li dir="ltr"><a href="https://www.zebra.com/us/en/blog/posts/2021/why-auto-manufacturers-should-take-a-second-look-at-new-machine-vision-systems.html" target="_blank">https://www.zebra.com/us/en/blog/posts/2021/why-auto-manufacturers-should-take-a-second-look-at-new-machine-vision-systems.html</a></li><li dir="ltr"><a href="https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry" target="_blank">https://www.automate.org/vision/industry-insights/machine-vision-in-the-automotive-industry</a></li><li dir="ltr"><a href="https://www.vision-systems.com/factory/article/14300295/machine-vision-and-imaging-trends-in-automation-a-view-from-the-trenches-for-2024" target="_blank">https://www.vision-systems.com/factory/article/14300295/machine-vision-and-imaging-trends-in-automation-a-view-from-the-trenches-for-2024</a></li><li dir="ltr"><a href="https://news.mit.edu/2019/deep-learning-point-clouds-1021" target="_blank">https://news.mit.edu/2019/deep-learning-point-clouds-1021</a></li><li dir="ltr"><a href="https://pubs.aip.org/aip/acp/article-abstract/2788/1/110005/2903916/An-overview-of-machine-vision-based-quality?redirectedFrom=fulltext" target="_blank">https://pubs.aip.org/aip/acp/article-abstract/2788/1/110005/2903916/An-overview-of-machine-vision-based-quality?redirectedFrom=fulltext</a></li><li dir="ltr"><a href="https://aimagazine.com/articles/machine-vision-enabling-the-factories-of-the-future" target="_blank">https://aimagazine.com/articles/machine-vision-enabling-the-factories-of-the-future</a></li></ol>

Discover the revolutionary impact of machine vision in manufacturing. Learn how integrating AI and machine vision enhances efficiency, quality, and safety in industrial automation, and explore the challenges and future advancements in this cutting-edge technology.

Transforming Manufacturing with Machine Vision Technology

Three-and-a-half&nbsp;years on from the Bluetooth Special Interest Group’s (SIG) announcement of a new specification that defined a high quality, low-power audio codec (Low Complexity Communications Codec or LC3) that promised to redefine audio streaming and wireless sound,&nbsp;<a href="https://www.nordicsemi.com/Products/Wireless/Bluetooth-LE-Audio" target="_blank" rel="noreferrer noopener">Bluetooth LE Audio</a>&nbsp;(LE Audio) is finally about to hit the commercial mainstream.The delay has due to the development time needed to shake out the bugs from what is a complex technology. As Chuck Sabin, Senior Director, Market Development at the SIG says, developing specifications is hard, particularly when there are billions of Bluetooth-enabled receiving devices out there. That’s a lot of interested parties invested in ensuring the specification is going to deliver so things work perfectly.&nbsp;With the due diligence now complete, and a year on from the adoption of the specification, the first commercial implementations of LE Audio are now coming to market. Within another three or four years, Sabin predicts—once the tech is natively adopted by iOS and Android smartphones and major wireless audio device vendors—LE Audio will be universal.&nbsp;&nbsp;When it is, wireless audio will become unrecognizable from anything we’ve previously experienced, and not just for the millions of us that use wireless headphones and earbuds, but more importantly for the hearing impaired as well.&nbsp;&nbsp;<h2>True Wireless Stereo&nbsp;and&nbsp;better battery life&nbsp;</h2>Two key advantages of LE Audio over Classic Audio that will drive mass consumer adoption are the noticeable improvements in audio quality and longer playtime.&nbsp;&nbsp;While Bluetooth LE supports multiple communication topologies for data transfer (for example, Bluetooth Mesh), it only supports point-to-point communication for audio streaming. LE Audio’s multi-stream audio functionality however enables multiple devices—for example a pair of earbuds—to receive their own unique audio stream. The left and right channels retain the original precise musical timing upon replay. The result is excellent stereo sound together with long battery life. That’s unlike the earbuds in use today that can only stream audio to a single device and therefore require an engineering work around to retransmit the second channel to a second earbud, at the cost of battery life.&nbsp;On the question of battery life, quantifying the expected reduction in power consumption over Classic Audio is difficult because it will depend on both implementation and product use. Product developers focused on providing the highest audio quality might choose to implement the LC3 codec for hi-fidelity rather than aiming for significant gains in playback time. Others may choose to focus entirely on extending battery life by trading-off sound quality. In the latter case, the battery life of your earbuds or Bluetooth speaker could double.&nbsp;<h2>Auracast&nbsp;broadcast audio promotes&nbsp;inclusion&nbsp;</h2>Another breakthrough feature of LE Audio is the addition of Auracast broadcast audio. Auracast allows an audio source device such as a smartphone, laptop, TV, or sound system to broadcast one or several audio streams to an unlimited number of Bluetooth audio receivers.&nbsp;&nbsp;There are several potential use cases for Auracast, but chief among them is the ability for it to become a high quality, lower cost augmented and assistive listening technology in venues where PA or hearing loop infrastructure is currently deployed. Such a use case opens up wider accessibility for the over 1.5 billion people living with hearing loss around the world. It can also enable multi-language audio streams in locations that support simultaneous translation services.&nbsp;&nbsp;<h2>Next gen SoCs&nbsp;optimized for LE Audio&nbsp;</h2>While LE Audio’s LC3 codec does demand more computational resource than Classic Audio’s SBC codec to compress and decompress audio, the latest generation of wireless SoCs can comfortably meet this requirement without compromising any power consumption advantages. For example, Nordic Semiconductor’s&nbsp;<a href="https://www.nordicsemi.com/Products/nRF5340" target="_blank">nRF5340</a> SoC features a dedicated Arm Cortex-M33 application processor, optimized for performance and with ample overhead to look after the computational demands of advanced audio codecs. The SoC incorporates a second M33 processor to supervise wireless connectivity and which is optimized for low power consumption.&nbsp;Looking forward to mass commercialization of LE Audio, Nordic’s fourth generation of wireless SoCs, the&nbsp;<a href="https://www.nordicsemi.com/Products/nRF54L15" target="_blank">nRF54L15</a>&nbsp;and&nbsp;<a href="https://www.nordicsemi.com/Products/nRF54H20" target="_blank">nRF54H20</a>&nbsp;are both now sampling, and represent the future of not just LE Audio, but for many applications demanding complex machine learning (ML) and support for sensor fusion at the edge. The&nbsp;nRF54H20 SoC features multiple Arm Cortex-M33 processors and multiple RISC-V coprocessors optimized for specific types of workloads. The level of integration this will provide product developers ensures tomorrow’s LE Audio solutions—from earbuds to hearing aids—will consume even less energy, be smaller, and simpler to bring to market.&nbsp;<h2>LE Audio and&nbsp;Auracast&nbsp;world first&nbsp;</h2>The first commercialized LE Audio solutions are already available. Earlier this year, global hearing technology solutions provider GN launched a&nbsp;<a href="https://www.nordicsemi.com/Nordic-news/2024/01/GN-Group-unveils-Nordic-powered-Bluetooth-LE-Audio-and-Auracast-broadcast-audio-TV-streamer" target="_blank">world first LE Audio and Auracast broadcast audio TV streamer</a>, enabling people suffering from hearing loss to stream hi-fidelity audio between their television and their hearing aids, with all the benefits of Auracast. Working seamlessly with the company’s new ReSound Nexia hearing aids or other LE Audio compatible solutions, the TV-Streamer+ integrates a Nordic’s LE Audio- and Auracast-capable SoC to deliver low power, low latency, and long range wireless connectivity with improved audio quality.&nbsp;&nbsp;For people with hearing loss the launch means they can enjoy immediate speech clarity at their preferred volume, while at the same time enjoying the benefits of Auracast. For developers and consumers, it means the LE Audio and Auracast are in the public eye.&nbsp;

Three-and-a-half years on from the Bluetooth Special Interest Group’s (SIG) announcement of a new specification that defined a high quality, low-power audio codec that promised to redefine audio streaming and wireless sound, Bluetooth LE Audio is finally about to hit the commercial mainstream.

Bluetooth LE Audio and Auracast going mainstream will redefine wireless audio

Researchers have developed a new training tool to help artificial intelligence (AI) programs better account for the fact that humans don’t always tell the truth when providing personal information. The new tool was developed for use in contexts when humans have an economic incentive to lie, such as applying for a mortgage or trying to lower their insurance premiums.“AI programs are used in a wide variety of business contexts, such as helping to determine how large of a mortgage an individual can afford, or what an individual’s insurance premiums should be,” says Mehmet Caner, co-author of a paper on the work. “These AI programs generally use mathematical algorithms driven solely by statistics to do their forecasting. But the problem is that this approach creates incentives for people to lie, so that they can get a mortgage, lower their insurance premiums, and so on.“We wanted to see if there was some way to adjust AI algorithms in order to account for these economic incentives to lie,” says Caner, who is the Thurman-Raytheon Distinguished Professor of Economics in North Carolina State University’s Poole College of Management.To address this challenge, the researchers developed a new set of training parameters that can be used to inform how the AI teaches itself to make predictions. Specifically, the new training parameters focus on recognizing and accounting for a human user’s economic incentives. In other words, the AI trains itself to recognize circumstances in which a human user might lie to improve their outcomes.In proof-of-concept simulations, the modified AI was better able to detect inaccurate information from users.“This effectively reduces a user’s incentive to lie when submitting information,” Caner says. “However, small lies can still go undetected. We need to do some additional work to better understand where the threshold is between a ‘small lie’ and a ‘big lie.’”The researchers are making the new AI training parameters publicly available, so that AI developers can experiment with them.“This work shows we can improve AI programs to reduce economic incentives for humans to lie,” Caner says. “At some point, if we make the AI clever enough, we may be able to eliminate those incentives altogether.”The paper, “<a href="https://www.tandfonline.com/doi/full/10.1080/07350015.2024.2316102" data-type="link" data-id="https://www.tandfonline.com/doi/full/10.1080/07350015.2024.2316102" target="_blank" rel="noreferrer noopener">Should Humans Lie to Machines? The Incentive Compatibility of Lasso and GLM Structured Sparsity Estimators</a>,” is published in the Journal of Business &amp; Economic Statistics. The paper was co-authored by Kfir Eliaz of Tel-Aviv University and the University of Utah.<hr>“Should Humans Lie to Machines? The Incentive Compatibility of Lasso and GLM Structured Sparsity Estimators”Authors: Mehmet Caner, North Carolina State University; and Kfir Eliaz, Tel-Aviv University and the University of UtahPublished: March 12,&nbsp;Journal of Business &amp; Economic StatisticsDOI: 10.1080/07350015.2024.2316102

Researchers have developed a new training tool to help artificial intelligence (AI) programs better account for the fact that humans don’t always tell the truth when providing personal information.

New Technique Helps AI Tell When Humans Are Lying

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/f153920a-aca0-4937-863a-2cc925d11434?dark=false" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Columbia engineers build Emo, a silicon-clad robotic face that makes eye contact and uses two AI models to anticipate and replicate a person’s smile before the person actually smiles -- a major advance in robots predicting human facial expressions accurately, improving interactions, and building trust between humans and robots.<section id="isPasted">What would you do if you walked up to a robot with a human-like head and it smiled at you first? You’d likely smile back and perhaps feel the two of you were genuinely interacting. But how does a robot know how to do this? Or a better question, how does it know to get you to smile back?While we’re getting accustomed to robots that are adept at verbal communication, thanks in part to advancements in large language models like ChatGPT, their nonverbal communication skills, especially facial expressions, have lagged far behind. Designing a robot that can not only make a wide range of facial expressions but also know when to use them has been a daunting task.&nbsp;</section><section><h2>Tackling the challenge</h2>The <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Creative Machines Lab</a> at <a href="http://engineering.columbia.edu/" rel="nofollow" target="_blank">Columbia Engineering</a> has been working on this challenge for more than five years. In a new study published today in <a href="https://www.science.org/doi/10.1126/scirobotics.adi4724" rel="nofollow" target="_blank">Science Robotics</a>, the group unveils Emo, a robot that anticipates facial expressions and executes them simultaneously with a human. It has even learned to predict a forthcoming smile about 840 milliseconds before the person smiles, and to co-express the smile simultaneously with the person.The team, led by <a href="https://www.engineering.columbia.edu/faculty/hod-lipson" rel="nofollow" target="_blank">Hod Lipson</a>, a leading researcher in the fields of artificial intelligence (AI) and robotics, faced two challenges: how to mechanically design an expressively versatile robotic face which involves complex hardware and actuation mechanisms, and knowing which expression to generate so that they appear natural, timely, and genuine. The team proposed training a robot to anticipate future facial expressions in humans and execute them simultaneously with a person. The timing of these expressions was critical -- delayed facial mimicry looks disingenuous, but facial co-expression feels more genuine since it requires correctly inferring the human's emotional state for timely execution.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzExNTg1NTM0ODUxLXl1aGFuZy1odWEtb2YtY3JlYXRpdmUtbWFjaGluZXMtbGFiLWZhY2UtdG8tZmFjZS13aXRoLWVtby5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Yuhang Hu of Creative Machines Lab face-to-face with Emo. Image: Courtesy of Creative Machines Lab<h2 id="isPasted">How Emo connects with you</h2>Emo is a human-like head with a face that is equipped with 26 actuators that enable a broad range of nuanced facial expressions. The head is covered with a soft silicone skin with a magnetic attachment system, allowing for easy customization and quick maintenance. For more lifelike interactions, the researchers integrated high-resolution cameras within the pupil of each eye, enabling Emo to make eye contact, crucial for nonverbal communication.The team developed two AI models: one that predicts human facial expressions by analyzing subtle changes in the target face and another that generates motor commands using the corresponding facial expressions.To train the robot how to make facial expressions, the researchers put Emo in front of the camera and let it do random movements. After a few hours, the robot learned the relationship between their facial expressions and the motor commands -- much the way humans practice facial expressions by looking in the mirror. This is what the team calls “self modeling” – similar to our human ability to imagine what we look like when we make certain expressions.Then the team ran videos of human facial expressions for Emo to observe them frame by frame. After training, which lasts a few hours, Emo could predict people’s facial expressions by observing tiny changes in their faces as they begin to form an intent to smile.“I think predicting human facial expressions accurately is a revolution in HRI. Traditionally, robots have not been designed to consider humans' expressions during interactions. Now, the robot can integrate human facial expressions as feedback,” said the study’s lead author Yuhang Hu, who is a PhD student at Columbia Engineering in Lipson’s lab. “When a robot makes co-expressions with people in real-time, it not only improves the interaction quality but also helps in building trust between humans and robots. In the future, when interacting with a robot, it will observe and interpret your facial expressions, just like a real person.”<iframe width="640" height="360" src="https://www.youtube.com/embed/pWTTzR_wXuQ?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-gtm-yt-inspected-9="true" id="556492795" title="Human-Robot Facial Co-expression">&nbsp;</iframe>Go inside the Creative Machines Lab to watch Emo's facial co-expression. Video Credit: Yuhang Hu, Columbia Engineering<h2 id="isPasted">What’s next</h2>The researchers are now working to integrate verbal communication, using a large language model like ChatGPT into Emo. As robots become more capable of behaving like humans, Lipson is well aware of the ethical considerations associated with this new technology.“Although this capability heralds a plethora of positive applications, ranging from home assistants to educational aids, it is incumbent upon developers and users to exercise prudence and ethical considerations,” says Lipson, James and Sally Scapa Professor of Innovation in the Department of <a href="https://www.me.columbia.edu/" rel="nofollow" target="_blank">Mechanical Engineering</a> at Columbia Engineering, co-director of the <a href="https://make.columbia.edu/" rel="nofollow" target="_blank">Makerspace</a> at Columbia, and a member of the <a href="https://datascience.columbia.edu/" rel="nofollow" target="_blank">Data Science Institute</a>. “But it’s also very exciting -- by advancing robots that can interpret and mimic human expressions accurately, we're moving closer to a future where robots can seamlessly integrate into our daily lives, offering companionship, assistance, and even empathy. Imagine a world where interacting with a robot feels as natural and comfortable as talking to a friend.”&nbsp;</section>

The study's lead author, Yuhang Hu, a PhD student at Columbia Engineering in Hod Lipson’s lab. Credit: John Abbott/Columbia Engineering

Columbia engineers build Emo, a silicon-clad robotic face that makes eye contact and uses two AI models to anticipate and replicate a person’s smile before the person actually smiles 

Robot, can you say 'Cheese'?

ANYmal has for some time had no problem coping with the stony terrain of Swiss hiking trails. Now researchers at ETH Zurich have taught this quadrupedal robot some new skills: it is proving rather adept at parkour, a sport based on using athletic manoeuvres to smoothly negotiate obstacles in an urban environment, which has become very popular. ANYmal is also proficient at dealing with the tricky terrain commonly found on building sites or in disaster areas.To teach ANYmal these new skills, two teams, both from the group led by ETH Professor Marco Hutter of the Department of Mechanical and Process Engineering, followed different approaches.<h2>Exhausting the mechanical options</h2>Working in one of the teams is ETH doctoral student Nikita Rudin, who does parkour in his free time. “Before the project started, several of my researcher colleagues thought that legged robots had already reached the limits of their development potential,” he says, “but I had a different opinion. In fact, I was sure that a lot more could be done with the mechanics of legged robots.”With his own parkour experience in mind, Rudin set out to further push the boundaries of what ANYmal could do. And he succeeded, by using machine learning to teach the quadrupedal robot new skills. ANYmal can now scale obstacles and perform dynamic manoeuvres to jump back down from them.<iframe width="640" height="360" src="https://www.youtube.com/embed/REvNnUzVDAA?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe>(Video: ETH Zurich / Nicole Davidson)In the process, ANYmal learned like a child would – through trial and error. Now, when presented with an obstacle, ANYmal uses its camera and artificial neural network to determine what kind of impediment it’s dealing with. It then performs movements that seem likely to succeed based on its previous training.Is that the full extent of what’s technically possible? Rudin suggests that this is largely the case for each individual new skill. But he adds that this still leaves plenty of potential improvements. These include allowing the robot to move beyond solving predefined problems and instead asking it to negotiate difficult terrain like rubble-strewn disaster areas.<h2>Combining new and traditional technologies</h2>Getting ANYmal ready for precisely that kind of application was the goal of the other project, conducted by Rudin’s colleague and fellow ETH doctoral student Fabian Jenelten. But rather than relying on machine learning alone, Jenelten combined it with a tried-and-tested approach used in control engineering known as model-based control. This provides an easier way of teaching the robot accurate manoeuvres, such as how to recognise and get past gaps and recesses in piles of rubble. In turn, machine learning helps the robot master movement patterns that it can then flexibly apply in unexpected situations. “Combining both approaches lets us get the most out of ANYmal,” Jenelten says.As a result, the quadrupedal robot is now better at gaining a sure footing on slippery surfaces or unstable boulders. ANYmal is soon also to be deployed on building sites or anywhere that is too dangerous for people – for instance to inspect a collapsed house in a disaster area.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzExNDk3MTQwNDIwLWltYWdlLmltYWdlZm9ybWF0LjEyODYuMTg0OTg5MDA4Ny5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">ANYmal on a civil defence training ground. (Photograph: ETH Zurich / Fabian Jenelten)<hr><h2 id="isPasted">References</h2>Hoeller D, Rudin N, Sako D, Hutter M: ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots, Science Robotics, 13 March 2024, doi: <a href="https://doi.org/10.1126/scirobotics.adi7566" target="_blank">external page10.1126/scirobotics.adi7566call_made</a>Jenelten F, He J, Farshidian F, Hutter M: DTC: Deep Tracking Control. Science Robotics 2024, 17: eadh5401, doi: <a href="https://doi.org/10.1126/scirobotics.adh5401" target="_blank">external page10.1126/scirobotics.adh5401call_made</a>

The quadrupedal robot ANYmal practises parkour in a hall at ETH Zurich. (Photograph: ETH Zurich / Nikita Rudin)

The quadrupedal robot ANYmal went back to school and has learned a lot. ETH Zurich researchers used machine learning to teach it new skills: the robot can now climb over obstacles and successfully negotiate pitfalls.

ANYmal can do parkour and walk across rubble

If a robot traveling to a destination has just two possible paths, it needs only to compare the routes’ travel time and probability of success. But if the robot is traversing a complex environment with many possible paths, choosing the best route amid so much uncertainty can quickly become an intractable problem.MIT researchers developed a method that could help this robot efficiently reason about the best routes to its destination. They created an algorithm for constructing roadmaps of an uncertain environment that balances the tradeoff between roadmap quality and computational efficiency, enabling the robot to quickly find a traversable route that minimizes travel time.The algorithm starts with paths that are certain to be safe and automatically finds shortcuts the robot could take to reduce the overall travel time. In simulated experiments, the researchers found that their algorithm can achieve a better balance between planning performance and efficiency in comparison to other baselines, which prioritize one or the other.This algorithm could have applications in areas like exploration, perhaps by helping a robot plan the best way to travel to the edge of a distant crater across the uneven surface of Mars. It could also aid a search-and-rescue drone in finding the quickest route to someone stranded on a remote mountainside.“It is unrealistic, especially in very large outdoor environments, that you would know exactly where you can and can’t traverse. But if we have just a little bit of information about our environment, we can use that to build a high-quality roadmap,” says Yasmin Veys, an electrical engineering and computer science (EECS) graduate student and lead author of a&nbsp;<a href="https://groups.csail.mit.edu/rrg/papers/veys_icra_24.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">paper on this technique</a>.Veys wrote the paper with Martina Stadler Kurtz, a graduate student in the MIT Department of Aeronautics and Astronautics, and senior author Nicholas Roy, an MIT professor of aeronautics and astronautics and a member of the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL). The research will be presented at the International Conference on Robotics and Automation.<h2>Generating graphs</h2>To study motion planning, researchers often think about a robot’s environment like a graph, where a series of “edges,” or line segments, represent possible paths between a starting point and a goal.Veys and her collaborators used a graph representation called the Canadian Traveler’s Problem (CTP), which draws its name from frustrated Canadian motorists who must turn back and find a new route when the road ahead is blocked by snow.In a CTP, each edge of the graph has a weight associated with it, which represents how long that path will take to traverse, and a probability of how likely it is to be traversable. The goal in a CTP is to minimize travel time to the destination.The researchers focused on how to automatically generate a CTP graph that effectively represents an uncertain environment.“If we are navigating in an environment, it is possible that we have some information, so we are not just going in blind. While it isn’t a detailed navigation plan, it gives us a sense of what we are working with. The crux of this work is trying to capture that within the CTP graph,” adds Kurtz.Their algorithm assumes this partial information — perhaps a satellite image — can be divided into specific areas (a lake might be one area, an open field another, etc.)Each area has a probability that the robot can travel across it. For instance, it is more likely a nonaquatic robot can drive across a field than through a lake, so the probability for a field would be higher.The algorithm uses this information to build an initial graph through open space, mapping out a conservative path that is slow but definitely traversable. Then it uses a metric the team developed to determine which edges, or shortcut paths through uncertain regions, should be added to the graph to cut down on the overall travel time.<h2>Selecting shortcuts</h2>By only selecting shortcuts that are likely to be traversable, the algorithm keeps the planning process from becoming needlessly complicated.“The quality of the motion plan is dependent on the quality of graph. If that graph doesn’t have good paths in it, then the algorithm can’t give you a good plan,” Veys explains.After testing the algorithm in more than 100 simulated experiments with increasingly complex environments, the researchers found that it could consistently outperform baseline methods that don’t consider probabilities. They also tested it using an aerial campus map of MIT to show that it could be effective in real-world, urban environments.In the future, they want to enhance the algorithm so it can work in more than two dimensions, which could enable its use for complicated robotic manipulation problems. They are also interested in studying the mismatch between CTP graphs and the real-world environments those graphs represent.“Robots that operate in the real world are plagued by uncertainty, whether in the available sensor data, prior knowledge about the environment, or about how other agents will behave. Unfortunately, dealing with these uncertainties incurs a high computational cost,” says Seth Hutchinson, professor and KUKA Chair for Robotics in the School of Interactive Computing at Georgia Tech, who was not involved with this research. “This work addresses these issues by proposing a clever approximation scheme that can be used to efficiently compute uncertainty-tolerant plans.”

MIT researchers developed an algorithm that can automatically select the best shortcuts for a robot to take on its way to a destination that will reduce the overall travel time while limiting the likelihood that the robot will meet an impassable obstacle. Image: Jose-Luis Olivares, MIT; iStock

A new algorithm reduces travel time by identifying shortcuts a robot could take on the way to its destination.

Researchers help robots navigate efficiently in uncertain environments

In the blink of an eye, the unruly, superheated plasma that drives a fusion reaction can lose its stability and escape the strong magnetic fields confining it within the donut-shaped fusion reactor. These getaways frequently spell the end of the reaction, posing a core challenge to developing fusion as a non-polluting, virtually limitless energy source.But a Princeton-led team composed of engineers, physicists, and data scientists from the University and the U.S. Department of Energy’s <a href="https://www.pppl.gov/" target="_blank">Princeton Plasma Physics La</a><a href="https://www.pppl.gov/" target="_blank">boratory</a> (PPPL) have harnessed the power of artificial intelligence to predict — and then avoid — the formation of a specific plasma problem in real time.In experiments at the <a href="https://www.ga.com/magnetic-fusion/diii-d" target="_blank">DIII-D National Fusion Facility</a> in San Diego, the researchers demonstrated their model, trained only on past experimental data, could forecast potential plasma instabilities known as tearing mode instabilities up to 300 milliseconds in advance. While that leaves no more than enough time for a slow blink in humans, it was plenty of time for the AI controller to change certain operating parameters to avoid what would have developed into a tear within the plasma’s magnetic field lines, upsetting its equilibrium and opening the door for a reaction-ending escape.“By learning from past experiments, rather than incorporating information from physics-based models, the AI could develop a final control policy that supported a stable, high-powered plasma regime in real time, at a real reactor,” said research leader <a href="https://engineering.princeton.edu/faculty/egemen-kolemen" target="_blank">Egemen Kolemen</a>, associate professor of <a href="https://mae.princeton.edu/" target="_blank">mechanical and aerospace engineering</a> and the <a href="https://acee.princeton.edu/" target="_blank">Andlinger Center for Energy and the Environment</a>, as well as staff research physicist at PPPL.The research opens the door for more dynamic control of a fusion reaction than current approaches, and it provides a foundation for using artificial intelligence to solve a broad range of plasma instabilities, which have long been obstacles to achieving a sustained fusion reaction. The team published <a href="https://www.nature.com/articles/s41586-024-07024-9" target="_blank">their findings</a> in Nature on February 21.“Previous studies have generally focused on either suppressing or mitigating the effects of these tearing instabilities after they occur in the plasma,” said first author <a href="https://physics.cau.ac.kr/BM/bm_1_view.php?idx=76" target="_blank">Jaemin Seo</a>, an assistant professor of physics at Chung-Ang University in South Korea who performed much of the work while a postdoctoral researcher in Kolemen’s group. “But our approach allows us to predict and avoid those instabilities before they ever appear.”<h2>Superheated plasma swirling in a donut-shaped device</h2>Fusion takes place when two atoms — usually light atoms like hydrogen — come together to form one heavier atom, releasing a large amount of energy in the process. The process powers the Sun, and, by extension, makes life on Earth possible.However, getting the two atoms to fuse is tricky, as it takes massive amounts of pressure and energy for the two atoms to overcome their mutual repulsion.Fortunately for the Sun, its massive gravitational pull and extremely high pressures at its core allow fusion reactions to proceed. To replicate a similar process on the Earth, scientists instead use extremely hot plasma and extremely strong magnets.In donut-shaped devices known as tokamaks — sometimes referred to as “stars in jars” — magnetic fields struggle to contain plasmas that reach above 100 million degrees Celsius, hotter than the center of the Sun.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1709522056008-magnetic_tearing_v3_simplified-copy.webp" style="width: 100%px;" class="fr-fic fr-dib">The researchers harnessed the power of AI to predict and avoid the formation of a tearing instability (left), which could quickly result in plasma disruption and the termination of the fusion reaction. (Courtesy of the researchers).While there are many types of plasma instabilities that can terminate the reaction, the Princeton team concentrated on solving tearing mode instabilities, a disturbance in which the magnetic field lines within a plasma actually break and create an opportunity for the plasma’s subsequent escape.“Tearing mode instabilities are one of the major causes of plasma disruption, and they will become even more prominent as we try to run fusion reactions at the high powers required to produce enough energy,” said Seo. “They are an important challenge for us to solve.”<h2>Fusing artificial intelligence and plasma physics</h2>Since tearing mode instabilities can form and derail a fusion reaction in milliseconds, the researchers turned to artificial intelligence for its ability to quickly process and act in response to new data.But the process to develop an effective AI controller was not as simple as trying out a few things on a tokamak, where time is limited, and the stakes are high.Co-author&nbsp;<a href="https://mae.princeton.edu/people/research-staff/jalalvand" target="_blank">Azarakhsh Jalalvand</a>, a research scholar in Kolemen’s group, compared teaching an algorithm to run a fusion reaction in a tokamak to teaching someone how to fly a plane.“You wouldn’t teach someone by handing them a set of keys and telling them to try their best,” Jalalvand said. “Instead, you’d have them practice on a very intricate flight simulator until they’ve learned enough to try out the real thing.”Like developing a flight simulator, the Princeton team used data from past experiments at the DIII-D tokamak to construct a deep neural network capable of predicting the likelihood of a future tearing instability based on real-time plasma characteristics.They used that neural network to train a reinforcement learning algorithm. Like a pilot trainee, the reinforcement learning algorithm could try out different strategies for controlling plasma, learning through trial and error which strategies worked and which did not within the safety of a simulated environment.“We don’t teach the reinforcement learning model all of the complex physics of a fusion reaction,” Jalalvand said. “We tell it what the goal is — to maintain a high-powered reaction — what to avoid — a tearing mode instability — and the knobs it can turn to achieve those outcomes. Over time, it learns the optimal pathway for achieving the goal of high power while avoiding the punishment of an instability.”While the model went through countless simulated fusion experiments, trying to find ways to maintain high power levels while avoiding instabilities, co-author&nbsp;<a href="https://www.pppl.gov/people/sangkyeun-kim" target="_blank">SangKyeun Kim</a> could observe and refine its actions.“In the background, we can see the intentions of the model,” said Kim, a staff research scientist at PPPL and former postdoctoral researcher in Kolemen’s group. “Some of the changes that the model wants are too rapid, so we work to smooth and calm the model. As humans, we arbitrate between what the AI wants to do and what the tokamak can accommodate.”Once they were confident in the AI controller’s abilities, they tested it during an actual fusion experiment at the D-III D tokamak, observing as the controller made real-time changes to certain tokamak parameters to avoid the onset of an instability. These parameters included changing the shape of the plasma and the strength of the beams inputting power into the reaction.“Being able to predict instabilities ahead of time can make it easier to run these reactions than current approaches, which are more passive,” said Kim. “We no longer have to wait for the instabilities to occur and then take quick corrective action before the plasma becomes disrupted.”<h2>Powering into the future</h2>While the researchers said the work is a promising proof-of-concept demonstrating how artificial intelligence can effectively control fusion reactions, it is only one of many next steps already ongoing in Kolemen’s group to advance the field of fusion research.The first step is to get more evidence of the AI controller in action at the DIII-D tokamak, and then expand the controller to function at other tokamaks.“We have strong evidence that the controller works quite well at DIII-D, but we need more data to show that it can work in a number of different situations,” said first author Seo. “We want to work toward something more universal.”A second line of research involves expanding the algorithm to handle many different control problems at the same time. While the current model uses a limited number of diagnostics to avoid one specific type of instability, the researchers could provide data on other types of instabilities and give access to more knobs for the AI controller to tune.“You could imagine one large reward function that turns many different knobs to simultaneously control for several types of instabilities,” said co-author&nbsp;<a href="https://mae.princeton.edu/people/graduate-students/shousha" target="_blank">Ricardo Shousha</a>, a postdoc at PPPL and former graduate student in Kolemen’s group who provided support for the experiments at DIII-D.And on the route to developing better AI controllers for fusion reactions, researchers might also gain more understanding of the underlying physics. By studying the AI controller’s decisions as it attempts to contain the plasma, which can be radically different than what traditional approaches might prescribe, artificial intelligence may be not only a tool to control fusion reactions but also a teaching resource.“Eventually, it may be more than just a one-way interaction of scientists developing and deploying these AI models,” said Kolemen. “By studying them in more detail, they may have certain things that they can teach us too.”<hr>The paper, “<a href="https://www.nature.com/articles/s41586-024-07024-9" target="_blank">Avoiding fusion plasma tearing instability with deep reinforcement learning</a>,” was published February 21 in Nature. In addition to Kolemen, Seo, Jalalvand, Kim, and Shousha, co-authors include Rory Conlin, Andrew Rothstein, Joseph Abbate and Josiah Wai of Princeton University, as well as Keith Erickson of PPPL.The work was supported by the U.S. Department of Energy’s Office of Fusion Energy Sciences, as well as the National Research Foundation of Korea (NRF). The authors also acknowledge the use of the DIII-D National Fusion Facility, a Department of Energy Office of Science user facility.

In the blink of an eye, the unruly, superheated plasma that drives a fusion reaction can lose its stability and escape the strong magnetic fields confining it within the donut-shaped fusion reactor. 

Engineers use AI to wrangle fusion power for the grid

In order to successfully 3D print a metal part to the exacting specifications that many in industry demand, process parameters, including printing speed, laser power, and layer thickness of the deposited material, must all be optimized.But to develop additive manufacturing process maps that ensure those optimal results, researchers have had to rely upon conventional methods—lab experiments that use ex situ materials characterization to test parts that have been printed using various parameters.Testing so many combinations of parameters in order to develop the optimal process can be both time consuming and expensive, especially considering the wide range of metals and alloys that can be used in additive manufacturing.David Guirguis, <a href="https://engineering.cmu.edu/directory/bios/beuth-jack.html" target="_blank">Jack Beuth</a>, and <a href="https://engineering.cmu.edu/directory/bios/tucker-conrad.html" target="_blank">Conrad Tucker</a> developed a system using ultra high-speed in-situ imaging and vision transformers that can not only optimize those process parameters but is also generalizable so that it can be applied to various metal alloys.Vision transformers are a form of machine learning that apply neural network architectures originally developed for natural language processing tasks to computer vision tasks such as image classification. And the video vision transformers take that a step further by using video sequences instead of still images to capture both spatial and temporal relationships that enable the model to learn complex patterns and dependencies in video data.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1709336830375-0223-em-ai-metal-alloys.gif" style="width: 100%px;" class="fr-fic fr-dib">Videos of molten pools of the titanium alloy Ti-6Al-4V captured at different printing regimes.The self-attention mechanism, which allows natural language processing models to weigh the importance of different words in a sequence, allows the model Guirguis created to weigh the importance of different parts of the input sequence for making predictions about the occurrence of defects.“We needed to automate the process, but it can’t be done with computer programming alone,” explained Guirguis, a postdoctoral associate in <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineering</a>. “In order to capture the patterns, we need to apply machine learning.”"We are excited to have developed an AI method that leverages temporal features in AM imaging data to detect different types of defects. Demonstrating the generalizability of the AI method using different AM metals is groundbreaking and reveals that the same trained AI model can be employed without costly retraining using additional data,” remarked Tucker, a professor of mechanical engineering.Guirguis says he is fortunate to have had such strong training in machine learning at Carnegie Mellon because it is more important than ever that mechanical engineers know how to apply both experimental and computational solutions to the problems they solve.<blockquote>We are excited to have developed an AI method that leverages temporal features in AM imaging data to detect different types of defects.<cite>Conrad Tucker<cite id="isPasted"><cite>Professor<cite>, Mechanical Engineering</cite></cite></cite><cite id="isPasted"> </cite></cite></blockquote>In this case, Guirguis was trying to overcome the primary limitation of in-situ imaging of the laser powder bed fusion (LPBF) additive manufacturing process. The technology uses a high-power laser as an energy source to melt and fuse powders in specific locations to form certain shapes, a re-coater then spreads a new layer of powder, and the process repeats until 3D objects are formed.But the molten metal seen by a camera during the printing process is saturated, so it’s not possible to see its physical features, which can identify possible defects that can deteriorate the mechanical properties and reduce fatigue life of the printed part.Guirguis used funding from the Army Research Laboratory to develop a high-speed imaging setup that can capture clear features of the molten pool and a machine learning model that could see the patterns associated with the defects they were trying to detect and prevent.He incorporated the temporal features of molten metal as it changed over time by using high-speed imaging and video vision transformers.By using the vision transformers to classify the different types of defects that can occur during the 3D printing process, Guirguis enhanced the algorithmic accuracy to greater than 90% depending on the material.“In additive manufacturing processing of a new alloy, the first goal is to find a ‘window’ of process variables yielding flaw-free parts,” explained Beuth, a professor of mechanical engineering. “Dave’s use of vision transformers to relate the variability in high-speed melt pool images to flaw formation can greatly reduce the time needed to find that window. &nbsp;It is a huge step forward.”The researchers developed an off-axial imaging setup using a high-speed video camera and magnification lens to capture the high-frequency oscillation in the melt-pool shape with video recorded with extremely high temporal resolution of over 50,000 frames per second. The videos were then classified into four categories: a desirable regime and printing regimes of the three different types of defects: keyholing, balling, and lack-of-fusion.Keyholing defects, which are characterized by unstable, deep, and narrow penetration, can lead to enclosed pores inside the printed parts and result in cracks that can degrade the fatigue life of the parts. The keyhole regime is typically characterized by fluctuations in the width and depth of the keyhole.With balling defects, known as humping in welding, the melted tracks exhibit a rough surface with a periodic ball cross-section shape and are associated with undercuts at the corners. In the balling regime, the molten pool elongates and disconnects, leaving behind peaks in the track.Lack-of-fusion defects, where the energy density is not sufficient to fully melt the powder, cause unmelted powder and irregular gaps between the melted tracks. Melt pools captured in the lack-of-fusion regime are very small with a low length-to-width ratio, as the energy density is very low, and the laser beam does not penetrate deeply into the material.To explore the generalizability of the method, they conducted single-bead experiments with different P-V combinations, covering the four printing regimes on stainless steel SS316L, titanium alloy Ti-6AL-4V, and Inconel alloy IN718. They performed a cross-dataset evaluation, where the model was trained on the recorded videos of one alloy and tested on the videos while the hyperparameters were kept unchanged.Their findings that video vision transformers with temporal embedding can enable in situ detection of melt-pool defects with a simple off-axial imaging setup and generate process maps that can potentially accelerate the qualification of printability and process development for newly developed 3D printed alloys were recently published in <a href="https://www.nature.com/articles/s41467-024-44783-5" rel="noopener" target="_blank">Nature Communications</a>.

Researchers use AI and high-speed in-situ imaging to optimize process parameters for 3D printing metal alloys.

AI accelerates process design for 3D printing metal alloys

An AI-generated depiction of a modern day chipset

Gaining a comprehensive understanding of the distinct capabilities and applications of TPUs and GPUs is crucial for developers and researchers aiming to navigate the complex and rapidly changing terrain of artificial intelligence.

machine learning

Profiles