Large Language Models such as OpenAI’s ChatGPT have already transformed the way many of us go about some of our daily tasks. These generative artificial intelligence chatbots are trained with language — hundreds of terabytes of text ‘scraped’ from across the Internet and with billions of parameters.Looking ahead, many believe the ‘engines’ that drive generative artificial intelligence will be multimodal models that are not just trained on text but also can process various other modalities of information, including images, video, sound, and modalities from other domains such as biological or atmospheric data.Yet, until recently, training a single model to handle a wide range of modalities – inputs – and tasks – outputs – faced significant challenges. For example, the training often led to a reduction in performance compared to single-task models and typically required careful strategies to reduce quality losses and maximize accuracy. In addition, training one network on different modalities - or inputs - such as language, images or videos that vary greatly, presented additional complexities, and essential information in certain modalities was often incorrectly ignored by the model.<h2>Multimodal Modeling</h2>In a multi-year project undertaken with support from Apple in California, EPFL researchers from the <a href="https://vilab.epfl.ch/">Visual Intelligence and Learning Laboratory</a> (VILAB) in the <a href="https://www.epfl.ch/schools/ic/">School of Computer and Communication Sciences</a> (IC) have developed 4M, for Massively Masked Multimodal Modeling, one of the world’s most advanced single neural networks to handle a wide and varied range of tasks and modalities.In their latest<a href="https://4m.epfl.ch/#fourm21_paper">&nbsp;research paper</a> on 4M, presented in December at NeurIPS 2024, the Annual Conference on Neural Information Processing Systems, the researchers describe how it expands the capabilities of existing models in multiple ways (see box below for more technical details).“With 4M, we now have a rich model that can interpret more than just language. But why does this matter? One common criticism of LLMs is that their knowledge is not grounded because the training data is limited to only language,” explained Assistant Professor Amir Zamir, Head of VILAB.“When we advance to multimodal modeling, we don’t have to limit ourselves to language. We bring in other modalities, including sensors. For example, we can communicate an orange through the word ‘orange,’ just like in language models, but also through a collection of pixels, meaning how the orange looks, or through the sense of touch, capturing how touching an orange feels. If you assemble various modalities, you have a more complete encapsulation of the physical reality that we are trying to model,” he continued.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1736298018585-bc5ffbec.jpg" style="width: 100%px;" class="fr-fic fr-dib">Trying to model the physical reality by assembling various modalities: the image shows a couple of oranges seen through the lens of multiple modalities, with each slice showing a different way one might perceive and understand this scene. The modalities from left to right represent surface normals (the color represents surface orientation), depth (distance to the camera, red=near, blue=far), RGB (the original image), segmentation (distinct objects and image regions), and edges (object or texture boundaries). 2025 EPFL/Visual Intelligence and Learning Laboratory - CC-BY-SA 4.0<h2 id="isPasted">Towards an open-source, generic model for wide use</h2>Despite these impressive advances, Zamir says the development of 4M has presented some intriguing challenges, including the model not developing a truly unified representation across the modalities, and he has his own theory as to why.“We think that secretly, under the hood, the models cheat and create a little ensemble of independent models. One set of parameters solves one problem, another set of parameters solves another, and collectively, they appear to solve the overall problem. But they’re not truly unifying their knowledge in a way that enables a compact joint representation of the environment that would be a good portal to the world.”The VILAB team is continuing to work on building more structure and unification into 4M, with the goal of developing an open-source, generic architecture, enabling experts in other domains to adapt it to their specific needs, such as climate modeling or biomedical research. The team also works on addressing other important aspects, such as boosting the scalability even further and methods for the specialization of models to deployment contexts.“The whole point of open sourcing is that people can tailor the model for themselves with their own data and their own specifications. 4M is coming at the right moment in time, and we are especially enthusiastic about other domains adopting this line of modeling for their specific use cases. We are excited to see where that leads. But there are still a lot of challenges, and there is still a lot to do,” said Oguzhan Fatih Kar and Roman Bachmann, Doctoral Assistants in VILAB and co-authors of the paper.Based on the team’s experience developing 4M and the intriguing problems that they continue to work on, Zamir believes there are some interesting questions around the future development of foundation models.“As humans, we have five key senses, and on top of that, we efficiently learn language, which adds labels and structure to the knowledge that was already grounded in these other senses. It’s the opposite with the current AI – we have language models without sensory access to the world but that are trained using colossal data and compute resources. Our goal is to study the role of multimodality and efficiently develop a grounded world model that can be effectively utilized for downstream uses.”<hr>4M expands the capabilities of existing models across several key axes, including:<ul><li>Modalities: 4M enables new capabilities like predicting tens of modalities from tens of others, cross-modal retrieval, controllable generation, and strong out-of-the-box performance. It has convincingly shown that a single model can solve tens of diverse tasks without any loss in performance compared to dedicated single-task models and the state-of-the-art.</li><li>Diversity: 4M supports varied modalities and more structured data, such as human poses, SAM instances, and metadata for controllable generation.</li><li>Tokenization: 4M investigates discrete tokenization of diverse modalities such as global image embeddings, human poses, and semantics.</li><li>Scale: The public model has been scaled to 3 billion parameters and trained on over 500 billion tokens.</li><li>Co-Training: 4M demonstrates co-training on vision and language modeling simultaneously.</li></ul>

A couple of oranges seen through the lens of multiple modalities, with each slice showing a different way one might perceive and understand this scene - 2025 EPFL - CC-BY-SA 4.0

EPFL researchers have developed 4M, a next-generation, open-sourced framework for training versatile and scalable multimodal foundation models that go beyond language.

An open-source training framework to advance multimodal AI

By adapting artificial intelligence models known as large language models, researchers have made great progress in their ability to predict a protein’s structure from its sequence. However, this approach hasn’t been as successful for antibodies, in part because of the hypervariability seen in this type of protein.To overcome that limitation, MIT researchers have developed a computational technique that allows large language models to predict antibody structures more accurately. Their work could enable researchers to sift through millions of possible antibodies to identify those that could be used to treat SARS-CoV-2 and other infectious diseases.“Our method allows us to scale, whereas others do not, to the point where we can actually find a few needles in the haystack,” says Bonnie Berger, the Simons Professor of Mathematics, the head of the Computation and Biology group in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and one of the senior authors of the new study. “If we could help to stop drug companies from going into clinical trials with the wrong thing, it would really save a lot of money.”The technique, which focuses on modeling the hypervariable regions of antibodies, also holds potential for analyzing entire antibody repertoires from individual people. This could be useful for studying the immune response of people who are super responders to diseases such as HIV, to help figure out why their antibodies fend off the virus so effectively.Bryan Bryson, an associate professor of biological engineering at MIT and a member of the Ragon Institute of MGH, MIT, and Harvard, is also a senior author of the paper, which&nbsp;<a href="https://www.pnas.org/doi/10.1073/pnas.2418918121" target="_blank">appears this week in the&nbsp;Proceedings of the National Academy of Sciences</a>. Rohit Singh, a former CSAIL research scientist who is now an assistant professor of biostatistics and bioinformatics and cell biology at Duke University, and Chiho Im ’22 are the lead authors of the paper. Researchers from Sanofi and ETH Zurich also contributed to the research.<h2>Modeling hypervariability</h2>Proteins consist of long chains of amino acids, which can fold into an enormous number of possible structures. In recent years, predicting these structures has become much easier to do, using artificial intelligence programs such as AlphaFold. Many of these programs, such as ESMFold and OmegaFold, are based on large language models, which were originally developed to analyze vast amounts of text, allowing them to learn to predict the next word in a sequence. This same approach can work for protein sequences — by learning which protein structures are most likely to be formed from different patterns of amino acids.However, this technique doesn’t always work on antibodies, especially on a segment of the antibody known as the hypervariable region. Antibodies usually have a Y-shaped structure, and these hypervariable regions are located in the tips of the Y, where they detect and bind to foreign proteins, also known as antigens. The bottom part of the Y provides structural support and helps antibodies to interact with immune cells.Hypervariable regions vary in length but usually contain fewer than 40 amino acids. It has been estimated that the human immune system can produce up to 1 quintillion different antibodies by changing the sequence of these amino acids, helping to ensure that the body can respond to a huge variety of potential antigens. Those sequences aren’t evolutionarily constrained the same way that other protein sequences are, so it’s difficult for large language models to learn to predict their structures accurately.“Part of the reason why language models can predict protein structure well is that evolution constrains these sequences in ways in which the model can decipher what those constraints would have meant,” Singh says. “It’s similar to learning the rules of grammar by looking at the context of words in a sentence, allowing you to figure out what it means.”To model those hypervariable regions, the researchers created two modules that build on existing protein language models. One of these modules was trained on hypervariable sequences from about 3,000 antibody structures found in the Protein Data Bank (PDB), allowing it to learn which sequences tend to generate similar structures. The other module was trained on data that correlates about 3,700 antibody sequences to how strongly they bind three different antigens.The resulting computational model, known as AbMap, can predict antibody structures and binding strength based on their amino acid sequences. To demonstrate the usefulness of this model, the researchers used it to predict antibody structures that would strongly neutralize the spike protein of the SARS-CoV-2 virus.The researchers started with a set of antibodies that had been predicted to bind to this target, then generated millions of variants by changing the hypervariable regions. Their model was able to identify antibody structures that would be the most successful, much more accurately than traditional protein-structure models based on large language models.Then, the researchers took the additional step of clustering the antibodies into groups that had similar structures. They chose antibodies from each of these clusters to test experimentally, working with researchers at Sanofi. Those experiments found that 82 percent of these antibodies had better binding strength than the original antibodies that went into the model.Identifying a variety of good candidates early in the development process could help drug companies avoid spending a lot of money on testing candidates that end up failing later on, the researchers say.“They don’t want to put all their eggs in one basket,” Singh says. “They don’t want to say, I’m going to take this one antibody and take it through preclinical trials, and then it turns out to be toxic. They would rather have a set of good possibilities and move all of them through, so that they have some choices if one goes wrong.”<h2>Comparing antibodies</h2>Using this technique, researchers could also try to answer some longstanding questions about why different people respond to infection differently. For example, why do some people develop much more severe forms of Covid, and why do some people who are exposed to HIV never become infected?Scientists have been trying to answer those questions by performing single-cell RNA sequencing of immune cells from individuals and comparing them — a process known as antibody repertoire analysis. Previous work has shown that antibody repertoires from two different people may overlap as little as 10 percent. However, sequencing doesn’t offer as comprehensive a picture of antibody performance as structural information, because two antibodies that have different sequences may have similar structures and functions.The new model can help to solve that problem by quickly generating structures for all of the antibodies found in an individual. In this study, the researchers showed that when structure is taken into account, there is much more overlap between individuals than the 10 percent seen in sequence comparisons. They now plan to further investigate how these structures may contribute to the body’s overall immune response against a particular pathogen.“This is where a language model fits in very beautifully because it has the scalability of sequence-based analysis, but it approaches the accuracy of structure-based analysis,” Singh says.

A new computational technique allows large language models to predict antibody structures more accurately. Image: MIT News; iStock

Using this model, researchers may be able to identify antibody drugs that can target a variety of infectious diseases.

A new computational model can predict antibody structures more accurately

GPUs excel in parallel processing for graphics and AI training with scalability, while NPUs focus on low-latency AI inference on edge devices, enhancing privacy by processing data locally. Together, they complement each other in addressing different stages of AI workloads efficiently.

NPU vs GPU: Understanding the Key Differences and Use Cases

Machine-learning models can fail when they try to make predictions for individuals who were underrepresented in the datasets they were trained on.For instance, a model that predicts the best treatment option for someone with a chronic disease may be trained using a dataset that contains mostly male patients. That model might make incorrect predictions for female patients when deployed in a hospital.To improve outcomes, engineers can try balancing the training dataset by removing data points until all subgroups are represented equally. While dataset balancing is promising, it often requires removing large amount of data, hurting the model’s overall performance.MIT researchers developed a new technique that identifies and removes specific points in a training dataset that contribute most to a model’s failures on minority subgroups. By removing far fewer datapoints than other approaches, this technique maintains the overall accuracy of the model while improving its performance regarding underrepresented groups.In addition, the technique can identify hidden sources of bias in a training dataset that lacks labels. Unlabeled data are far more prevalent than labeled data for many applications.This method could also be combined with other approaches to improve the fairness of machine-learning models deployed in high-stakes situations. For example, it might someday help ensure underrepresented patients aren’t misdiagnosed due to a biased AI model.“Many other algorithms that try to address this issue assume each datapoint matters as much as every other datapoint. In this paper, we are showing that assumption is not true. There are specific points in our dataset that are contributing to this bias, and we can find those data points, remove them, and get better performance,” says Kimia Hamidieh, an electrical engineering and computer science (EECS) graduate student at MIT and co-lead author of a&nbsp;<a href="https://arxiv.org/pdf/2406.16846" target="_blank">paper on this technique</a>.She wrote the paper with co-lead authors Saachi Jain PhD ’24 and fellow EECS graduate student Kristian Georgiev; Andrew Ilyas MEng ’18, PhD ’23, a Stein Fellow at Stanford University; and senior authors Marzyeh Ghassemi, an associate professor in EECS and a member of the Institute of Medical Engineering Sciences and the Laboratory for Information and Decision Systems, and Aleksander Madry, the Cadence Design Systems Professor at MIT. The research will be presented at the Conference on Neural Information Processing Systems.<h2>Removing bad examples</h2>Often, machine-learning models are trained using huge datasets gathered from many sources across the internet. These datasets are far too large to be carefully curated by hand, so they may contain bad examples that hurt model performance.Scientists also know that some data points impact a model’s performance on certain downstream tasks more than others.The MIT researchers combined these two ideas into an approach that identifies and removes these problematic datapoints. They seek to solve a problem known as worst-group error, which occurs when a model underperforms on minority subgroups in a training dataset.The researchers’ new technique is driven by prior work in which they introduced a method, called&nbsp;<a href="https://arxiv.org/pdf/2303.14186">TRAK</a>, that identifies the most important training examples for a specific model output.For this new technique, they take incorrect predictions the model made about minority subgroups and use TRAK to identify which training examples contributed the most to that incorrect prediction.“By aggregating this information across bad test predictions in the right way, we are able to find the specific parts of the training that are driving worst-group accuracy down overall,” Ilyas explains.Then they remove those specific samples and retrain the model on the remaining data.Since having more data usually yields better overall performance, removing just the samples that drive worst-group failures maintains the model’s overall accuracy while boosting its performance on minority subgroups.<h2>A more accessible approach</h2>Across three machine-learning datasets, their method outperformed multiple techniques. In one instance, it boosted worst-group accuracy while removing about 20,000 fewer training samples than a conventional data balancing method. Their technique also achieved higher accuracy than methods that require making changes to the inner workings of a model.Because the MIT method involves changing a dataset instead, it would be easier for a practitioner to use and can be applied to many types of models.It can also be utilized when bias is unknown because subgroups in a training dataset are not labeled. By identifying datapoints that contribute most to a feature the model is learning, they can understand the variables it is using to make a prediction.“This is a tool anyone can use when they are training a machine-learning model. They can look at those datapoints and see whether they are aligned with the capability they are trying to teach the model,” says Hamidieh.Using the technique to detect unknown subgroup bias would require intuition about which groups to look for, so the researchers hope to validate it and explore it more fully through future human studies.They also want to improve the performance and reliability of their technique and ensure the method is accessible and easy-to-use for practitioners who could someday deploy it in real-world environments.“When you have tools that let you critically look at the data and figure out which datapoints are going to lead to bias or other undesirable behavior, it gives you a first step toward building models that are going to be more fair and more reliable,” Ilyas says.

MIT researchers developed an AI debiasing technique that improves the fairness of a machine-learning model by boosting its performance for subgroups that are underrepresented in its training data, while maintaining its overall accuracy. Credit: José-Luis Olivares, iStock

A new technique identifies and removes the training examples that contribute most to a machine-learning model’s failures.

Researchers reduce bias in AI models while preserving or improving accuracy

If someone advises you to “know your limits,” they’re likely suggesting you do things like exercise in moderation. To a robot, though, the motto represents learning constraints, or limitations of a specific task within the machine’s environment, to do chores safely and correctly.For instance, imagine asking a robot to clean your kitchen when it doesn’t understand the physics of its surroundings. How can the machine generate a practical multistep plan to ensure the room is spotless? Large language models (LLMs) can get them close, but if the model is only trained on text, it’s likely to miss out on key specifics about the robot’s physical constraints, like how far it can reach or whether there are nearby obstacles to avoid. Stick to LLMs alone, and you’re likely to end up cleaning pasta stains out of your floorboards.To guide robots in executing these open-ended tasks, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) used vision models to see what’s near the machine and model its constraints. The team’s strategy involves an LLM sketching up a plan that’s checked in a simulator to ensure it’s safe and realistic. If that sequence of actions is infeasible, the language model will generate a new plan, until it arrives at one that the robot can execute.This trial-and-error method, which the researchers call “Planning for Robots via Code for Continuous Constraint Satisfaction” (PRoC3S), tests long-horizon plans to ensure they satisfy all constraints, and enables a robot to perform such diverse tasks as writing individual letters, drawing a star, and sorting and placing blocks in different positions. In the future, PRoC3S could help robots complete more intricate chores in dynamic environments like houses, where they may be prompted to do a general chore composed of many steps (like “make me breakfast”).“LLMs and classical robotics systems like task and motion planners can’t execute these kinds of tasks on their own, but together, their synergy makes open-ended problem-solving possible,” says PhD student Nishanth Kumar SM ’24, co-lead author of a new paper about PRoC3S. “We’re creating a simulation on-the-fly of what’s around the robot and trying out many possible action plans. Vision models help us create a very realistic digital world that enables the robot to reason about feasible actions for each step of a long-horizon plan.”The team’s work was presented this past month in a paper shown at the Conference on Robot Learning (CoRL) in Munich, Germany.<iframe width="640" height="360" src="https://www.youtube.com/embed/hBq0EeFSsPo?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Teaching a robot its limits for open-ended chores. MIT CSAILThe researchers’ method uses an LLM pre-trained on text from across the internet. Before asking PRoC3S to do a task, the team provided their language model with a sample task (like drawing a square) that’s related to the target one (drawing a star). The sample task includes a description of the activity, a long-horizon plan, and relevant details about the robot’s environment.But how did these plans fare in practice? In simulations, PRoC3S successfully drew stars and letters eight out of 10 times each. It also could stack digital blocks in pyramids and lines, and place items with accuracy, like fruits on a plate. Across each of these digital demos, the CSAIL method completed the requested task more consistently than comparable approaches like <a href="https://arxiv.org/pdf/2403.11552">“LLM3”</a> and&nbsp;<a href="https://arxiv.org/pdf/2209.07753">“Code as Policies”</a>. The CSAIL engineers next brought their approach to the real world. Their method developed and executed plans on a robotic arm, teaching it to put blocks in straight lines. PRoC3S also enabled the machine to place blue and red blocks into matching bowls and move all objects near the center of a table.Kumar and co-lead author Aidan Curtis SM ’23, who’s also a PhD student working in CSAIL, say these findings indicate how an LLM can develop safer plans that humans can trust to work in practice. The researchers envision a home robot that can be given a more general request (like “bring me some chips”) and reliably figure out the specific steps needed to execute it. PRoC3S could help a robot test out plans in an identical digital environment to find a working course of action — and more importantly, bring you a tasty snack.For future work, the researchers aim to improve results using a more advanced physics simulator and to expand to more elaborate longer-horizon tasks via more scalable data-search techniques. Moreover, they plan to apply PRoC3S to mobile robots such as a quadruped for tasks that include walking and scanning surroundings.“Using foundation models like ChatGPT to control robot actions can lead to unsafe or incorrect behaviors due to hallucinations,” says The AI Institute researcher Eric Rosen, who isn’t involved in the research. “PRoC3S tackles this issue by leveraging foundation models for high-level task guidance, while employing AI techniques that explicitly reason about the world to ensure verifiably safe and correct actions. This combination of planning-based and data-driven approaches may be key to developing robots capable of understanding and reliably performing a broader range of tasks than currently possible.”Kumar and Curtis’ co-authors are also CSAIL affiliates: MIT undergraduate researcher Jing Cao and MIT Department of Electrical Engineering and Computer Science professors Leslie Pack Kaelbling and Tomás Lozano-Pérez. Their work was supported, in part, by the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research, the Army Research Office, MIT Quest for Intelligence, and The AI Institute.

PhD students Aidan Curtis (left) and Nishanth Kumar. To help robots execute open-ended tasks safely, the researchers used vision models to see what’s near the machine and model its constraints. Their “PRoC3S” strategy has an LLM sketch up an action plan that’s checked in a simulator to ensure it will work in the real world. Credits: Mike Grimmett/MIT CSAIL

The “PRoC3S” method helps an LLM create a viable action plan by testing each step in a simulation. This strategy could eventually aid in-home robots to complete more ambiguous chore requests.

Teaching a robot its limits, to complete open-ended tasks safely

Professional piano players spend countless hours at the keys, perfecting their craft. For people with smaller hands, this dedication can take a physical toll. Repeated stretching to reach distant keys in a chord can strain muscles and joints and may lead to tendonitis, carpal tunnel syndrome, and other injuries.Researchers at Stanford are working to understand the forces involved in the hand movements of elite-level piano players, and how those change with different hand sizes and keyboard widths. On Dec. 5 at SIGGRAPH Asia 2024, Stanford Engineering researchers presented an AI-trained model that can recreate the hand movements required to play complicated pieces of music and the physical stresses those hands are under. This is the first step in an effort to reduce the risk of long-term injuries in piano players and test solutions – such as narrower keyboards – that could make piano playing more equitable for everyone.“We would never expect a world-class athlete to compete with equipment that does not fit their body. Yet we ask pianists, particularly women, to adapt to a one-size-fits-all design that was never built with them in mind,” said<a href="https://profiles.stanford.edu/elizabeth-schumann" data-extlink="">&nbsp;Elizabeth Schumann</a>, the Billie Bennet Achilles Director of Keyboard Studies at the&nbsp;<a href="https://humsci.stanford.edu/" data-extlink="">School of Humanities and Sciences</a> and co-author on the paper. “We can use research like this to reshape the future of performance and make it more sustainable.”&nbsp;<h2>Capturing elite performance</h2>The modern piano keyboard was designed in the 19th century with the average European man in mind. Today, an estimated 87% of adult women and 24% of adult men have hands that are smaller than ideal for the standard piano. Using traditional methods to study the impacts of smaller hands would require following a cohort of piano players for decades, and it could still be difficult to quantify injury risks or test solutions. Instead, the researchers have recorded the hand movements of professional piano players and are using artificial intelligence to predict how different sizes of hands would move to play new music.Schumann and her colleagues recruited 15 elite-level pianists to play a total of 10 hours of music while cameras filmed their hands from every angle. The researchers couldn’t put any sensors on the players without potentially interfering with their performance, so they used advanced computer vision techniques to combine the videos and reconstruct the players’ hand motions in three dimensions. They used additional processing to make sure that those movements were perfectly synced with the audio.“The quality of data that we were able to achieve is unprecedented,” said&nbsp;<a href="https://profiles.stanford.edu/c-karen-liu" data-extlink="">Karen Liu</a>, a professor of computer science at Stanford and lead author on the paper. “It’s not just that the reconstructed 3D motions are accurate and clean; the quality of the performance itself is truly remarkable. We were able to capture diverse movements from a group of talented musicians performing at a high level – I have not seen a dataset like that in the past.”Roucheng Wang, a graduate student in Liu’s lab, and Pei Xu, a postdoctoral researcher, used this dataset to train a model to accurately generate new piano-playing data. When they gave the model sheet music it had never seen, such as Beethoven’s “Für Elise,” the model could provide the correct 3D hand movements needed to play the piece, using a simulated piano to generate sound.“I was just so stunned at how accurately this model could simulate elite-level technique,” Schumann said. “It’s really incredible.”<iframe width="640" height="360" src="https://www.youtube.com/embed/sGQNCEl7pJQ?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>When researchers gave the AI-trained piano playing model sheet music it had never seen, such as Beethoven’s “Für Elise,” the model could provide the correct 3D hand movements needed to play the piece. | Ruocheng Wang<h2 id="isPasted">From robot joints to real muscles</h2>This model is just the first step for Liu, Schumann, and their colleagues. At the moment, the model can physically simulate hand movements, but it doesn’t simulate the muscles and tendons that create those movements or the strain that they are subjected to.“We are not at the biomechanics level yet, where the actuation of the hands is generated by muscles – currently our model is more like a robot that directly generates torque at joints,” Liu said. “Eventually, our model needs to be able to predict the tension level of muscles and the potential of injury.”The researchers are currently working to add these layers to their model so that they can use it to help piano players – especially those with small hands – avoid injuries. They are also adapting their work to model the movements of other dexterous musicians. They have applied some of the same techniques to an existing dataset of hand motions of highly skilled guitar players. Their results, which were also presented recently at SIGGRAPH Asia 2024, showed that they could successfully model the distinct left- and right-hand movements needed to play a variety of both popular and classical music.“With high-quality data, we could model the 3D movements needed for other types of music performance as well,” Liu said. “This isn’t a model that is replacing people – we’re working with musicians to help understand and solve problems. This project is about advancing people, and AI is just a tool for that.”

Stanford researchers collected the first large-scale 3D hand motion dataset containing 10 hours and 153 pieces of piano music performed by 15 elite-level pianists, along with synchronized audio and key pressing events. | The Movement Lab

Researchers at Stanford Engineering have developed an AI-trained model to accurately recreate the hand movements of elite-level pianists and the physical stresses they endure while playing.

AI could help reduce injury risk in pianists

According to <a href="https://www.statista.com/outlook/tmo/artificial-intelligence/ai-robotics/worldwide" target="_blank" id="isPasted">Statista</a>, the global AI robotics market will reach $17 billion in 2024 and will keep growing by 24% annually. Today, robots deliver packages, conduct geological surveys, and entertain visitors of parks and exhibitions – however, that’s not the limit of their abilities. In this article, we get into the technologies at the intersection of AI and robotics, as well as the research done in this field at ITMO, with Prof. Ivan Borisov, a researcher at ITMO’s International Laboratory of Biomechatronics and Energy-Efficient Robotics (BE2R Lab).<h1 dir="ltr" id="isPasted">Why mix AI and robotics</h1>Robotics is an interdisciplinary field that brings together mechanics, electronics, and control systems. In order to plan a robot’s route and manage its movements, experts need to have a good command of mathematics and algorithms. Without correctly functioning optimization algorithms, a robot won’t be able to plan its movement trajectory and steps, especially if there are obstacles in its way.A control algorithm is based on an accurate mathematical model of a robot, which describes its state, geometry, and dynamics. Using this model, the algorithm calculates the robot’s movements and interactions with the environment. In other words, the more precise the mathematical model is, the more efficient the actual robot will be.To build an accurate mathematical model, it is essential to consider key geometric parameters (such as mass, inertia, friction, drive power, torque, as well as the performance metrics of drives and sensors). For a robot isolated from its environment, creating an accurate model is straightforward. However, the task becomes fundamentally more complex when the robot is required to physically interact with objects in its environment, especially when this environment is unknown.<blockquote>“We can, for example, build an accurate model of a manipulator robot welding a car. It has a fixed base and predetermined environment – we know what will happen to it. That’s why we can make accurate models of the robot itself and its surroundings, calculate the welding machine’s trajectory, and make the robot simply follow a precalculated trajectory. However, if we need a mobile robot, one that will walk or move on wheels in a city, or a kitchen manipulator robot, the conventional paradigm would require a model of its surroundings. The case is, though, that we cannot create an accurate model for an undetermined, a priori unknown, and constantly changing environment,” says Prof.&nbsp;Ivan Borisov, a researcher at ITMO’s International Laboratory of Biomechatronics and Energy-Efficient Robotics (BE2R Lab).</blockquote><iframe width="640" height="360" src="https://www.youtube.com/embed/TbyHKQKYVHg?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>AI technologies can be used to address the challenges of designing robots as systems. For this purpose, researchers turn to gradient and global optimization algorithms, as well as neural network approaches, including training on large datasets and reinforcement learning. Thanks to these methods, it’s possible to automate the design process (by delegating the filtering of solutions to algorithms), solve the task of motion policy synthesis, and enable robots to perceive their environment.<h1 dir="ltr">How AI is facilitating robot design</h1>One robotics task that AI is already helping to solve is the design of entire robots or their individual parts, such as mechanisms or drives.Previously, the quality of a design depended solely on the skills of its creator: if the same task was given to several engineers, they would all solve it differently. As a result, there would be various designs, each with hardly guaranteed optimality.These days, this problem is being addressed through the use of numerical and generative design. The main goal is to automate the design process and help specialists find optimal robot designs more quickly by exploring all possible solutions. Numerical design can be based on both optimization algorithms and neural network approaches, such as reinforcement learning or supervised learning. It helps identify the optimal parameters to enhance a robot’s performance, while generative design helps create mechanical structures and optimize them.Reinforcement learning was proven effective in the industry. For example, it was used by Disney Research in collaboration with ETH Zurich to <a href="https://la.disneyresearch.com/publication/design-and-control-of-a-bipedal-robotic-character/" target="_blank">merge</a> the animation of a droid from Star Wars and a physical model of a bipedal robot.&nbsp;Researchers from the BE2R Lab have <a href="https://news.itmo.ru/en/science/cyberphysics/news/13989/" target="_blank">created</a> the open-source repository <a href="https://github.com/aimclub/rostok?tab=readme-ov-file" target="_blank">Rostok</a>; with it, it’s possible to use generative design to create single-drive adaptive grabbing tools. Taking into account the parameters given by an engineer and the robot’s design limitations, the technology balances between them, presenting a greater number of various solutions. This makes the design process quicker (compared to human performance) and mathematically justified. Each solution is tested in a simulator and graded on performance. The higher the grade, the more likely the algorithm is to recommend a particular solution to engineers.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1733879540331-1399600.jpg" style="width: 100%px;" class="fr-fic fr-dib">Grabbing robot design. Photo courtesy of ITMO scientistsThe researchers used a similar approach to improve their previous <a href="https://news.itmo.ru/en/news/13232/" target="_blank">design</a> – a mobile jumping robot. Earlier, the robot relied on bearings, making its structure more rigid and less mobile; during a fall, part of the impact energy damaged its solid links. Using global optimization genetic algorithms, the scientists <a href="https://news.itmo.ru/en/science/cyberphysics/news/13925/" target="_blank">found out</a> that it would be better to use soft links instead of bearings. By absorbing all the energy from a fall, such links protect the robot’s structure and allow for partial recouping of energy from the impact. Thanks to this change, quadrupeds, anthropomorphic robots, and collaborative manipulators can be more energy-efficient, withstand shock loads, and adapt to uneven surfaces.&nbsp;Both solutions were revealed at the International Conference on Intelligent Robots and Systems (IROS 2024), a major conference that <a href="https://portal.core.edu.au/conf-ranks/?search=IROS&by=all&source=all&sort=dsource&page=1" target="_blank">holds the A rank</a> (one of the highest) in the CORE conference ranking.. In 2024, the event was held in Abu Dhabi on October 14-18.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1733879566211-1399601.jpg" style="width: 100%px;" class="fr-fic fr-dib">Jumping robot prototype. Photo courtesy of ITMO scientists<h1 dir="ltr" id="isPasted">AI for robotic vision</h1>Computer vision is a key element in robotic design, as it helps robots build a 3D map of their surroundings and plan routes around obstacles. However, without AI solutions to these tasks are too expensive, time-consuming, and inefficient.In the 1980s, Prof. Hans Moravec from Carnegie Mellon University explained why vision became one of the hardest “capacities” to acquire for robots. It’s summarized in a paradox subsequently named after him: contrary to popular assumption, higher cognitive processes (such as playing chess or go) require relatively little computational resources and are thus available to robots; at the same time lower-level sensorimotor operations (vision, perception, and body coordination) require immense computational resources.&nbsp;With AI, it’s possible to overcome Moravec’s paradox. For that, engineers first assemble a dataset (or take an existing one) wherein images are clearly marked by features and categories: for example, some are cats, and some are humans. To map out its surroundings and plan routes, robots use LiDARs, cameras, and other sensory technologies, while trained networks help them “make sense” of the surrounding objects. If a robot encounters something unknown along the way, it might try to classify this object using similar images from its dataset.AI can also help robots navigate complex environments with mirrors or glass doors. Their surfaces reflect or permit light through, which means that robots can misinterpret obstacles or distances to them.&nbsp;<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1733879591092-1399604.jpg" style="width: 100%px;" class="fr-fic fr-dib">Sergey Kolyubin with the team of ITMO's Faculty of Control Systems and Robotics. Photo by ITMO.NEWSHeaded by Sergey Kolyubin, researchers from ITMO’s BE2R Lab have&nbsp;<a href="https://ieeexplore.ieee.org/abstract/document/10342329" target="_blank" id="isPasted">developed</a> a 3D mapping system based on deep neural networks and the camera’s ego-motion (its movement relative to surrounding objects). Having tested the system in a space with glass partitions, the researchers concluded that it calculates distance to objects and creates 3D maps of spaces more accurately than other neural network-based solutions with RGB-D cameras that calculate distances to objects and their colors.

According to , the global AI robotics market will reach $17 billion in 2024 and will keep growing by 24% annually. Today, robots deliver packages, conduct geological surveys, and entertain visitors of parks and exhibitions – however, that’s not the limit of their abilities.

Higher, Faster, Stronger: How AI Makes Robots Smarter and More Efficient

<a href="https://scienceexchange.caltech.edu/topics/neuroscience/neuroscience-experts/brain-machine-interfaces-richard-andersen">Brain–machine interfaces</a> (BMIs) have enabled a handful of test participants who are unable to move or speak to communicate simply by thinking. An implanted device picks up the neural signals associated with a particular thought and converts them into control signals that are fed into a computer or a robotic limb. For example, a quadriplegic person is asked to think about moving a cursor on a computer screen. Once the BMI is trained to recognize the neurological activity as this intent, the person's thought is transmitted via the BMI to carry out the action of moving the cursor. BMIs currently in an experimental phase may also consist of robotic limbs that can execute manual tasks as instructed by a disabled person's thoughts alone.The hardware necessary for this incredible feat is a computer—either freestanding or within a robotic device—and an implant in the brain of the person who is using BMI technology to communicate their intent through their thoughts. At Caltech, researchers use implants that consist of arrays of 100 microelectrodes mounted on a 4x4 mm chip. The microelectrodes are typically 1.5 mm long and penetrate into the brain's cortex, where they can record the activity of individual neurons.Unfortunately, the performance of these microelectrode arrays is not consistent and degrades over time. To overcome this challenge, Caltech's&nbsp;<a href="https://www.ee.caltech.edu/people/azita">Azita Emami</a>, the Andrew and Peggy Cherng Professor of Electrical Engineering and Medical Engineering and director of the Center for Sensing to Intelligence (S2I), and her colleagues have used machine learning to effectively interpret the neuronal signals picked up by older implants."Not only do we observe day-to-day variations, but over time the performance of brain–computer interfaces degrade for a variety of reasons," Emami says. "There may be a small movement of the implant or its electrodes. The electrodes themselves may deteriorate or become encapsulated in brain tissue. Some people think that over time the neurons move away from the implant because they react to it as a foreign object in the brain. For whatever reason, the signals we receive become noisier."When a BMI is first set up, the microelectrode array produces a signal characterized by strong action potentials that appear like spikes in the recordings. Once this strong signal is no longer detected by the microelectrode array—that is, when the feedback from the array becomes noisier and neural spikes are no longer clearly detected—it is a far trickier task to link a pattern of neural activity from more distant neurons to a specific intent that can be successfully transmitted to a computer or other device. Researchers have tried to identify alternative signals, such as so-called threshold crossings or local field potentials recorded from distant neurons. One approach has been to use wavelets that measure small oscillations in neuronal activity. But the success of wavelets and other methods has been limited.Now Emami and her colleagues have found that by applying machine learning, BMIs can be trained to interpret data from neural activity even after the signal from an implant has become less clear.Benyamin Haghi, formerly a graduate student in the Emami lab explains: "Where before we relied on counting neural spikes, we have now created a neural network that automatically extracts information from the entire neural signal, from all the little dips and picks and changes in the signal, and converts this into the intent of the patient." Emami adds, "Over time the BMI has been trained on both a signal that is neural activity and a signal that looks like noise, and is therefore able to interpret the user's intent."Emami describes the experience of one participant, JJ, who lost mobility due to a vehicle accident. "When we first started working with him, his implant was three years old, and it had already degraded. We were thinking of removing the implant, but with our new algorithm, he's happy with continuing to use the system he already has. JJ can move a cursor very precisely on a grid, just as he did when the implant was new. He can play video games and control a computer environment that mimics driving."The team's algorithm is called FENet, for Feature Extraction Network. Remarkably, it can be trained on data from one patient and then used successfully in another. "This means that there is some fundamental type of information in the neural data that we are picking up," Emami says. Not only that, FENet can generalize across different brain regions and types of electrodes and be easily incorporated into existing BMIs.Richard Andersen, the James G. Boswell Professor of Neuroscience and leadership chair and director of the T&amp;C Chen Brain-Machine Interface Center, says "FENet has already extended our clinical study with JJ by two years. BMI research is a perfect field for interdisciplinary research, in this case melding the disciplines of engineering, computer science, and neuroscience."Today's BMIs require a manual connection from the electrode implant to a connector that then links with wires that lead to a microsystem that processes the raw data before sending it to a computer—the visual interface the patient manipulates. "This is quite a cumbersome system," Emami says. "Our goal now, after the creation of FENet to better interpret brain signals, is to miniaturize the system so that one day it can be a wearable or an implant that communicates wirelessly with the computer."<hr>This research was published in Nature Biomedical Engineering under the title "<a href="https://rdcu.be/d2D2L">Enhanced control of a brain-machine interface by tetraplegic participants via neural-network-mediated feature extraction</a>." Co-authors include Andersen; Emami; Haghi; Albert Yan Huang from the Emami lab; members of professional staff Tyson Aflalo and Spencer Kellis, postdoc Jorge A. Gamez de Leon, and graduate student Charles Guan from the Andersen lab; and Nader Pouratian from UCLA Neurosurgery. Funding was provided by the National Institutes of Health, Caltech's S2I, the T&amp;C Chen Brain-Machine Interface Center at Caltech, the Boswell Foundation, the Braun Foundation, and the Heritage Medical Research Institute.

Brain–machine interfaces (BMIs) have enabled a handful of test participants who are unable to move or speak to communicate simply by thinking.

Improving Brain-Machine Interfaces with Machine Learning

In 2018, Google DeepMind's AlphaZero program taught itself the games of chess, shogi, and Go using machine learning and a special algorithm to determine the best moves to win a game within a defined grid. Now, a team of Caltech researchers has developed an analogous algorithm for autonomous robots—a planning and decision-making control system that helps freely moving robots determine the best movements to make as they navigate the real world."Our algorithm actually strategizes and then explores all the possible and important motions and chooses the best one through dynamic simulation, like playing many simulated games involving moving robots," says&nbsp;<a href="https://www.eas.caltech.edu/people/sjchung">Soon-Jo Chung</a>, Caltech's Bren Professor of Control and Dynamical Systems and a senior research scientist at JPL, which Caltech manages for NASA. "The breakthrough innovation here is that we have derived a very efficient way of finding that optimal safe motion that typical optimization-based methods would never find."The team describes the technique, which they call Spectral Expansion Tree Search (SETS), in the December&nbsp;<a href="https://www.science.org/doi/10.1126/scirobotics.ado1010">cover article</a> of the journal&nbsp;Science Robotics.Many robots can move quite freely and in any direction. Consider, for example, a humanoid robot designed to assist an elderly person in a home. Such a robot should be able to move in many different ways and, essentially, in any direction within the space as it encounters obstacles or unexpected events while completing its tasks. That robot's set of movements, obstacles, and challenges will be very different from those of a self-driving car, for example.How, then, can a single algorithm guide different robotic systems to make the best decisions to move through their surroundings?"You don't want a designer to have to go in and handcraft these motions and say, 'This is the discrete set of moves the robot should be able to do,'" says John Lathrop, a graduate student in control and dynamical systems at Caltech and co-lead author of the new paper. "To overcome this, we came up with SETS."SETS uses control theory and linear algebra to find natural motions that use a robotic platform's capabilities to its fullest extent in a physical setting.The basic underlying concept is based on a Monte Carlo Tree Search, a decision-making algorithm also used by Google's AlphaZero. Here, Monte Carlo essentially means something random, and tree search refers to navigating a branching structure that represents the relationships of data in a system. In such a tree, a root branches off to so-called child nodes that are connected by edges. Using Monte Carlo Tree Search for a game like Go, possible moves are represented as new nodes, and the tree grows larger as more random samples of possible trajectories are attempted. The algorithm plays out the possible moves to see the final outcomes of the different nodes and then selects the one that offers the best outcome based on a point valuation.The problem, Lathrop explains, is that when using this branching tree structure for continuous dynamical systems such as robots operating in the physical world, the total number of trajectories in the tree grows exponentially. "For some problems, trying to simulate every single possibility and then figure out which one is best would take years, maybe hundreds of years," he says.To overcome this, SETS takes advantage of an exploration/exploitation trade-off. "We want to try simulating trajectories that we haven't investigated before—that's exploration," Lathrop says. "And we want to continue looking down paths that have previously yielded high reward—that's exploitation. By balancing the exploration and the exploitation, the algorithm is able to quickly converge on the optimal solution among all possible trajectories."For example, if a robot starts out calculating a couple of possible actions that it determines would cause it to smash into a wall, there is no need for it to investigate any of the other nodes on that branch of the tree."This exploration/exploitation tradeoff and search over the robot's natural motions enables our robots to think, move, and adapt to new information in real-time," says Benjamin Rivière (PhD '24), a postdoctoral scholar research associate in mechanical and civil engineering at Caltech and co-lead author of the paper.SETS can run an entire tree search in about a tenth of a second. During that time, it can simulate thousands to tens of thousands of possible trajectories, select the best one, and then act. The loop continues over and over, giving the robotic system the ability to make many decisions each second.A key feature of the SETS algorithm is that it can be applied to essentially any robotic platform. The features and capabilities do not have to be programmed individually. In the new paper, Chung and his colleagues demonstrate the algorithm's successful utility in three completely different experimental settings—something that is very rare in robotics papers.In the first, a quadrotor drone was able to observe four hovering white balls while avoiding four orange balls, all while navigating an airfield rife with randomly occurring, dangerous air currents, or thermals. The drone experiment was conducted at Caltech's&nbsp;<a href="https://cast.caltech.edu/">Center for Autonomous Systems and Technologies</a> (CAST). In the second, the algorithm augmented a human driver of a tracked ground vehicle to navigate a narrow and winding track without hitting the siderails. And in the final setup, SETS helped a pair of tethered spacecraft capture and redirect a third agent, which could represent another spacecraft, an asteroid or another object.A team of Caltech students and researchers are currently applying a version of the SETS algorithm to an Indy car that will participate in the Indy Autonomous Challenge at the Consumer Electronics Show (CES) in Las Vegas on January 9.The work was supported by the Defense Advanced Research Projects Agency's Learning Introspective Control (LINC) program, the Aerospace Corporation, and Supernal, and is in part based on work supported by the National Science Foundation Graduate Research Fellowship Program.

A team of Caltech researchers has developed an analogous algorithm for autonomous robots—a planning and decision-making control system that helps freely moving robots determine the best movements to make as they navigate the real world.

Helping Robots Make Good Decisions in Real Time

In my last blog on AI and Machine Learning (ML) I emphasized that AI and ML would transform the IoT, not in 10 or even five years from now, but on product roadmaps coming to market within the next couple of years. I’d like to ‘up’ the urgency on that prediction. AI and ML aren’t a couple of years away, they are here. Not a tool of the future, but a dynamic force driving innovation and change now. &nbsp;Any question about whether there is going to be a big role for AI and ‘Edge AI’—where AI and ML is performed locally rather than sending data up to the Cloud—in the IoT has been dispelled.&nbsp;<h2>The ML and TinyML difference for Edge AI</h2>When we talk about AI in the IoT, I want to make it clear that, for the foreseeable future, we are really talking about ML and the stripped-down TinyML variant.&nbsp;ML adopts a model-based approach that relies on learning, trial and error from captured data. There’s a learning component (‘training’) that happens where there is an abundance of compute power like in the Cloud, and an on-the-fly predictive component (‘inferencing’) that in many cases can be so lightweight that it can run on small, embedded devices.&nbsp;As data is gathered, one can continuously re-do the training part and update the predictive part to get more accurate. As such, any attached sensor will continuously get better at monitoring what it is supposed to be monitoring.&nbsp;AI in the IoT also needs to be performed as close to the sensors as possible, all the way to the edge, on increasingly capable devices in terms of compute power. But devices that will still, by and large, be battery powered. And so much more resource-constrained than if they were plugged into the mains power grid.&nbsp;&nbsp;<h2>Three types of AI learning for the IoT&nbsp;</h2>We are also primarily talking about three types of IoT AI: supervised learning, reinforcement learning, and unsupervised learning. And the three main uses cases are ‘three Vs’ as Arm coined the term: vibration (anomaly detection and movement classification), voice (using audio recognition as a sensor), and vision (using image recognition as a sensor).&nbsp;The only difference being how much the ML model can learn on its own: running from very little (supervised) all the way through to autonomous learning (unsupervised) when given the right model to start with. Right now, ML can be run locally at the edge on IoT platform devices like Nordic’s latest&nbsp;<a href="https://www.nordicsemi.com/Products/nRF54H20">nRF54 Series</a>. You can optimize ML on a constrained device by using ML accelerators, achieving performance that can rival much more powerful devices and far outperform traditional MCUs. Once you have trained the basic model, which will typically take a day on a desktop PC, the inferencing side of things can accelerate hugely using AI-optimized algorithms.&nbsp;Nordic’s nRF54 Series was built for running optimized ML on constrained edge devices in just this way. It is around 25 percent more powerful than market-leading microprocessors and consumes 20 percent less energy than the lowest energy microprocessors in AI and ML edge computing applications.&nbsp;This means Nordic has enabled the advantages of edge computing, AI, and ML to be accessible to the broadest range of customers and applications.&nbsp;<h2>The edge computing,&nbsp;AI&nbsp;and ML advantage&nbsp;</h2>There are five primary advantages to customers. The first is latency. If you need real-time or near real-time responsiveness you must compute and take decisions locally. This also enables your AI and ML to perform quicker decision making.&nbsp;&nbsp;The next benefit is bandwidth. Edge computing reduces the reliance of having to maintain continuous network or Cloud connectivity that incur huge data overheads and data costs.&nbsp;The third benefit is privacy. Transmitting sensitive or personal data to the Cloud raises privacy concerns. Local edge processing and storage means better control over data confidentiality. This will help minimize the potential for security breaches.&nbsp;The fourth benefit of operating at the edge is cost. Cloud-based AI is significantly more expensive because it costs money to transmit a lot of data to and from the Cloud, and then pay for all that Cloud computing capability on top. Processing at the edge, in comparison, is free of such recurring operating costs.&nbsp;The fifth, and perhaps most important benefit, is energy efficiency. The world simply does not have enough resources for everything to be run in the Cloud. Smarter edge devices minimize power consumption, making them more environmentally friendly and beneficial for combating climate change.&nbsp;&nbsp;<h2>Ultra-low power connectivity leadership</h2>No company in the world does ultra-low power connectivity like Nordic. And no SoC does energy-efficient edge AI computing like the Nordic nRF54 Series. The latest advances in AI and ML acceleration that&nbsp;<a href="https://www.nordicsemi.com/Nordic-news/2023/08/Nordic-to-acquire-AI-ML-technology-in-the-US">Nordic’s Atlazo acquisition</a>&nbsp;has brought to our internal R&amp;D capabilities will only further extend our leadership in edge AI and ML.&nbsp;In my view, this development signifies a pivotal transformation for Nordic, elevating its identity far above being just a wireless connectivity or Bluetooth company within the IoT landscape, but also offering a seamless integration platform for AI in the IoT.&nbsp;

AI in the IoT also needs to be performed as close to the sensors as possible, all the way to the edge, on increasingly capable devices in terms of compute power. 

AI and machine learning goes mainstream

While generative artificial intelligence models for images can streamline artistic processes by enabling creators to produce lifelike 2D images from text prompts, these models are not designed to generate 3D shapes. To bridge the gap, a recently developed technique called&nbsp;<a href="https://dreamfusion3d.github.io/">Score Distillation</a> leverages 2D image generation models to create 3D shapes, but its output often ends up blurry or cartoonish.MIT researchers explored the relationships and differences between the algorithms used to generate 2D images and 3D shapes, identifying the root cause of lower-quality 3D models. From there, they crafted a simple fix to Score Distillation, which enables the generation of sharp, high-quality 3D shapes that are closer in quality to the best model-generated 2D images.Some other methods try to fix this problem by retraining or fine-tuning the generative AI model, which can be expensive and time-consuming.By contrast, the MIT researchers’ technique achieves 3D shape quality on par with or better than these approaches without additional training or complex postprocessing. Moreover, by identifying the cause of the problem, the researchers have improved mathematical understanding of Score Distillation and related techniques, enabling future work to further improve performance.“Now we know where we should be heading, which allows us to find more efficient solutions that are faster and higher-quality,” says Artem Lukoianov, an electrical engineering and computer science (EECS) graduate student who is lead author of a paper on this technique. “In the long run, our work can help facilitate the process to be a co-pilot for designers, making it easier to create more realistic 3D shapes.”Lukoianov’s co-authors are Haitz Sáez de Ocáriz Borde, a graduate student at Oxford University; Kristjan Greenewald, a research scientist in the MIT-IBM Watson AI Lab; Vitor Campagnolo Guizilini, a scientist at the Toyota Research Institute; Timur Bagautdinov, a research scientist at Meta; and senior authors Vincent Sitzmann, an assistant professor of EECS at MIT who leads the Scene Representation Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and Justin Solomon, an associate professor of EECS and leader of the CSAIL Geometric Data Processing Group. The research will be presented at the Conference on Neural Information Processing Systems.<h2>From 2D images to 3D shapes</h2>Diffusion models, such as DALL-E, are a type of generative AI model that can produce lifelike images from random noise. To train these models, researchers add noise to images and then teach the model to reverse the process and remove the noise. The models use this learned “denoising” process to create images based on a user’s text prompts.But diffusion models underperform at directly generating realistic 3D shapes because there are not enough 3D data to train them. To get around this problem, researchers developed a technique called <a href="https://dreamfusion3d.github.io/">Score Distillation Sampling</a> (SDS) in 2022 that uses a pretrained diffusion model to combine 2D images into a 3D representation.The technique involves starting with a random 3D representation, rendering a 2D view of a desired object from a random camera angle, adding noise to that image, denoising it with a diffusion model, then optimizing the random 3D representation so it matches the denoised image. These steps are repeated until the desired 3D object is generated.However, 3D shapes produced this way tend to look blurry or oversaturated.“This has been a bottleneck for a while. We know the underlying model is capable of doing better, but people didn’t know why this is happening with 3D shapes,” Lukoianov says.The MIT researchers explored the steps of SDS and identified a mismatch between a formula that forms a key part of the process and its counterpart in 2D diffusion models. The formula tells the model how to update the random representation by adding and removing noise, one step at a time, to make it look more like the desired image.Since part of this formula involves an equation that is too complex to be solved efficiently, SDS replaces it with randomly sampled noise at each step. The MIT researchers found that this noise leads to blurry or cartoonish 3D shapes.<h2>An approximate answer</h2>Instead of trying to solve this cumbersome formula precisely, the researchers tested approximation techniques until they identified the best one. Rather than randomly sampling the noise term, their approximation technique infers the missing term from the current 3D shape rendering.“By doing this, as the analysis in the paper predicts, it generates 3D shapes that look sharp and realistic,” he says.In addition, the researchers increased the resolution of the image rendering and adjusted some model parameters to further boost 3D shape quality.In the end, they were able to use an off-the-shelf, pretrained image diffusion model to create smooth, realistic-looking 3D shapes without the need for costly retraining. The 3D objects are similarly sharp to those produced using other methods that rely on ad hoc solutions.“Trying to blindly experiment with different parameters, sometimes it works and sometimes it doesn’t, but you don’t know why. We know this is the equation we need to solve. Now, this allows us to think of more efficient ways to solve it,” he says.Because their method relies on a pretrained diffusion model, it inherits the biases and shortcomings of that model, making it prone to hallucinations and other failures. Improving the underlying diffusion model would enhance their process.In addition to studying the formula to see how they could solve it more effectively, the researchers are interested in exploring how these insights could improve image editing techniques.This work is funded, in part, by the Toyota Research Institute, the U.S. National Science Foundation, the Singapore Defense Science and Technology Agency, the U.S. Intelligence Advanced Research Projects Activity, the Amazon Science Hub, IBM, the U.S. Army Research Office, the CSAIL Future of Data program, the Wistron Corporation, and the MIT-IBM Watson AI Laboratory.

Creating realistic 3D models for applications like virtual reality, filmmaking, and engineering design can be a cumbersome process requiring lots of manual trial and error.

A new way to create realistic 3D shapes using generative AI

The deep neural network models that power today’s most demanding machine-learning applications have grown so large and complex that they are pushing the limits of traditional electronic computing hardware.Photonic hardware, which can perform machine-learning computations with light, offers a faster and more energy-efficient alternative. However, there are some types of neural network computations that a photonic device can’t perform, requiring the use of off-chip electronics or other techniques that hamper speed and efficiency.Building on a decade of research, scientists from MIT and elsewhere have developed a new photonic chip that overcomes these roadblocks. They demonstrated a fully integrated photonic processor that can perform all the key computations of a deep neural network optically on the chip.The optical device was able to complete the key computations for a machine-learning classification task in less than half a nanosecond while achieving more than 92 percent accuracy — performance that is on par with traditional hardware.The chip, composed of interconnected modules that form an optical neural network, is fabricated using commercial foundry processes, which could enable the scaling of the technology and its integration into electronics.In the long run, the photonic processor could lead to faster and more energy-efficient deep learning for computationally demanding applications like lidar, scientific research in astronomy and particle physics, or high-speed telecommunications.“There are a lot of cases where how well the model performs isn’t the only thing that matters, but also how fast you can get an answer. Now that we have an end-to-end system that can run a neural network in optics, at a nanosecond time scale, we can start thinking at a higher level about applications and algorithms,” says Saumil Bandyopadhyay ’17, MEng ’18, PhD ’23, a visiting scientist in the Quantum Photonics and AI Group within the Research Laboratory of Electronics (RLE) and a postdoc at NTT Research, Inc., who is the lead author of a paper on the new chip.Bandyopadhyay is joined on the paper by Alexander Sludds ’18, MEng ’19, PhD ’23; Nicholas Harris PhD ’17; Darius Bunandar PhD ’19; Stefan Krastanov, a former RLE research scientist who is now an assistant professor at the University of Massachusetts at Amherst; Ryan Hamerly, a visiting scientist at RLE and senior scientist at NTT Research; Matthew Streshinsky, a former silicon photonics lead at Nokia who is now co-founder and CEO of Enosemi; Michael Hochberg, president of Periplous, LLC; and Dirk Englund, a professor in the Department of Electrical Engineering and Computer Science, principal investigator of the Quantum Photonics and Artificial Intelligence Group and of RLE, and senior author of the paper. The research&nbsp;<a href="https://www.nature.com/articles/s41566-024-01567-z" target="_blank">appears today in&nbsp;Nature Photonics</a>.<h2>Machine learning with light</h2>Deep neural networks are composed of many interconnected layers of nodes, or neurons, that operate on input data to produce an output. One key operation in a deep neural network involves the use of linear algebra to perform matrix multiplication, which transforms data as it is passed from layer to layer.But in addition to these linear operations, deep neural networks perform nonlinear operations that help the model learn more intricate patterns. Nonlinear operations, like activation functions, give deep neural networks the power to solve complex problems.In 2017, Englund’s group, along with researchers in the lab of Marin Soljačić, the Cecil and Ida Green Professor of Physics,&nbsp;<a href="https://news.mit.edu/2017/new-system-allows-optical-deep-learning-0612" target="_blank">demonstrated an optical neural network on a single photonic chip</a> that could perform matrix multiplication with light.But at the time, the device couldn’t perform nonlinear operations on the chip. Optical data had to be converted into electrical signals and sent to a digital processor to perform nonlinear operations.“Nonlinearity in optics is quite challenging because photons don’t interact with each other very easily. That makes it very power consuming to trigger optical nonlinearities, so it becomes challenging to build a system that can do it in a scalable way,” Bandyopadhyay explains.They overcame that challenge by designing devices called nonlinear optical function units (NOFUs), which combine electronics and optics to implement nonlinear operations on the chip.The researchers built an optical deep neural network on a photonic chip using three layers of devices that perform linear and nonlinear operations.<h2>A fully-integrated network</h2>At the outset, their system encodes the parameters of a deep neural network into light. Then, an array of programmable beamsplitters, which was&nbsp;demonstrated&nbsp;in the 2017 paper, performs matrix multiplication on those inputs.The data then pass to programmable NOFUs, which implement nonlinear functions by siphoning off a small amount of light to photodiodes that convert optical signals to electric current. This process, which eliminates the need for an external amplifier, consumes very little energy.“We stay in the optical domain the whole time, until the end when we want to read out the answer. This enables us to achieve ultra-low latency,” Bandyopadhyay says.Achieving such low latency enabled them to efficiently train a deep neural network on the chip, a process known as in situ&nbsp;training that typically consumes a huge amount of energy in digital hardware.“This is especially useful for systems where you are doing in-domain processing of optical signals, like navigation or telecommunications, but also in systems that you want&nbsp;to learn in real time,” he says.The photonic system achieved more than 96 percent accuracy during training tests and more than 92 percent accuracy during inference, which is comparable to traditional hardware. In addition, the chip performs key computations in less than half a nanosecond. &nbsp; &nbsp;&nbsp;“This work demonstrates that computing — at its essence, the mapping of inputs to outputs — can be compiled onto new architectures of linear and nonlinear physics that enable a fundamentally different scaling law of computation versus effort needed,” says Englund.The entire circuit was fabricated using the same infrastructure and foundry processes that produce CMOS computer chips. This could enable the chip to be manufactured at scale, using tried-and-true techniques that introduce very little error into the fabrication process.Scaling up their device and integrating it with real-world electronics like cameras or telecommunications systems will be a major focus of future work, Bandyopadhyay says. In addition, the researchers want to explore algorithms that can leverage the advantages of optics to train systems faster and with better energy efficiency.

Researchers demonstrated a fully integrated photonic processor that can perform all key computations of a deep neural network optically on the chip, which could enable faster and more energy-efficient deep learning for computationally demanding applications like lidar or high-speed telecommunications. Image: Sampson Wilcox, Research Laboratory of Electronics.

This new device uses light to perform the key operations of a deep neural network on a chip, opening the door to high-speed processors that can learn in real-time.

Photonic processor could enable ultrafast AI computations with extreme energy efficiency

ChatGPT exploded onto the public scene in late 2022 attracting more than 100-million users in just its first month. Since then, there have been rising examples of how AI may transform society in the coming years, from employment and communication to education.In higher education, AI assistants are being increasingly used by students. While these tools provide opportunities for improved teaching and education, they also pose significant challenges for assessment and learning outcomes yet, until now, there has been no comprehensive study of their potential impact on the assessment methods that educational institutions use.As outlined in their&nbsp;<a href="https://www.pnas.org/doi/full/10.1073/pnas.2414955121" target="_blank" rel="noopener">new paper published in the Proceedings of the National Academy of Sciences</a> (PNAS), researchers from EPFL’s&nbsp;<a href="https://www.epfl.ch/schools/ic/">School of Computer and Communication Sciences</a> have conducted a large-scale study across 50 EPFL courses to measure the current performance of Large Language Models on higher education course assessments. The selected courses were sampled from 9 Bachelor’s, Master’s, and online programs, covering a broad spectrum of STEM disciplines, including computer science, mathematics, biology, chemistry, physics, and material sciences."We were lucky that a large consortium of EPFL professors, teachers, and teaching assistants helped us collect the largest data set to date of course materials, assessments, and exams to get a diverse array of materials across our degree programs,” explained Assistant Professor Antoine Bosselut, Head of the&nbsp;<a href="https://nlp.epfl.ch/">Natural Language Processing Laboratory</a> (NLP) and member of the&nbsp;<a href="https://ai.epfl.ch/">EPFL AI Center</a>. “This data was curated into a format that we thought would most resemble the ways that students would actually give this information to models and then we generated responses from the models and saw how well they answered.”Focused on GPT-3.5 and GPT-4, the researchers used eight prompting strategies to produce responses and found that GPT-4 answers an average of 65.8% of questions correctly, and can even produce the correct answer across at least one prompting strategy for 85.1% of questions.“We were surprised at the results, nobody expected that the AI assistants would achieve such a high percentage of correct answers across so many courses. Importantly, the 65% of questions answered correctly was achieved using the most basic, no knowledge prompt strategy, so anyone, without understanding anything technically could achieve this. With some subject knowledge, which is typical, it was possible to achieve an 85% success rate and that was really the shock,” said Anna Sotnikova, a scientist with the NPL and co-author on the paper.<blockquote>We were surprised at the results, nobody expected that the AI assistants would achieve such a high percentage of correct answers across so many courses.<footer>Anna Sotnikova, a scientist at the Natural Language Processing Laboratory</footer></blockquote><h2>AI's&nbsp;impact on&nbsp;student&nbsp;learning and&nbsp;skill&nbsp;development</h2>The researchers theoretically grounded the problems associated with these AI systems being used by students around vulnerability – on the one hand, assessment vulnerability or whether traditionally used assessments can be ‘gamed’ by these systems and on the other hand, educational vulnerability, that is, could these systems be used to circumvent the typical cognitive paths that students take to learn the academic skills they need.In this context, the researchers believe that the results of the study bring up clear questions around how to ensure that students are able to learn the core concepts needed in order to grasp more complex topics later on.“The fear is that if these models are as capable as what we indicate, students that use them might shortcut the process through which they would learn new concepts. This could build weaker foundations for certain skills earlier on, making it harder to learn more complex concepts later. Maybe this needs to be a debate about what we should be teaching in the first place in order to come up with the best synergies of the technologies we have and what students will do in decades to come,” said Bosselut.Another key point of the development of AI assistants is that they are not going to get worse, they will only get better. In this research, which was completed a year ago, one single model was used for all subjects and, for example, it had particular trouble with mathematics questions. Now there are specific models for math. The conclusion, the researchers say, is that if the study was rerun today, the numbers would be even higher.<h2>Emphasizing&nbsp;complex&nbsp;assessments and&nbsp;adapting education</h2>“In the short term we should push for harder assessments - not in the sense of question difficulty, but in the sense of the complexity of the assessment itself where multiple skills have to be taken from different concepts that are learned throughout the course during the semester and that are brought together in a holistic assessment,” Bosselut suggested. “The models aren’t yet really designed to plan and work in this kind of way and in the end, we actually think this project-based learning is better for students anyway.”“AI challenges higher education institutions in many ways, for example: which new skills are required for future graduates, which are becoming obsolete, how can we provide feedback at scale and how do we measure knowledge? These kinds of questions come up in almost every management meeting at EPFL and what matters most is that our teams initiate projects which provide evidence-based answers to as many of them as we can,” said Pierre Dillenbourg, Vice President for Academic Affairs at EPFL.<blockquote>AI challenges higher education institutions in many ways.<footer>Pierre Dillenbourg, Vice President for Academic Affairs at EPFL</footer></blockquote>Longer-term it’s clear that education systems will need to adapt and the researchers want to bring this ongoing project closer towards educators, aligning studies and then recommendations with what they will find useful.“This is only the beginning and I think a good analogy to LLMs right now are calculators when they were introduced, there were a similar set of concerns that children would no longer learn mathematics. Now, in the earlier phases of education calculators are usually not allowed but by high school and above, they are expected, taking care of lower level work while students are learning more advanced skills that rely upon them,” added Beatriz Borges, a PhD student in the NLP and co-author of the paper.“I think that we will see a similar, gradual adaptation and shift to an understanding of both what these systems can do for us and what we can't rely on them to do. Ultimately, we include practical suggestions to better support students, teachers, administrators, and everyone else during this transition, while also while also helping to reduce some of the risks and vulnerabilities outlined in the paper,” she concluded.The&nbsp;<a href="https://ai.epfl.ch/">EPFL AI Center</a>&nbsp;brings together 80 laboratories and professors and has 1000 affiliates, leading&nbsp;the way towards trustworthy, accessible, and inclusive AI.

EPFL research investigating the potential impact on education of AI assistants has found that systems like GPT-4 can answer up to 85% of university assessment questions correctly.

Could ChatGPT get an engineering degree?

Visualizing the potential impacts of a hurricane on people’s homes before it hits can help residents prepare and decide whether to evacuate.MIT scientists have developed a method that generates satellite imagery from the future to depict how a region would look after a potential flooding event. The method combines a generative artificial intelligence model with a physics-based flood model to create realistic, birds-eye-view images of a region, showing where flooding is likely to occur given the strength of an oncoming storm.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732838043094-MIT-AI-Flood-01-press_0.jpg" style="width: 832px;" class="fr-fic fr-dib">Caption:A generative AI model visualizes how floods in Texas would look like in satellite imagery. The original photo is on the left, and the AI generated image is in on the right.&nbsp; Credit: Pre-flood images from Maxar Open Data Program via Gupta et al., CVPR Workshop Proceedings. Generated images from Lütjen et al., IEEE TGRS.As a test case, the team applied the method to Houston and generated satellite images depicting what certain locations around the city would look like after a storm comparable to Hurricane Harvey, which hit the region in 2017. The team compared these generated images with actual satellite images taken of the same regions after Harvey hit. They also compared AI-generated images that did not include a physics-based flood model.The team’s physics-reinforced method generated satellite images of future flooding that were more realistic and accurate. The AI-only method, in contrast, generated images of flooding in places where flooding is not physically possible.The team’s method is a proof-of-concept, meant to demonstrate a case in which generative AI models can generate realistic, trustworthy content when paired with a physics-based model.&nbsp;In order to apply the method to other regions to depict flooding from future storms, it will need to be trained on many more satellite images to learn how flooding would look in other regions.“The idea is: One day, we could use this before a hurricane, where it provides an additional visualization layer for the public,” says Björn Lütjens, a postdoc in MIT’s Department of Earth, Atmospheric and Planetary Sciences, who led the research while he was a doctoral student in MIT’s Department of Aeronautics and Astronautics (AeroAstro). “One of the biggest challenges is encouraging people to evacuate when they are at risk. Maybe this could be another visualization to help increase that readiness.”To illustrate the potential of the new method, which they have dubbed the “Earth Intelligence Engine,” the team has made it <a href="https://climate-viz.github.io/" target="_blank">available</a> as an online resource for others to try.The researchers <a href="https://ieeexplore.ieee.org/document/10758300" target="_blank">report their results today in the journal IEEE Transactions on Geoscience and Remote Sensing</a>.&nbsp;The study’s MIT co-authors include Brandon Leshchinskiy; Aruna Sankaranarayanan; and Dava Newman, professor of AeroAstro and director of the MIT Media Lab; along with collaborators from multiple institutions.<h2>Generative adversarial images</h2>The new study is an extension of the team’s efforts to apply generative AI tools to visualize future climate scenarios.“Providing a hyper-local perspective of climate seems to be the most effective way to communicate our scientific results,” says Newman, the study’s senior author. “People relate to their own zip code, their local environment where their family and friends live. Providing local climate simulations becomes intuitive, personal, and relatable.”For this study, the authors use a conditional generative adversarial network, or GAN, a type of machine learning method that can generate realistic images using two competing, or “adversarial,” neural networks. The first “generator” network is trained on pairs of real data, such as satellite images before and after a hurricane. The second “discriminator” network is then trained to distinguish between the real satellite imagery and the one synthesized by the first network.Each network automatically improves its performance based on feedback from the other network. The idea, then, is that such an adversarial push and pull should ultimately produce synthetic images that are indistinguishable from the real thing. Nevertheless, GANs can still produce “hallucinations,” or factually incorrect features in an otherwise realistic image that shouldn’t be there.“Hallucinations can mislead viewers,” says Lütjens, who began to wonder whether such hallucinations could be avoided, such that generative AI tools can be trusted to help inform people, particularly in risk-sensitive scenarios. “We were thinking: How can we use these generative AI models in a climate-impact setting, where having trusted data sources is so important?”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732838099417-MIT-AI-Flood-02-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">Here, the AI model visualizes how flood in North Carolina would look like in satellite imagery. These new visualizations of physics-based flood models might facilitate more intuitive communication of climate risks.&nbsp; Credit: Pre-flood images from Maxar Open Data Program via Gupta et al., CVPR Workshop Proceedings. Generated images from Lütjen et al., IEEE TGRS.<h2>Flood hallucinations</h2>In their new work, the researchers considered a risk-sensitive scenario in which generative AI is tasked with creating satellite images of future flooding that could be trustworthy enough to inform decisions of how to prepare and potentially evacuate people out of harm’s way.Typically, policymakers can get an idea of where flooding might occur based on visualizations in the form of color-coded maps. These maps are the final product of a pipeline of physical models that usually begins with a hurricane track model, which then feeds into a wind model that simulates the pattern and strength of winds over a local region.&nbsp;This is combined with a flood or storm surge model that forecasts how wind might push any nearby body of water onto land. A hydraulic model then maps out where flooding will occur based on the local flood infrastructure and generates a visual, color-coded&nbsp;map of flood elevations over a particular region.“The question is: Can visualizations of satellite imagery add another level to this, that is a bit more tangible and emotionally engaging than a color-coded map of reds, yellows, and blues, while still being trustworthy?” Lütjens says.The team first tested how generative AI alone would produce satellite images of future flooding. They trained a GAN on actual satellite images taken by&nbsp;satellites as they passed over Houston before and after Hurricane Harvey. When they tasked the generator to produce new flood images of the same regions, they found that the images resembled typical satellite imagery, but a closer look revealed hallucinations in some images, in the form of floods where flooding should not be possible (for instance, in locations at higher elevation).To reduce hallucinations and increase the trustworthiness of the AI-generated images, the team paired the GAN with a physics-based flood model that incorporates real, physical parameters and phenomena, such as an approaching hurricane’s trajectory, storm surge, and flood patterns.&nbsp;With this physics-reinforced method, the team generated satellite images around Houston that depict the same flood extent, pixel by pixel, as forecasted by the flood model.“We show a tangible way to combine machine learning with physics for a use case that’s risk-sensitive, which requires us to analyze the complexity of Earth’s systems and project future actions and possible scenarios to keep people out of harm’s way,” Newman says. “We can’t wait to get our generative AI tools into the hands of decision-makers at the local community level, which could make a significant difference and perhaps save lives.”<hr>The research was supported, in part, by the MIT Portugal Program, the DAF-MIT Artificial Intelligence Accelerator, NASA, and Google Cloud.

The method could help communities visualize and prepare for approaching storms.

New AI tool generates realistic satellite images of future flooding

Cutting back on animal protein in our diets can save on resources and greenhouse gas emissions. But convincing meat-loving consumers to switch up their menu is a challenge. Looking at this problem from a mechanical engineering angle, Stanford engineers are pioneering a new approach to food texture testing that could pave the way for faux filets that fool even committed carnivores.In a <a href="https://www.nature.com/articles/s41538-024-00330-6" data-extlink="">new paper</a> in Science of Food, the team demonstrated that a combination of mechanical testing and machine learning can describe food texture with striking similarity to human taste testers. Such a method could speed up the development of new and better plant-based meats. The team also found that some plant-based products are already nailing the texture of the meats they’re mimicking.“We were surprised to find that today’s plant-based products can reproduce the whole texture spectrum of animal meats,” said <a href="https://profiles.stanford.edu/ellen-kuhl" data-extlink="">Ellen Kuhl</a>, professor of mechanical engineering and senior author of the study. Meat substitutes have come a long way from when tofu was the only option, she added.Industrial animal agriculture contributes to climate change, pollution, habitat loss, and antibiotic resistance. That burden on the planet can be eased by swapping animal proteins for plant proteins in diets. One study estimated that plant-based meats, on average, have <a href="https://www.sciencedirect.com/science/article/pii/S0921344922006632" data-extlink="">half the environmental impact</a> as animal meat. But many meat eaters are reluctant to change; <a href="https://www.frontiersin.org/journals/sustainable-food-systems/articles/10.3389/fsufs.2019.00011/full" data-extlink="">only about a third of Americans</a> in one survey indicated they were “very likely” or “extremely likely” to buy plant-based alternatives.“People love meat,” said <a href="https://livingmatter.stanford.edu/people/skyler-st-pierre" data-extlink="">Skyler St. Pierre</a>, a PhD student in mechanical engineering and lead author of the paper. “If we want to convince the hardcore meat eaters that alternatives are worth trying, the closer we can mimic animal meat with plant-based products, the more likely people might be open to trying something new.”To successfully mimic animal meat, food scientists analyze the texture of plant-based meat products. Unfortunately, traditional food testing methods are not standardized and the results are rarely made available to science and to the public, said St. Pierre. This makes it harder for scientists to collaborate and create new recipes for alternatives.<h2>New food texture tests</h2>The research grew out of a class project by St. Pierre. Looking for affordable materials to use in mechanical tests, he turned to hot dogs and tofu. Over the summer of 2023, undergraduate researchers joined in to test the foods and learn how engineers depict material responses to stress, loading, and stretching.Realizing how this work could aid the development of plant-based meats, the Stanford team debuted a three-dimensional food test. They put eight products to the test: animal and plant-based hot dog, animal and plant-based sausage, animal and plant-based turkey, and extra firm and firm tofu. They mounted bits of meat into a machine that pulled, pushed, and sheared on the samples. “These three loading modes represent what you do when you chew,” said Kuhl, who is also the Catherine Holman Johnson Director of <a href="http://biox.stanford.edu/" data-extlink="">Stanford Bio-X</a> and the Walter B. Reinhold Professor in the School of Engineering.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732580654113-20241108_edible_foods2_0.jpg.webp" style="width: 100%px;" class="fr-fic fr-dib">A mechanical test of plant-based meat can help quantify food texture. | Kurt HickmanThen, they used machine learning to process the data from these tests: They designed a new type of neural network that takes the raw data from the tests and produces equations that explain the properties of the meats.To see if these equations can explain the perception of texture, the team carried out a taste test survey. The tasters—who first completed surveys on their openness to new foods and their attachment to meat—ate samples of the eight products and rated them on five-point scale for 12 categories: soft, hard, brittle, chewy, gummy, viscous, springy, sticky, fibrous, fatty, moist, and meat-like.&nbsp;<h2>Impressive hot dogs and sausages</h2>In the mechanical testing, the plant-based hot dog and sausage behaved very similarly in the pulling, pushing, and shear tests to their animal counterparts, and showed similar stiffnesses. Meanwhile, the plant-based turkey was twice as stiff as animal turkey, and the tofu was much softer than the meat products. Strikingly, the human testers also ranked the stiffness of the hot dogs and sausages very similarly to the mechanical tests. “What’s really cool is that the ranking of the people was almost identical to the ranking of the machine,” said Kuhl. “That’s great because now we can use the machine to have a quantitative, very reproducible test.”The findings suggest that new, data-driven methods hold promise for speeding up the process of developing tasty plant-based products. “Instead of using a trial-and-error approach to improve the texture of plant-based meat, we could envision using generative artificial intelligence to scientifically generate recipes for plant-based meat products with precisely desired properties,” the authors wrote in the paper.But artificial intelligence recipe development, like other AIs, needs lots of data. That’s why the team is&nbsp;<a href="https://github.com/LivingMatterLab/CANN" data-extlink="">sharing their data</a> online, making it open for other researchers to view and add to. “Historically, some researchers, and especially companies, don’t share their data and that’s a really big barrier to innovation,” said St. Pierre. Without sharing information and working together, he added, “how are we going to come up with a steak mimic together?”The team is continuing to test foods and build a public database. This summer, St. Pierre oversaw undergraduates testing veggie and meat deli slices. The researchers also plan to test engineered fungi developed by&nbsp;<a href="https://profiles.stanford.edu/349659" data-extlink="">Vayu Hill-Maini</a>, who recently joined Stanford as an assistant professor of bioengineering. “If anybody has an artificial or a plant-based meat they want to test,” said Kuhl, “we’re so happy to test it to see how it stacks up.”Additional Stanford co-authors of the paper include mechanical engineering professor&nbsp;<a href="https://profiles.stanford.edu/marc-levenston" data-extlink="">Marc Levenston</a>, associate professor of mechanical engineering; postdoctoral researcher Kevin Linka; graduate students Ethan C. Darwin, Divya Adil, Reese A. Dunne, and Valerie A. Perez Medina; visiting researcher María Parra Vallecillo; and undergraduate researchers Magaly C. Aviles, Archer Date, Yanav Lall.Kuhl is also a professor, by courtesy, of bioengineering in the Schools of Engineering and&nbsp;<a href="https://med.stanford.edu/" data-extlink="">Medicine</a>, and a member of the&nbsp;<a href="http://cvi.stanford.edu/" data-extlink="">Cardiovascular Institute</a>, the&nbsp;<a href="https://humanperformance.stanford.edu/" data-extlink="">Wu Tsai Human Performance Alliance</a>, the&nbsp;<a href="https://icme.stanford.edu/" data-extlink="">Institute for Computational and Mathematical Engineering</a>, and the&nbsp;<a href="https://neuroinstitute.stanford.edu/" data-extlink="">Wu Tsai Neurosciences Institute</a>. Levenston is also a member of Bio-X, the Cardiovascular Institute, the Wu Tsai Human Performance Alliance and the Maternal &amp; Child Health Research Institute.

Is it possible to close the texture gap between plant-based meat and animal meat? Ellen Kuhl and her lab are trying. From left: Skyler St. Pierre, Marc Levenston, Ellen Kuhl, Reese Dunne, Ethan Darwin, Valerie Perez Medina, and Divya Adil pose with the meat and plant-based meat they analyzed. | Kurt Hickman

Stanford engineers are testing foods using an approach that combines a three-dimensional mechanical technique and AI to mimic our sensory experience.

Can AI help improve plant-based meats?

Matroid-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid

Visio Nerf-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Visio Nerf-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Visio Nerf-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Visio Nerf-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Visio Nerf-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Pulsar Fusion-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Pulsar Fusion-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Pulsar Fusion-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Pulsar Fusion-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Pulsar Fusion-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Fusion Bionic GmbH-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Fusion Bionic GmbH-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Fusion Bionic GmbH-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Fusion Bionic GmbH-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Fusion Bionic GmbH-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

AILIVERSE-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

AILIVERSE-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

AILIVERSE-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

AILIVERSE-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

AILIVERSE-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Xsens-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:53 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:58 GMT+0300 (Eastern European Summer Time)

Movella-Mon Oct 30 2023 14:34:24 GMT+0200 (Eastern European Standard Time)

Movella-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Computer Vision

Fine-tuning large language models adapts pre-trained models to specific tasks or domains using tailored datasets, while Retrieval-Augmented Generation (RAG) combines retrieval systems with generative models to dynamically incorporate external, up-to-date knowledge into outputs.

RAG vs Fine-Tuning: Differences, Benefits, and Use Cases Explained

Large Language Model (LLM) training involves teaching AI models to understand and generate human-like text by processing vast amounts of data, significantly enhancing their language comprehension and production capabilities.

LLM Training: Mastering the Art of Language Model Development

CUDA Cores and Tensor Cores are specialized units within NVIDIA GPUs; the former are designed for a wide range of general GPU tasks, while the latter are specifically optimized to accelerate AI and deep learning through efficient matrix operations.

Tensor Cores vs CUDA Cores: The Powerhouses of GPU Computing from Nvidia

An AI-generated depiction of a modern day chipset

Gaining a comprehensive understanding of the distinct capabilities and applications of TPUs and GPUs is crucial for developers and researchers aiming to navigate the complex and rapidly changing terrain of artificial intelligence.

TPU vs GPU in AI: A Comprehensive Guide to Their Roles and Impact on Artificial Intelligence

Clustering - A Common Unsupervised Learning Technique

Understanding What is Unsupervised Learning, the Mechanisms, Types, and Applications of Various Algorithms and Challenges it presents in Machine Learning

Computer Vision

Profiles