Cutting back on animal protein in our diets can save on resources and greenhouse gas emissions. But convincing meat-loving consumers to switch up their menu is a challenge. Looking at this problem from a mechanical engineering angle, Stanford engineers are pioneering a new approach to food texture testing that could pave the way for faux filets that fool even committed carnivores.In a <a href="https://www.nature.com/articles/s41538-024-00330-6" data-extlink="">new paper</a> in Science of Food, the team demonstrated that a combination of mechanical testing and machine learning can describe food texture with striking similarity to human taste testers. Such a method could speed up the development of new and better plant-based meats. The team also found that some plant-based products are already nailing the texture of the meats they’re mimicking.“We were surprised to find that today’s plant-based products can reproduce the whole texture spectrum of animal meats,” said <a href="https://profiles.stanford.edu/ellen-kuhl" data-extlink="">Ellen Kuhl</a>, professor of mechanical engineering and senior author of the study. Meat substitutes have come a long way from when tofu was the only option, she added.Industrial animal agriculture contributes to climate change, pollution, habitat loss, and antibiotic resistance. That burden on the planet can be eased by swapping animal proteins for plant proteins in diets. One study estimated that plant-based meats, on average, have <a href="https://www.sciencedirect.com/science/article/pii/S0921344922006632" data-extlink="">half the environmental impact</a> as animal meat. But many meat eaters are reluctant to change; <a href="https://www.frontiersin.org/journals/sustainable-food-systems/articles/10.3389/fsufs.2019.00011/full" data-extlink="">only about a third of Americans</a> in one survey indicated they were “very likely” or “extremely likely” to buy plant-based alternatives.“People love meat,” said <a href="https://livingmatter.stanford.edu/people/skyler-st-pierre" data-extlink="">Skyler St. Pierre</a>, a PhD student in mechanical engineering and lead author of the paper. “If we want to convince the hardcore meat eaters that alternatives are worth trying, the closer we can mimic animal meat with plant-based products, the more likely people might be open to trying something new.”To successfully mimic animal meat, food scientists analyze the texture of plant-based meat products. Unfortunately, traditional food testing methods are not standardized and the results are rarely made available to science and to the public, said St. Pierre. This makes it harder for scientists to collaborate and create new recipes for alternatives.<h2>New food texture tests</h2>The research grew out of a class project by St. Pierre. Looking for affordable materials to use in mechanical tests, he turned to hot dogs and tofu. Over the summer of 2023, undergraduate researchers joined in to test the foods and learn how engineers depict material responses to stress, loading, and stretching.Realizing how this work could aid the development of plant-based meats, the Stanford team debuted a three-dimensional food test. They put eight products to the test: animal and plant-based hot dog, animal and plant-based sausage, animal and plant-based turkey, and extra firm and firm tofu. They mounted bits of meat into a machine that pulled, pushed, and sheared on the samples. “These three loading modes represent what you do when you chew,” said Kuhl, who is also the Catherine Holman Johnson Director of <a href="http://biox.stanford.edu/" data-extlink="">Stanford Bio-X</a> and the Walter B. Reinhold Professor in the School of Engineering.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1732580654113-20241108_edible_foods2_0.jpg.webp" style="width: 100%px;" class="fr-fic fr-dib">A mechanical test of plant-based meat can help quantify food texture. | Kurt HickmanThen, they used machine learning to process the data from these tests: They designed a new type of neural network that takes the raw data from the tests and produces equations that explain the properties of the meats.To see if these equations can explain the perception of texture, the team carried out a taste test survey. The tasters—who first completed surveys on their openness to new foods and their attachment to meat—ate samples of the eight products and rated them on five-point scale for 12 categories: soft, hard, brittle, chewy, gummy, viscous, springy, sticky, fibrous, fatty, moist, and meat-like.&nbsp;<h2>Impressive hot dogs and sausages</h2>In the mechanical testing, the plant-based hot dog and sausage behaved very similarly in the pulling, pushing, and shear tests to their animal counterparts, and showed similar stiffnesses. Meanwhile, the plant-based turkey was twice as stiff as animal turkey, and the tofu was much softer than the meat products. Strikingly, the human testers also ranked the stiffness of the hot dogs and sausages very similarly to the mechanical tests. “What’s really cool is that the ranking of the people was almost identical to the ranking of the machine,” said Kuhl. “That’s great because now we can use the machine to have a quantitative, very reproducible test.”The findings suggest that new, data-driven methods hold promise for speeding up the process of developing tasty plant-based products. “Instead of using a trial-and-error approach to improve the texture of plant-based meat, we could envision using generative artificial intelligence to scientifically generate recipes for plant-based meat products with precisely desired properties,” the authors wrote in the paper.But artificial intelligence recipe development, like other AIs, needs lots of data. That’s why the team is&nbsp;<a href="https://github.com/LivingMatterLab/CANN" data-extlink="">sharing their data</a> online, making it open for other researchers to view and add to. “Historically, some researchers, and especially companies, don’t share their data and that’s a really big barrier to innovation,” said St. Pierre. Without sharing information and working together, he added, “how are we going to come up with a steak mimic together?”The team is continuing to test foods and build a public database. This summer, St. Pierre oversaw undergraduates testing veggie and meat deli slices. The researchers also plan to test engineered fungi developed by&nbsp;<a href="https://profiles.stanford.edu/349659" data-extlink="">Vayu Hill-Maini</a>, who recently joined Stanford as an assistant professor of bioengineering. “If anybody has an artificial or a plant-based meat they want to test,” said Kuhl, “we’re so happy to test it to see how it stacks up.”Additional Stanford co-authors of the paper include mechanical engineering professor&nbsp;<a href="https://profiles.stanford.edu/marc-levenston" data-extlink="">Marc Levenston</a>, associate professor of mechanical engineering; postdoctoral researcher Kevin Linka; graduate students Ethan C. Darwin, Divya Adil, Reese A. Dunne, and Valerie A. Perez Medina; visiting researcher María Parra Vallecillo; and undergraduate researchers Magaly C. Aviles, Archer Date, Yanav Lall.Kuhl is also a professor, by courtesy, of bioengineering in the Schools of Engineering and&nbsp;<a href="https://med.stanford.edu/" data-extlink="">Medicine</a>, and a member of the&nbsp;<a href="http://cvi.stanford.edu/" data-extlink="">Cardiovascular Institute</a>, the&nbsp;<a href="https://humanperformance.stanford.edu/" data-extlink="">Wu Tsai Human Performance Alliance</a>, the&nbsp;<a href="https://icme.stanford.edu/" data-extlink="">Institute for Computational and Mathematical Engineering</a>, and the&nbsp;<a href="https://neuroinstitute.stanford.edu/" data-extlink="">Wu Tsai Neurosciences Institute</a>. Levenston is also a member of Bio-X, the Cardiovascular Institute, the Wu Tsai Human Performance Alliance and the Maternal &amp; Child Health Research Institute.

Is it possible to close the texture gap between plant-based meat and animal meat? Ellen Kuhl and her lab are trying. From left: Skyler St. Pierre, Marc Levenston, Ellen Kuhl, Reese Dunne, Ethan Darwin, Valerie Perez Medina, and Divya Adil pose with the meat and plant-based meat they analyzed. | Kurt Hickman

Stanford engineers are testing foods using an approach that combines a three-dimensional mechanical technique and AI to mimic our sensory experience.

Can AI help improve plant-based meats?

Artificial intelligence (AI) is rapidly transforming healthcare by leveraging machine learning techniques to analyze medical images, patient data, and genetic information with unprecedented speed and accuracy, significantly improving early disease detection and enhancing diagnostic precision.


AI-Powered Health Diagnostics

Optical sensors are the “eyes” of systems in industrial applications. However, just as with humans, in industrial applications a powerful peripheral system must ensure that image processing permanently functions reliably, without errors, and at the required performance level.As optical sensors become increasingly powerful, the number of applications in which they can be used is also increasing. Along with the associated quality management and documentation system, the monitoring and evaluation electronics in state-of-the art production and documentation systems produce a flood of data. This mass of data must be transmitted, collected, and processed reliably. The underlying technical structures of machine vision systems must remain fully functional even under demanding conditions, such as permanent mechanical stress due to shocks and vibrations. Data connectors in particular, which are important interfaces within the system, also have to meet these requirements so that there is no wear on the male and female contacts (cf. fretting corrosion). Otherwise, contact interruptions will lead to downtimes in production or process errors that require time-consuming and cost-intensive measures.Increasingly compact systems in smaller spaces, following the trend of miniaturization, pose additional challenges for device developers of machine vision systems. These include electromagnetic interference pulses that can influence the functionality of the individual components. The reduced structures within the system architecture also jeopardize stable system integrity. The increasingly powerful components, installed increasingly closer together, also bring with them a thermal problem that may not only have an impact on the operation of the sensors, but may also make device approval more difficult. Miniaturized interfaces and device-integrated connection technology that provide secure shielding and data transmission are active solutions to these potential problems. The optimized positioning of components on the printed circuit board in accordance with electromagnetic and thermal considerations during device development also ensures reliability. Simulations of heat generation, signal integrity, and impedance are helpful, especially when non-standard components are used.&nbsp;<h2>Standards for optimized image processing</h2>There are numerous transmission standards, including Camera Link, CoaXPress, GigE Vision, and USB 3, to suit the respective machine vision applications and associated interfaces. All factors must be coordinated and work reliably together to ensure optimized image processing.&nbsp;Phoenix Contact is synonymous with high quality in the field of industrial connection technology. The portfolio of the company, founded in 1923, includes tried and tested data connectors such as D-SUB, RJ45, USB, HDMI, coaxial, and fiber-optic connection technology.&nbsp;With its commitment to quality and innovation, Phoenix Contact is also an expert partner for new components, such as the Single Pair Ethernet (SPE), RJ45 Industrial (RJ45-IND), and the connection system for shielded and unshielded Finepitch series board-to-board connectors.<h3>RJ45 Industrial</h3>The RJ45 Industrial (RJ45-IND) product family was developed especially for industrial applications. This powerful product family has been further developed and optimized based on a large amount of experience gained with the standard RJ45 products in cooperation with customers from different industries. With this development, Phoenix Contact has created a product range featuring an optimized 360° shielding concept, for example.To counter electromagnetic interference, the housing shield springs and shield pins for the printed circuit board have been raised. This also improves the mechanical strength of the socket on the printed circuit board.The connection between RJ45-IND connectors or patch cables and RJ45-IND female printed circuit board (PCB) connectors has been decisively optimized. Shocks and vibrations are now counteracted by the so-called “perfect match” of the connector system. The development also included optimizing the contact depth of the female PCB connector from 8.3 m to 9 mm, thus enlarging the contact zone of the connector system. There have also been decisive improvements in the tolerances in the connection of the socket and plug, meaning that their contact connection is even better matched. On the connector side, the contact pressure was increased further through the inclusion of a compression spring.&nbsp;The temperature resistance of the female PCB connector has been improved through the use of a reinforced housing, now utilizing fiberglass technology. Temperatures from -40 up to +105 C are now possible without any problems.&nbsp;The RJ45-IND patch cables, which are suitable for drag chain applications and robots, are also particularly robust. A major advantage and unique selling point of the Phoenix Contact solution is the IDC fast connection, which enables the one-piece connector to be assembled quickly and easily without the use of a special tool.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzMyNTc3NjczNzg5LUJpbGQgM19SSjQ1IElORCByZWNodHMgKDEpLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">Single Pair EthernetThe IEEE 802.3 task force has developed the transmission standards for Single Pair Ethernet (SPE), defining the standards 10Base-T1, 100Base-T1, 1000Base-T1, and MultiGigBASE-T1.&nbsp;The SPE standards enable data rates of 10 Mbps up to 1 Gbps at distances of 15 to 1,000 m. The MultiGigBASE-T1 standard goes so far as to enable data rates of up to 10 Gbps at distances of up to 15 m. The power transmission for the SPE standards 100BASE-T1 and 1000BASE-T1 are defined by the IEEE 802.3 bu standard (PoDL – Power over Data Lines).&nbsp;The load outputs of the end devices (power devices) are divided into 10 performance classes from 0.5 to 50 W.&nbsp;With regard to industrial image processing, SPE is an exciting field of application not just because of the potential to transmit data and power. In particular, the SPE portfolio from Phoenix Contact plays to its strengths here with its compact and 50 W consistent design. For example, the SPE-IP20 socket saves about 50 percent of space compared to a standardized RJ45 jack.&nbsp;Phoenix Contact also has a consistent SPE product portfolio in which all components have been matched and tested. This portfolio provides compact IP20 connectors (in accordance with IEC&nbsp;63171-2) and IP67-protected M8 connectors (in accordance with IEC&nbsp;63171-5). A brand new addition to the portfolio is the SPE-M12 hybrid connector from the Onepair series.The advantages of the SPE connection technology include the uniform mating face and the possibility to achieve a high degree of common part usage. This means that adapters are no longer necessary and no additional costs are incurred.Various SPE components are available for the developers of machine vision sensors. These include new semiconductors (SPE PHYs and PoDL controllers), SPE magnetics, connection technology in accordance with IEC 63171-2 and -5, and SPE-compatible cables in accordance with IEC 61156.&nbsp;Tip: With a membership in the System Alliance (<a href="http://www.singlepairethernet.com/">www.singlepairethernet.com/</a>), it is possible to gain detailed information about Single Pair Ethernet and become familiar with specific solutions for industrial applications.<h3>Board-to-board connectors</h3>Contact systems are to be preferred over simply designed male and female strips, in particular in demanding applications – such as for industrial image processing, for example. They are sophisticated yet at the same time robust.&nbsp;With its board-to-board connectors, the Finepitch product series from Phoenix Contact offers a professional contact system for space-saving PCB connections. Various numbers of positions as well as stack heights provide a great deal of flexibility for developers. The system features high-speed data transmission rates of up to 52&nbsp;Gbps. The innovative ScaleX double contacts (FP&nbsp;0,8) are designed as hermaphroditic, meaning that vibration-resistant connections can be created in very confined spaces.The shielded version available as an option also prevents electromagnetic interferences.As a valuable service offering, Phoenix Contact experts provide support during the entire design-in process as well as design-in support. This means that device developers of machine vision systems can quickly obtain preconfigured 2D/3D data for their CAD software.&nbsp;Phoenix Contact also provides individual thermal and housing simulations based on the finite element method (FEM). Free simulations for data and signal integrity complete the comprehensive service offering. With this support, customers can realize stable communication right from the start and throughout the entire development process.&nbsp;Phoenix Contact rounds off its services with a free sample service and reliable logistics.&nbsp;<h2>Summary&nbsp;</h2>The developers of modern machine vision systems must consider various important factors during the product development process to ensure that reliable and high-performance systems are ready for the market in the environment of high data transmission rates and miniaturization. The cost factor also plays a decisive role. Phoenix Contact is a reliable partner, providing both products and services. In addition to the proven data connectors and innovative product solutions – such as for SPE, for example – Phoenix Contact provides simulations that help to realize stable device communication and reliable systems with cost-efficiency. This creates a high-performance periphery for reliable industrial image processing.Written by: Dipl.-Ing. Verena Neuhaus, Product Management Data Connectors, Business Unit Field Device Connectors, Phoenix Contact GmbH &amp; Co. KG, Blomberg, GermanyThomas Schulze, M.Sc., Product Management Board-to-Board Connectors, Business Unit PCB Connectors, Phoenix Contact GmbH &amp; Co. KG, Blomberg, Germany

Optical sensors are the “eyes” of industrial systems, relying on robust peripherals for reliable image processing. As sensors grow more powerful, challenges like data management, mechanical stress, and thermal issues in compact designs demand innovative solutions.

Image data in industrial applications: Ensuring reliability and performance during transmission

Now, researchers have introduced a technique for compressing an LLM’s reams of data, which could increase privacy, save energy and lower costs.The&nbsp;<a href="https://arxiv.org/abs/2405.18886">new algorithm</a>, developed by engineers at Princeton and Stanford Engineering, works by trimming redundancies and reducing the precision of an LLM’s layers of information. This type of leaner LLM could be stored and accessed locally on a device like a phone or laptop and could provide performance nearly as accurate and nuanced as an uncompressed version.“Any time you can reduce the computational complexity, storage and bandwidth requirements of using AI models, you can enable AI on devices and systems that otherwise couldn’t handle such compute- and memory-intensive tasks,” said study coauthor&nbsp;<a href="https://engineering.princeton.edu/faculty/andrea-goldsmith">Andrea Goldsmith</a>, dean of Princeton’s School of Engineering and Applied Science and Arthur LeGrand Doty Professor of&nbsp;<a href="https://ece.princeton.edu/">Electrical and Computer Engineering</a>.“When you use ChatGPT, whatever request you give it goes to the back-end servers of OpenAI, which process all of that data, and that is very expensive,” said coauthor&nbsp;<a href="https://sites.google.com/view/rajarshi-saha">Rajarshi Saha</a>, a Stanford Engineering Ph.D. student. “So, you want to be able to do this LLM inference using consumer GPUs [graphics processing units], and the way to do that is by compressing these LLMs.” Saha’s graduate work is coadvised by Goldsmith and coauthor&nbsp;<a href="https://stanford.edu/~pilanci">Mert Pilanci</a>, an assistant professor at Stanford Engineering.The researchers will present&nbsp;<a href="https://arxiv.org/abs/2405.18886">their new algorithm</a> CALDERA, which stands for Calibration Aware Low precision DEcomposition with low Rank Adaptation, at the Conference on Neural Information Processing Systems (NeurIPS) in December. Saha and colleagues began this compression research not with LLMs themselves, but with the large collections of information that are used to train LLMs and other complex AI models, such as those used for image classification. This technique, a forerunner to the new LLM compression approach, was&nbsp;<a href="https://openreview.net/forum?id=rxsCTtkqA9">published</a> in 2023.Training data sets and AI models are both composed of matrices, or grids of numbers that are used to store data. In the case of LLMs, these are called weight matrices, which are numerical representations of word patterns learned from large swaths of text.“We proposed a generic algorithm for compressing large data sets or large matrices,” said Saha. “And then we realized that nowadays, it’s not just the data sets that are large, but the models being deployed are also getting large. So, we could also use our algorithm to compress these models.”While the team’s algorithm is not the first to compress LLMs, its novelty lies in an innovative combination of two properties, one called “low-precision,” the other “low-rank.” As digital computers store and process information as bits (zeros and ones), “low-precision” representation reduces the number of bits, speeding up storage and processing while improving energy efficiency. On the other hand, “low-rank” refers to reducing redundancies in the LLM weight matrices.“Using both of these properties together, we are able to get much more compression than either of these techniques can achieve individually,” said Saha.The team tested their technique using Llama 2 and Llama 3, open-source large language models released by Meta AI, and found that their method, which used low-rank and low-precision components in tandem with each other, can be used to improve other methods which use just low-precision. The improvement can be up to 5%, which is significant for metrics that measure uncertainty in predicting word sequences.They evaluated the performance of the compressed language models using several sets of benchmark tasks for LLMs. The tasks included determining the logical order of two statements, or answering questions involving physical reasoning, such as how to separate an egg white from a yolk or how to make a cup of tea.“I think it’s encouraging and a bit surprising that we were able to get such good performance in this compression scheme,” said Goldsmith, who moved to Princeton from Stanford Engineering in 2020. “By taking advantage of the weight matrix rather than just using a generic compression algorithm for the bits that are representing the weight matrix, we were able to do much better.”Using an LLM compressed in this way could be suitable for situations that don’t require the highest possible precision. Moreover, the ability to fine-tune compressed LLMs on edge devices like a smartphone or laptop enhances privacy by allowing organizations and individuals to adapt models to their specific needs without sharing sensitive data with third-party providers. This reduces the risk of data breaches or unauthorized access to confidential information during the training process. To enable this, the LLMs must initially be compressed enough to fit on consumer-grade GPUs.Saha also cautioned that running LLMs on a smartphone or laptop could hog the device’s memory for a period of time. “You won’t be happy if you are running an LLM and your phone drains out of charge in an hour,” said Saha. Low-precision computation can help reduce power consumption, he added. “But I wouldn’t say that there’s one single technique that solves all the problems. What we propose in this paper is one technique that is used in combination with techniques proposed in prior works. And I think this combination will enable us to use LLMs on mobile devices more efficiently and get more accurate results.”The paper, “<a href="https://arxiv.org/abs/2405.18886">Compressing Large Language Models using Low Rank and Low Precision Decomposition</a>,” will be presented at the Conference on Neural Information Processing Systems (NeurIPS) in December 2024. In addition to Goldsmith, Saha and Pilanci, coauthors include Stanford Engineering researchers Naomi Sagan and Varun Srivastava. This work was supported in part by the U.S. National Science Foundation, the U.S. Army Research Office, and the Office of Naval Research.

Large language models (LLMs) are increasingly automating tasks like translation, text classification and customer service. But tapping into an LLM’s power typically requires users to send their requests to a centralized server — a process that’s expensive, energy-intensive and often slow.

Leaner large language models could enable efficient local use on phones and laptops

Fine-tuning large language models adapts pre-trained models to specific tasks or domains using tailored datasets, while Retrieval-Augmented Generation (RAG) combines retrieval systems with generative models to dynamically incorporate external, up-to-date knowledge into outputs.

RAG vs Fine-Tuning: Differences, Benefits, and Use Cases Explained

For roboticists, one challenge towers above all others: generalization – the ability to create machines that can adapt to any environment or condition. Since the 1970s, the field has evolved from writing sophisticated programs to using deep learning, teaching robots to learn directly from human behavior. But a critical bottleneck remains: data quality. To improve, robots need to encounter scenarios that push the boundaries of their capabilities, operating at the edge of their mastery. This process traditionally requires human oversight, with operators carefully challenging robots to expand their abilities. As robots become more sophisticated, this hands-on approach hits a scaling problem: the demand for high-quality training data far outpaces humans' ability to provide it.Now, a team of MIT CSAIL researchers have developed a novel approach to robot training that could significantly accelerate the deployment of adaptable, intelligent machines in real-world environments. The new system, called "LucidSim," uses recent advances in generative AI and physics simulators to create diverse and realistic virtual training environments, helping robots achieve expert-level performance in difficult tasks without any real-world data.<iframe width="640" height="360" src="https://www.youtube.com/embed/fgnJQrvTj70?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>LucidSim combines physics simulation with generative AI models, addressing one of the most persistent challenges in robotics: transferring skills learned in simulation to the real world. “A fundamental challenge in robot learning has long been the ‘sim-to-real gap’ – the disparity between simulated training environments and the complex, unpredictable real world,” says MIT CSAIL postdoctoral associate Ge Yang, a lead researcher on LucidSim. “Previous approaches often relied on depth sensors, which simplified the problem but missed crucial real-world complexities.”The multipronged system is a blend of different technologies. At its core, LucidSim uses large language models to generate various structured descriptions of environments. These descriptions are then transformed into images using generative models. To ensure that these images reflect real-world physics, an underlying physics simulator is used to guide the generation process.&nbsp;<h2>The Birth of an Idea: From Burritos to Breakthroughs</h2>The inspiration for LucidSim came from an unexpected place: a conversation outside Beantown Taqueria in Cambridge. ”We wanted to teach vision-equipped robots how to improve using human feedback. &nbsp;But then, we realized we didn't have a pure vision-based policy to begin with,” says Alan Yu, an undergraduate student at MIT and co-lead on LucidSim. "We kept talking about it as we walked down the street, and then we stopped outside the taqueria for about half an hour. That's where we had our moment."To cook up their data, the team generated realistic images by extracting depth maps, which provide geometric information, and semantic masks, which label different parts of an image, from the simulated scene. They quickly realized, however, that with tight control on the composition of the image content, the model would produce similar images that weren’t different from each other using the same prompt. So, they devised a way to source diverse text prompts from ChatGPT.&nbsp;This approach, however, only resulted in a single image. To make short, coherent videos which serve as little “experiences” for the robot, the scientists hacked together some image magic into another novel technique the team created, called “Dreams In Motion (DIM).” The system computes the movements of each pixel between frames, to warp a single generated image into a short, multi-frame video. Dreams In Motion does this by considering the 3D geometry of the scene and the relative changes in the robot’s perspective.&nbsp;"We outperform domain randomization, a method developed in 2017 that applies random colors and patterns to objects in the environment, which is still considered the go-to method these days," says Yu. &nbsp;"While this technique generates diverse data, it lacks realism. LucidSim addresses both diversity and realism problems. It’s exciting that even without seeing the real world during training, the robot can recognize and navigate obstacles in real environments."&nbsp;The team is particularly excited about the potential of applying LucidSim to domains outside quadruped locomotion and parkour, their main testbed. One example is mobile manipulation, where a mobile robot is tasked to handle objects in an open area, and also, color perception is critical. “Today, these robots still learn from real-world demonstrations,” says Yang. “Although collecting demonstrations is easy, scaling a real-world robot teleoperation setup to thousands of skills is challenging because a human has to physically set up each scene. We hope to make this easier, thus qualitatively more scalable, by moving data collection into a virtual environment.”&nbsp;The team put LucidSim to the test against an alternative, where an expert teacher demonstrates the skill for the robot to learn from. The results were surprising: robots trained by the expert struggled, succeeding only 15 percent of the time – and even quadrupling the amount of expert training data barely moved the needle. But when robots collected their own training data through LucidSim, the story changed dramatically. Just doubling the dataset size catapulted success rates to 88 percent. "And giving our robot more data monotonically improves its performance – eventually, the student becomes the expert," says Yang.“One of the main challenges in sim-to-real transfer for robotics is achieving visual realism in simulated environments,” says Stanford University Assistant Professor of Electrical Engineering Shuran Song, who wasn’t involved in the research. “The LucidSim framework provides an elegant solution by using generative models to create diverse, highly realistic visual data for any simulation. This work could significantly accelerate the deployment of robots trained in virtual environments to real-world tasks.”From the streets of Cambridge to the cutting edge of robotics research, LucidSim is paving the way toward a new generation of intelligent, adaptable machines – ones that learn to navigate our complex world without ever setting foot in it.<hr>Yu and Yang wrote the paper with four fellow CSAIL affiliates: mechanical engineering postdoc Ran Choi; undergraduate researcher Yajvan Ravan; John Leonard, Samuel C. Collins Professor of Mechanical and Ocean Engineering in the MIT Department of Mechanical Engineering; and MIT Associate Professor Phillip Isola. Their work was supported, in part, by a Packard Fellowship, a Sloan Research Fellowship, the Office of Naval Research, Singapore’s Defence Science and Technology Agency, Amazon, MIT Lincoln Laboratory, and the National Science Foundation Institute for Artificial Intelligence and Fundamental Interactions. The researchers will present their work at the Conference on Robot Learning (CoRL) in early November.

A team of MIT CSAIL researchers have developed a novel approach to robot training that could significantly accelerate the deployment of adaptable, intelligent machines in real-world environments.

Can robots learn from machine dreams?

A less wasteful way to train large language models, such as the GPT series, finishes in the same amount of time for up to 30% less energy, according to a new study from the University of Michigan.The approach could save enough energy to power <a href="https://www.eia.gov/energyexplained/use-of-energy/electricity-use-in-homes.php" target="_blank" rel="noreferrer noopener">1.1 million U.S. homes</a> in 2026, based on Wells Fargo’s projections of <a href="https://www.marketwatch.com/story/ai-could-demand-a-shocking-amount-of-electricity-check-out-this-chart-e91e306d" target="_blank" rel="noreferrer noopener">AI power demand</a>. It could also take a bite out of the International Monetary Fund’s prediction that data centers could account for <a href="https://www.imf.org/en/Blogs/Articles/2024/08/15/carbon-emissions-from-ai-and-crypto-are-surging-and-tax-policy-can-help" target="_blank" rel="noreferrer noopener">1.2% of the world’s carbon emissions by 2027</a>—and the <a href="https://theconversation.com/ais-excessive-water-consumption-threatens-to-drown-out-its-environmental-contributions-225854">water demands</a> that come with that energy use.Some experts say that these costs could be outweighed by environmental benefits. They argue that AI could be a <a href="https://news.un.org/en/story/2023/11/1143187" target="_blank" rel="noreferrer noopener">“game changer” for fighting climate change</a> by identifying ways to optimize supply chains and the grid, manage our energy needs, and improve research on climate change. Still, that doesn’t excuse squandering energy, and some of the power used to train AI has zero impact on training time and model accuracy.“Why spend something when there’s no point?” said <a href="https://eecs.engin.umich.edu/people/chowdhury-mosharaf/" target="_blank" rel="noreferrer noopener">Mosharaf Chowdhury</a>, an associate professor of computer science and engineering and the corresponding author of the study presented at the <a href="https://sigops.org/s/conferences/sosp/2024/index.html" target="_blank" rel="noreferrer noopener">30th Symposium on Operating Systems Principles</a> last Monday.“We can’t keep building bigger and bigger data centers because we won’t have the power to run them,” said Chowdhury. “If we can reduce the energy consumed by AI, we can reduce AI’s carbon footprint and cooling requirements and allow for more computation to fit within our current energy constraints.”The energy waste is created when AI training is unequally divided between GPUs, which are computer processors specialized for large data and graphics applications. Although it opens the door for waste, splitting the work is necessary for processing huge datasets.“AI models today are so large, they cannot fit inside a single computer processor,” said <a href="https://jaewonchung.me/about" target="_blank" rel="noreferrer noopener">Jae-Won Chung</a>, a doctoral student in computer science and engineering and the first author of the study. “They need to be divided into tens of thousands of processors to be trained, but dividing the models in perfectly equal sizes across all processors is practically impossible.”The training jobs are so difficult to evenly split up because some tasks need to be grouped together on the same processor—like how each installment of a book series will be grouped together in an organized shelf. Depending on how the tasks are grouped, some processors might get stuck with the AI-training equivalent of the Encyclopedia Britannica while others get assigned a fantasy trilogy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731630851359-AI_waste_Infographic-768x1470.jpg" style="width: 100%px;" class="fr-fic fr-dib">Infographic credit: Rachel Voigt, Michigan Engineering.Because current training methods run each processor at top speed, processors with a lighter load will finish their calculations before other processors. This doesn’t speed up training, which isn’t complete until every processor finishes its job—but it is wasteful because faster calculations require more energy. In addition, problems such as faulty hardware or network delays create energy waste by slowing down a single processor’s computing speed.To save energy, the researchers developed a software tool, called&nbsp;<a href="https://ml.energy/zeus/research_overview/perseus/" target="_blank" rel="noreferrer noopener">Perseus</a>, that identifies a critical path, or a series of subtasks that will take the longest time to complete. Then, Perseus slows down processors that aren’t on the critical path so that they all finish their jobs around the same time—eliminating unnecessary power use.“Reducing the power cost of AI can have important implications for equitable AI access,” said Chowdhury. “If a country doesn’t have enough power to run a big model, they might need to use services from far away, or be stuck running smaller, less accurate models. This gap could further perpetuate disparity between different communities.”The team tested Perseus by training GPT-3, three other large language models and one computer vision model.Perseus is an open-sourced tool available as part of&nbsp;<a href="https://ml.energy/zeus/" target="_blank" rel="noreferrer noopener">Zeus</a>, a tool for measuring and optimizing AI energy consumption.&nbsp;

Smarter use of processor speeds saves energy without compromising training speed and performance.

Up to 30 percent of the power used to train AI is wasted. Here's how to fix it.

In this episode, we explore how AI co-pilots are equipping doctors with powerful tools to enhance decision-making and patient care and discuss how it could impact you - a potential patient - in the not distant future.

Podcast: Will AI Assistants Make Your Doctor Better?

Large Language Model (LLM) training involves teaching AI models to understand and generate human-like text by processing vast amounts of data, significantly enhancing their language comprehension and production capabilities.

LLM Training: Mastering the Art of Language Model Development

In the classic cartoon “The Jetsons,” Rosie the robotic maid seamlessly switches from vacuuming the house to cooking dinner to taking out the trash. But in real life, training a general-purpose robot remains a major challenge.Typically, engineers collect data that are specific to a certain robot and task, which they use to train the robot in a controlled environment. However, gathering these data is costly and time-consuming, and the robot will likely struggle to adapt to environments or tasks it hasn’t seen before.To train better general-purpose robots, MIT researchers developed a versatile technique that combines a huge amount of heterogeneous&nbsp;data from many of sources into one system that can teach any robot a wide range of tasks.Their method involves aligning data from varied domains, like simulations and real robots, and multiple modalities, including vision sensors and robotic arm position encoders, into a shared “language” that a generative AI model can process.By combining such an enormous amount of data, this approach can be used to train a robot to perform a variety of tasks without the need to start training it from scratch each time.This method could be faster and less expensive than traditional techniques because it requires far fewer task-specific data. In addition, it outperformed training from scratch by more than 20 percent in simulation and real-world experiments.“In robotics, people often claim that we don’t have enough training data. But in my view, another big problem is that the data come from so many different domains, modalities, and robot hardware. Our work shows how you’d be able to train a robot with all of them put together,” says Lirui Wang, an electrical engineering and computer science (EECS) graduate student and lead author of a <a href="https://arxiv.org/pdf/2409.20537" target="_blank">paper on this technique</a>.Wang’s co-authors include fellow EECS graduate student Jialiang Zhao; Xinlei Chen, a research scientist at Meta; and senior author Kaiming He, an associate professor in EECS and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). The research will be presented&nbsp;at the Conference on Neural Information Processing Systems.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730247515325-MIT-HeteroGenRobot-02-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">A figure shows how the new technique aligns data from varied domains, like simulation and real robots, and multiple modalities, including vision sensors and robotic arm position encoders, into a shared “language” that a generative AI model can process. Image: Courtesy of the researchers<h2>Inspired by LLMs</h2>A robotic “policy” takes in sensor observations, like camera images or proprioceptive measurements that track the speed and position a robotic arm, and then tells a robot how and where to move.Policies are typically trained using imitation learning, meaning a human demonstrates actions or teleoperates a robot to generate data, which are fed into an AI model that learns the policy. Because this method uses a small amount of task-specific data, robots often fail when their environment or task changes.To develop a better approach, Wang and his collaborators drew inspiration from large language models like GPT-4.These models are pretrained using an enormous amount of diverse language data and then fine-tuned by feeding them a small amount of task-specific data. Pretraining on so much data helps the models adapt to perform well on a variety of tasks.“In the language domain, the data are all just sentences. In robotics, given all the heterogeneity in the data, if you want to pretrain in a similar manner, we need a different architecture,” he says.Robotic data take many forms, from camera images to language instructions to depth maps. At the same time, each robot is mechanically unique, with a different number and orientation of arms, grippers, and sensors. Plus, the environments where data are collected vary widely.The MIT researchers developed a new architecture called Heterogeneous Pretrained Transformers (HPT) that unifies data from these varied modalities and domains.They put a machine-learning model known as a transformer into the middle of their architecture, which processes vision and proprioception inputs. A transformer is the same type of model that forms the backbone of large language models.The researchers align data from vision and proprioception into the same type of input, called a token, which the transformer can process. Each input is represented with the same fixed number of tokens.Then the transformer maps all inputs into one shared space, growing into a huge, pretrained model as it processes and learns from more data. The larger the transformer becomes, the better it will perform.A user only needs to feed HPT a small amount of data on their robot’s design, setup, and the task they want it to perform. Then HPT transfers the knowledge the transformer grained during pretraining to learn the new task.<h2>Enabling dexterous motions</h2>One of the biggest challenges of developing HPT was building the massive dataset to pretrain the transformer, which included 52 datasets with more than 200,000 robot trajectories in four categories, including human demo videos and simulation.The researchers also needed to develop an efficient way to turn raw proprioception signals from an array of sensors into data the transformer could handle.“Proprioception is key to enable a lot of dexterous motions. Because the number of tokens is in our architecture always the same, we place the same importance on proprioception and vision,” Wang explains.When they tested HPT, it improved robot performance by more than 20 percent on simulation and real-world tasks, compared with training from scratch each time. Even when the task was very different from the pretraining data, HPT still improved performance.“This paper provides a novel approach to training a single policy across multiple robot embodiments. This enables training across diverse datasets, enabling robot learning methods to significantly scale up the size of datasets that they can train on.&nbsp;It also allows the model to quickly adapt to new&nbsp;robot embodiments, which is important as new robot designs are continuously being produced,” says David Held, associate professor at the Carnegie Mellon University Robotics Institute, who was not involved with this work.In the future, the researchers want to study how data diversity could boost the performance of HPT. They also want to enhance HPT so it can process unlabeled data like GPT-4 and other large language models.“Our dream is to have a universal robot brain that you could download and use for your robot without any training at all. While we are just in the early stages, we are going to keep pushing hard and hope scaling leads to a breakthrough in robotic policies, like it did with large language models,” he says.

Researchers filmed multiple instances of a robotic arm feeding co-author Jialiang Zhao's adorable dog, Momo. The videos were included in datasets to train the robot. Image: Courtesy of the researchers

Inspired by large language models, researchers develop a training technique that pools diverse data to teach robots new skills.

A faster, better way to train general-purpose robots

In recent years, the field of robotics has seen remarkable advancements in gripper technology that pushes the boundaries of dexterity and manipulation. Shadow Robot is at the forefront of this evolution as a pioneer in innovative robotics development. Shadow’s approach to the innovative solutions has caught the attention of industry leaders, including Google DeepMind, a leading AI research lab focused on developing artificial intelligence capable of performing a wide range of tasks. This collaboration has led to the creation of DEX-EE, a groundbreaking gripper that may revolutionize robotic manipulation and machine learning research.<h2 dir="ltr">The Challenge from Google DeepMind</h2>In 2017, Google DeepMind approached Shadow Robot with a big challenge. The DeepMind project was conducting research on generalized intelligence and the role of embodiment in learning—how physical interaction with the world plays a part in learning, especially for tasks that involve understanding and manipulating objects. The study required a robot with high sensitivity and a robust design.For their research, the DeepMind team needed a design that could reliably run long-term and perform physically demanding experiments without constraints within its workspace. The study required a robot that could strike its environment repeatedly to learn how to manipulate objects. This meant that the robot needed to be resilient, sensor-rich and provide large volumes of high-quality data, while operating for prolonged periods without interruption for repairs and without the need to restart experiments.The challenge pushed the boundaries of existing robotic capabilities because it required a solution that was both robust and highly precise. Google DeepMind approached Shadow Robot for this challenge. Shadow Robot has partnered with numerous research projects and the company’s flagship Shadow Dexterous Hand has contributed to major advancements in the field of robotics.&nbsp;Read more about the Shadow Dexterous Hand and its contributions in our <a href="https://www.wevolver.com/article/dexterous-robotic-hands-part-1-unraveling-the-history-technology-and-applications-of-dexterous-robotic-hands">three-part Shadow Dexterous Hand series</a>.<h3 dir="ltr">DEX-EE: Shadow Robot’s Innovative Solution</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730204504132-Untitled+design+%285%29.png" style="width: 100%px;" class="fr-fic fr-dib">In response to the challenge from Google DeepMind, Shadow Robot developed DEX-EE, a groundbreaking dexterous robot hand, as well as a major advancement in the field of machine learning robots. DEX-EE is designed to perform with dynamic and controlled motion in a robust and reliable package. The hand enables long-running experiments without interruptions due to hardware failure.One of DEX-EE’s most innovative features is its high-speed network sensors, which provide rich data including position, force, and inertial measurement. Each finger has hundreds of channels of tactile sensing, and Shadow’s advanced fingertip sensors have hundreds of taxels (short for tactile pixel—the individual sensing elements that make up a tactile sensor), each with a massive dynamic range. With this wealth of sensory data, researchers can gather huge amounts of data from dexterous robotic tasks.DEX-EE was developed with an intensive focus on reliability and robustness. Through iterative design and testing over five years, Shadow created a new dexterous hand that could withstand the physical demands of Google DeepMind’s research. As a result of this focused development, the DEX-EE is a functional, reliable, and data-rich research tool. DEX-EE’s capabilities position it as an ideal hardware platform for dexterous robotic research into cognition and manipulation.<h2 dir="ltr">DEX-EE Technical Specifications and Advancements</h2>DEX-EE is a significant leap in dexterous robotic hand technology. The robot has high-bandwidth torque and position control loops, which enable delicate and precise fingertip dexterity, while torque and inertial measurement throughout the hand make it sensitive to interactions with its environment.The hand also features tactile fingertip sensors based on stereo-camera technology, which gives the hand abundant 3D data about the environment in a robust package. Additional, multi-taxel, 3-DOF sensors on the middle and proximal fingers provide crucial information during grasping and manipulation tasks.Compared to earlier models of the Shadow Hand, DEX-EE offers greater reliability and repairability. It was designed for easy maintenance with minimal training, incorporates fail-safes and a shutdown routing that reduces instances of failure and downtime. The hand’s advanced sensor networks provide rich data, including position, force, and inertial measurement, with hundreds of channels of tactile sensing per finger. Shadow’s innovative fingertip sensors have hundreds of taxels each, with a dynamic range that significantly surpasses the capabilities of other dexterous hands.DEX-EE is also integrated with ROS (Robot Operating System), making it versatile as a research tool. This combination of robustness, advanced sensing capabilities, and ease of integration positions DEX-EE as an ideal platform for dexterous manipulation research that is capable of withstanding the demands of long-term machine learning experiments.<h2 dir="ltr" id="isPasted">Applications in Machine Learning and AI Research</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730197463018-shadowrobot-grip.png" style="width: 100%px;" class="fr-fic fr-dib">Image: DEX-EE performs machine learning experiments with high sensitivity and robustness | Source:&nbsp;<a href="https://www.shadowrobot.com/dex-ee/" target="_blank" rel="noopener noreferrer">shadowrobot.com</a>DEX-EE represents the next major advancement in dexterous manipulation research because it provides a robust platform for complex machine learning experiments like Google DeepMind. The advanced sensor networks and tactile capabilities of DEX-EE enable researchers to gather enormous amounts of data during experiments and push the boundaries of what’s possible in robotic learning.Google DeepMind has leveraged DEX-EE’s capabilities in their latest AI system DemoStart, which helps robots learn to perform complex tasks that require dexterous movement. With these systems, Google DeepMind has achieved remarkable results, like inserting a plug into a socket, tightening a bolt, and other tasks that were previously challenging for robots to master.DEX-EE’s durability allows for long-running experiments without interruptions, making it ideal for reinforcement learning protocols that involve repeated physical interactions with the environment. This has already helped researchers explore new frontiers in embodied AI, investigating how physical interactions contribute to learning and the development of generalized intelligence.The hand’s integration with ROS further enhances its versatility as a research tool because it allows for easier implementation of new and existing learning algorithms and control strategies. With this combination of robustness, advanced sensing, and flexibility, DEX-EE is positioned to be a transformative platform for exploring the future of dexterous manipulation in AI and robotics.<h2 dir="ltr">Shadow Robot's Unique Approach to Robotics Development</h2>Shadow Robot continues to distinguish itself as a robotics company through problem-solving approaches and focus on custom solutions. Rather than offering off-the-shelf products that don’t meet a particular need, Shadow engages deeply with clients to understand their specific issues and develop tailored solutions.This collaborative approach has been exemplified in the development of DEX-EE, where Shadow worked closely with Google DeepMind over five years. Shadow’s willingness to rethink fundamental design principles demonstrates their commitment to innovation and their ability to work at the cutting edge of technology.Shadow has positioned themselves as a research and hardware company that is capable of tackling any robotics challenge. They have established a unique niche in the industry, bridging academic research and practical, real-world applications.<h2 dir="ltr">Leading Innovation in Custom Robotics Solutions</h2>Shadow Robot stands out as a leader in collaborative robotics and demonstrates a commitment to addressing practical needs through innovative design and partnership. Shadow’s work on DEX-EE, developed alongside Google DeepMind, emphasizes the company’s ability to meet the changing demands of cutting-edge research. Shadow continues to push the boundaries of dexterity and manipulation by fostering collaboration and by focusing on tailored solutions that meet practical needs. In the evolving field of robotics, Shadow’s dedication to understanding and responding to the unique challenges positions the company as an important player in robotics research.<blockquote>DEX-EE marks a significant milestone for Shadow Robot. While designed for our core research market, it embodies our vision of reaching beyond this to real-world application research. We remain committed to our fundamental research customers, but DEX-EE represents our expanding horizons. It's a step towards a future where our technology not only advances scientific understanding but also addresses practical challenges across industries, allowing us to make a broader impact while continuing to innovate within the research community.</blockquote>Gavin Cassidy, Head of Product Development at Shadow Robot.For more information about Shadow’s collaborative problem solving and innovative robotics, check out the <a href="https://www.shadowrobot.com/">Shadow Robot website</a>. The <a href="https://www.wevolver.com/article/dexterous-robotic-hands-part-1-unraveling-the-history-technology-and-applications-of-dexterous-robotic-hands">Shadow Robot Dexterous Hand series</a> also provides an in-depth look at Shadow’s commitment to advancing the field of robotics.<h3 dir="ltr">References</h3><ol><li>&nbsp;Shadow Robot. “Collaborative Working: A Case Study”. June 13, 2024. Accessed from <a href="https://www.shadowrobot.com/blog/collaboration-case-study/">https://www.shadowrobot.com/blog/collaboration-case-study/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “DEX-EE Overview” The most robust dexterous robot hand on the market”. June 6, 2024. Accessed from <a href="https://www.shadowrobot.com/blog/shadow-robot-hand-overview/">https://www.shadowrobot.com/blog/shadow-robot-hand-overview/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “Introducing DEX-EE the most robust dexterous robot hand on the market’. Accessed from <a href="https://www.shadowrobot.com/dex-ee/">https://www.shadowrobot.com/dex-ee/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “DEX-EE Overview: The most dexterous robot hand on the market”. <a href="https://www.shadowrobot.com/blog/shadow-robot-hand-overview/">https://www.shadowrobot.com/blog/shadow-robot-hand-overview/</a>&nbsp;</li><li>&nbsp;Google DeepMind. “Our latest advances in robot dexterity”. 12 September 2024. Accessed from <a href="https://deepmind.google/discover/blog/advances-in-robot-dexterity/">https://deepmind.google/discover/blog/advances-in-robot-dexterity/</a>&nbsp;</li></ol>

Discover how Shadow Robot’s DEX-EE gripper is advancing robotic dexterity with AI-driven precision.

Gripping the Future: Shadow Robot's DEX-EE and the Next Generation of Dexterous Manipulation

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries. There are also full-sequence diffusion models like Sora, which convert words into dazzling, realistic visuals by successively “denoising” an entire video sequence.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have proposed a simple change to the diffusion training scheme that makes this sequence denoising considerably more flexible.When applied to fields like computer vision and robotics, the next-token and full-sequence diffusion models have capability trade-offs. Next-token models can spit out sequences that vary in length. However, they make these generations while being unaware of desirable states in the far future — such as steering its sequence generation toward a certain goal 10 tokens away — and thus require additional mechanisms for long-horizon (long-term) planning. Diffusion models can perform such future-conditioned sampling, but lack the ability of next-token models to generate variable-length sequences.Researchers from CSAIL want to combine the strengths of both models, so they created a sequence model training technique called “Diffusion Forcing.” The name comes from “Teacher Forcing,” the conventional training scheme that breaks down full sequence generation into the smaller, easier steps of next-token generation (much like a good teacher simplifying a complex concept).<iframe width="640" height="360" src="https://www.youtube.com/embed/m4rhzR-h-JU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Diffusion Forcing found common ground between diffusion models and teacher forcing: They both use training schemes that involve predicting masked (noisy) tokens from unmasked ones. In the case of diffusion models, they gradually add noise to data, which can be viewed as fractional masking. The MIT researchers’ Diffusion Forcing method trains neural networks to cleanse a collection of tokens, removing different amounts of noise within each one while simultaneously predicting the next few tokens. The result: a flexible, reliable sequence model that resulted in higher-quality artificial videos and more precise decision-making for robots and AI agents.By sorting through noisy data and reliably predicting the next steps in a task, Diffusion Forcing can aid a robot in ignoring visual distractions to complete manipulation tasks. It can also generate stable and consistent video sequences and even guide an AI agent through digital mazes. This method could potentially enable household and factory robots to generalize to new tasks and improve AI-generated entertainment.“Sequence models aim to condition on the known past and predict the unknown future, a type of binary masking. However, masking doesn’t need to be binary,” says lead author, MIT electrical engineering and computer science (EECS) PhD student, and CSAIL member Boyuan Chen. “With Diffusion Forcing, we add different levels of noise to each token, effectively serving as a type of fractional masking. At test time, our system can “unmask” a collection of tokens and diffuse a sequence in the near future at a lower noise level. It knows what to trust within its data to overcome out-of-distribution inputs.”In several experiments, Diffusion Forcing thrived at ignoring misleading data to execute tasks while anticipating future actions.When implemented into a robotic arm, for example, it helped swap two toy fruits across three circular mats, a minimal example of a family of long-horizon tasks that require memories. The researchers trained the robot by controlling it from a distance (or teleoperating it) in virtual reality. The robot is trained to mimic the user’s movements from its camera. Despite starting from random positions and seeing distractions like a shopping bag blocking the markers, it placed the objects into its target spots.To generate videos, they trained Diffusion Forcing on “Minecraft” game play and colorful digital environments created within Google’s <a href="https://github.com/google-deepmind/lab">DeepMind Lab Simulator</a>. When given a single frame of footage, the method produced more stable, higher-resolution videos than comparable baselines like a Sora-like full-sequence diffusion model and ChatGPT-like next-token models. These approaches created videos that appeared inconsistent, with the latter sometimes failing to generate working video past just 72 frames.Diffusion Forcing not only generates fancy videos, but can also serve as a motion planner that steers toward desired outcomes or rewards. Thanks to its flexibility, Diffusion Forcing can uniquely generate plans with varying horizon, perform tree search, and incorporate the intuition that the distant future is more uncertain than the near future. In the task of solving a 2D maze, Diffusion Forcing outperformed six baselines by generating faster plans leading to the goal location, indicating that it could be an effective planner for robots in the future.Across each demo, Diffusion Forcing acted as a full sequence model, a next-token prediction model, or both. According to Chen, this versatile approach could potentially serve as a powerful backbone for a “world model,” an AI system that can simulate the dynamics of the world by training on billions of internet videos. This would allow robots to perform novel tasks by imagining what they need to do based on their surroundings. For example, if you asked a robot to open a door without being trained on how to do it, the model could produce a video that’ll show the machine how to do it.The team is currently looking to scale up their method to larger datasets and the latest transformer models to improve performance. They intend to broaden their work to build a ChatGPT-like robot brain that helps robots perform tasks in new environments without human demonstration.“With Diffusion Forcing, we are taking a step to bringing video generation and robotics closer together,” says senior author Vincent Sitzmann, MIT assistant professor and member of CSAIL, where he leads the Scene Representation group. “In the end, we hope that we can use all the knowledge stored in videos on the internet to enable robots to help in everyday life. Many more exciting research challenges remain, like how robots can learn to imitate humans by watching them even when their own bodies are so different from our own!”<hr>Chen and Sitzmann wrote the paper alongside recent MIT visiting researcher Diego Martí Monsó, and CSAIL affiliates: Yilun Du, a EECS graduate student; Max Simchowitz, former postdoc and incoming Carnegie Mellon University assistant professor; and Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at the Toyota Research Institute, and CSAIL member. Their work was supported, in part, by the U.S. National Science Foundation, the Singapore Defence Science and Technology Agency, Intelligence Advanced Research Projects Activity via the U.S. Department of the Interior, and the Amazon Science Hub. They will present their research at NeurIPS in December.

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries.

Combining next-token prediction and video diffusion in computer vision and robotics

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(0:50) - <a href="https://www.wevolver.com/article/ai-helps-detect-and-monitor-infrastructure-defects">Infrastructure Defect Detecting AI</a>Become a founding reader of our newsletter: <a href="http://read.thenextbyte.com/">http://read.thenextbyte.com/</a><hr><h2 data-attrid="title" data-local-attribute="d3bn" data-ved="2ahUKEwiBhLnmu4aAAxVmwjgGHcDWC38Q3B0oAXoECEgQEQ" id="isPasted">Transcript</h2>What's up, folks? As you know, cracks in buildings and roads and bridges can cause some serious damage if they're not found early enough. Today, we're talking about a team from EPFL that's gonna fix this problem using AI that can help detect cracks automatically and let us know when we need to do repairs. I think it's interesting, so let's jump into it.I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background.&nbsp;Daniel:&nbsp;What's up friends? Like we said, today we're talking about the future of crack detection in infrastructure and why that's so important. We're not just talking about like cracks and buildings and roads and, sometimes it makes them look ugly or ivy can grow in them. But the problem is that they can cause serious damage if not found early enough. The traditional methods we have for inspecting and looking for these cracks are really slow, really costly, really hard to do in dangerous areas. And there's a lot of human error involved with it as well. So, we're talking about some awesome research here where they're using AI to try and help find and monitor cracks, but they're even taking kind of a unique AI approach as well that doesn't require detailed pixel by pixel labeling either. So, I'm pretty stoked about this one.Farbod:&nbsp;Dude, same. I feel like infrastructure improvement has been talked about for so long. I remember being in middle school and politicians were promising the next infrastructure build and how like, our infrastructure is aging and we need to replace it. Well, before this part, I was just kind of curious. I'm like, let's pull up some stats just about the US alone, right? I don't know if you knew this, but a freeway bridge inspection can take, can cost about $5,000 and take about eight hours to do. And that doesn't take into account the detour traffic causing people pains to get to work, et cetera, et cetera. And on top of that, the way we do grading of these cracks tends to be kind of subjective. So, the person that's doing it might grade it more, I don't know, liberally than the person that did last year. So that person might've thought things were going bad. This person says everything's good. So that kind of creates a weird situation where we can't track how bad it's getting super well.Daniel:&nbsp;Or can you trust the data that comes from these expectations? When you spend this much money and you spend this much time, you may not be even able to trust the data that comes out of it.Farbod: And to just make matters worse, there's something like one out of every three bridges in the US needs repair or replacement. So, one, you don't know if things are actually bad. Two, it seems like at least one third are kind of bad, maybe more. So not super comfortable about driving over a bridge at this point, but I saw something interesting about drones. Drones have had great improvement over the past couple of years. People were saying on average it's about 40% cheaper. So that makes it more affordable for local governments to take them on. But still, I'm guessing no one is super excited about shelling out over $3,000 to inspect every bridge they got. So yeah, sticky situation to be in.Daniel:&nbsp;Well, and just a quick throwback here. I would say like in the broader realm of infrastructure detection, both of our capstone projects, when we were graduating our undergrad program was related to using robotics and automation to try and detect issues with infrastructure before they progress to something that becomes a catastrophic failure. And that's not to say that we're experts in this topic at all. I think we were just at the tip of the iceberg here, but it is interesting to note that in separate graduating classes for about, and I both had this capstone project because it's such a big problem and we're still sitting here saying, maybe now we finally got some AI solution, which can help with this in a more robust manner, which is exciting for both of us, I think on a personal note, to see this field start to progress.Farbod: Dude, absolutely. Back when we were young men, which we're not anymore, at least my team, we were toying around with the idea of using AI, but even then, there wasn't a whole lot of excitement around it. Specifically, we were looking at corrosion, but now it looks like the game has completely shifted. We're going to EPFL for this episode, but if we can transition to the secret sauce, these folks are talking about how AI can actually completely remove the human element here and in the best way possible. We've been talking about subjectivity, how that can result in variance from person to person, and if a crack is getting better or worse. These folks are saying, hey, we can just train a very simple model that can tell us if there's a crack or if there's no crack, like binary. Once it learns how to detect the crack well, then it's just a matter of saying “cool, where are you seeing the crack and how is it changing over time?” Which in the world of like, you know, computer vision is a fairly simple and straightforward analysis. And the main application they're looking at for this is railroads, right? They're saying, railroads are used very often if there's cracks that grow that can cause catastrophic failures. So how do we maintain this and like monitor it on a regular basis? Well, they're like trains are passing through this bridges and stuff. What if they were the ones that were doing the analysis completely autonomously, no added costs, no added time, no requirement for shut down.Daniel: It's not sending people on harnesses to rappel from these. Honestly, you should just check on the link in the show notes, click for no other reason than to see the beautiful photo of this train on a railroad bridge going across a canyon in Switzerland where, and I wanted to confirm this, I wasn't 100% sure. The Matterhorn-Gothard bond network. I'm pretty sure Nellie and I took this train on our honeymoon, which is pretty cool.Farbod: Wow. That's special.Daniel:&nbsp;If not something very similar because all of Switzerland is full of beautiful trains and bridges and mountains, but I'm pretty sure we went on this exact railroad section where they're starting testing, but yeah, if for no other reason, check out this photo, but for me to get back to my main point, this is a bridge suspended several stories above the bottom of a canyon. It would cost a lot and you'd have to pay someone a lot to be able to visually inspect all of that manually, like as a person by hand. And what they're trying to do now is instead send a drone, but not in the form of the quadcopter drone that you think of probably when we say drone. The solution is, send a drone rail car, right? An instrumented rail car down this railroad and try and collect a bunch of data on, where are their cracks? Are they propagating? Are they staying the same? Which ones need to be addressed? Continuing to understand and map the ever evolving, let's say ecosystem around all these bridges and this infrastructure and understand when there's issues and indicate whether they need further inspection or whether they need immediate repair, et cetera.Farbod:&nbsp;Dude, totally agree. And the solution seems so simple when you say it out loud, but it really is, I don't know, I guess the best solutions are the simple ones. And what it made me think of is obviously this is great for the railroad system. But then we've been talking about like as a commuter that's going in a tunnel. That one tunnel in Boston I always hate it gives me the creeps when I'm going through it or when I'm going through bridges to Maryland and stuff like that. It would be great if there was something that could do the analysis there and in my mind, I'm like we're definitely a long way away from having autonomous vehicles doing these trips for us. But Uber and Lyft, they're drivers that are operating every single day that are doing these normal routes that people take to go to work and whatnot. I wonder if there was an opportunity there to say, hey, while you're doing this, mount this device on the back of your car so that whenever your GPS detects you're going from the beginning of the tunnel to the back of the tunnel, you do a scan and then that data uploads to a cloud somewhere and we can do analysis over time to see if we need to patch any cracks or what?Daniel: I'm like wondering if you read my mind here. I went to buses, because I'm like, the department of transportation in a city may also own the buses, right? So, they're in charge of the bridges. They're also in charge of the buses. Wonder if they can just put a bunch of instruments on top of a bus. In my mind, it's like the closest analogy to a rail car. Put a bunch of this infrastructure on top of a bus and this instrumentation, see if you can analyze the infrastructure around us using that same method as well. But I also want to jump a little bit into the specific AI methods they're using. And I'm not an expert here by any means, but I took some notes from the paper that I think are interesting. That there have previously tried to do AI classification of images, but it's too computationally intensive to justify the cost. So, if you were to use AI to try and evaluate millions of photos of an entire bridge. It may actually cost more than actually sending a human down there to do it. Because one of the things that I had to do is classify each pixel by pixel, trying to understand whether it was light or if it's dark and look at contrast to understand what is a crack, then it classifies a crack and that took a lot of computational energy. One of the things that they did is they, they train their AI ahead of time with images labeled as crack or no crack. And then it uses explainable AI is what they say is their unique method here to try and look at a new photo and circle, basically highlight a cracked area and create a map showing where cracks are and where they aren't. And essentially what it's doing is it's just taking photo after photo after photo and trying to trace the area around where it thinks the crack is. And over a period of time, it's trying to look at the same photo from the same location the last time this train came through and look at whether this crack got bigger or if it stayed the same. And if cracks are serially getting bigger and bigger and bigger from these images, it warrants sending someone down there, an expert down there to take a look at it. And one of the things that they mentioned is this cut the processing and labeling effort, as opposed to previous AI methods by about 90%, which makes it cheaper and faster and worthwhile to implement in an autonomous manner. It's not quite as accurate as these full AI models, but they said it can still help directionally track crack growth. And they think that they can hone it essentially and get rid of more of the area that they're experiencing right now. But essentially the game plan I guess is to go test this out in the field, get more learnings, understand more but in practice even if it has the same performance that it has today you can use this as kind of like a non-invasive constant monitoring where it looks at the rails, it looks at the ties, it looks at the retaining walls and tries to understand if there's anything wrong with the railroad. And you could have this on every train or one of every 10 trains passing through this railroad. Even though I think right now it's got 35% error, which again, it sounds high compared to most AI models, but if you're telling me, you can tell me 65% of the time, whether a crack's growing or not. If I've got two or three trains that pass through over a series of a week and show that this crack indicates that this crack may be growing, the playbook here is not to say, all right, let's shut down the bridge and tear it down. The playbook here could be, let's go send one of those expert inspectors that we used to have doing this job full time and had to shut down the railroad to do now that we know that there's something that's worth further evaluation, let's send one of those crack experts down there to understand what's going on and see whether it needs repair.Farbod: And this is a great example, at least in my opinion of complimentary automation, right? Like here's a task that was bad for people to do manually. And some places are like in America, it's just straight up not done at times. Now you have automation that's taking care of the tedious part. And then you have the human that's taking on the challenging part, which is, hey, we're like 90% sure there is a crack, please go out there and now characterize it and give us feedback on if it's catastrophic or if, you know, it's a simple crack that needs to be fixed, etc, etc. So, I feel like this is the best use of every technology or every resource, think of the human expert as a resource. And this is kind of going against the AI is going to get rid of all jobs mentality.Daniel: Well, and I think what we, the intended end here is to be able to do continuous monitoring, collect data all the time to understand which regions are susceptible to cracks, and then use that as a heat map essentially to show someone who's again, got expertise in this field to go, this is the area that you need to inspect. This is the area that you need to monitor. This is the area that may need potential repair as opposed to the other portions of track that maybe didn't need as much attention and they were getting it before because you had to go check every single mile of this thing manually. And maybe you used to only check one, every single mile of this thing manually, one out of every five years. And now we could have a train passing on it five times a day that tells us how the railroad's doing. So, I appreciate here, like you said, it's a complimentary solution between automation and humans. But it also gets us an immediate benefit, which is a lot more data, a lot more consistently on how our infrastructure is doing and when it needs to be repaired.Farbod: Totally agree, man. You want to do us a favor and wrap up the episode?Daniel:&nbsp;I will. What if we could stop big cracks in buildings and roads before they cause big problems, I'm thinking of bridges collapsing or bridges needing to be shut down because they need repair or rip, cars getting derailed. These are big problems and we're talking today about a new AI method that helps find and track these cracks automatically without needing a ton of expensive detailed data. Essentially, they're getting the job done with 90% less effort. That's what their new tech promises. And they use this system called explainable AI that spots cracks with simple image labels, saving time and money, keeping an eye on crack growth to let experts know where they need to address cracks and where we might need repairs in the future. I think it's a lot like, I don't know, Google Maps navigation or some sort of traffic heat map, but it tells us where most areas are prone to cracks and where we need repairs so that folks know where they need to focus their efforts.Farbod:&nbsp;Money.Daniel:&nbsp;Thanks, my dude.Farbod:&nbsp;You killed it, dude. All right, folks, thank you so much for listening. And as always, we'll catch you in the next one.Daniel:&nbsp;Peace.<hr>As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank" rel="nofollow">&nbsp;Apple</a> podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank" rel="nofollow">&nbsp;Spotify</a>, and most of your favorite podcast platforms!--The Next Byte: We're two engineers on a mission to simplify complex science &amp; technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, &amp; much more), all while keeping it entertaining &amp; engaging along the way.

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems

Podcast: Saving America's Collapsing Infrastructure

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/35ecb899-3a2c-45e8-9d17-7a8c87d944ba?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Most doctors go into medicine because they want to help patients. But today’s health care system requires that doctors spend hours each day on other work — searching through electronic health records (EHRs), writing documentation, coding and billing, prior authorization, and utilization management — often surpassing the time they spend caring for patients. The situation leads to physician burnout, administrative inefficiencies, and worse overall care for patients.Ambience Healthcare is working to change that with an AI-powered platform that automates routine&nbsp;tasks for clinicians before, during, and after patient visits."We build co-pilots to give clinicians AI superpowers," says Ambience CEO Mike Ng MBA ’16, who co-founded the company with Nikhil Buduma ’17. "Our platform is embedded directly into&nbsp;EHRs to free up clinicians to&nbsp;focus on what matters most, which is providing the best possible patient care."Ambience’s suite of products handles pre-charting and real-time AI scribing, and assists with navigating the thousands of rules to select the right insurance billing codes. The platform can also send after-visit summaries to patients and their families in different languages to keep everyone informed and on the same page.Ambience is already being used across roughly 40 large institutions such as UCSF Health, the Memorial Hermann Health System, St. Luke’s Health System, John Muir Health, and more.&nbsp;Clinicians leverage Ambience in dozens of languages and more than&nbsp;100 specialties and subspecialties, in settings&nbsp;like the emergency department, hospital inpatient settings,&nbsp;and the oncology ward.The founders say clinicians using Ambience save two to three hours per day on documentation,&nbsp;report lower levels of burnout, and develop higher-quality relationships with their patients.<h2>From problem to product to platform</h2>Ng worked in finance until getting an up-close look at the health care system after he fractured his back in 2012. He was initially misdiagnosed and put on the wrong care plan, but he learned a lot about the U.S. health system in the process, including how the majority of clinicians’ days are spent documenting visits, selecting billing codes, and completing other administrative tasks. The average clinician only spends 27 percent of their time on direct patient care.In 2014, Ng decided to enter the MIT Sloan School of Management. During his first week, he attended the “t=0” celebration of entrepreneurship hosted by the Martin Trust Center for MIT Entrepreneurship, where he met Buduma.&nbsp;The pair became fast friends, and they ended up taking classes together including 15.378 (Building an Entrepreneurial Venture) and 15.392 (Scaling Entrepreneurial Ventures).“MIT was an incredible training ground to evaluate what makes a great company and learn the foundations of building a successful company,” Ng says.Buduma had gone through his own journey to discover problems with the health care system. After immigrating to the U.S. from India as a child and battling persistent health issues, he had watched his parents struggle to navigate the U.S. medical&nbsp;system. While completing his bachelor’s degree at MIT, he was also steeped in the AI research community and wrote an early textbook on modern AI and deep learning.In 2016, Ng and Buduma&nbsp;founded&nbsp;their first company in San Francisco — Remedy Health — which operated its own AI-powered health care platform. In the process of hiring clinicians, taking care of patients, and implementing technology themselves, they developed an even deeper appreciation for the challenges that health care organizations face.During that time, they also got an inside look at advances in AI. Google’s Chief Scientist Jeff Dean, a major investor in Remedy and now in Ambience, led a research group inside of Google Brain to invent the transformer architecture. Ng and Buduma say they were among the first to put transformers into production to support their own clinicians at Remedy. Subsequently, several of their friends and housemates went on to start the large language model group within OpenAI. Their friends’ work formed the research foundations that ultimately led to ChatGPT. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;“It was very clear that we were at this inflection point where we were going to have this class of general-purpose models that were going to get exponentially better,” Buduma says. “But I think we also noticed a big gap between those general-purpose models versus what actually would be robust enough to work in a clinic. Mike and I decided in 2020 that there should be a team that specifically focused on fine-tuning these models for health&nbsp;care and medicine.”The founders started Ambience by building an AI-powered scribe that works on phones and laptops to record the details of doctor-patient visits in a HIPAA-compliant system that preserves patient privacy. They quickly saw that the models needed to be fine-tuned for each area&nbsp;of medicine, and they slowly expanded specialty coverage one by one in a multiyear process.The founders also realized their scribes needed to fit within back-office operations like insurance coding and billing.“Documentation isn’t just for the clinician, it's also for the revenue cycle team,” Buduma says. “We had to go back and rewrite all of our algorithms to be coding-aware. There are literally tens of thousands of coding rules that change every year and differ by specialty and contract type.”From there, the founders built out models for clinicians to make referrals and to send comprehensive summaries of visits to patients.“In most care settings before Ambience, when a patient and their family left the clinic, whatever the patient and their family wrote down was what they remembered from the visit,” Buduma says. “That’s one of the features that physicians love most, because they are trying to create the best experience for patients and their families. By the time that patient is in the parking lot, they already have a really robust, high-quality summary of exactly what you talked about and all the shared decision-making around your visit in their portal.”<h2>Democratizing health care</h2>By improving physician productivity, the founders believe they’re helping the health care&nbsp;system manage a chronic shortage of clinicians that’s expected to grow in coming years.“In health care, access is still a huge problem,” Ng says. “Rural Americans have a 40 percent higher risk of preventable hospitalization, and half of that is attributed to a lack of access to specialty care.”With Ambience already helping health systems manage razor-thin margins by streamlining administrative tasks,&nbsp;the founders have a longer-term vision to help&nbsp;increase access to the best clinical information across the country.“There’s a really exciting opportunity to make expertise at some of the major academic medical centers more democratized across the U.S.,” Ng says. “Right now, there’s just not enough specialists in the U.S. to support our rural populations. We hope to help scale the knowledge of the leading specialists in the country through an AI infrastructure layer, especially as these models become more clinically intelligent.”

Ambience is being used across roughly 40 health systems in the U.S. by clinicians in over 100 subspecialties. Image: iStock

machine learning

Profiles