Computer Vision

Fine-tuning large language models adapts pre-trained models to specific tasks or domains using tailored datasets, while Retrieval-Augmented Generation (RAG) combines retrieval systems with generative models to dynamically incorporate external, up-to-date knowledge into outputs.

RAG vs Fine-Tuning: Differences, Benefits, and Use Cases Explained

For roboticists, one challenge towers above all others: generalization – the ability to create machines that can adapt to any environment or condition. Since the 1970s, the field has evolved from writing sophisticated programs to using deep learning, teaching robots to learn directly from human behavior. But a critical bottleneck remains: data quality. To improve, robots need to encounter scenarios that push the boundaries of their capabilities, operating at the edge of their mastery. This process traditionally requires human oversight, with operators carefully challenging robots to expand their abilities. As robots become more sophisticated, this hands-on approach hits a scaling problem: the demand for high-quality training data far outpaces humans' ability to provide it.Now, a team of MIT CSAIL researchers have developed a novel approach to robot training that could significantly accelerate the deployment of adaptable, intelligent machines in real-world environments. The new system, called "LucidSim," uses recent advances in generative AI and physics simulators to create diverse and realistic virtual training environments, helping robots achieve expert-level performance in difficult tasks without any real-world data.<iframe width="640" height="360" src="https://www.youtube.com/embed/fgnJQrvTj70?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>LucidSim combines physics simulation with generative AI models, addressing one of the most persistent challenges in robotics: transferring skills learned in simulation to the real world. “A fundamental challenge in robot learning has long been the ‘sim-to-real gap’ – the disparity between simulated training environments and the complex, unpredictable real world,” says MIT CSAIL postdoctoral associate Ge Yang, a lead researcher on LucidSim. “Previous approaches often relied on depth sensors, which simplified the problem but missed crucial real-world complexities.”The multipronged system is a blend of different technologies. At its core, LucidSim uses large language models to generate various structured descriptions of environments. These descriptions are then transformed into images using generative models. To ensure that these images reflect real-world physics, an underlying physics simulator is used to guide the generation process.&nbsp;<h2>The Birth of an Idea: From Burritos to Breakthroughs</h2>The inspiration for LucidSim came from an unexpected place: a conversation outside Beantown Taqueria in Cambridge. ”We wanted to teach vision-equipped robots how to improve using human feedback. &nbsp;But then, we realized we didn't have a pure vision-based policy to begin with,” says Alan Yu, an undergraduate student at MIT and co-lead on LucidSim. "We kept talking about it as we walked down the street, and then we stopped outside the taqueria for about half an hour. That's where we had our moment."To cook up their data, the team generated realistic images by extracting depth maps, which provide geometric information, and semantic masks, which label different parts of an image, from the simulated scene. They quickly realized, however, that with tight control on the composition of the image content, the model would produce similar images that weren’t different from each other using the same prompt. So, they devised a way to source diverse text prompts from ChatGPT.&nbsp;This approach, however, only resulted in a single image. To make short, coherent videos which serve as little “experiences” for the robot, the scientists hacked together some image magic into another novel technique the team created, called “Dreams In Motion (DIM).” The system computes the movements of each pixel between frames, to warp a single generated image into a short, multi-frame video. Dreams In Motion does this by considering the 3D geometry of the scene and the relative changes in the robot’s perspective.&nbsp;"We outperform domain randomization, a method developed in 2017 that applies random colors and patterns to objects in the environment, which is still considered the go-to method these days," says Yu. &nbsp;"While this technique generates diverse data, it lacks realism. LucidSim addresses both diversity and realism problems. It’s exciting that even without seeing the real world during training, the robot can recognize and navigate obstacles in real environments."&nbsp;The team is particularly excited about the potential of applying LucidSim to domains outside quadruped locomotion and parkour, their main testbed. One example is mobile manipulation, where a mobile robot is tasked to handle objects in an open area, and also, color perception is critical. “Today, these robots still learn from real-world demonstrations,” says Yang. “Although collecting demonstrations is easy, scaling a real-world robot teleoperation setup to thousands of skills is challenging because a human has to physically set up each scene. We hope to make this easier, thus qualitatively more scalable, by moving data collection into a virtual environment.”&nbsp;The team put LucidSim to the test against an alternative, where an expert teacher demonstrates the skill for the robot to learn from. The results were surprising: robots trained by the expert struggled, succeeding only 15 percent of the time – and even quadrupling the amount of expert training data barely moved the needle. But when robots collected their own training data through LucidSim, the story changed dramatically. Just doubling the dataset size catapulted success rates to 88 percent. "And giving our robot more data monotonically improves its performance – eventually, the student becomes the expert," says Yang.“One of the main challenges in sim-to-real transfer for robotics is achieving visual realism in simulated environments,” says Stanford University Assistant Professor of Electrical Engineering Shuran Song, who wasn’t involved in the research. “The LucidSim framework provides an elegant solution by using generative models to create diverse, highly realistic visual data for any simulation. This work could significantly accelerate the deployment of robots trained in virtual environments to real-world tasks.”From the streets of Cambridge to the cutting edge of robotics research, LucidSim is paving the way toward a new generation of intelligent, adaptable machines – ones that learn to navigate our complex world without ever setting foot in it.<hr>Yu and Yang wrote the paper with four fellow CSAIL affiliates: mechanical engineering postdoc Ran Choi; undergraduate researcher Yajvan Ravan; John Leonard, Samuel C. Collins Professor of Mechanical and Ocean Engineering in the MIT Department of Mechanical Engineering; and MIT Associate Professor Phillip Isola. Their work was supported, in part, by a Packard Fellowship, a Sloan Research Fellowship, the Office of Naval Research, Singapore’s Defence Science and Technology Agency, Amazon, MIT Lincoln Laboratory, and the National Science Foundation Institute for Artificial Intelligence and Fundamental Interactions. The researchers will present their work at the Conference on Robot Learning (CoRL) in early November.

A team of MIT CSAIL researchers have developed a novel approach to robot training that could significantly accelerate the deployment of adaptable, intelligent machines in real-world environments.

Can robots learn from machine dreams?

A less wasteful way to train large language models, such as the GPT series, finishes in the same amount of time for up to 30% less energy, according to a new study from the University of Michigan.The approach could save enough energy to power <a href="https://www.eia.gov/energyexplained/use-of-energy/electricity-use-in-homes.php" target="_blank" rel="noreferrer noopener">1.1 million U.S. homes</a> in 2026, based on Wells Fargo’s projections of <a href="https://www.marketwatch.com/story/ai-could-demand-a-shocking-amount-of-electricity-check-out-this-chart-e91e306d" target="_blank" rel="noreferrer noopener">AI power demand</a>. It could also take a bite out of the International Monetary Fund’s prediction that data centers could account for <a href="https://www.imf.org/en/Blogs/Articles/2024/08/15/carbon-emissions-from-ai-and-crypto-are-surging-and-tax-policy-can-help" target="_blank" rel="noreferrer noopener">1.2% of the world’s carbon emissions by 2027</a>—and the <a href="https://theconversation.com/ais-excessive-water-consumption-threatens-to-drown-out-its-environmental-contributions-225854">water demands</a> that come with that energy use.Some experts say that these costs could be outweighed by environmental benefits. They argue that AI could be a <a href="https://news.un.org/en/story/2023/11/1143187" target="_blank" rel="noreferrer noopener">“game changer” for fighting climate change</a> by identifying ways to optimize supply chains and the grid, manage our energy needs, and improve research on climate change. Still, that doesn’t excuse squandering energy, and some of the power used to train AI has zero impact on training time and model accuracy.“Why spend something when there’s no point?” said <a href="https://eecs.engin.umich.edu/people/chowdhury-mosharaf/" target="_blank" rel="noreferrer noopener">Mosharaf Chowdhury</a>, an associate professor of computer science and engineering and the corresponding author of the study presented at the <a href="https://sigops.org/s/conferences/sosp/2024/index.html" target="_blank" rel="noreferrer noopener">30th Symposium on Operating Systems Principles</a> last Monday.“We can’t keep building bigger and bigger data centers because we won’t have the power to run them,” said Chowdhury. “If we can reduce the energy consumed by AI, we can reduce AI’s carbon footprint and cooling requirements and allow for more computation to fit within our current energy constraints.”The energy waste is created when AI training is unequally divided between GPUs, which are computer processors specialized for large data and graphics applications. Although it opens the door for waste, splitting the work is necessary for processing huge datasets.“AI models today are so large, they cannot fit inside a single computer processor,” said <a href="https://jaewonchung.me/about" target="_blank" rel="noreferrer noopener">Jae-Won Chung</a>, a doctoral student in computer science and engineering and the first author of the study. “They need to be divided into tens of thousands of processors to be trained, but dividing the models in perfectly equal sizes across all processors is practically impossible.”The training jobs are so difficult to evenly split up because some tasks need to be grouped together on the same processor—like how each installment of a book series will be grouped together in an organized shelf. Depending on how the tasks are grouped, some processors might get stuck with the AI-training equivalent of the Encyclopedia Britannica while others get assigned a fantasy trilogy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731630851359-AI_waste_Infographic-768x1470.jpg" style="width: 100%px;" class="fr-fic fr-dib">Infographic credit: Rachel Voigt, Michigan Engineering.Because current training methods run each processor at top speed, processors with a lighter load will finish their calculations before other processors. This doesn’t speed up training, which isn’t complete until every processor finishes its job—but it is wasteful because faster calculations require more energy. In addition, problems such as faulty hardware or network delays create energy waste by slowing down a single processor’s computing speed.To save energy, the researchers developed a software tool, called&nbsp;<a href="https://ml.energy/zeus/research_overview/perseus/" target="_blank" rel="noreferrer noopener">Perseus</a>, that identifies a critical path, or a series of subtasks that will take the longest time to complete. Then, Perseus slows down processors that aren’t on the critical path so that they all finish their jobs around the same time—eliminating unnecessary power use.“Reducing the power cost of AI can have important implications for equitable AI access,” said Chowdhury. “If a country doesn’t have enough power to run a big model, they might need to use services from far away, or be stuck running smaller, less accurate models. This gap could further perpetuate disparity between different communities.”The team tested Perseus by training GPT-3, three other large language models and one computer vision model.Perseus is an open-sourced tool available as part of&nbsp;<a href="https://ml.energy/zeus/" target="_blank" rel="noreferrer noopener">Zeus</a>, a tool for measuring and optimizing AI energy consumption.&nbsp;

Smarter use of processor speeds saves energy without compromising training speed and performance.

Up to 30 percent of the power used to train AI is wasted. Here's how to fix it.

In this episode, we explore how AI co-pilots are equipping doctors with powerful tools to enhance decision-making and patient care and discuss how it could impact you - a potential patient - in the not distant future.

Podcast: Will AI Assistants Make Your Doctor Better?

Manufacturing defect rates have risen with metal fabrication facilities facing unprecedented challenges in maintaining quality control amid increasing operational complexity.&nbsp;As production lines become more sophisticated and quality standards more stringent, traditional inspection methods must be revised for modern manufacturing demands.Matroid is transforming this landscape by democratizing computer vision technology, making advanced automation accessible to industry professionals without coding expertise. Through its innovative platform, Matroid enables manufacturing teams to implement sophisticated defect detection and process monitoring solutions that can be deployed within minutes rather than months.The impact is significant in welding applications, where Matroid's computer vision technology can evaluate multiple quality parameters simultaneously - from porosity and spatter to burn-through - with the precision of a weld engineering expert.&nbsp;This scalable approach to quality control is proving essential as manufacturers face mounting pressure to maintain precision while managing costs.We will explore in detail how Matroid's computer vision technology is changing the landscape of metal manufacturing, examining its technical capabilities, practical applications, and the tangible benefits it brings to production environments. &nbsp;For decision-makers, we'll look at how this technology sets new standards in automated quality control and process optimization.<h2 dir="ltr">Industry Challenges and Opportunities in Metal Manufacturing</h2>The metal manufacturing industry faces unprecedented pressures, with many organizations' quality control costs consuming <a href="https://www.weboccult.com/blog/computer-vision-for-quality-control-in-manufacturing-a-comprehensive-guide">15-20%</a> of annual sales revenue. Traditional manual inspection methods, while common, are increasingly inadequate for modern manufacturing demands, showing inconsistency rates increase due to operator fatigue and subjective assessments.According to the US. Census Bureau <a href="https://www.census.gov/library/stories/2021/09/manufacturing-continues-to-be-among-top-five-largest-employment-sectors.html">study</a> shows that production facilities struggle with a critical shortage of skilled workers, with nearly one-fourth of the manufacturing workforce aged 55 or older. This demographic shift and early retirement trends post-pandemic have created significant gaps in quality control expertise and operational efficiency.<h3 dir="ltr">The Role of Automation and Computer Vision</h3>Computer vision technology is emerging as a transformative solution, capable of analyzing images and video from any imaging technology, spanning resolutions from low (VGA) to ultra-high (gigapixel) and across various spectra, including visible light, infrared, and x-ray. &nbsp;This enables the detection of the smallest and most challenging defects in manufacturing processes, enhancing quality control and operational efficiency.Unlike traditional rule-based inspection systems, modern computer vision &amp; AI solutions can detect and classify multiple defect types simultaneously, from surface imperfections to welding anomalies.Automated inspection systems are valuable in welding applications, where they can evaluate multiple quality parameters - including porosity, spatter, burn-through, edge welds, and cold welds - with consistent precision.&nbsp;Computer vision &amp; AI address the skilled labor shortage and significantly improve inspection accuracy, with some implementations showing up to <a href="https://cloud.google.com/blog/products/ai-machine-learning/improve-manufacturing-quality-control-with-visual-inspection-ai">10x</a> improvement in defect detection compared to conventional methods, which leads to dramatic cost savings in cost per part and rework efforts and also improves production throughput.<h2 dir="ltr">Matroid's Vision and Technological Focus</h2>Founded in 2016 by Stanford professor and founding member of Databricks Reza Zadeh, Matroid has established itself as a leader in enterprise-level computer vision &amp; AI software platform, backed by <a href="https://growjo.com/company/Matroid">$33.5M</a> in funding from prominent investors, including NEA, Energize Ventures, and Intel Capital.&nbsp;The Matroid <a href="https://www.matroid.com/">platform</a> domain experts from manufacturing to rapidly deploy and adapt custom detectors that can leverage camera systems' imagery/ video types to detect and classify defects, monitor manual assembly operations, verify SOPs are followed, and provide continuous cycle-time analysis with unprecedented precision across various production environments.<h3 dir="ltr">Custom Detectors and Real-Time Monitoring</h3>Matroid's platform distinguishes itself through its no-code software to build and deploy detector systems, allowing manufacturing teams to develop specialized detection models through an intuitive point-and-click interface.&nbsp;These vision detectors can simultaneously evaluate multiple quality parameters in metal processing applications - from weld porosity and spatter to surface defects and assembly errors - while maintaining consistent accuracy across varying conditions.The platform's camera-agnostic system integrates seamlessly with existing infrastructure, supporting everything from standard industrial cameras to specialized equipment like borescopes, microscopes, and X-ray systems.&nbsp;Real-time alerts and analytics are delivered directly with production systems (PLCs, SCADA, MES, ERP) and mobile devices. This enables immediate response to the right people and systems at the right time to detect anomalies and maintain digital traceability throughout manufacturing.<h2 dir="ltr">How Matroid’s Computer Vision Powers Welding Applications</h2><iframe src="https://www.youtube.com/embed/20FpAGGAqmk?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" style="width: 640px; height: 460px;"></iframe> <h3 dir="ltr">Detecting Defects in Metal Manufacturing</h3>Matroid's computer vision enables quality control oversight across the entire metal manufacturing process. The <a href="https://www.matroid.com/automated-visual-anomoly-detection/">system</a> identifies structural defects at the casting stage, like cracks, porosity, cold flow, and hot tears.&nbsp;Within the stamping processes, Matroid detectors are used to replace manual visual inspection by automating the inspection of stamped parts for tears and splits. &nbsp;This gives near real-time feedback to production engineers on how the presses are performing and gives insights into potential adjustments to prevent further defects.&nbsp;In the final inspection, the platform performs comprehensive checks to ensure consistently high quality, effectively reducing returns and warranty claims.<h3 dir="ltr">Computer Vision in Welding</h3>Matroid's technology monitors operations such as spot welding, MIG welding, and continuous seam welding through real-time monitoring and instant defect detection. The system evaluates multiple quality parameters simultaneously, identifying common issues like porosity, spatter, cold welds, edge welds, and burn-through, just as a weld engineering expert would.&nbsp;The platform's deep learning capabilities enable it to detect defects as a person could through visual media (images/video). The solutions are designed for rugged &amp; dynamic manufacturing environments.&nbsp;<h3 dir="ltr">Root Cause Analysis and Real-Time Adjustments</h3>Matroid's <a href="https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control">solution</a> provides detailed defect classification and analysis through its integrated system. The system immediately alerts operators when issues are detected through direct integration with production systems, QMS, and mobile devices. &nbsp;Production teams can take corrective action when they have near real-time classification information and analytics; for example, when burn-through is detected on welds, the welding engineer can make more effective adjustments to robotic welding systems faster. &nbsp;This reduces rework and defect risks and improves production throughput.&nbsp;The platform operates 24/7, providing continuous monitoring and real-time insights into production operations. Through its historical data analysis capabilities, manufacturing teams can capture actionable insights and implement zero-defect initiatives.<h2 dir="ltr">Real-World Applications and Benefits&nbsp;</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1731927333487-1731927333487.png" id="isPasted" class="fr-fic fr-dii"><h3 dir="ltr" id="isPasted">Traditional Methods vs. Computer Vision &amp; AI for Welding</h3>Traditional welding inspection methods, relying on manual or basic automated systems, provide only binary pass/fail results. These systems need help to meet modern manufacturing demands, where quality teams need detailed defect analysis and real-time process adjustments.&nbsp;Manual inspections introduce delays between defect occurrence and detection, while inspector fatigue and varying experience levels lead to inconsistent results. Computer vision &amp; AI solutions powered by deep learning technology transform this landscape by enabling real-time identification and classification of specific weld defects.&nbsp;Matroid's system automatically detects and categorizes issues such as porosity, burn-through, and spatter, immediately notifying weld engineers for rapid response. This immediate feedback loop allows manufacturing teams to quickly identify root causes and implement real-time process adjustments.The deployment of edge or on-premises installations ensures minimal latency for real-time monitoring while maintaining data security. This approach enables manufacturing teams to shift from reactive quality control to proactive process optimization.&nbsp;Through consistent quality assessment and automated defect classification, companies can reduce inspection time, minimize waste, and improve overall welding quality. The system's integration with existing production processes creates a seamless transition from traditional inspection methods to advanced computer vision &amp; AI solutions.<h2 dir="ltr">The Deployment of Matroid’s Solution</h2><h3 dir="ltr">Easy Integration and Deployment</h3>Matroid's computer vision &amp; AI platform integrates seamlessly with existing manufacturing infrastructure through multiple deployment options. The system can be implemented as downloadable software for on-premises installation or as a Cloud software service, allowing manufacturers to choose the deployment method that best suits their operational needs.&nbsp;This flexibility enables manufacturing teams to maintain data security while leveraging advanced detection capabilities. &nbsp;Matroid’s platform makes integration into various control systems, MES, QMS, ERP, and databases easy so that production can work with real-time data and has enhanced traceability of the products produced.&nbsp;<h3 dir="ltr">Compatibility with Multiple Camera Technologies</h3>The platform's camera-agnostic approach sets it apart in the manufacturing sector. Matroid works with existing and new vision systems, supporting any image-or-video-based camera regardless of resolution (VGA - giga-pixel) or spectra (visible, infrared, X-ray, etc.).This allows manufacturers to reuse their existing machine vision cameras and use the right imaging technology for the application at hand while maintaining the ability to detect visual defects. The system connects effortlessly with industrial cameras, supporting continuous 24/7 monitoring and real-time insights into manufacturing processes.<h2 dir="ltr">Future of Welding with Computer Vision &amp; AI</h2>Integrating computer vision technology in welding operations represents a significant leap forward in manufacturing quality control. Matroid's platform transforms traditional inspection methods by enabling real-time defect detection and detailed classification, which enables immediate process adjustments.&nbsp;By combining deep learning capabilities with enterprise-grade deployment options, manufacturers can achieve consistent quality while reducing operational costs. The platform's ability to empower domain experts in manufacturing, work with existing infrastructure, and support multiple camera technologies makes it a practical solution for facilities of all sizes.<h2 dir="ltr">Ready to make your welding operations efficient and reliable with advanced computer vision?</h2>Discover how Matroid's platform can enhance your quality control processes and drive real-time improvements. Visit Matroid's <a href="https://www.matroid.com/">website</a> to explore their innovative solutions for metal manufacturing.<h2 dir="ltr">References</h2><ol><li dir="ltr"><a href="https://info.matroid.com/visual-inspection-computer-vision-software">https://info.matroid.com/visual-inspection-computer-vision-software</a></li><li dir="ltr"><a href="https://www.dayjob.com.au/metal-fabrication-challenges-with-sustainability/">https://www.dayjob.com.au/metal-fabrication-challenges-with-sustainability/</a></li><li dir="ltr"><a href="https://www.matroid.com/automated-visual-anomoly-detection/">https://www.matroid.com/automated-visual-anomoly-detection/</a></li><li dir="ltr"><a href="https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control">https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control</a></li><li dir="ltr"><a href="https://www.weboccult.com/blog/computer-vision-for-quality-control-in-manufacturing-a-comprehensive-guide">https://www.weboccult.com/blog/computer-vision-for-quality-control-in-manufacturing-a-comprehensive-guide</a></li><li dir="ltr"><a href="https://cloud.google.com/blog/products/ai-machine-learning/improve-manufacturing-quality-control-with-visual-inspection-ai">https://cloud.google.com/blog/products/ai-machine-learning/improve-manufacturing-quality-control-with-visual-inspection-ai</a></li><li dir="ltr"><a href="https://www.matroid.com/automated-visual-anomoly-detection/">https://www.matroid.com/automated-visual-anomoly-detection/</a></li><li dir="ltr"><a href="https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control">https://www.qualitymag.com/articles/98221-advances-in-computer-vision-offer-a-clearer-path-to-reliable-quality-control</a></li></ol>

Discover how computer vision is solving quality challenges in metal manufacturing through enhanced defect detection, proactive process monitoring, and flexible integration solutions.

AI-Powered Welding: How Computer Vision Solutions Drive Precision and Efficiency in Metal Manufacturing

Large Language Model (LLM) training involves teaching AI models to understand and generate human-like text by processing vast amounts of data, significantly enhancing their language comprehension and production capabilities.

LLM Training: Mastering the Art of Language Model Development

In the classic cartoon “The Jetsons,” Rosie the robotic maid seamlessly switches from vacuuming the house to cooking dinner to taking out the trash. But in real life, training a general-purpose robot remains a major challenge.Typically, engineers collect data that are specific to a certain robot and task, which they use to train the robot in a controlled environment. However, gathering these data is costly and time-consuming, and the robot will likely struggle to adapt to environments or tasks it hasn’t seen before.To train better general-purpose robots, MIT researchers developed a versatile technique that combines a huge amount of heterogeneous&nbsp;data from many of sources into one system that can teach any robot a wide range of tasks.Their method involves aligning data from varied domains, like simulations and real robots, and multiple modalities, including vision sensors and robotic arm position encoders, into a shared “language” that a generative AI model can process.By combining such an enormous amount of data, this approach can be used to train a robot to perform a variety of tasks without the need to start training it from scratch each time.This method could be faster and less expensive than traditional techniques because it requires far fewer task-specific data. In addition, it outperformed training from scratch by more than 20 percent in simulation and real-world experiments.“In robotics, people often claim that we don’t have enough training data. But in my view, another big problem is that the data come from so many different domains, modalities, and robot hardware. Our work shows how you’d be able to train a robot with all of them put together,” says Lirui Wang, an electrical engineering and computer science (EECS) graduate student and lead author of a <a href="https://arxiv.org/pdf/2409.20537" target="_blank">paper on this technique</a>.Wang’s co-authors include fellow EECS graduate student Jialiang Zhao; Xinlei Chen, a research scientist at Meta; and senior author Kaiming He, an associate professor in EECS and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL). The research will be presented&nbsp;at the Conference on Neural Information Processing Systems.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730247515325-MIT-HeteroGenRobot-02-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">A figure shows how the new technique aligns data from varied domains, like simulation and real robots, and multiple modalities, including vision sensors and robotic arm position encoders, into a shared “language” that a generative AI model can process. Image: Courtesy of the researchers<h2>Inspired by LLMs</h2>A robotic “policy” takes in sensor observations, like camera images or proprioceptive measurements that track the speed and position a robotic arm, and then tells a robot how and where to move.Policies are typically trained using imitation learning, meaning a human demonstrates actions or teleoperates a robot to generate data, which are fed into an AI model that learns the policy. Because this method uses a small amount of task-specific data, robots often fail when their environment or task changes.To develop a better approach, Wang and his collaborators drew inspiration from large language models like GPT-4.These models are pretrained using an enormous amount of diverse language data and then fine-tuned by feeding them a small amount of task-specific data. Pretraining on so much data helps the models adapt to perform well on a variety of tasks.“In the language domain, the data are all just sentences. In robotics, given all the heterogeneity in the data, if you want to pretrain in a similar manner, we need a different architecture,” he says.Robotic data take many forms, from camera images to language instructions to depth maps. At the same time, each robot is mechanically unique, with a different number and orientation of arms, grippers, and sensors. Plus, the environments where data are collected vary widely.The MIT researchers developed a new architecture called Heterogeneous Pretrained Transformers (HPT) that unifies data from these varied modalities and domains.They put a machine-learning model known as a transformer into the middle of their architecture, which processes vision and proprioception inputs. A transformer is the same type of model that forms the backbone of large language models.The researchers align data from vision and proprioception into the same type of input, called a token, which the transformer can process. Each input is represented with the same fixed number of tokens.Then the transformer maps all inputs into one shared space, growing into a huge, pretrained model as it processes and learns from more data. The larger the transformer becomes, the better it will perform.A user only needs to feed HPT a small amount of data on their robot’s design, setup, and the task they want it to perform. Then HPT transfers the knowledge the transformer grained during pretraining to learn the new task.<h2>Enabling dexterous motions</h2>One of the biggest challenges of developing HPT was building the massive dataset to pretrain the transformer, which included 52 datasets with more than 200,000 robot trajectories in four categories, including human demo videos and simulation.The researchers also needed to develop an efficient way to turn raw proprioception signals from an array of sensors into data the transformer could handle.“Proprioception is key to enable a lot of dexterous motions. Because the number of tokens is in our architecture always the same, we place the same importance on proprioception and vision,” Wang explains.When they tested HPT, it improved robot performance by more than 20 percent on simulation and real-world tasks, compared with training from scratch each time. Even when the task was very different from the pretraining data, HPT still improved performance.“This paper provides a novel approach to training a single policy across multiple robot embodiments. This enables training across diverse datasets, enabling robot learning methods to significantly scale up the size of datasets that they can train on.&nbsp;It also allows the model to quickly adapt to new&nbsp;robot embodiments, which is important as new robot designs are continuously being produced,” says David Held, associate professor at the Carnegie Mellon University Robotics Institute, who was not involved with this work.In the future, the researchers want to study how data diversity could boost the performance of HPT. They also want to enhance HPT so it can process unlabeled data like GPT-4 and other large language models.“Our dream is to have a universal robot brain that you could download and use for your robot without any training at all. While we are just in the early stages, we are going to keep pushing hard and hope scaling leads to a breakthrough in robotic policies, like it did with large language models,” he says.

Researchers filmed multiple instances of a robotic arm feeding co-author Jialiang Zhao's adorable dog, Momo. The videos were included in datasets to train the robot. Image: Courtesy of the researchers

Inspired by large language models, researchers develop a training technique that pools diverse data to teach robots new skills.

A faster, better way to train general-purpose robots

In recent years, the field of robotics has seen remarkable advancements in gripper technology that pushes the boundaries of dexterity and manipulation. Shadow Robot is at the forefront of this evolution as a pioneer in innovative robotics development. Shadow’s approach to the innovative solutions has caught the attention of industry leaders, including Google DeepMind, a leading AI research lab focused on developing artificial intelligence capable of performing a wide range of tasks. This collaboration has led to the creation of DEX-EE, a groundbreaking gripper that may revolutionize robotic manipulation and machine learning research.<h2 dir="ltr">The Challenge from Google DeepMind</h2>In 2017, Google DeepMind approached Shadow Robot with a big challenge. The DeepMind project was conducting research on generalized intelligence and the role of embodiment in learning—how physical interaction with the world plays a part in learning, especially for tasks that involve understanding and manipulating objects. The study required a robot with high sensitivity and a robust design.For their research, the DeepMind team needed a design that could reliably run long-term and perform physically demanding experiments without constraints within its workspace. The study required a robot that could strike its environment repeatedly to learn how to manipulate objects. This meant that the robot needed to be resilient, sensor-rich and provide large volumes of high-quality data, while operating for prolonged periods without interruption for repairs and without the need to restart experiments.The challenge pushed the boundaries of existing robotic capabilities because it required a solution that was both robust and highly precise. Google DeepMind approached Shadow Robot for this challenge. Shadow Robot has partnered with numerous research projects and the company’s flagship Shadow Dexterous Hand has contributed to major advancements in the field of robotics.&nbsp;Read more about the Shadow Dexterous Hand and its contributions in our <a href="https://www.wevolver.com/article/dexterous-robotic-hands-part-1-unraveling-the-history-technology-and-applications-of-dexterous-robotic-hands">three-part Shadow Dexterous Hand series</a>.<h3 dir="ltr">DEX-EE: Shadow Robot’s Innovative Solution</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730204504132-Untitled+design+%285%29.png" style="width: 100%px;" class="fr-fic fr-dib">In response to the challenge from Google DeepMind, Shadow Robot developed DEX-EE, a groundbreaking dexterous robot hand, as well as a major advancement in the field of machine learning robots. DEX-EE is designed to perform with dynamic and controlled motion in a robust and reliable package. The hand enables long-running experiments without interruptions due to hardware failure.One of DEX-EE’s most innovative features is its high-speed network sensors, which provide rich data including position, force, and inertial measurement. Each finger has hundreds of channels of tactile sensing, and Shadow’s advanced fingertip sensors have hundreds of taxels (short for tactile pixel—the individual sensing elements that make up a tactile sensor), each with a massive dynamic range. With this wealth of sensory data, researchers can gather huge amounts of data from dexterous robotic tasks.DEX-EE was developed with an intensive focus on reliability and robustness. Through iterative design and testing over five years, Shadow created a new dexterous hand that could withstand the physical demands of Google DeepMind’s research. As a result of this focused development, the DEX-EE is a functional, reliable, and data-rich research tool. DEX-EE’s capabilities position it as an ideal hardware platform for dexterous robotic research into cognition and manipulation.<h2 dir="ltr">DEX-EE Technical Specifications and Advancements</h2>DEX-EE is a significant leap in dexterous robotic hand technology. The robot has high-bandwidth torque and position control loops, which enable delicate and precise fingertip dexterity, while torque and inertial measurement throughout the hand make it sensitive to interactions with its environment.The hand also features tactile fingertip sensors based on stereo-camera technology, which gives the hand abundant 3D data about the environment in a robust package. Additional, multi-taxel, 3-DOF sensors on the middle and proximal fingers provide crucial information during grasping and manipulation tasks.Compared to earlier models of the Shadow Hand, DEX-EE offers greater reliability and repairability. It was designed for easy maintenance with minimal training, incorporates fail-safes and a shutdown routing that reduces instances of failure and downtime. The hand’s advanced sensor networks provide rich data, including position, force, and inertial measurement, with hundreds of channels of tactile sensing per finger. Shadow’s innovative fingertip sensors have hundreds of taxels each, with a dynamic range that significantly surpasses the capabilities of other dexterous hands.DEX-EE is also integrated with ROS (Robot Operating System), making it versatile as a research tool. This combination of robustness, advanced sensing capabilities, and ease of integration positions DEX-EE as an ideal platform for dexterous manipulation research that is capable of withstanding the demands of long-term machine learning experiments.<h2 dir="ltr" id="isPasted">Applications in Machine Learning and AI Research</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1730197463018-shadowrobot-grip.png" style="width: 100%px;" class="fr-fic fr-dib">Image: DEX-EE performs machine learning experiments with high sensitivity and robustness | Source:&nbsp;<a href="https://www.shadowrobot.com/dex-ee/" target="_blank" rel="noopener noreferrer">shadowrobot.com</a>DEX-EE represents the next major advancement in dexterous manipulation research because it provides a robust platform for complex machine learning experiments like Google DeepMind. The advanced sensor networks and tactile capabilities of DEX-EE enable researchers to gather enormous amounts of data during experiments and push the boundaries of what’s possible in robotic learning.Google DeepMind has leveraged DEX-EE’s capabilities in their latest AI system DemoStart, which helps robots learn to perform complex tasks that require dexterous movement. With these systems, Google DeepMind has achieved remarkable results, like inserting a plug into a socket, tightening a bolt, and other tasks that were previously challenging for robots to master.DEX-EE’s durability allows for long-running experiments without interruptions, making it ideal for reinforcement learning protocols that involve repeated physical interactions with the environment. This has already helped researchers explore new frontiers in embodied AI, investigating how physical interactions contribute to learning and the development of generalized intelligence.The hand’s integration with ROS further enhances its versatility as a research tool because it allows for easier implementation of new and existing learning algorithms and control strategies. With this combination of robustness, advanced sensing, and flexibility, DEX-EE is positioned to be a transformative platform for exploring the future of dexterous manipulation in AI and robotics.<h2 dir="ltr">Shadow Robot's Unique Approach to Robotics Development</h2>Shadow Robot continues to distinguish itself as a robotics company through problem-solving approaches and focus on custom solutions. Rather than offering off-the-shelf products that don’t meet a particular need, Shadow engages deeply with clients to understand their specific issues and develop tailored solutions.This collaborative approach has been exemplified in the development of DEX-EE, where Shadow worked closely with Google DeepMind over five years. Shadow’s willingness to rethink fundamental design principles demonstrates their commitment to innovation and their ability to work at the cutting edge of technology.Shadow has positioned themselves as a research and hardware company that is capable of tackling any robotics challenge. They have established a unique niche in the industry, bridging academic research and practical, real-world applications.<h2 dir="ltr">Leading Innovation in Custom Robotics Solutions</h2>Shadow Robot stands out as a leader in collaborative robotics and demonstrates a commitment to addressing practical needs through innovative design and partnership. Shadow’s work on DEX-EE, developed alongside Google DeepMind, emphasizes the company’s ability to meet the changing demands of cutting-edge research. Shadow continues to push the boundaries of dexterity and manipulation by fostering collaboration and by focusing on tailored solutions that meet practical needs. In the evolving field of robotics, Shadow’s dedication to understanding and responding to the unique challenges positions the company as an important player in robotics research.<blockquote>DEX-EE marks a significant milestone for Shadow Robot. While designed for our core research market, it embodies our vision of reaching beyond this to real-world application research. We remain committed to our fundamental research customers, but DEX-EE represents our expanding horizons. It's a step towards a future where our technology not only advances scientific understanding but also addresses practical challenges across industries, allowing us to make a broader impact while continuing to innovate within the research community.</blockquote>Gavin Cassidy, Head of Product Development at Shadow Robot.For more information about Shadow’s collaborative problem solving and innovative robotics, check out the <a href="https://www.shadowrobot.com/">Shadow Robot website</a>. The <a href="https://www.wevolver.com/article/dexterous-robotic-hands-part-1-unraveling-the-history-technology-and-applications-of-dexterous-robotic-hands">Shadow Robot Dexterous Hand series</a> also provides an in-depth look at Shadow’s commitment to advancing the field of robotics.<h3 dir="ltr">References</h3><ol><li>&nbsp;Shadow Robot. “Collaborative Working: A Case Study”. June 13, 2024. Accessed from <a href="https://www.shadowrobot.com/blog/collaboration-case-study/">https://www.shadowrobot.com/blog/collaboration-case-study/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “DEX-EE Overview” The most robust dexterous robot hand on the market”. June 6, 2024. Accessed from <a href="https://www.shadowrobot.com/blog/shadow-robot-hand-overview/">https://www.shadowrobot.com/blog/shadow-robot-hand-overview/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “Introducing DEX-EE the most robust dexterous robot hand on the market’. Accessed from <a href="https://www.shadowrobot.com/dex-ee/">https://www.shadowrobot.com/dex-ee/</a>&nbsp;</li><li>&nbsp;Shadow Robot. “DEX-EE Overview: The most dexterous robot hand on the market”. <a href="https://www.shadowrobot.com/blog/shadow-robot-hand-overview/">https://www.shadowrobot.com/blog/shadow-robot-hand-overview/</a>&nbsp;</li><li>&nbsp;Google DeepMind. “Our latest advances in robot dexterity”. 12 September 2024. Accessed from <a href="https://deepmind.google/discover/blog/advances-in-robot-dexterity/">https://deepmind.google/discover/blog/advances-in-robot-dexterity/</a>&nbsp;</li></ol>

Discover how Shadow Robot’s DEX-EE gripper is advancing robotic dexterity with AI-driven precision.

Gripping the Future: Shadow Robot's DEX-EE and the Next Generation of Dexterous Manipulation

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries. There are also full-sequence diffusion models like Sora, which convert words into dazzling, realistic visuals by successively “denoising” an entire video sequence.Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have proposed a simple change to the diffusion training scheme that makes this sequence denoising considerably more flexible.When applied to fields like computer vision and robotics, the next-token and full-sequence diffusion models have capability trade-offs. Next-token models can spit out sequences that vary in length. However, they make these generations while being unaware of desirable states in the far future — such as steering its sequence generation toward a certain goal 10 tokens away — and thus require additional mechanisms for long-horizon (long-term) planning. Diffusion models can perform such future-conditioned sampling, but lack the ability of next-token models to generate variable-length sequences.Researchers from CSAIL want to combine the strengths of both models, so they created a sequence model training technique called “Diffusion Forcing.” The name comes from “Teacher Forcing,” the conventional training scheme that breaks down full sequence generation into the smaller, easier steps of next-token generation (much like a good teacher simplifying a complex concept).<iframe width="640" height="360" src="https://www.youtube.com/embed/m4rhzR-h-JU?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Diffusion Forcing found common ground between diffusion models and teacher forcing: They both use training schemes that involve predicting masked (noisy) tokens from unmasked ones. In the case of diffusion models, they gradually add noise to data, which can be viewed as fractional masking. The MIT researchers’ Diffusion Forcing method trains neural networks to cleanse a collection of tokens, removing different amounts of noise within each one while simultaneously predicting the next few tokens. The result: a flexible, reliable sequence model that resulted in higher-quality artificial videos and more precise decision-making for robots and AI agents.By sorting through noisy data and reliably predicting the next steps in a task, Diffusion Forcing can aid a robot in ignoring visual distractions to complete manipulation tasks. It can also generate stable and consistent video sequences and even guide an AI agent through digital mazes. This method could potentially enable household and factory robots to generalize to new tasks and improve AI-generated entertainment.“Sequence models aim to condition on the known past and predict the unknown future, a type of binary masking. However, masking doesn’t need to be binary,” says lead author, MIT electrical engineering and computer science (EECS) PhD student, and CSAIL member Boyuan Chen. “With Diffusion Forcing, we add different levels of noise to each token, effectively serving as a type of fractional masking. At test time, our system can “unmask” a collection of tokens and diffuse a sequence in the near future at a lower noise level. It knows what to trust within its data to overcome out-of-distribution inputs.”In several experiments, Diffusion Forcing thrived at ignoring misleading data to execute tasks while anticipating future actions.When implemented into a robotic arm, for example, it helped swap two toy fruits across three circular mats, a minimal example of a family of long-horizon tasks that require memories. The researchers trained the robot by controlling it from a distance (or teleoperating it) in virtual reality. The robot is trained to mimic the user’s movements from its camera. Despite starting from random positions and seeing distractions like a shopping bag blocking the markers, it placed the objects into its target spots.To generate videos, they trained Diffusion Forcing on “Minecraft” game play and colorful digital environments created within Google’s <a href="https://github.com/google-deepmind/lab">DeepMind Lab Simulator</a>. When given a single frame of footage, the method produced more stable, higher-resolution videos than comparable baselines like a Sora-like full-sequence diffusion model and ChatGPT-like next-token models. These approaches created videos that appeared inconsistent, with the latter sometimes failing to generate working video past just 72 frames.Diffusion Forcing not only generates fancy videos, but can also serve as a motion planner that steers toward desired outcomes or rewards. Thanks to its flexibility, Diffusion Forcing can uniquely generate plans with varying horizon, perform tree search, and incorporate the intuition that the distant future is more uncertain than the near future. In the task of solving a 2D maze, Diffusion Forcing outperformed six baselines by generating faster plans leading to the goal location, indicating that it could be an effective planner for robots in the future.Across each demo, Diffusion Forcing acted as a full sequence model, a next-token prediction model, or both. According to Chen, this versatile approach could potentially serve as a powerful backbone for a “world model,” an AI system that can simulate the dynamics of the world by training on billions of internet videos. This would allow robots to perform novel tasks by imagining what they need to do based on their surroundings. For example, if you asked a robot to open a door without being trained on how to do it, the model could produce a video that’ll show the machine how to do it.The team is currently looking to scale up their method to larger datasets and the latest transformer models to improve performance. They intend to broaden their work to build a ChatGPT-like robot brain that helps robots perform tasks in new environments without human demonstration.“With Diffusion Forcing, we are taking a step to bringing video generation and robotics closer together,” says senior author Vincent Sitzmann, MIT assistant professor and member of CSAIL, where he leads the Scene Representation group. “In the end, we hope that we can use all the knowledge stored in videos on the internet to enable robots to help in everyday life. Many more exciting research challenges remain, like how robots can learn to imitate humans by watching them even when their own bodies are so different from our own!”<hr>Chen and Sitzmann wrote the paper alongside recent MIT visiting researcher Diego Martí Monsó, and CSAIL affiliates: Yilun Du, a EECS graduate student; Max Simchowitz, former postdoc and incoming Carnegie Mellon University assistant professor; and Russ Tedrake, the Toyota Professor of EECS, Aeronautics and Astronautics, and Mechanical Engineering at MIT, vice president of robotics research at the Toyota Research Institute, and CSAIL member. Their work was supported, in part, by the U.S. National Science Foundation, the Singapore Defence Science and Technology Agency, Intelligence Advanced Research Projects Activity via the U.S. Department of the Interior, and the Amazon Science Hub. They will present their research at NeurIPS in December.

In the current AI zeitgeist, sequence models have skyrocketed in popularity for their ability to analyze data and predict what to do next. For instance, you’ve likely used next-token prediction models like ChatGPT, which anticipate each word (token) in a sequence to form answers to users’ queries.

Combining next-token prediction and video diffusion in computer vision and robotics

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/b921c890-089c-4f02-b05d-59a0c9850785?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(0:50) - <a href="https://www.wevolver.com/article/ai-helps-detect-and-monitor-infrastructure-defects">Infrastructure Defect Detecting AI</a>Become a founding reader of our newsletter: <a href="http://read.thenextbyte.com/">http://read.thenextbyte.com/</a><hr><h2 data-attrid="title" data-local-attribute="d3bn" data-ved="2ahUKEwiBhLnmu4aAAxVmwjgGHcDWC38Q3B0oAXoECEgQEQ" id="isPasted">Transcript</h2>What's up, folks? As you know, cracks in buildings and roads and bridges can cause some serious damage if they're not found early enough. Today, we're talking about a team from EPFL that's gonna fix this problem using AI that can help detect cracks automatically and let us know when we need to do repairs. I think it's interesting, so let's jump into it.I'm Daniel, and I'm Farbod. And this is the NextByte Podcast. Every week, we explore interesting and impactful tech and engineering content from Wevolver.com and deliver it to you in bite sized episodes that are easy to understand, regardless of your background.&nbsp;Daniel:&nbsp;What's up friends? Like we said, today we're talking about the future of crack detection in infrastructure and why that's so important. We're not just talking about like cracks and buildings and roads and, sometimes it makes them look ugly or ivy can grow in them. But the problem is that they can cause serious damage if not found early enough. The traditional methods we have for inspecting and looking for these cracks are really slow, really costly, really hard to do in dangerous areas. And there's a lot of human error involved with it as well. So, we're talking about some awesome research here where they're using AI to try and help find and monitor cracks, but they're even taking kind of a unique AI approach as well that doesn't require detailed pixel by pixel labeling either. So, I'm pretty stoked about this one.Farbod:&nbsp;Dude, same. I feel like infrastructure improvement has been talked about for so long. I remember being in middle school and politicians were promising the next infrastructure build and how like, our infrastructure is aging and we need to replace it. Well, before this part, I was just kind of curious. I'm like, let's pull up some stats just about the US alone, right? I don't know if you knew this, but a freeway bridge inspection can take, can cost about $5,000 and take about eight hours to do. And that doesn't take into account the detour traffic causing people pains to get to work, et cetera, et cetera. And on top of that, the way we do grading of these cracks tends to be kind of subjective. So, the person that's doing it might grade it more, I don't know, liberally than the person that did last year. So that person might've thought things were going bad. This person says everything's good. So that kind of creates a weird situation where we can't track how bad it's getting super well.Daniel:&nbsp;Or can you trust the data that comes from these expectations? When you spend this much money and you spend this much time, you may not be even able to trust the data that comes out of it.Farbod: And to just make matters worse, there's something like one out of every three bridges in the US needs repair or replacement. So, one, you don't know if things are actually bad. Two, it seems like at least one third are kind of bad, maybe more. So not super comfortable about driving over a bridge at this point, but I saw something interesting about drones. Drones have had great improvement over the past couple of years. People were saying on average it's about 40% cheaper. So that makes it more affordable for local governments to take them on. But still, I'm guessing no one is super excited about shelling out over $3,000 to inspect every bridge they got. So yeah, sticky situation to be in.Daniel:&nbsp;Well, and just a quick throwback here. I would say like in the broader realm of infrastructure detection, both of our capstone projects, when we were graduating our undergrad program was related to using robotics and automation to try and detect issues with infrastructure before they progress to something that becomes a catastrophic failure. And that's not to say that we're experts in this topic at all. I think we were just at the tip of the iceberg here, but it is interesting to note that in separate graduating classes for about, and I both had this capstone project because it's such a big problem and we're still sitting here saying, maybe now we finally got some AI solution, which can help with this in a more robust manner, which is exciting for both of us, I think on a personal note, to see this field start to progress.Farbod: Dude, absolutely. Back when we were young men, which we're not anymore, at least my team, we were toying around with the idea of using AI, but even then, there wasn't a whole lot of excitement around it. Specifically, we were looking at corrosion, but now it looks like the game has completely shifted. We're going to EPFL for this episode, but if we can transition to the secret sauce, these folks are talking about how AI can actually completely remove the human element here and in the best way possible. We've been talking about subjectivity, how that can result in variance from person to person, and if a crack is getting better or worse. These folks are saying, hey, we can just train a very simple model that can tell us if there's a crack or if there's no crack, like binary. Once it learns how to detect the crack well, then it's just a matter of saying “cool, where are you seeing the crack and how is it changing over time?” Which in the world of like, you know, computer vision is a fairly simple and straightforward analysis. And the main application they're looking at for this is railroads, right? They're saying, railroads are used very often if there's cracks that grow that can cause catastrophic failures. So how do we maintain this and like monitor it on a regular basis? Well, they're like trains are passing through this bridges and stuff. What if they were the ones that were doing the analysis completely autonomously, no added costs, no added time, no requirement for shut down.Daniel: It's not sending people on harnesses to rappel from these. Honestly, you should just check on the link in the show notes, click for no other reason than to see the beautiful photo of this train on a railroad bridge going across a canyon in Switzerland where, and I wanted to confirm this, I wasn't 100% sure. The Matterhorn-Gothard bond network. I'm pretty sure Nellie and I took this train on our honeymoon, which is pretty cool.Farbod: Wow. That's special.Daniel:&nbsp;If not something very similar because all of Switzerland is full of beautiful trains and bridges and mountains, but I'm pretty sure we went on this exact railroad section where they're starting testing, but yeah, if for no other reason, check out this photo, but for me to get back to my main point, this is a bridge suspended several stories above the bottom of a canyon. It would cost a lot and you'd have to pay someone a lot to be able to visually inspect all of that manually, like as a person by hand. And what they're trying to do now is instead send a drone, but not in the form of the quadcopter drone that you think of probably when we say drone. The solution is, send a drone rail car, right? An instrumented rail car down this railroad and try and collect a bunch of data on, where are their cracks? Are they propagating? Are they staying the same? Which ones need to be addressed? Continuing to understand and map the ever evolving, let's say ecosystem around all these bridges and this infrastructure and understand when there's issues and indicate whether they need further inspection or whether they need immediate repair, et cetera.Farbod:&nbsp;Dude, totally agree. And the solution seems so simple when you say it out loud, but it really is, I don't know, I guess the best solutions are the simple ones. And what it made me think of is obviously this is great for the railroad system. But then we've been talking about like as a commuter that's going in a tunnel. That one tunnel in Boston I always hate it gives me the creeps when I'm going through it or when I'm going through bridges to Maryland and stuff like that. It would be great if there was something that could do the analysis there and in my mind, I'm like we're definitely a long way away from having autonomous vehicles doing these trips for us. But Uber and Lyft, they're drivers that are operating every single day that are doing these normal routes that people take to go to work and whatnot. I wonder if there was an opportunity there to say, hey, while you're doing this, mount this device on the back of your car so that whenever your GPS detects you're going from the beginning of the tunnel to the back of the tunnel, you do a scan and then that data uploads to a cloud somewhere and we can do analysis over time to see if we need to patch any cracks or what?Daniel: I'm like wondering if you read my mind here. I went to buses, because I'm like, the department of transportation in a city may also own the buses, right? So, they're in charge of the bridges. They're also in charge of the buses. Wonder if they can just put a bunch of instruments on top of a bus. In my mind, it's like the closest analogy to a rail car. Put a bunch of this infrastructure on top of a bus and this instrumentation, see if you can analyze the infrastructure around us using that same method as well. But I also want to jump a little bit into the specific AI methods they're using. And I'm not an expert here by any means, but I took some notes from the paper that I think are interesting. That there have previously tried to do AI classification of images, but it's too computationally intensive to justify the cost. So, if you were to use AI to try and evaluate millions of photos of an entire bridge. It may actually cost more than actually sending a human down there to do it. Because one of the things that I had to do is classify each pixel by pixel, trying to understand whether it was light or if it's dark and look at contrast to understand what is a crack, then it classifies a crack and that took a lot of computational energy. One of the things that they did is they, they train their AI ahead of time with images labeled as crack or no crack. And then it uses explainable AI is what they say is their unique method here to try and look at a new photo and circle, basically highlight a cracked area and create a map showing where cracks are and where they aren't. And essentially what it's doing is it's just taking photo after photo after photo and trying to trace the area around where it thinks the crack is. And over a period of time, it's trying to look at the same photo from the same location the last time this train came through and look at whether this crack got bigger or if it stayed the same. And if cracks are serially getting bigger and bigger and bigger from these images, it warrants sending someone down there, an expert down there to take a look at it. And one of the things that they mentioned is this cut the processing and labeling effort, as opposed to previous AI methods by about 90%, which makes it cheaper and faster and worthwhile to implement in an autonomous manner. It's not quite as accurate as these full AI models, but they said it can still help directionally track crack growth. And they think that they can hone it essentially and get rid of more of the area that they're experiencing right now. But essentially the game plan I guess is to go test this out in the field, get more learnings, understand more but in practice even if it has the same performance that it has today you can use this as kind of like a non-invasive constant monitoring where it looks at the rails, it looks at the ties, it looks at the retaining walls and tries to understand if there's anything wrong with the railroad. And you could have this on every train or one of every 10 trains passing through this railroad. Even though I think right now it's got 35% error, which again, it sounds high compared to most AI models, but if you're telling me, you can tell me 65% of the time, whether a crack's growing or not. If I've got two or three trains that pass through over a series of a week and show that this crack indicates that this crack may be growing, the playbook here is not to say, all right, let's shut down the bridge and tear it down. The playbook here could be, let's go send one of those expert inspectors that we used to have doing this job full time and had to shut down the railroad to do now that we know that there's something that's worth further evaluation, let's send one of those crack experts down there to understand what's going on and see whether it needs repair.Farbod: And this is a great example, at least in my opinion of complimentary automation, right? Like here's a task that was bad for people to do manually. And some places are like in America, it's just straight up not done at times. Now you have automation that's taking care of the tedious part. And then you have the human that's taking on the challenging part, which is, hey, we're like 90% sure there is a crack, please go out there and now characterize it and give us feedback on if it's catastrophic or if, you know, it's a simple crack that needs to be fixed, etc, etc. So, I feel like this is the best use of every technology or every resource, think of the human expert as a resource. And this is kind of going against the AI is going to get rid of all jobs mentality.Daniel: Well, and I think what we, the intended end here is to be able to do continuous monitoring, collect data all the time to understand which regions are susceptible to cracks, and then use that as a heat map essentially to show someone who's again, got expertise in this field to go, this is the area that you need to inspect. This is the area that you need to monitor. This is the area that may need potential repair as opposed to the other portions of track that maybe didn't need as much attention and they were getting it before because you had to go check every single mile of this thing manually. And maybe you used to only check one, every single mile of this thing manually, one out of every five years. And now we could have a train passing on it five times a day that tells us how the railroad's doing. So, I appreciate here, like you said, it's a complimentary solution between automation and humans. But it also gets us an immediate benefit, which is a lot more data, a lot more consistently on how our infrastructure is doing and when it needs to be repaired.Farbod: Totally agree, man. You want to do us a favor and wrap up the episode?Daniel:&nbsp;I will. What if we could stop big cracks in buildings and roads before they cause big problems, I'm thinking of bridges collapsing or bridges needing to be shut down because they need repair or rip, cars getting derailed. These are big problems and we're talking today about a new AI method that helps find and track these cracks automatically without needing a ton of expensive detailed data. Essentially, they're getting the job done with 90% less effort. That's what their new tech promises. And they use this system called explainable AI that spots cracks with simple image labels, saving time and money, keeping an eye on crack growth to let experts know where they need to address cracks and where we might need repairs in the future. I think it's a lot like, I don't know, Google Maps navigation or some sort of traffic heat map, but it tells us where most areas are prone to cracks and where we need repairs so that folks know where they need to focus their efforts.Farbod:&nbsp;Money.Daniel:&nbsp;Thanks, my dude.Farbod:&nbsp;You killed it, dude. All right, folks, thank you so much for listening. And as always, we'll catch you in the next one.Daniel:&nbsp;Peace.<hr>As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.To learn more about this show, please visit our<a href="https://www.wevolver.com/profile/the.next.byte">&nbsp;shows page</a>. By following the page, you will get automatic updates by email when a new show is published. Be sure to give us a follow and review on<a href="https://podcasts.apple.com/nl/podcast/the-next-byte/id1548255861?l=en" target="_blank" rel="nofollow">&nbsp;Apple</a> podcasts,<a href="https://open.spotify.com/show/6lPPsRtegnivsRUEsQ09R7?si=IPMiah7xT_apJtPjqvXMlA" target="_blank" rel="nofollow">&nbsp;Spotify</a>, and most of your favorite podcast platforms!--The Next Byte: We're two engineers on a mission to simplify complex science &amp; technology, making it easy to understand. In each episode of our show, we dive into world-changing tech (such as AI, robotics, 3D printing, IoT, &amp; much more), all while keeping it entertaining &amp; engaging along the way.

In this episode, we explore how AI technology is revolutionizing the detection and monitoring of infrastructure defects and learn how these innovative solutions are enhancing safety, improving maintenance, and preventing failures in critical infrastructure systems

Podcast: Saving America's Collapsing Infrastructure

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/35ecb899-3a2c-45e8-9d17-7a8c87d944ba?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Most doctors go into medicine because they want to help patients. But today’s health care system requires that doctors spend hours each day on other work — searching through electronic health records (EHRs), writing documentation, coding and billing, prior authorization, and utilization management — often surpassing the time they spend caring for patients. The situation leads to physician burnout, administrative inefficiencies, and worse overall care for patients.Ambience Healthcare is working to change that with an AI-powered platform that automates routine&nbsp;tasks for clinicians before, during, and after patient visits."We build co-pilots to give clinicians AI superpowers," says Ambience CEO Mike Ng MBA ’16, who co-founded the company with Nikhil Buduma ’17. "Our platform is embedded directly into&nbsp;EHRs to free up clinicians to&nbsp;focus on what matters most, which is providing the best possible patient care."Ambience’s suite of products handles pre-charting and real-time AI scribing, and assists with navigating the thousands of rules to select the right insurance billing codes. The platform can also send after-visit summaries to patients and their families in different languages to keep everyone informed and on the same page.Ambience is already being used across roughly 40 large institutions such as UCSF Health, the Memorial Hermann Health System, St. Luke’s Health System, John Muir Health, and more.&nbsp;Clinicians leverage Ambience in dozens of languages and more than&nbsp;100 specialties and subspecialties, in settings&nbsp;like the emergency department, hospital inpatient settings,&nbsp;and the oncology ward.The founders say clinicians using Ambience save two to three hours per day on documentation,&nbsp;report lower levels of burnout, and develop higher-quality relationships with their patients.<h2>From problem to product to platform</h2>Ng worked in finance until getting an up-close look at the health care system after he fractured his back in 2012. He was initially misdiagnosed and put on the wrong care plan, but he learned a lot about the U.S. health system in the process, including how the majority of clinicians’ days are spent documenting visits, selecting billing codes, and completing other administrative tasks. The average clinician only spends 27 percent of their time on direct patient care.In 2014, Ng decided to enter the MIT Sloan School of Management. During his first week, he attended the “t=0” celebration of entrepreneurship hosted by the Martin Trust Center for MIT Entrepreneurship, where he met Buduma.&nbsp;The pair became fast friends, and they ended up taking classes together including 15.378 (Building an Entrepreneurial Venture) and 15.392 (Scaling Entrepreneurial Ventures).“MIT was an incredible training ground to evaluate what makes a great company and learn the foundations of building a successful company,” Ng says.Buduma had gone through his own journey to discover problems with the health care system. After immigrating to the U.S. from India as a child and battling persistent health issues, he had watched his parents struggle to navigate the U.S. medical&nbsp;system. While completing his bachelor’s degree at MIT, he was also steeped in the AI research community and wrote an early textbook on modern AI and deep learning.In 2016, Ng and Buduma&nbsp;founded&nbsp;their first company in San Francisco — Remedy Health — which operated its own AI-powered health care platform. In the process of hiring clinicians, taking care of patients, and implementing technology themselves, they developed an even deeper appreciation for the challenges that health care organizations face.During that time, they also got an inside look at advances in AI. Google’s Chief Scientist Jeff Dean, a major investor in Remedy and now in Ambience, led a research group inside of Google Brain to invent the transformer architecture. Ng and Buduma say they were among the first to put transformers into production to support their own clinicians at Remedy. Subsequently, several of their friends and housemates went on to start the large language model group within OpenAI. Their friends’ work formed the research foundations that ultimately led to ChatGPT. &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;“It was very clear that we were at this inflection point where we were going to have this class of general-purpose models that were going to get exponentially better,” Buduma says. “But I think we also noticed a big gap between those general-purpose models versus what actually would be robust enough to work in a clinic. Mike and I decided in 2020 that there should be a team that specifically focused on fine-tuning these models for health&nbsp;care and medicine.”The founders started Ambience by building an AI-powered scribe that works on phones and laptops to record the details of doctor-patient visits in a HIPAA-compliant system that preserves patient privacy. They quickly saw that the models needed to be fine-tuned for each area&nbsp;of medicine, and they slowly expanded specialty coverage one by one in a multiyear process.The founders also realized their scribes needed to fit within back-office operations like insurance coding and billing.“Documentation isn’t just for the clinician, it's also for the revenue cycle team,” Buduma says. “We had to go back and rewrite all of our algorithms to be coding-aware. There are literally tens of thousands of coding rules that change every year and differ by specialty and contract type.”From there, the founders built out models for clinicians to make referrals and to send comprehensive summaries of visits to patients.“In most care settings before Ambience, when a patient and their family left the clinic, whatever the patient and their family wrote down was what they remembered from the visit,” Buduma says. “That’s one of the features that physicians love most, because they are trying to create the best experience for patients and their families. By the time that patient is in the parking lot, they already have a really robust, high-quality summary of exactly what you talked about and all the shared decision-making around your visit in their portal.”<h2>Democratizing health care</h2>By improving physician productivity, the founders believe they’re helping the health care&nbsp;system manage a chronic shortage of clinicians that’s expected to grow in coming years.“In health care, access is still a huge problem,” Ng says. “Rural Americans have a 40 percent higher risk of preventable hospitalization, and half of that is attributed to a lack of access to specialty care.”With Ambience already helping health systems manage razor-thin margins by streamlining administrative tasks,&nbsp;the founders have a longer-term vision to help&nbsp;increase access to the best clinical information across the country.“There’s a really exciting opportunity to make expertise at some of the major academic medical centers more democratized across the U.S.,” Ng says. “Right now, there’s just not enough specialists in the U.S. to support our rural populations. We hope to help scale the knowledge of the leading specialists in the country through an AI infrastructure layer, especially as these models become more clinically intelligent.”

Ambience is being used across roughly 40 health systems in the U.S. by clinicians in over 100 subspecialties. Image: iStock

Alumni-founded Ambience Healthcare automates routine tasks for clinicians before, during, and after patient visits.

Equipping doctors with AI co-pilots

Robots, combined with artificial intelligence (AI), are transforming industries such as manufacturing, healthcare, and research. 

AI-enabled robots and the importance of force sensing

Edge AI is the process of running artificial intelligence (AI) and machine learning (ML) algorithms on computing devices at the periphery of a network, rather than on large cloud servers. By doing this, we enable a wide range of possibilities to solve unique problems. The benefits of edge AI are the same as those found in edge computing: decreased network bandwidth usage, reduced network latency, less energy consumption, improved reliability, and enhanced data privacy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739292008339-1.png" style="width: 100%px;" class="fr-fic fr-dib"><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable">&nbsp;</iframe>We are announcing the release of an Edge AI course to help you become familiar with edge computing and machine learning. The course is entirely free and hosted on our site with embedded YouTube videos. At the end of each section, you are encouraged to take a quiz to test your knowledge. Once you complete all 10 &nbsp;episodes, you can take a comprehensive test and earn a free digital certificate.Start the course here: <a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/intro-to-edge-ai" rel="noreferrer">Introduction to Edge AI</a>The videos are also accessible as a playlist on YouTube:<iframe width="640" height="360" src="https://www.youtube.com/embed/z5lIrWTebOY??list=PL7VEa1KauMQppUExiIOfxxk_2ta94EEpd&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe>We divide the course into 10 parts plus an introduction page:<ol><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/intro-to-edge-ai" rel="noreferrer">Introduction to edge AI</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-computing" rel="noreferrer">What is edge computing?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-machine-learning" rel="noreferrer">What is machine learning (ML)?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-ai" rel="noreferrer">What is edge AI?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/how-to-choose-an-edge-ai-device" rel="noreferrer">How to choose an edge AI device</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/edge-ai-lifecycle" rel="noreferrer">Edge AI lifecycle</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-mlops" rel="noreferrer">What is edge MLOps?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-impulse" rel="noreferrer">What is Edge Impulse?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/case-study-izoelektro" rel="noreferrer">Case study: Izoelektro smart grid monitoring</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/test-and-certification" rel="noreferrer">Test and certification</a></li></ol>The course is high-level to help you become familiar with the concepts and vocabulary around edge AI. There are no hands-on exercises or programming required. If you would like to dive into code and hardware, check out our <a href="https://www.coursera.org/learn/introduction-to-embedded-machine-learning">Introduction to Embedded Machine Learning course on Coursera</a>.Edge AI is a fast-growing field enabling new technologies, especially in healthcare and industrial settings. For example, Izoelektro created a <a href="https://edgeimpulse.com/case-studies/smart-grid-monitoring">power grid monitoring system</a>, and the <a href="https://edgeimpulse.com/case-studies/oura-health-monitoring">Oura Ring</a> tracks sleep with incredible precision.We are excited to see what you will build with edge AI!<hr><h3 id="isPasted"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739292036074-1.png" style="width: 100%px;" class="fr-fic fr-dib"></h3><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable"></iframe>

Edge AI is the process of running artificial intelligence (AI) and machine learning (ML) algorithms on computing devices at the periphery of a network, rather than on large cloud servers.

Get Certified with our new Introduction to Edge AI Course

Similar to visual data, collecting and curating sound data that accurately reflects real-world scenarios is a major hurdle in training effective machine learning models. The time, effort, and costs involved can be substantial, posing significant barriers, especially for projects with limited resources. As our mission is to lower the barriers of creating and utilizing ML and AI tools, we aim to drastically reduce these types of impediments, making the process faster, cheaper, and as reliable as possible.Edge AI enables smart devices to perform machine learning tasks right at the source of data collection, using small models that are optimized for performing scoped tasks as long as they are trained on quality data. That’s why we emphasize that quality is key: "Garbage in, garbage out."<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739291965210-1.png" style="width: 100%px;" class="fr-fic fr-dib"><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable">&nbsp;</iframe><h3>Improving sound datasets with generative AI</h3>To enhance your sound datasets, we've been working on integrating Edge Impulse with <a href="http://elevenlabs.io/">ElevenLabs.io</a>. Edge Impulse excels in crafting and optimizing models for edge computing. Meanwhile, ElevenLabs.io offers advanced capabilities for creating highly realistic sound effects. This integration enables you to expand your datasets with sounds that are difficult or expensive to record naturally, using state-of-the-art generative techniques provided by ElevenLabs.io. This integration not only saves time and money but also boosts the accuracy and reliability of the models we deploy on edge devices.&nbsp;It is accessible now in the new <a href="https://www.edgeimpulse.com/blog/announcing-synthetic-data-integration/" rel="noreferrer">Synthetic data tab</a> inside the Data Acquisition segment of Edge Impulse for Enterprise users. Sign up for a <a href="https://studio.edgeimpulse.com/trial-signup" rel="noreferrer">free Enterprise Trial</a> to test it out.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI4ODkyNTgxOTIzLWltYWdlMi0xICgxKS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Find the synthetic audio generator inside Edge Impulse<h3 id="isPasted">Case Study: Detecting the Sound of Breaking Glass</h3><iframe width="640" height="360" src="https://www.youtube.com/embed/vnyFk58qpf4?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe>In the attached video, we focus on a practical application: Train a model to detect the sound of glass breaking. This simple walk-through shows:&nbsp;<ul><li>How to generate an audio dataset using a text-to-sound generative AI models directly in Edge Impulse</li><li>An example of a trained model</li><li>A demonstration of the model inference</li></ul>This use case could be applied in smart security systems or for safety enhancements in industrial environments.If you have ever faced the challenge of collecting high-quality sound data that is prohibitively expensive or scarce, we hope this integration will make your lives easier!<hr><h3 id="isPasted"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739291988224-1.png" style="width: 100%px;" class="fr-fic fr-dib"></h3><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable"></iframe>

Similar to visual data, collecting and curating sound data that accurately reflects real-world scenarios is a major hurdle in training effective machine learning models.

That Sounds Great: Create Ultra-Realistic Audio Datasets with ElevenLabs.io

Recently we launched new Edge Impulse functionality that shows just one of the powerful ways that LLMs can be utilized on the edge: <a href="https://www.edgeimpulse.com/blog/llm-knowledge-distillation-gpt-4o/" rel="noreferrer">using GPT-4o to train a 2,000,000x smaller model</a>, that runs directly on device.Our initial demo showed how this tool could be used to label images for binary classification (presence/absence) but there are more complex labeling tasks possible.In this post and video we explore how visual data from a satellite could be labeled both for cloud type and for coverage, generating two useful edge models from the same dataset very quickly:<iframe width="640" height="360" src="https://www.youtube.com/embed/iXdyrDrY9Zs?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe><h3 id="isPasted">The challenge of AI for remote applications</h3>Satellites are resource constrained — power, compute and storage must all be managed carefully as expansion is rarely possible. Another issue is data transfer to Earth, which can be costly and has bandwidth restrictions. Current LLMs can only run in large data centers due to their unwieldy size (billions of parameters) so to utilize their power, raw data must be sent over the air to be computed.Edge AI is perfectly positioned to address these issues. By developing a model that can run on the device where data collection occurs, you can reduce data transmission to just the desired insight. For example, if you’re only interested in seeing clear images but a particular satellite image is obscured by clouds, it doesn't need to be sent down to earth for inspection. These methods can also be useful for improving existing edge models — sending raw data for classifications on-device that return an “unsure” or “anomaly” result to an LLM for labeling can be a good way to improve model performance over time.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739291889089-1.png" style="width: 100%px;" class="fr-fic fr-dib"><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable">&nbsp;</iframe><h3>Cloud type and coverage labeling examples</h3>To show the power of LLM labeling tools we have created two models from one raw dataset by designing two different LLM prompts for labeling. In the above video, I accessed an existing <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BSETKZ" rel="noreferrer">Harvard cloud dataset</a>. Using ChatGPT, I can ask questions about the images in the dataset, such as what type of cloud is shown or how much of the frame is covered by clouds. The latest LLMs excel at this type of image analysis, but running this procedure has high latency, and incurs significant cloud costs, especially when in a production environment with many source cameras and a high frame-rate.Here's how we can create a smaller model to perform this same tasks, on-device:<h3>Project walkthrough</h3><ol><li>Data collection: I downloaded the open <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BSETKZ" rel="noreferrer">Harvard cloud satellite-image dataset</a> (originally curated for cloud removal with spatiotemporal networks, a more complex task). Taking only the RGB images, I then uploaded them without labels into an Edge Impulse project. This unlabeled dataset forms the bases for both the cloud type classification and cloud coverage models we want to create.</li><li>Designing a prompt: I first experimented with labeling a few images in ChatGPT itself, noting the setting specific output options that worked best for each tasks. • Cloud Classification: My final prompt was “What cloud type is visible in this satellite picture? Respond only with “cirrus”, “cirrocumulus”, “cirrostratus”, “altocumulus”, “altostratus”, “nimbostratus”, “stratocumulus”, “stratus”, “cumulus”, “cumulonimbus”. If there is no cloud respond with "no cloud" or "unsure" if you're not sure. <img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI4ODkxNzgwOTUyLVVudGl0bGVkLS00LS0yLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 100%px;" class="fr-fic fr-dib">•&nbsp;Cloud Coverage: My final prompt was “Look at this image from a satellite and respond with a percentage of how much of the image is obscured by clouds. Respond only with a percentage not including "%" or "unsure" if the image is corrupted, or it is not an image from a satellite.”</li></ol><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI4ODkxODAzMzMxLVVudGl0bGVkLS01LS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">3. Labeling with GPT-4o: Using the new transformation block "Label image data using GPT-4o" inside each Edge Impulse project, I asked GPT-4o to label the images with my prompt. The block discards any blurry or uncertain images, providing a clean dataset. It also provides natural language reasoning for every labeling decision — a powerful tool for debugging model performance.4. Model Training: For each model type, I designed an impulse with the desired output format. I used a Transfer Learning Image classifier block for the cloud classification task and a Regression block for the cloud coverage task: •&nbsp;Cloud Classification: I trained a small MobileNetV1&nbsp;classification model with these images. The model was only 1.6MB in size, significantly smaller than the original LLM. •&nbsp;Cloud Coverage: I trained a small convolutional neural network regression model with these images. The model was only 49.1K in size, significantly smaller than the original LLM.5. Deployment: This model could be deployed on MCU hardware from Cortex M4 upwards, for example an Arduino Nicla Vision. The model accurately detects cloud types on-device, in real-time, without requiring cloud services.<h3>Results</h3>These two models were created in an incredibly short time. Data labeling is often the most time-consuming and error-prone part of training a model for the edge. By utilizing LLMs you can create your first-pass model much more quickly, and the power of having natural language reasons for each label makes debugging model datasets even easier.LLMs are still flawed and don’t perform perfectly for every use-case, but they are a great tool to assist subject matter experts in developing models faster. As generative AI continues to accelerate in popularity and complexity these techniques will only get better.<h3>References</h3><ul><li>Jan Jongboom <a href="https://www.edgeimpulse.com/blog/llm-knowledge-distillation-gpt-4o/" rel="noreferrer">Bringing Large Language Models to the Edge with GPT-4o and NVIDIA TAO</a></li><li>Met Office <a href="https://www.metoffice.gov.uk/weather/learn-about/weather/types-of-weather/clouds/cloud-names-classifications" rel="noreferrer">Cloud names and classifications</a></li><li>Edge Impulse <a href="https://studio.edgeimpulse.com/public/430153/latest" rel="noreferrer">GPT-4o Cloud Type Experiment</a></li><li>Edge Impulse <a href="https://studio.edgeimpulse.com/public/417680/latest" rel="noreferrer">GPT-4o Cloud Coverage Experiment</a></li><li>Harvard Dataverse <a href="https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/BSETKZ" rel="noreferrer">Dataset From: Cloud Removal from Satellite Images using Spatiotemporal Generator Networks</a></li></ul><hr><h3 id="isPasted"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739291937897-1.png" style="width: 100%px;" class="fr-fic fr-dib"></h3><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable"></iframe>

Recently we launched new Edge Impulse functionality that shows just one of the powerful ways that LLMs can be utilized on the edge: using GPT-4o to train a 2,000,000x smaller model, that runs directly on device.

Using LLMs to Analyze and Label Satellite Imagery in Edge Impulse

AI Infrastructure Engineering: Building Enterprise-Scale Machine Learning Systems

This article covers advanced architectural patterns, performance optimisation strategies, and critical operational considerations for building production-grade AI infrastructure systems!

Specialized hardware accelerators like Neural Processing Units (NPUs) and Tensor Processing Units (TPUs) have revolutionized artificial intelligence (AI) hardware by providing capabilities for accelerating machine learning tasks. Both are designed to optimize performance for tasks like machine learning, deep learning, and neural network computations, but they cater to distinct use cases and requirements.&nbsp;This article discusses the difference in architectures, performance, practical applications, and challenges of these powerful processing units, offering actionable insights for professionals.Recommended reading:&nbsp;<a href="https://www.wevolver.com/article/tpu-vs-gpu-in-ai-a-comprehensive-guide-to-their-roles-and-impact-on-artificial-intelligence" target="_blank">TPU vs GPU in AI: A Comprehensive Guide to Their Roles and Impact on Artificial Intelligence</a><h2 dir="ltr">Introduction</h2>The rapid advancement of AI has led to an increasing demand for specialized hardware capable of efficiently handling complex computations. NPUs and TPUs have emerged as pivotal technologies in this space, each tailored to optimize tasks involving complex machine learning algorithms in unique ways. NPUs, with their low-power design, enhance edge computing performance, making them indispensable for IoT and mobile devices. TPUs, on the other hand, excel in scaling deep learning workloads across massive datasets, driving breakthroughs in cloud-based AI.This article aims to provide professionals with a detailed comparison of these technologies, offering practical insights into their architectures, applications, and limitations. Understanding these distinctions can guide informed decision-making and optimize AI deployments for diverse requirements.<h2 dir="ltr">What Are Neural Processing Units (NPUs)?</h2>NPUs are specialized processors designed to optimize computations for neural networks, enabling efficient processing in real-time. With their ability to efficiently handle workloads like image recognition, voice processing, and natural language tasks, NPUs are an integral part of modern central processing units (CPUs) and System-on-Chip (SoC) designs. Companies like Huawei, Samsung, and Apple are leading the way in integrating NPUs into their devices to enhance AI capabilities without sacrificing power efficiency.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzM3NTY2MzI0NTQ0LWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib" alt="apple-neural-engine-devices">Apple's NPU, branded as the Neural Engine, is a key component of Apple's A-series and M-series chips. It is designed specifically for accelerating on-device machine learning and AI tasksRecommended reading:&nbsp;<a href="https://www.wevolver.com/article/npu-vs-gpu-understanding-the-key-differences-and-use-cases" target="_blank">NPU vs GPU: Understanding the Key Differences and Use Cases</a><h2 dir="ltr">What Are Tensor Processing Units (TPUs)?</h2>TPUs are hardware AI accelerator application-specific integrated circuits (ASICs) developed by Google to accelerate tensor-based computations, which are foundational to deep learning. Their design is optimized for mathematical operations like matrix-multiplication and parallel computing, making them ideal for tasks such as training large-scale neural networks and performing batch processing.Here is a block diagram showcasing the architecture of TPUs:<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzM3NTY2Mzc3NTczLWltYWdlNS5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib" alt="TPU-architecture-simplified-block-diagram">A simplified block diagram of a TPUThe TPU architecture is designed to optimize AI workloads by integrating specialized components for efficient computation, storage, and communication. At the core is the Matrix Multiply Unit, which performs rapid large-scale matrix operations critical for AI tasks. Data flows through the Unified Buffer, a local storage unit for managing activations, ensuring quick access and reducing dependency on external memory. The Control Unit orchestrates data movement and synchronizes operations between components. The PCIe Interface and Host Interface facilitate communication between the TPU and external systems, enabling smooth data transfer and coordination. The DRAM Ports provide access to external memory for managing large datasets that exceed the buffer's capacity. The Activation Pipeline processes intermediate results, applying transformations essential for refining AI computations. Additionally, the Accumulator stores partial computation results temporarily for aggregation and further processing. Finally, the Miscellaneous I/O manages input/output tasks, ensuring seamless integration with the broader system. This balanced architecture ensures high efficiency and performance for AI tasks in data centers.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzM3NTY2MzkwMzg5LWltYWdlNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib" alt="Google-TPU-v4">Google TPUs are primarily used in servers within data centers to accelerate large-scale AI workloads, such as training and deploying deep learning models.Recommended reading:&nbsp;<a href="https://www.wevolver.com/article/tensor-cores-vs-cuda-cores" target="_blank">Tensor Cores vs CUDA Cores: The Powerhouses of GPU Computing from Nvidia</a><h2 dir="ltr">Comparing the Performance of NPUs vs TPUs</h2>NPUs and TPUs exhibit distinct performance characteristics, making them suitable for different AI workloads. Following section compares both the AI chips based on different parameters.<h3 dir="ltr">Speed and Latency</h3>NPUs&nbsp;are designed for&nbsp;real-time, low-latency&nbsp;tasks, making them ideal for&nbsp;edge computing&nbsp;scenarios where immediate responses are required. For instance,&nbsp;NPUs&nbsp;enable instant image processing in&nbsp;smartphones&nbsp;and&nbsp;IoT devicesTPUs often excel in speed and scalability for large datasets, leveraging their specialized architecture for tensor computations. They are optimized for throughput rather than latency, making them suitable for batch data processing tasks like training massive deep learning models. In contrast, NPUs are designed for real-time, low-latency tasks on the edge.&nbsp;<h3 dir="ltr">Power and Energy Efficiency</h3>Power consumption is a critical factor for both NPUs and TPUs, but their optimization differs. NPUs provide better efficiency for edge computing and low-power applications, optimizing computing tasks with limited computational resources. &nbsp;This ensures extended battery life for small devices while performing complex AI tasks. TPUs, while efficient for their scale, are designed for environments like data centers where power availability is less constrained but efficiency remains essential for handling extensive computations.<h3 dir="ltr">Scalability and Flexibility</h3>NPUs&nbsp;are less scalable compared to&nbsp;TPUs, as they are tailored for specific edge&nbsp;AI&nbsp;applications. However, their&nbsp;flexibility&nbsp;in&nbsp;real-time&nbsp;processing tasks makes them invaluable for&nbsp;localized AI workloads.&nbsp;TPUs, on the other hand, are built for&nbsp;scalability, capable of handling&nbsp;large-scale&nbsp;AI&nbsp;models and&nbsp;datasets&nbsp;in&nbsp;cloud-based&nbsp;or&nbsp;distributed environments.<h2 dir="ltr">Architectural Differences</h2><h3 dir="ltr">Core Design</h3>NPUs&nbsp;are engineered to handle&nbsp;neural network activations&nbsp;and&nbsp;real-time&nbsp;matrix computations with specialized cores that focus on efficient, localized processing. These cores typically feature lightweight architectures optimized for low power consumption and latency-sensitive tasks.&nbsp;TPUs, in contrast, utilize systolic array architectures with a high density of computational cores designed for parallel matrix multiplications.<h3 dir="ltr">Programming Frameworks and Ecosystem</h3>NPUs are compatible with general-purpose AI frameworks such as TensorFlow, PyTorch, and ONNX, allowing developers to integrate AI tasks into various devices. As they are integrated within larger processors, they are more general-purpose and are compatible with a number of different computing tasks. TPUs, in contrast, are tightly integrated with Google’s TensorFlow ecosystem, making them an ideal choice for developers already working within Google’s AI infrastructure. The trade-off lies in flexibility versus specialization.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzM3NTY2NDEzOTA4LWltYWdlMS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib" alt="intel-core-ultra-gen-1-series-with-NPUs">Intel Core Ultra processors integrate NPUs to deliver efficient and enhanced performance for modern AI desktop applications.<h2 dir="ltr">Use Cases and&nbsp;Practical Applications</h2>NPUs and TPUs cater to distinct scenarios based on their specialized capabilities.NPUs find their strengths in edge computing, enabling AI tasks directly on devices like smartphones, cameras, and IoT sensors. Examples of&nbsp;NPU Applications&nbsp;include:<ul><li dir="ltr">Real-time object detection in edge devices, such as security cameras.</li><li dir="ltr">Voice recognition in smart assistants operating on low-power devices.</li><li dir="ltr">AI-powered healthcare devices for monitoring patient vitals and diagnosing conditions.</li><li dir="ltr">Industrial <a href="https://www.wevolver.com/article/building-automation-systems">automation systems</a> requiring real-time decision-making capabilities.</li></ul>TPUs are the backbone of AI operations on large amounts of data. Examples of&nbsp;TPU Applications&nbsp;include:<ul><li dir="ltr">Training large-scale machine learning models in cloud environments.</li><li dir="ltr">Batch processing for natural language understanding tasks in virtual assistants.</li><li dir="ltr">Optimizing recommendation systems in large-scale e-commerce platforms.</li><li dir="ltr">Enabling real-time analytics in large datasets for financial and scientific domains.</li></ul><h2 dir="ltr">Challenges and Limitations of NPUs and TPUs</h2><h3 dir="ltr">Engineering Challenges in Implementation</h3>Deploying NPUs and TPUs presents several technical challenges, including:<ul><li dir="ltr">Compatibility Issues: Existing software ecosystems, such as libraries and frameworks, often require extensive adaptation to fully leverage the capabilities of NPUs and TPUs.</li><li dir="ltr">Hardware Integration:&nbsp;Ensuring seamless integration with existing system architectures can be complex, particularly in legacy systems.</li><li dir="ltr">Optimization Challenges:&nbsp;AI models often need to be specifically optimized to achieve peak performance on these specialized processors.</li><li dir="ltr">Cost-Effectiveness:&nbsp;While powerful, the initial investment in NPU or TPU hardware can be prohibitive for smaller organizations.</li></ul><h3 dir="ltr">Limitations in Real-World Applications</h3>Despite their capabilities, NPUs and TPUs face limitations in specific environments:<ul><li dir="ltr">Latency-Sensitive Systems: TPUs, designed for high-throughput batch processing, may not be ideal for real-time applications that demand low latency.</li><li dir="ltr">Hybrid Requirements: Certain applications, such as autonomous vehicles, may require a combination of TPU and graphics processing unit (GPU) capabilities to balance precision and speed effectively.</li><li dir="ltr">Scalability Constraints:&nbsp;While TPUs scale well in cloud environments, NPUs are less adaptable to large-scale deployments due to their edge-specific design.</li></ul>Here is a table summarising the discussion on NPUs and TPUs:<div dir="ltr" align="left"><table><tbody><tr><td>Feature</td><td>NPUs</td><td>TPUs</td></tr><tr><td>Primary Use Case</td><td>Real-time AI tasks on edge devices (e.g., smartphones, laptops, IoT, robotics)</td><td>Large-scale AI workloads in data centers (e.g., training deep learning models)</td></tr><tr><td>Key Applications</td><td>Image recognition, natural language processing, voice recognition, augmented reality</td><td>Language models, natural language processing, large-scale image classification</td></tr><tr><td>Architectural Focus</td><td>Low-latency, power-efficient processing for real-time AI tasks</td><td>High-performance parallel processing and matrix multiplication for deep learning</td></tr><tr><td>Core Design</td><td>Lightweight architectures optimized for low power and real-time processing</td><td>Systolic array architectures optimized for massive parallel tensor operations</td></tr><tr><td>Scalability</td><td>Limited scalability, designed for specific edge AI tasks</td><td>High scalability for training large datasets and deploying AI models</td></tr><tr><td>Power Consumption</td><td>Optimized for energy efficiency in portable devices</td><td>Designed for efficiency in data centers with less strict power constraints</td></tr><tr><td>Programming Ecosystem</td><td>Supports general-purpose AI frameworks (e.g., TensorFlow, PyTorch, ONNX)</td><td>Tightly integrated with TensorFlow and Google Cloud services</td></tr><tr><td>Manufacturers</td><td>Companies like Apple (Neural Engine), Huawei, Samsung, and others</td><td>Google</td></tr><tr><td>Integration</td><td>Embedded in SoC designs for smartphones and IoT devices</td><td>Used as standalone accelerators in server environments</td></tr><tr><td>Key Strengths</td><td>Real-time processing, low latency, energy efficiency</td><td>Exceptional performance for large-scale AI training and inference</td></tr><tr><td>Key Limitations</td><td>Less suited for large-scale AI workloads</td><td>Higher power requirements, less suited for edge applications</td></tr></tbody></table></div><h2 dir="ltr">Conclusion</h2>NPUs and TPUs play complementary roles in the advancement of AI technology, each excelling in specific environments. While TPUs offer unparalleled performance for large-scale machine learning tasks in cloud environments, NPUs are perfect for real-time inference and energy-efficient edge applications.&nbsp;<h2 dir="ltr">Frequently Asked Questions</h2><ol><li dir="ltr">What Is the Primary Difference Between NPUs and TPUs? NPUs are specialized for low-power, real-time tasks, making them ideal for edge devices. TPUs are optimized for large-scale tensor computations, excelling in cloud-based machine learning.</li><li dir="ltr">What Are the Key Roles of NPUs and TPUs in AI? NPUs focus on real-time, low-power AI processing in edge devices like smartphones and IoT systems, enabling efficient inference for tasks such as image recognition and voice commands. TPUs specialize in high-performance, large-scale tensor computations, excelling in data centers for AI training and deployment.</li><li dir="ltr">How Do NPUs and TPUs Compare to GPUs? NPUs are specialized for efficient AI/ML tasks, particularly inference, and are optimized for low-power edge devices like smartphones. TPUs excel in tensor-based operations for deep learning, offering high-speed training and inference, primarily in Google's ecosystem. GPUs, versatile in design, handle a wide range of workloads, including graphics rendering, ML training, and scientific computing, making them ideal for parallel processing. While GPUs are general-purpose, NPUs and TPUs focus on AI-centric optimizations.</li></ol>Recommended reading:&nbsp;<a href="https://www.wevolver.com/article/understanding-nvidia-cuda-cores-a-comprehensive-guide" target="_blank">Understanding Nvidia CUDA Cores: A Comprehensive Guide</a><h2 dir="ltr">References</h2>[1] Google. Cloud TPU documentation. Available from: <a href="https://cloud.google.com/tpu/docs" target="_blank">https://cloud.google.com/tpu/docs</a>[2] Apple. Deploying Transformers on the Apple Neural Engine. Available from: <a href="https://machinelearning.apple.com/research/neural-engine-transformers" target="_blank">https://machinelearning.apple.com/research/neural-engine-transformers</a>[3] Van Belle JL. Why neural processing units (NPUs) are the next Big Thing in AI. 2024. doi:10.13140/RG.2.2.35619.87845. Available from: <a href="https://www.researchgate.net/publication/383184868_Why_neural_processing_units_NPUs_are_the_next_Big_Thing_in_AI" target="_blank">https://www.researchgate.net/publication/383184868_Why_neural_processing_units_NPUs_are_the_next_Big_Thing_in_AI</a>[4] Sanmartín D, Prohaska V. Exploring TPUs for AI Applications. IE Robotics &amp; AI Club, IE University, Madrid, Spain. arXiv preprint arXiv:2309.08918. Available from:<a href="https://arxiv.org/pdf/2309.08918" target="_blank">&nbsp;https://arxiv.org/pdf/2309.08918</a>.

NPUs are integrated units that excel in real-time AI tasks on edge devices like smartphones and IoT systems with low power consumption. TPUs are standalone processors designed for large-scale AI workloads in data centers, delivering exceptional performance in deep learning tasks.

NPU vs TPU: Understanding the Key Differences in AI Hardware Accelerators

GPUs excel in parallel processing for graphics and AI training with scalability, while NPUs focus on low-latency AI inference on edge devices, enhancing privacy by processing data locally. Together, they complement each other in addressing different stages of AI workloads efficiently.

NPU vs GPU: Understanding the Key Differences and Use Cases

CUDA Cores and Tensor Cores are specialized units within NVIDIA GPUs; the former are designed for a wide range of general GPU tasks, while the latter are specifically optimized to accelerate AI and deep learning through efficient matrix operations.

Computer Vision

Technical Guides

AI Infrastructure Engineering: Building Enterprise-Scale Machine Learning Systems...

NPU vs TPU: Understanding the Key Differences in AI Hardware Accelerators

NPU vs GPU: Understanding the Key Differences and Use Cases

RAG vs Fine-Tuning: Differences, Benefits, and Use Cases Explained

LLM Training: Mastering the Art of Language Model Development

Tensor Cores vs CUDA Cores: The Powerhouses of GPU Computing from Nvidia