Edge AI is the process of running artificial intelligence (AI) and machine learning (ML) algorithms on computing devices at the periphery of a network, rather than on large cloud servers. By doing this, we enable a wide range of possibilities to solve unique problems. The benefits of edge AI are the same as those found in edge computing: decreased network bandwidth usage, reduced network latency, less energy consumption, improved reliability, and enhanced data privacy.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739292008339-1.png" style="width: 100%px;" class="fr-fic fr-dib"><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable">&nbsp;</iframe>We are announcing the release of an Edge AI course to help you become familiar with edge computing and machine learning. The course is entirely free and hosted on our site with embedded YouTube videos. At the end of each section, you are encouraged to take a quiz to test your knowledge. Once you complete all 10 &nbsp;episodes, you can take a comprehensive test and earn a free digital certificate.Start the course here: <a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/intro-to-edge-ai" rel="noreferrer">Introduction to Edge AI</a>The videos are also accessible as a playlist on YouTube:<iframe width="640" height="360" src="https://www.youtube.com/embed/z5lIrWTebOY??list=PL7VEa1KauMQppUExiIOfxxk_2ta94EEpd&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe>We divide the course into 10 parts plus an introduction page:<ol><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/intro-to-edge-ai" rel="noreferrer">Introduction to edge AI</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-computing" rel="noreferrer">What is edge computing?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-machine-learning" rel="noreferrer">What is machine learning (ML)?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-ai" rel="noreferrer">What is edge AI?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/how-to-choose-an-edge-ai-device" rel="noreferrer">How to choose an edge AI device</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/edge-ai-lifecycle" rel="noreferrer">Edge AI lifecycle</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-mlops" rel="noreferrer">What is edge MLOps?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/what-is-edge-impulse" rel="noreferrer">What is Edge Impulse?</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/case-study-izoelektro" rel="noreferrer">Case study: Izoelektro smart grid monitoring</a></li><li><a href="https://edge-impulse.gitbook.io/docs/concepts/edge-ai/test-and-certification" rel="noreferrer">Test and certification</a></li></ol>The course is high-level to help you become familiar with the concepts and vocabulary around edge AI. There are no hands-on exercises or programming required. If you would like to dive into code and hardware, check out our <a href="https://www.coursera.org/learn/introduction-to-embedded-machine-learning">Introduction to Embedded Machine Learning course on Coursera</a>.Edge AI is a fast-growing field enabling new technologies, especially in healthcare and industrial settings. For example, Izoelektro created a <a href="https://edgeimpulse.com/case-studies/smart-grid-monitoring">power grid monitoring system</a>, and the <a href="https://edgeimpulse.com/case-studies/oura-health-monitoring">Oura Ring</a> tracks sleep with incredible precision.We are excited to see what you will build with edge AI!<hr><h3 id="isPasted"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739292036074-1.png" style="width: 100%px;" class="fr-fic fr-dib"></h3><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable"></iframe>

Edge AI is the process of running artificial intelligence (AI) and machine learning (ML) algorithms on computing devices at the periphery of a network, rather than on large cloud servers.

Get Certified with our new Introduction to Edge AI Course

Similar to visual data, collecting and curating sound data that accurately reflects real-world scenarios is a major hurdle in training effective machine learning models. The time, effort, and costs involved can be substantial, posing significant barriers, especially for projects with limited resources. As our mission is to lower the barriers of creating and utilizing ML and AI tools, we aim to drastically reduce these types of impediments, making the process faster, cheaper, and as reliable as possible.Edge AI enables smart devices to perform machine learning tasks right at the source of data collection, using small models that are optimized for performing scoped tasks as long as they are trained on quality data. That’s why we emphasize that quality is key: "Garbage in, garbage out."<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739291965210-1.png" style="width: 100%px;" class="fr-fic fr-dib"><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable">&nbsp;</iframe><h3>Improving sound datasets with generative AI</h3>To enhance your sound datasets, we've been working on integrating Edge Impulse with <a href="http://elevenlabs.io/">ElevenLabs.io</a>. Edge Impulse excels in crafting and optimizing models for edge computing. Meanwhile, ElevenLabs.io offers advanced capabilities for creating highly realistic sound effects. This integration enables you to expand your datasets with sounds that are difficult or expensive to record naturally, using state-of-the-art generative techniques provided by ElevenLabs.io. This integration not only saves time and money but also boosts the accuracy and reliability of the models we deploy on edge devices.&nbsp;It is accessible now in the new <a href="https://www.edgeimpulse.com/blog/announcing-synthetic-data-integration/" rel="noreferrer">Synthetic data tab</a> inside the Data Acquisition segment of Edge Impulse for Enterprise users. Sign up for a <a href="https://studio.edgeimpulse.com/trial-signup" rel="noreferrer">free Enterprise Trial</a> to test it out.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI4ODkyNTgxOTIzLWltYWdlMi0xICgxKS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Find the synthetic audio generator inside Edge Impulse<h3 id="isPasted">Case Study: Detecting the Sound of Breaking Glass</h3><iframe width="640" height="360" src="https://www.youtube.com/embed/vnyFk58qpf4?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;&nbsp;</iframe>In the attached video, we focus on a practical application: Train a model to detect the sound of glass breaking. This simple walk-through shows:&nbsp;<ul><li>How to generate an audio dataset using a text-to-sound generative AI models directly in Edge Impulse</li><li>An example of a trained model</li><li>A demonstration of the model inference</li></ul>This use case could be applied in smart security systems or for safety enhancements in industrial environments.If you have ever faced the challenge of collecting high-quality sound data that is prohibitively expensive or scarce, we hope this integration will make your lives easier!<hr><h3 id="isPasted"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1739291988224-1.png" style="width: 100%px;" class="fr-fic fr-dib"></h3><iframe width="540" height="375" src="https://c8056756.sibforms.com/serve/MUIFAMXbzNzNSyMeYsMybLI-54_IoJGam-Ow5Rafkxl9--KaPuomcDrglOqbvtjakyD2bXs-rydfKR0xW02WlzYWFyZRaIxw6KHBvpL-qZHhww0M5OsSm7r0LrA90cGrXNvcfbrPS_J7fBOttsX1qray3uaPjy5ps39DeMLyY0mPjNw-CYJyrnWFs9IZWDxJfo_GrULjpgCTCRnD" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable"></iframe>

Similar to visual data, collecting and curating sound data that accurately reflects real-world scenarios is a major hurdle in training effective machine learning models.

That Sounds Great: Create Ultra-Realistic Audio Datasets with ElevenLabs.io

‘Industry 4.0’—the next&nbsp;stage of the Industrial Revolution—leans heavily on the Industrial Internet of Things (IIoT) to support large scale automation of traditional manufacturing practices. In turn, today’s IIoT is founded on wireless machine-to-machine (M2M) communication,&nbsp;<a href="https://www.nordicsemi.com/Products/Technologies/Edge-AI" target="_blank" rel="noreferrer noopener">edge computing</a>, and Machine Learning (ML).&nbsp;Wireless networks&nbsp;using these technologies&nbsp;link machine tools,&nbsp;programmable logic controllers (PLCs), sensors, gateways, and the Cloud, enabling every part of the factory to gather and process data and share information with every other part as well as the Internet. This&nbsp;deep&nbsp;mine of information is enabling engineers to revolutionize how everything is manufactured - from cans to cars, screws to smartphones, and jigsaws to jet turbines.&nbsp;But there is always room for improvement. For example, machines can be further enhanced to improve yield and quality while using less material, while engineers can design products for greater sustainability. A new generation of IIoT solutions promises to make this happen.&nbsp;<h2>IIoT&nbsp;and M2M&nbsp;increase manufacturing flexibility&nbsp;</h2>Wireless technology not only cuts the cost of implementing IIoT connectivity it also makes it easy to reconfigure networks as the factory adapts and expands. Such technology not only changes the way products are made but also how they’re designed.&nbsp;&nbsp;Now, connectivity links the front office to the factory floor using M2M communications, allowing computer aided design (CAD) tools to talk to machine tools to directly program them to make parts. And machine tools can speak to CAD to let them know where the bottlenecks are in the manufacturing process such that products can be redesigned for simpler manufacture without compromising function.&nbsp;The results are an increase in productivity and reduction in product failures - bringing sustained cost savings and environmental benefits.&nbsp;<h2>Edge computing and&nbsp;ML&nbsp;optimize&nbsp;production in&nbsp;smart factories&nbsp;</h2>Once mass production is up-and-running, wireless technology makes the modern factory a highly controlled and optimized environment. Things rarely go wrong. But rarely doesn’t mean never. Today’s wireless&nbsp;<a href="https://www.nordicsemi.com/Products/Wireless/Bluetooth-Low-Energy" target="_blank" rel="noreferrer noopener">Bluetooth LE</a>&nbsp;Systems-on-Chip (SoCs) from Nordic incorporate powerful embedded processors to perform edge processing, looking for the infrequent trends that indicate things might be changing for the worse. This edge computing is increasingly reliant on ML models constantly adapting to spot deviations in sensor trends and forwarding the key information for further action.&nbsp;&nbsp;Such models can help to pre-empt issues that might arise due to these external factors—for instance, an increase in humidity caused by workers arriving for the day, air flow from open windows and doors, and changes in temperature throughout the day and night—and adjust machine settings in advance of those factors&nbsp;impacting&nbsp;manufacturing processes. And dedicated vibration and acoustic sensors can&nbsp;monitor&nbsp;the machine tools to make sure they are in the best of health. Any unusual vibration, temperature increase or increase in power consumption can be reported ahead of a breakdown for early maintenance – thus preventing an unscheduled and expensive hiatus in production.&nbsp;<h2>Tech to&nbsp;power the industrial future&nbsp;</h2>But while modern factories are impressive examples of the power of connectivity, standing still is not an option. The Japanese concept of Kaizen (“improvement”) epitomizes this strategy. Kaizen guides continuous effort across the factory to improve programs and processes while eliminating waste and redundancy. Kaizen has proved so successful that it has spread throughout the world.&nbsp;To continue the improvement process, factories will need even better wireless technology than they have today. Nordic is working hard and investing deeply to build and supply the next generation of IIoT technology. Several multiprotocol SoCs and Systems-in-Package (SiPs) from Nordic’s product range already support TinyML, a framework for running embedded ML on constrained devices. But there’s much more to come.&nbsp;The company’s new&nbsp;<a href="https://www.nordicsemi.com/Products/nRF54H20">nRF54H20 SoC</a>, for example, integrates multiple Arm Cortex-M33 processors, clocked up to 320 MHz, together with multiple RISC-V coprocessors. The higher clock speeds of the SoC’s application processors allow for reduced processing latency for AI operations, when compared to Nordic’s nRF5340 SoC.&nbsp;&nbsp;Better yet the nRF54H20 brings increased efficiency—cutting the power consumption required to run equivalent operations—because it is fabricated using the cutting-edge 22-nm process node. In addition, Nordic’s next-generation multiprotocol radio offers improvements in both TX power and RX sensitivity, enabling efficient and reliable transfer of information from embedded ML applications to Cloud connected devices, or between different nodes in a network. This makes the SoC ideal for next generation IIoT applications.&nbsp;&nbsp;The nRF54H20 will be supported in Nordic’s&nbsp;<a href="https://www.nordicsemi.com/Products/Technologies/Edge-AI/ML-Studio?lang=en#infotabs">ML Studio</a>&nbsp;alongside the nRF52, nRF53, nRF91 Series products already supported.&nbsp;&nbsp;<h2>The beginning of a revolution&nbsp;</h2>Wireless tech is&nbsp;booming&nbsp;in&nbsp;IIoT&nbsp;applications.&nbsp;According to&nbsp;ABI Research, the Bluetooth-enabled industrial devices market&nbsp;alone&nbsp;is expected to grow from 143 million annual unit shipments in 2023 to over 611 million by 2028, achieving a CAGR of 34 percent over the five-year forecast&nbsp;period[1]. But even with this impressive growth things are just getting&nbsp;started; Industry 4.0 is set to have a larger impact on global manufacturing than all three&nbsp;previous&nbsp;major leaps in the industrial revolution combined.&nbsp;<hr><h3>References </h3>1. https://www.abiresearch.com/market-research/product/7783366-how-can-bluetooth-technology-enable-digita/&nbsp;

‘Industry 4.0’—the next stage of the Industrial Revolution—leans heavily on the Industrial Internet of Things (IIoT) to support large scale automation of traditional manufacturing practices.

Machine Learning is transforming the Industrial Internet of Things

Simulating digital actions to teach robots new tasks is also time-consuming for humans, though. Cutting those minutes in half, MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) PhD student Lirui Wang and his colleagues’ new “GenSim2” framework uses multimodal and reasoning large language models (LLMs that process and produce text, images, and other media) to supersize training data for robots. The researchers combined the powers of multimodal LLM GPT-4V (which can draw better inferences about text and images) and reasoning LLM OpenAI o1 (which can “think” before answering) to take ten real-world videos of tasks and generate 100 new, simulated action videos.GenSim2 can then convert task names into task descriptions and then to task code, which can be simulated into a sequence of actions for a robot to execute. The approach could eventually assist home robots with tasks like figuring out each step needed to reheat your breakfast, including opening a microwave and placing bread in a toaster. It could also help in manufacturing and logistics settings one day, where a machine may need to transport new materials in several steps. This framework is a sequel to Wang’s earlier work,&nbsp;<a href="https://www.csail.mit.edu/news/using-llms-code-new-tasks-robots" target="_blank">“GenSim,”</a> which used LLMs to encode new pick-and-place tasks for robots. He wanted to expand his approach to more dexterous activities with more complex object categories, like opening a box or closing a safe.&nbsp; “To plan these more complicated chores in robotics, we need to figure out how to solve them,” says Wang. “This planning problem was not present in GenSim, since the tasks were much simpler, so we only needed “blind” LLMs. With GenSim2, we integrated the logic model GPT-4V, which teaches multimodal models to “see” by analyzing image inputs with better reasoning skills. Now, we can code the simulation task, and then generate plans in seconds.”<h2 dir="ltr">The nuts and bolts of GenSim2</h2>First, you prompt an LLM like GPT-4 to generate a novel task plan like “place a ball in a box,” including images, assets, and keypoints (or specific points in an image). From there, GPT-4V reviews these details and concisely encodes which poses and actions are needed to execute the task. Humans can provide feedback about this plan to GPT-4V, and then it will refine its outline. Finally, a motion planner simulates those actions into videos, generating new training data for the robot.To convert these plans into actions, the researchers also designed a new architecture called the “proprioceptive point-cloud transformer” (PPT). PPT converts language, point cloud (data points within a 3D space), and proprioception inputs into a final action sequence. This allows a robot to learn to imitate video simulations and generalize to objects it hasn’t seen before.<h2 dir="ltr">Lights, camera, action plan!</h2>GenSim2’s souped-up approach generated data for 100 articulated tasks with 200 objects. Among these, the system simulated 50 long-horizon tasks, such as securing gold in a safe and preparing breakfast. Compared to the generative robotic agent and baseline “<a href="https://arxiv.org/abs/2311.01455" target="_blank">RoboGen</a>,” GenSim2 had a 20% better success rate with generating and planning primitive tasks, while also being more reliable with long-horizon ones. The researchers note that having multimodal models that can reason about visual inputs gave them the edge.Another intriguing finding: it only took humans about four minutes on average to verify robotic plans — half of how long it took them to design a task manually. Human efforts included labeling keypoints in the motion planner and giving feedback to help the multimodal language model improve its plans.In real-world experiments, GenSim2 successfully helped plan tasks for a robot, like opening a laptop and closing a drawer. When it trained on both simulation and real data to develop its robotic policy, the framework had a better success rate than either one standalone. This reduces the required effort to collect large amounts of data in the real world. While GenSim2 is a more intricate, advanced follow-up to its predecessor, the researchers note that they’d like it to plan and simulate robotic tasks with even less human intervention. Currently, it struggles to reliably create and code meaningful tasks on its own.&nbsp;Wang also notes that while it’s a step forward in achieving automated task generation, the researchers intend to make the system more advanced. To do this, they plan to increase task complexity and diversity through advanced multimodal agents and generate 3D assets.“Scaling up robot data has been a major challenge in creating generalizable robot foundation models,” says Yunzhu Li, Assistant Professor of Computer Science at Columbia University, who wasn’t involved in the paper. “GenSim2 addresses this by developing a scalable framework for data and action generation, using a combination of simulation, GPT-4, and sim-to-real transfer. I’m excited to see how this work could spark a “GPT moment” for robotics by effectively expanding the data available for robots.” Wang wrote the paper with Tsinghua University assistant professor Huazhe Xu and PhD student Pu Hua, Shanghai Jiao Tong University professor Weinan Zhang and PhD students Minghuan Liu and Yunfeng Lin, and University of California at San Diego student researcher Annabella Macaluso. Their work was supported, in part, by Amazon and the company’s Greater Boston Tech Initiative. The researchers will present their work at the Conference on Robot Learning next month in Munich, Germany.

For robots, simulation is a great teacher for learning long-horizon (multi-step) tasks — especially compared to how long it takes to collect real-world training data.

Multimodal and reasoning LLMs supersize training data for dexterous robotic tasks

Imagine you’re tasked with sending a team of football players onto a field to assess the condition of the grass (a likely task for them, of course). If you pick their positions randomly, they might cluster together in some areas while completely neglecting others. But if you give them a strategy, like spreading out uniformly across the field, you might get a far more accurate picture of the grass condition.Now, imagine needing to spread out not just in two dimensions, but across tens or even hundreds. That's the challenge MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers are getting ahead of. They've developed an AI-driven approach to “low-discrepancy sampling,” a method that improves simulation accuracy by distributing data points more uniformly across space.A key novelty lies in using graph neural networks (GNNs), which allow points to “communicate” and self-optimize for better uniformity. Their approach marks a pivotal enhancement for simulations in fields like robotics, finance, and computational science, particularly in handling complex, multidimensional problems critical for accurate simulations and numerical computations.“In many problems, the more uniformly you can spread out points, the more accurately you can simulate complex systems,” says T. Konstantin Rusch, lead author of the new paper and MIT CSAIL postdoc. “We've developed a method called Message-Passing Monte Carlo (MPMC) to generate uniformly spaced points, using geometric deep learning techniques. This further allows us to generate points that emphasize dimensions which are particularly important for a problem at hand, a property that is highly important in many applications. The model’s underlying graph neural networks lets the points 'talk' with each other, achieving far better uniformity than previous methods.”Their work was&nbsp;<a href="https://www.pnas.org/doi/full/10.1073/pnas.2409913121" target="_blank">published in the September issue of the&nbsp;Proceedings of the National Academy of Sciences</a>.<h2 dir="ltr">Take me to Monte Carlo</h2>The idea of Monte Carlo methods is to learn about a system by simulating it with random sampling. Sampling is the selection of a subset of a population to estimate characteristics of the whole population. Historically, it was already used in the 18th century, &nbsp;when mathematician Pierre-Simon Laplace employed it to estimate the population of France without having to count each individual.Low-discrepancy sequences, which are sequences with low discrepancy, i.e., high uniformity, such as Sobol’, Halton, and Niederreiter, have long been the gold standard for quasi-random sampling, which exchanges random sampling with low-discrepancy sampling. They are widely used in fields like computer graphics and computational finance, for everything from pricing options to risk assessment, where uniformly filling spaces with points can lead to more accurate results.&nbsp; The MPMC framework suggested by the team transforms random samples into points with high uniformity. This is done by processing the random samples with a GNN that minimizes a specific discrepancy measure.One big challenge of using AI for generating highly uniform points is that the usual way to measure point uniformity is very slow to compute and hard to work with. To solve this, the team switched to a quicker and more flexible uniformity measure called L2-discrepancy. For high-dimensional problems, where this method isn’t enough on its own, they use a novel technique that focuses on important lower-dimensional projections of the points. This way, they can create point sets that are better suited for specific applications.The implications extend far beyond academia, the team says. In computational finance, for example, simulations rely heavily on the quality of the sampling points. “With these types of methods, random points are often inefficient, but our GNN-generated low-discrepancy points lead to higher precision,” says Rusch. “For instance, we considered a classical problem from computational finance in 32 dimensions, where our MPMC points beat previous state-of-the-art quasi-random sampling methods by a factor of four to 24.”<h2 dir="ltr">Robots in Monte Carlo</h2>In robotics, path and motion planning often rely on sampling-based algorithms, which guide robots through real-time decision-making processes. The improved uniformity of MPMC could lead to more efficient robotic navigation and real-time adaptations for things like autonomous driving or drone technology. “In fact, in a recent preprint, we demonstrated that our MPMC points achieve a fourfold improvement over previous low-discrepancy methods when applied to real-world robotics motion planning problems,” says Rusch.“Traditional low-discrepancy sequences were a major advancement in their time, but the world has become more complex, and the problems we're solving now often exist in 10, 20, or even 100-dimensional spaces,” says Daniela Rus, CSAIL director and MIT professor of electrical engineering and computer science. “We needed something smarter, something that adapts as the dimensionality grows. GNNs are a paradigm shift in how we generate low-discrepancy point sets. Unlike traditional methods, where points are generated independently, GNNs allow points to 'chat' with one another so the network learns to place points in a way that reduces clustering and gaps — common issues with typical approaches.”Going forward, the team plans to make MPMC points even more accessible to everyone, addressing the current limitation of training a new GNN for every fixed number of points and dimensions.“Much of applied mathematics uses continuously varying quantities, but computation typically allows us to only use a finite number of points,” says Art B. Owen, Stanford University professor of statistics, who wasn’t involved in the research. “The century-plus-old field of discrepancy uses abstract algebra and number theory to define effective sampling points. This paper uses graph neural networks to find input points with low discrepancy compared to a continuous distribution. That approach already comes very close to the best-known low-discrepancy point sets in small problems and is showing great promise for a 32-dimensional integral from computational finance. We can expect this to be the first of many efforts to use neural methods to find good input points for numerical computation.”Rusch and Rus wrote the paper with University of Waterloo researcher Nathan Kirk, Oxford University’s DeepMind Professor of AI and former CSAIL affiliate Michael Bronstein, and University of Waterloo Statistics and Actuarial Science Professor Christiane Lemieux. Their research was supported, in part, by the AI2050 program at Schmidt Futures, Boeing, the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator, the Swiss National Science Foundation, Natural Science and Engineering Research Council of Canada, and an EPSRC Turing AI World-Leading Research Fellowship.

MIT CSAIL researchers developed an AI-driven approach to “low-discrepancy sampling,” a method that improves simulation accuracy by distributing data points more uniformly across space.

How AI is improving simulations with smarter sampling techniques

Imagine having to straighten up a messy kitchen, starting with a counter littered with sauce packets. If your goal is to wipe the counter clean, you might sweep up the packets as a group. If, however, you wanted to first pick out the mustard packets before throwing the rest away, you would sort more discriminately, by sauce type. And if, among the mustards, you had a hankering for Grey Poupon, finding this specific brand would entail a more careful search.MIT engineers have developed a method that enables robots to make similarly intuitive, task-relevant decisions.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764894016-MIT-Scene-Tasks-01-press_0.jpg" style="width: 100%px;" class="fr-fic fr-dib">From left to right: team members Lukas Schmid, Nathan Hughes, Dominic Maggio, Yun Chang, and Luca Carlone. Credit: Andy RyanThe team’s new approach, named Clio, enables a robot to identify the parts of a scene that matter, given the tasks at hand. With Clio, a robot takes in a list of tasks described in natural language and, based on those tasks, it then determines the level of granularity required to interpret its surroundings and “remember” only the parts of a scene that are relevant.In real experiments ranging from a cluttered cubicle to a five-story building on MIT’s campus, the team used Clio to automatically segment a scene at different levels of granularity, based on a set of tasks specified in natural-language prompts such as “move rack of magazines” and “get first aid kit.”The team also ran Clio in real-time on a quadruped robot. As the robot explored an office building, Clio identified and mapped only those parts of the scene that related to the robot’s tasks (such as retrieving a dog toy while ignoring piles of office supplies), allowing the robot to grasp the objects of interest.Clio is named after the Greek muse of history, for its ability to identify and remember only the elements that matter for a given task. The researchers envision that Clio would be useful in many situations and environments in which a robot would have to quickly survey and make sense of its surroundings in the context of its given task.“Search and rescue is the motivating application for this work, but Clio can also power domestic robots and robots working on a factory floor alongside humans,” says Luca Carlone, associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator in the Laboratory for Information and Decision Systems (LIDS), and director of the MIT SPARK Laboratory. “It’s really about helping the robot understand the environment and what it has to remember in order to carry out its mission.”The team details their results in a <a href="https://dspace.mit.edu/handle/1721.1/157072" target="_blank">study appearing today</a> in the journal Robotics and Automation Letters. Carlone’s co-authors include members of the SPARK Lab: Dominic Maggio, Yun Chang, Nathan Hughes, and Lukas Schmid; and members of MIT Lincoln Laboratory: Matthew Trang, Dan Griffith, Carlyn Dougherty, and Eric Cristofalo.<h2>Open fields</h2>Huge advances in the fields of computer vision and natural language processing have enabled robots to identify objects in their surroundings. But until recently, robots were only able to do so in “closed-set” scenarios, where they are programmed to work in a carefully curated and controlled environment, with a finite number of objects that the robot has been pretrained to recognize.In recent years, researchers have taken a more “open” approach to enable robots to recognize objects in more realistic settings. In the field of open-set recognition, researchers have leveraged deep-learning tools to build neural networks that can process billions of images from the internet, along with each image’s associated text (such as a friend’s Facebook picture of a dog, captioned “Meet my new puppy!”).From millions of image-text pairs, a neural network learns from, then identifies, those segments in a scene that are characteristic of certain terms, such as a dog. A robot can then apply that neural network to spot a dog in a totally new scene.But a challenge still remains as to how to parse a scene in a useful way that is relevant for a particular task.“Typical methods will pick some arbitrary, fixed level of granularity for determining how to fuse segments of a scene into what you can consider as one ‘object,’” Maggio says. “However, the granularity of what you call an ‘object’ is actually related to what the robot has to do. If that granularity is fixed without considering the tasks, then the robot may end up with a map that isn’t useful for its tasks.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727764931766-MIT-Scene-Tasks-03-press.jpg" style="width: 100%px;" class="fr-fic fr-dib">Clio quickly segments a scene such as a cluttered cubicle (pictured), and feeds the segments through an “information bottleneck” algorithm that focuses on the segments that only represent target objects relevant to the task at hand. Colored cubes represent segment regions that Clio has picked identified as task-relevant objects that a robot can focus on while ignoring the rest of the scene. Credit: Courtesy of the researchers<h2>Information bottleneck</h2>With Clio, the MIT team aimed to enable robots to interpret their surroundings with a level of granularity that can be automatically tuned to the tasks at hand.For instance, given a task of moving a stack of books to a shelf, the robot should be able to &nbsp;determine that the entire stack of books is the task-relevant object. Likewise, if the task were to move only the green book from the rest of the stack, the robot should distinguish the green book as a single target object and disregard the rest of the scene — including the other books in the stack.The team’s approach combines state-of-the-art computer vision and large language models comprising neural networks that make connections among millions of open-source images and semantic text. They also incorporate mapping tools that automatically split an image into many small segments, which can be fed into the neural network to determine if certain segments are semantically similar. The researchers then leverage an idea from classic information theory called the “information bottleneck,” which they use to compress a number of image segments in a way that picks out and stores segments that are semantically most relevant to a given task.“For example, say there is a pile of books in the scene and my task is just to get the green book. In that case we push all this information about the scene through this bottleneck and end up with a cluster of segments that represent the green book,” Maggio explains. “All the other segments that are not relevant just get grouped in a cluster which we can simply remove. And we’re left with an object at the right granularity that is needed to support my task.”The researchers demonstrated Clio in different real-world environments.“What we thought would be a really no-nonsense experiment would be to run Clio in my apartment, where I didn’t do any cleaning beforehand,” Maggio says.The team drew up a list of natural-language tasks, such as “move pile of clothes” and then applied Clio to images of Maggio’s cluttered apartment. In these cases, Clio was able to quickly segment scenes of the apartment and feed the segments through the Information Bottleneck algorithm to identify those segments that made up the pile of clothes.They also ran Clio on Boston Dynamic’s quadruped robot, Spot. They gave the robot a list of tasks to complete, and as the robot explored and mapped the inside of an office building, Clio ran in real-time on an on-board computer mounted to Spot, to pick out segments in the mapped scenes that visually relate to the given task. The method generated an overlaying map showing just the target objects, which the robot then used to approach the identified objects and physically complete the task.“Running Clio in real-time was a big accomplishment for the team,” Maggio says. “A lot of prior work can take several hours to run.”Going forward, the team plans to adapt Clio to be able to handle higher-level tasks and build upon recent advances in photorealistic visual scene representations.“We’re still giving Clio tasks that are somewhat specific, like ‘find deck of cards,’” Maggio says. “For search and rescue, you need to give it more high-level tasks, like ‘find survivors,’ or ‘get power back on.’ So, we want to get to a more human-level understanding of how to accomplish more complex tasks.”This research was supported, in part, by the U.S. National Science Foundation, the Swiss National Science Foundation, MIT Lincoln Laboratory, the U.S. Office of Naval Research, and the U.S. Army Research Lab Distributed and Collaborative Intelligent Systems and Technology Collaborative Research Alliance.

MIT’s Clio runs in real-time to map task-relevant objects in a robot’s surroundings, allowing the bot (Boston Dynamic’s quadruped robot Spot, pictured) carry out a natural language task (“pick up orange backpack”). Credit: Courtesy of the researchers

A new method called Clio enables robots to quickly map a scene and identify the items they need to complete a given set of tasks.

Helping robots zero in on the objects that matter

<h2 id="isPasted">Prospects of Beyond 5G/6G and technology to achieve it</h2><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727661978457-img-0001+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Mobile communications systems are dramatically enhanced on B5G/6GBeyond 5G (5th generation and later mobile communications system: hereinafter “B5G”) / 6G (6th generation mobile communications system) is expected to be applied and spread in popularity as a new form of information communications infrastructure from the 2030s. In addition to enhancing 5G (5th generation mobile communications system) features such as ultra-high speed, large capacity, low latency, and simultaneous multiple connections, B5G/6G is expected to have ultralow power consumption as well as features such as expandability of covered communication areas (coverage), network autonomy, ultra safety, and reliability.On this website, as technologies applied to achieve the characteristics of B5G/6G, we have covered future technology trends such as high-speed communication and sensing through <a href="https://article.murata.com/en-global/article/thz-communications-and-sensing-expected-for-b5g-6g" target="_blank">terahertz waves</a> (and subterahertz waves), as well as significant coverage expansions through&nbsp;<a href="https://article.murata.com/en-global/article/what-is-a-ntn-of-haps-and-satellites" target="_blank">Non-Terrestrial Networks (NTN)</a>. This article explains the prospects of power consumption reduction in communications equipment as well as communications resource optimization in multiple simultaneous connections through autonomy that apply AI (Artificial Intelligence) and ML (Machine Learning), which are expected to further enhance 5G.<h2>AI/ML in B5G/6G as well as their importance</h2>AI (Artificial Intelligence) and ML (Machine Learning), which are technologies that are crucial to 5G and next-generation mobile communications systems, are both computer science terms. Let’s review what they are about and touch on their differences.<h3>Overview and differences of AI and ML</h3>AI refers to the comprehensive ability that enables computers and machines to imitate human actions. On the other hand, ML is an application of AI, and refers to the education process applied to systems that enable them to compare and classify massive amounts of data (big data) using a neural network*1. Repeating this process eliminates the explicit use of programs, enabling computers and machines to learn to provide predictions as well as perform automatic processing.*1 A neural network is a mathematical model that was modeled based on the workings of the human brain and applied in a simplified format on a computer. It is comprised of input and output layers as well as an intermediate layer in between.<h3>The importance of AI/ML in B5G/6G</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727662038930-img-0002+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Prepare a network assumed to make use of AI/ML for B5G/6GConsidering the advanced features expected from B5G/6G such as ultra-high speed, large capacity, low latency, simultaneous multiple connections, ultralow power consumption as well as autonomous predictions and control which were mentioned at the beginning of this article, it will be difficult to use them with fixed network settings or at times having to optimize and control the network manually. Additionally, issues such as increased power consumption and environmental load that occur due to communications infrastructure enhancement cannot be overlooked.As AI/ML are technologies required to solve these kinds of issues, research has been performed to control and optimize communications network management in anticipation of the enhancement of 5G, which began formally being applied for commercial use around 2020, as well as the age of B5G/6G, which is assumed to come around 2030. Furthermore, while 5G applies AI/ML technologies simply as additional features, B5G/6G is an AI-native mobile communications system that assumes the use of AI/ML.<h2>The introduction objective and utilization prospects of AI/ML in B5G/6G</h2>As previously mentioned, AI/ML are considered as solutions for future mobile communications system issues. Here, we will explain the objectives of AI/ML introduction as well as several desired use examples.<h3>What we aim to achieve by introducing AI/ML to communications network management</h3>The following are examples of the main objectives we wish to achieve by introducing AI/ML to communications network management for 5G and B5G/6G.<ul><li>Throughput*2 enhancement Efficient wireless communications through communications resource management and optimization</li><li>Energy saving Reduction of power consumption through efficient network operation</li><li>Communications trouble prediction and response Prompt prediction and autonomous response to communications latencies and connection failures</li></ul>*2 Throughput is the amount of data that, for example, devices such as computers can process per unit time and indicates the effective transmission rating per unit time for communication lines.Below, we will explain the prospects of future AI/ML use foreseeing these objectives.<h3>Communications optimization through autonomy using AI/ML</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727662107374-img-0003+%281%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Achieving desired specifications for B5G/6G through AI/ML introductionTo communicate large capacities at high speed on more and more devices, B5G/6G is required to optimize communications resource use for both the sender and receiver in real time, and perform it at a higher level. For example, for technologies such as beamforming*3 for next-generation communications satellites of ground base stations and <a href="https://article.murata.com/en-global/article/what-is-a-ntn-of-haps-and-satellites" target="_blank">Non-Terrestrial Networks (NTN)</a> as well as&nbsp;<a href="https://article.murata.com/en-global/article/thz-communications-and-sensing-expected-for-b5g-6g" target="_blank">terahertz waves</a> (and subterahertz waves) with a short communications distance and attenuate due to liquids such as rain, the performance may vary depending on condition changes such as positional relationships between each device and base station as well as the weather and radio wave demand. Here, we teach data related to weather and radio wave demand per time zone and location using AI/ML, and have it make predictions swiftly. Based on this, by optimizing communications equipment control, we will be capable of making communications resource use more efficient and reduce power consumption through learning capacity and autonomy as well as making stability improvements for communications and connectivity.*3 Beamforming is a technology that controls radio waves to be sent collectively in a specific direction. When outputting radio waves from multiple antennas, by adjusting the phase and power intensity of each radio wave, the signal is strengthened in one direction while being canceled out and weakened in another direction. This characteristic is utilized for the control.Additionally, introduction of IoT devices and systems using wireless communications are steadily increasing following the transition of society into a smart society that includes smart cities, smart factories, and smart homes. It is also quite common that cellular circuits are applied for cloud servers and data communications. For example, the use of AI (Edge AI) to autonomously send and receive required data by having multiple IoT equipment determine what to filter, such as when sending limited data when the image sensor recognizes specific conditions, can be considered as a valid AI use technique to reduce the burden for power consumption and communications resources.<h3>Learning enhancement as well as prediction and autonomy responses for communications latency and failures through digital twins</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1727662153437-img-0004+%282%29.jpg" style="width: 100%px;" class="fr-fic fr-dib">Learning enhancement through digital twins promotes AI/ML autonomyLearning enhancement and simulations using digital twins*4 are examples of use methods for AI/ML that are expected to be applied in the age of B5G/6G. For example, when constructing smart cities, they can be applied to generate simulations by applying various conditions in virtual spaces and then performing proof of concept (PoC) for equipment and systems in the real world.Related articles:&nbsp;<a href="https://article.murata.com/en-global/article/examples-of-diverse-applications-of-3d-urban-models" target="_blank">3D City Models Accelerate the Creation of Smart Cities and Digital Twins, Data-Driven Efforts in Electronics</a>Applying possible communications latencies and communications interferences to AI/ML learning enhancement and simulations in communications networks of virtual spaces will be effective in AI predictive detection of network failures. We can assume a communications system with high autonomy by applying these learned features to real world scenarios.*4 Digital twins are virtual replicas of environments and systems, similar to those in the real world, generated in virtual spaces on a computer.Murata Manufacturing Co., Ltd. develops and manufactures wireless connectivity modules as well as edge AI modules that can be applied to image and audio recognition. For more details, access the following link for related products. In addition to our existing products, we are actively promoting R&amp;D for new products that will contribute to the development of future wireless communications technologies.

Beyond 5G (5th generation and later mobile communications system: hereinafter “B5G”) / 6G (6th generation mobile communications system) is expected to be applied and spread in popularity as a new form of information communications infrastructure from the 2030s.

Communications network management applying AI & ML: Technological Trends Leading to Beyond 5G & 6G

<section id="isPasted">Robots are widely used in the automotive industry and have started entering new application domains such as logistics in the last few years. However, current robots still face many limitations. They typically perform a single action or a fixed sequence of actions, repeating them the same way each time. To achieve greater efficiency and open new possibilities, robots must develop more human-like skills, such as fast physical interaction, spatial understanding, and fast adaptation to changes. We spoke with Alessandro Saccon, Associate Professor in nonlinear control and robotics at the department of Mechanical Engineering at the TU/e. He recently completed the<a href="http://i.am/" target="_blank" rel="noreferrer">&nbsp;</a><a href="http://i.am/" target="_blank" rel="noreferrer">I.AM project</a> that explicitly focuses on the advancement of fast physical interactions.Source:<a href="https://innovationorigins.com/en/" target="_blank" rel="noreferrer">&nbsp;Innovation Origins</a></section><section id="c393798" tabindex="-1"><h2>Why are impact-aware robots so important for humanity?</h2>“Certain jobs are not particularly suited for humans from a safety or ergonomic perspective. For instance, when handling 20-kilogram luggage at airports, working in unsafe areas of a nuclear plant, or dealing with disaster scenarios, you might prefer a machine instead. There are also various plans to send them to space for planet exploration. However, robots still statically interact with the environments when compared to us: the execution of certain key tasks is not yet possible or the execution is too slow. That’s why, in our project, we aimed to develop impact-aware robots. That means a robot has to learn to predict and react to what happens when it comes into fast contact with heavy objects in the environment.”<h2>What makes those robots different from the traditional robots we've known forever?</h2>“Typical robots are not designed to interact dynamically with their environment; making fast contact with surroundings is generally avoided at all costs. There is a very large number of scientific papers in the robotics literature whose focus is&nbsp;collision avoidance. In the I.AM project, we targeted instead&nbsp;collision exploitation. We looked into how the robots can, for example, pick up heavy objects quickly while ensuring that the execution of this type of motion remains reliable, despite disturbances and perception inaccuracies. An object might be heavier than the robot anticipated, or it assumes that an object is at a certain location, but it’s slightly off—maybe even by a few centimeters. How do you make these movements robust despite such uncertainties? That’s one of the things that we have been researching in depth.”<h2>Practically speaking, what main activities were involved in your project?</h2>“The project employed first-principle physics calculations, using basic concepts such as mass and friction, along with software simulations to identify discrepancies between mathematical models and real-world events. Although simulations can never perfectly replicate robot behavior, we improved and most importantly understood how to still use these algorithms for controlling the robots. We did this by taking real-time measurements of robots interacting with various objects in different scenarios. It’s an iterative cycle where you develop and implement a theoretical robot control algorithm in a simulation, evaluate the results, and compare them with real-world outcomes.”<h2>Can you highlight some key findings from this project?</h2>“We discovered how we can make a robot reliably and swiftly grab a heavy object with two arms, by developing a new control algorithm that respects the natural impact dynamics. We also understood how to use software simulations to obtain predictions that can be used for this purpose or other impact tasks.”While working on this project, I also further appreciated how complex movements and spatial perception come so naturally to us humans. We academic researchers are now working very hard on hardware development, spatial perception, and planning—especially the ability to understand the environment in real time and quickly decide what to do next, also in case of failure. This is one of the grand challenges in modern robotics. These actions are natural and intuitive for us, yet we don’t fully understand how we do them nor how we should build a machine with similar abilities.”<h2>Were there also companies involved in this project?</h2>Yes, for example, one of our partners was logistic process automation specialist VanderLande. This is a well-known and large company from the Netherlands that operates worldwide in various fields, such as airports, warehousing, and parcel handling. They provided a lot of real-life use cases and insights on what are the current bottlenecks in the field, the so-called market ``pains’’. One of the great aspects of this collaboration was having a shared lab on TU/e campus, which facilitated hands-on testing and close cooperation. Students and researchers loved it. We have done various comparisons between real and software-simulated impact experiments as well as developed new models of suction grippers to be used for motion control and planning.”<h2>Would you say that the Netherlands is specialized in robotics?</h2>“The Netherlands as a whole is definitely making significant strides when it comes to robotics. The country has long been known for contributing to medical robotics, robot learning, and mobile robotics, to cite a few. I like to think that via this project and its international collaborations, we have also made significant strides in impact-aware robotics. This area of research, which we could say originated here, has garnered global attention and recognition. Our project has played a key role in bringing the subject to the forefront, and it's gratifying to see that our work has been well-received and acknowledged, including in recent publications."<h2>The project is done. What does the future look like for you?</h2>“I will continue investigating and exploring new opportunities, including national and European funding. I'm considering follow-up projects focusing on areas that we couldn’t address during this one, like fast planning and perception. I’m also still in close contact with local and international companies to explore further collaboration opportunities. Some of the many students involved in the project –they are the real heroes!– have been hired in companies that were partners in the project: things are growing in the right direction. The visibility this project has generated is both challenging and exciting at the same time. While it means juggling more tasks, I’m also looking forward to seeing what the future holds.”<h3>European research project I.AM</h3>Europe is at the forefront of the market for torque-controlled robots designed to handle physical interactions with their environment. The recently completed I.AM project, in which Alessandro Saccon and his colleagues were involved, built on this technology and reinforced European leadership by enabling robots to utilize intentional impacts for manipulation tasks. I.AM specifically targeted impact-aware manipulation in logistics—a rapidly growing field in robotics expected to expand significantly in the coming years. Three other research institutes and four companies from the Netherlands, Germany, Switzerland, Sweden, and France collaborated on this four-year project.</section>

To achieve greater efficiency, robots must develop more human-like skills, such as fast physical interaction. That was the focus of the I.AM project.

Revolutionizing robotics: teaching robots to touch and interact like humans

Curious about how AI predicts trends or creates new content? In this piece, we dive into Predictive AI vs. Generative AI, exploring their core differences and real-world impact. Discover how these technologies shape everything from recommendations to creativity.

Generative AI vs Predictive AI: Unveiling the Titans of Artificial Intelligence

In this episode, we chat with Karsten Roscher - a renowned AI expert at the Fraunhofer Institute for Cognitive Systems IKS - about the importance of safety in AI products, the current challenges facing experts, and what the future holds.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/2a32914a-cb70-4e6b-83c9-39693967722c?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><hr><h3 id="isPasted">Episode Notes</h3>(2:00) - Karsten’s journey from undergraduate research to Department Head at Fraunhofer Institute for Cognitive Systems IKS(7:40) - Why Karsten &amp; his team are fixated on AI safety&nbsp;(10:40) - How the team impacts policy decisions(17:00) - The ways the average person can detect the outcomes of the team’s work(18:56) - How to detect what an AI doesn’t know &amp; why it’s so hard(28:40) - Robuscope: Fraunhofer’s free AI testing tool(34:46) - Why being benchmark driven can be bad for AI development(41:10) - How comfortable Karsten feels about using Autonomous vehicles &amp; AI in general(49:55) - How to balance safety and progress with AI progress(56:04) - Karsten &amp; SAE International’s machine learning &amp; safety course(58:30) - Karsten’s unintentional road to becoming a teacher &nbsp;This episode is part of a series brought to you by&nbsp;<a href="https://www.sae.org/" target="_blank">SAE International</a> to explore the leading edge of mobility with the support of experts from industry &amp; academia. Other episodes in this series include&nbsp;<a href="https://pod.link/wevolver/episode/a9e88f02f379c03548ffdbedbb563429" target="_blank">Episode 190</a>, where we talked about Karsten’s free AI evaluator tool, Robuscope!You can check out the information about the course Karsten mentioned for safety &amp; machine learning by clicking&nbsp;<a href="https://www.wevolver.com/article/mastering-machine-learning-for-critical-applications" target="_blank">HERE</a>.&nbsp;Enjoyed listening to Karsten? Want to hear more from him? Checkout his links below:<ul><li><a href="https://www.linkedin.com/in/karsten-roscher-17a2907/" target="_blank">Karsten’s LinkedIn</a></li><li><a href="https://safe-intelligence.fraunhofer.de/en/" target="_blank">Safe Intelligence Blog</a></li></ul>Become a founding reader of our newsletter:&nbsp;<a href="http://read.thenextbyte.com/" target="_blank">http://read.thenextbyte.com/</a>&nbsp;As always, you can find these and other interesting &amp; impactful engineering articles on Wevolver.com.

In this episode, we chat with Karsten Roscher - a renowned AI expert at the Fraunhofer Institute for Cognitive Systems IKS - about the importance of safety in AI products, the current challenges facing experts, and what the future holds. 

Podcast: Karsten Roscher & The Need For Safe AI (Interview)

Large language models (LLMs) such as GPT-4 have become indispensable for tasks like text generation, coding, operating chatbots, translation and others. At their heart, LLMs work by predicting the next word in a sentence based on the previous words – a simple but powerful idea that drives much of their functionality. But what happens when we ask these models to predict backward — to go “backwards in time” and determine the previous word from the subsequent ones?The question led&nbsp;<a href="https://www.epfl.ch/labs/csft/" target="_blank">Professor Clément Hongler</a> at EPFL and&nbsp;<a href="https://jeremiewenger.com/" target="_blank" rel="noopener">Jérémie Wenger</a> of Goldsmiths (London) to explore whether LLMs could construct a story backward, starting from the end. Working with Vassilis Papadopoulos, a machine learning researcher at EPFL, they discovered something surprising: LLMs are consistently less accurate when predicting backward than forward.<h2>A fundamental asymmetry</h2>The researchers tested LLMs of different architectures and sizes, including Generative Pre-trained Transformers (GPT), Gated Recurrent Units (GRU), and Long Short-Term Memory (LSTM) neural networks. Every one of them showed the “Arrow of Time” bias, revealing a fundamental asymmetry in how LLMs process text.Hongler explains: “The discovery shows that while LLMs are quite good both at predicting the next word and the previous word in a text, they are always slightly worse backwards rather than forward: their performance at predicting the previous word is always a few percent worse than at predicting the next word. This phenomenon is universal across languages, and can be observed with any large language model.”The work is also connected to the work of Claude Shannon, the father of Information Theory, in his seminal&nbsp;<a href="https://ieeexplore.ieee.org/document/6773263" target="_blank" rel="noopener">1951 paper</a>. Shannon explored whether predicting the next letter in a sequence was as easy as predicting the previous one. He discovered that although both tasks should theoretically be equally difficult, humans found backward prediction more challenging – though the performance difference was minimal.<h2>Intelligent agents</h2>“In theory, there should be no difference between the forward and backward directions, but LLMs appear to be somehow sensitive to the time direction in which they process text,” says Hongler. “Interestingly, this is related to a deep property of the structure of language that could only be discovered with the emergence of Large Language Models in the last five years.”The researchers link this property to the presence of intelligent agents processing information, meaning that it could be used as a tool to detect intelligence or life, and help design more powerful LLMs. Finally, it could point out new directions to the long-standing quest to understand the passage of time as an emergent phenomenon in physics.The work was presented at the prestigious&nbsp;<a href="https://icml.cc/" target="_blank" rel="noopener">International Conference on Machine Learning</a> (2024) and&nbsp;<a href="https://arxiv.org/html/2401.17505v4" target="_blank" rel="noopener">is currently available on arXiv</a>.<h2>From theater to math</h2>The study itself has a fascinating backstory, which Hongler relates: “In 2020, with Jérémie [Wenger], we were collaborating with&nbsp;<a href="https://www.manufacture.ch/en" target="_blank" rel="noopener">The Manufacture theater school</a> to make a chatbot that would play alongside actors to do improv; in improv, you often want to continue the story, while knowing what the end should look like. “In order to make stories that would finish in a specific manner, we got the idea to train the chatbot to speak ‘backwards’, allowing it to generate a story given its end – e.g., if the end is ‘they lived happily ever after’, the model could tell you how it happened. So, we trained models to do that, and noticed they were a little worse backwards than forwards. “With Vassilis [Papadopoulos], we later realized that this was a profound feature of language, and that it was a completely general new phenomenon, which has deep links with the passage of time, intelligence, and the notion of causality. Quite cool for some theater project!” Hongler’s excitement with this work stems in good part from the unexpected surprises that came along the way: “Only time could tell that something that started as a theater project would end up giving us new tools to understand so many things about the world.”<hr>ReferencesVassilis Papadopoulos, Jérémie Wenger, Clément Hongler. Arrows of Time for Large Language Models. arXiv:&nbsp;<a href="https://arxiv.org/html/2401.17505v4" target="_blank" rel="noopener">2401.17505v4</a>

Researchers have found that AI large language models, like GPT-4, are better at predicting what comes next than what came before in a sentence. This “Arrow of Time” effect could reshape our understanding of the structure of natural language, and the way these models understand it.

Large Language Models feel the direction of time

A GPU microchip for accelerated AI processing

Unleashing the Power of GPU, understanding the essential strategies for efficient Deep Learning Model Training without compromising on memory and hardware resources in Graphical Processing Units

Maximize GPU Utilization for Model Training: Unlocking Peak Performance

<h2 id="isPasted"><iframe id="66c33d8585cc0ed55e5ec302" width="250" class="specs-iframe" height="440" src="https://wevolver.com/iframe/specs/renesas-electronics-ra8m1-voice-kit?iframe=true" frameborder="0" scrolling="no" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Enhancing Security Using the Latest Technology</h2>Security is a topic on everyone’s minds today, and as technology gets more advanced, it can be easier to bypass the older systems that others have built. Large-scale system hacks are highly publicized and can affect a wide range of people, but smaller scale hacks can be just as devastating. Therefore, it’s essential to use the best technology to protect what matters most, such as your home and family. Face recognition can provide a more effective home security solution than traditional keys and keypads. Physical keys can be duplicated, stolen, lost, or forgotten, and key codes can be guessed, forgotten, or shared with unauthorized users. Accurate face recognition technology has none of those disadvantages, and owners can easily manage the authorization of each person and log their entries.With some of these legacy security systems becoming easier to bypass, we need to constantly develop new solutions using the latest technology to keep the highest level of security. This is where the smart door lock, utilizing machine learning and 3D facial recognition from AMD technology partner Makarena Labs, comes into play. In this blog, we discuss how smart door technology is bringing peace of mind to the front door and more with a superior level of security.<h2>Addressing Current Security Problems in Smart Door Locks</h2>Smart door locks can provide increased protection for homes and other consumer dwelling spaces, like garages or sheds, but previous versions of smart locks can be easily tricked. Due to the incorporated technology—generally a standard CMOS sensor—previous smart locks can be fooled into opening by using 2D pictures of an authorized person’s face. This results in a false recognition and the system will allow an unauthorized person entry, which is a major security breach that needs to be addressed. &nbsp;AMD, in partnership with Makarena, have developed a capable solution, based on the AMD <a href="https://www.mouser.com/new/xilinx/xilinx-zynq-7000-socs/" target="_blank">Zynq™ 7000 SoC</a>, that provides a fully functional and robust design using 3D facial recognition. This advanced facial recognition is achieved through a camera that uses IR Time-of-Flight (TOF) or Lidar technology to bounce a pattern off the face of the person attempting to gain entry, while the sensors in the smart lock capture the reflection (Figure 1).<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI1ODUzMjUwMjExLUZpZ3VyZV8xX01ha2FyZW5hX0xhYnNfU21hcnRfTG9jay5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 100%px;" class="fr-fic fr-dib">Figure 1: This smart lock design, developed by AMD and their partner Makarena, is based on the AMD Zynq 7000 SoC, incorporates IR TOF or Lidar technology to detect and create a 3D map of the face, and compares the 3D map to the authorized users in the system, only unlocking if a match is found. (Source: AMD)Using the reflection captured by the sensors, the system creates a 3D map of the face and compares it to the authorized users in the system, only unlocking if a match is found (Figure 2). The CMOS sensor provides an added layer of accuracy and simplifies user management for the homeowner by providing images of each authorized user.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNzI1ODUzMjcwNDY5LUZpZ3VyZTJfRGV0ZWN0aW9uX2F1dGhvcml6YXRpb25fcHJvY2Vzcy53ZWJwIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 100%px;" class="fr-fic fr-dib">Figure 2: The smart door lock’s detection and authorization process ensures accuracy while simplifying user management. (Source: AMD)This advanced smart door lock system also works in complete darkness and can recognize authorized people even while they are wearing glasses, hats, or facial covers.The AMD Zynq 7000 SoC design can be easily configured and customized by OEMs with help on hand from Makarena Labs, providing a lock that can store a family’s authorized faces and accurately authorize or reject them in under a second. For applications that need to store a larger number of faces, such as a neighborhood’s security gate, the higher performance AMD&nbsp;<a href="https://www.mouser.com/new/xilinx/xilinx-zynq-ultrascale-mpsocs/" target="_blank">Zynq UltraScale+™ MPSoC</a> can support fast recognition even with hundreds of authorized faces. Both parts include robust security engines, which OEM customers can configure to load only authentically signed, encrypted firmware.If you're interested in hearing more about the smart lock solution, please reach out to sales@makarenalabs.com.AMD, and the AMD Arrow logo, UltraScale+, Zynq, and combinations thereof are trademarks of Advanced Micro Devices, Inc.The <a href="https://community.amd.com/t5/adaptive-computing/smart-door-locks-enhancing-security-with-machine-learning/ba-p/583320" rel="noreferrer nofollow" target="_blank">Smart Door Locks – Enhancing Security with Machine Learning</a> blog was first published on amd.com and was reposted here with permission.<hr>Xilinx develops highly flexible and adaptive processing platforms that enable rapid innovation across a variety of technologies, from the endpoint to the edge to the cloud. Specializing in programmable logic devices, Xilinx is the semiconductor company that invented the Field Programmable Gate Array (FPGA), the hardware programmable System on Chip (SoC), and the Adaptive Compute Acceleration Platform (ACAP).

In this blog, we discuss how smart door technology is bringing peace of mind to the front door and more with a superior level of security.

Smart Door Locks - Enhancing Security with Machine Learning

In this episode, we explore how Robuscope is reimagining the future of AI testing, offering groundbreaking solutions for improving the accuracy and reliability of artificial intelligence systems. 

Podcast: Here's a Free Tool To Test Your AI

The nationwide rush to integrate AI into manufacturing passed through Pittsburgh in August, bringing representatives from industry, government, and academia together for the <a href="https://www.aimanufacturingconference.com/" rel="noopener" target="_blank">2024 AIMST conferenceOpens in new window</a>. The dual track conference, now in its eighth year, included sessions for AI in Manufacturing and SCADA (Supervisory Control and Data Acquisition) Technology.After being held in Dallas for the past three years, organizers opted to come to Pittsburgh in part due to the strength of manufacturing research and activity at Carnegie Mellon and its <a href="https://engineering.cmu.edu/mfi/mill19/index.html" rel="noopener" target="_blank">Mill 19Opens in new window</a> partner, the <a href="https://arminstitute.org/" rel="noopener" target="_blank">ARM (Advanced Robotics for Manufacturing) InstituteOpens in new window</a>.“We are very happy to be in Pittsburgh,” said conference organizer, Robert Schaudt.When considering the move to Pittsburgh, Schaudt said his first call was to <a href="https://engineering.cmu.edu/directory/bios/fedder-gary.html" target="_blank">Gary Fedder</a>, who is the faculty director of Carnegie Mellon’s <a href="https://engineering.cmu.edu/mfi/index.html" rel="noopener" target="_blank">Manufacturing Futures InstituteOpens in new window</a> and a university professor in <a href="https://www.ece.cmu.edu/" rel="noopener" target="_blank">electrical and computer engineeringOpens in new window</a>. Fedder was also instrumental in establishing the ARM Institute, which is now one of 17 Manufacturing USA institutes.Schaudt said the conference attracts those who want to learn more about new technology and discover what is possible by attending presentations and panel discussions with representatives from organizations that include NIST, NASA, Ford, General Motors, and US Steel.Fedder was one of three keynote speakers during the conference. In his presentation, Research Driving the Future of AI in Manufacturing, he shared examples of how AI and machine learning are being used to advance robotics, additive manufacturing, and digital twin technology.<blockquote>AI is pervasive at Carnegie Mellon. It is aiding nearly all our manufacturing research.<cite>Gary Fedder, Faculty Fedder, Manufacturing Futures Institute</cite></blockquote>“AI is pervasive at Carnegie Mellon. It is aiding nearly all our manufacturing research,” said Fedder.<a href="https://engineering.cmu.edu/directory/bios/tang-pingbo.html" target="_blank">Pingbo Tang</a>, an associate professor in Carnegie Mellon’s department of <a href="https://www.cee.engineering.cmu.edu/index.html" rel="noopener" target="_blank">Civil and Environmental EngineeringOpens in new window</a>, was also invited to be a presenter. During his Human-AI Co-Evolution in Lean Adaptation of Modular Building Production Systems,&nbsp;he discussed how AI agents can learn from production engineers and increasingly are able to help humans identify and resolve time and resource waste in producing modular building products with diverse customizations.The ARM Institute’s Chief Strategy Officer Suzy Teele, who attended the conference, said the conference did an impressive job bringing together experts and practitioners to share actionable information and scenarios for the use of AI in manufacturing, including the panel discussion with ARM Institute consortium members from Boeing and GrayMatter Robotics.<blockquote>There is so much hype and confusion around AI today, and this conference provided foundational knowledge and real-world examples to help attendees plan their own strategies for AI.<cite>Suzy Teele, Chief Strategy Officer, Advanced Robotics for Manufacturing Institute</cite></blockquote>“There is so much hype and confusion around AI today, and this conference provided foundational knowledge and real-world examples to help attendees plan their own strategies for AI,” said Teele.

machine learning

Profiles