Imagine that dinner could be prepared by a robot in your own home with just the push of a button. It sounds like something straight out of a far-flung future. Now, researchers at CMU are laying the groundwork for this technology with artificial intelligence.&nbsp;Alison Bartch, Atharva Dikshit, and Abraham George, under the guidance of <a href="https://engineering.cmu.edu/directory/bios/barati-farimani-amir.html" target="_blank">Amir Barati Farimani</a>, assistant professor of <a href="https://www.meche.engineering.cmu.edu/index.html" rel="noopener" target="_blank">mechanical engineeringOpens in new window</a>, trained a robot arm on different fruit slices and shapes to help improve its accuracy with cutting and moving real pieces of food on a cutting board.&nbsp;“So I’m cooking and have a cluttered cutting board. I’m reasoning about how to chop and how to select the objects that I want to cut, without cutting things that I don’t want,” said Bartch, a Ph.D. candidate in mechanical engineering. “It’s something that humans are pretty good at. And it’s a surprisingly difficult problem for robots.”<blockquote>It’s something that humans are pretty good at. And it’s a surprisingly difficult problem for robots.<cite>Alison Bartch<cite id="isPasted">,&nbsp;Ph.D. student<cite id="isPasted">, Mechanical Engineering</cite></cite> </cite></blockquote>The researchers combined two vision foundational models—models trained on large visual data sets—to help the robot arm recognize the shape and the type of fruit and vegetable slices. The AI would then determine what motion to use on these slices to produce the correct results with 70% accuracy.One of the models was only trained on whole fruits and vegetables, so the researchers needed to fine-tune the AI to more accurately recognize slices with a relatively small amount of newly collected data. According to Bartch, their fine-tuned framework can potentially be used for other related research; the combined foundational models, however, can be used for a variety of other AI projects in the future.“That’s just one of the things in a lot of robotics and machine learning—a lot of the data sets are already collected,” said George, also a Ph.D. candidate in mechanical engineering.&nbsp;Bartch said that the research focuses more on the “practicalities” of cutting food, such as producing the correct number of slices and moving pieces out of the way, making this research more applicable to a real kitchen than other research.“For this work, we focused more on the pipeline surrounding it, as opposed to the actual chop action,” Bartch said. “You want to pick which object to cut. You want to clear the scene before you cut. After you cut, detecting where the objects are then and continuing to plan. A lot of existing work focused on cutting assumed the position of the vegetable and that there is nothing else in the environment.”&nbsp;However, it will still be a while before AI will be chopping vegetables in homes. According to Bartch, there is still a lot of research and fine-tuning needed, and their robot still needs some human input. It’s difficult to replicate human reasoning in machines, even with tasks many humans find simple.“I think there is definitely a really long way to go before it’s in a person’s house.” Bartch said. “Because reasoning about cooking is really difficult. When you want to go into more practical situations, you need the system to be able to reason when the food is done.Atharva Dikshit, a master’s student in mechanical engineering, also contributed to this research. This research was submitted to the Journal of Intelligent and Robotic Systems.

Researchers at CMU combined two vision foundational models—models trained on large visual data sets—to help a robot arm recognize the shape and the type of fruit and vegetable slices.

Chop, chop: Improving food prep with the power of AI

Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know. The technique involves quantifying the fuzziness of human language and using that measurement to tell robots when to ask for further directions. Telling a robot to pick up a bowl from a table with only one bowl is fairly clear. But telling a robot to pick up a bowl when there are five bowls on the table generates a much higher degree of uncertainty — and triggers the robot to ask for clarification.Because tasks are typically more complex than a simple “pick up a bowl” command, the engineers use large language models (LLMs) — the technology behind tools such as ChatGPT — to gauge uncertainty in complex environments. LLMs are bringing robots powerful capabilities to follow human language, but LLM outputs are still frequently unreliable, said <a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of <a href="https://mae.princeton.edu/" target="_blank">mechanical and aerospace engineering</a> at Princeton and the senior author of a <a href="https://openreview.net/forum?id=4ZK8ODNyFXx" target="_blank">study</a> outlining the new method.“Blindly following plans generated by an LLM could cause robots to act in an unsafe or untrustworthy manner, and so we need our LLM-based robots to know when they don’t know,” said Majumdar.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1701898003603-knowno-ambiguities_WEB-2048x871.jpg" style="width: 766px;" class="fr-fic fr-dib">The researchers tested their method on a simulated robotic arm and on two types of robot hardware. The most complex experiments (left) involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. Another set of experiments used a tabletop robotic arm (right) tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm (center) added an additional layer of ambiguity. Images by the researchersThe system also allows a robot’s user to set a target degree of success, which is tied to a particular uncertainty threshold that will lead a robot to ask for help. For example, a user would set a surgical robot to have a much lower error tolerance than a robot that’s cleaning up a living room.“We want the robot to ask for enough help such that we reach the level of success that the user wants. But meanwhile, we want to minimize the overall amount of help that the robot needs,” said&nbsp;<a href="https://allenzren.github.io/" target="_blank">Allen Ren</a>, a graduate student in mechanical and aerospace engineering at Princeton and the study’s lead author. Ren received a&nbsp;<a href="https://www.corl2023.org/awards" target="_blank">best student paper award</a> for his Nov. 8 presentation at the Conference on Robot Learning in Atlanta. The new method produces high accuracy while reducing the amount of help required by a robot compared to other methods of tackling this issue.The researchers tested their method on a simulated robotic arm and on two types of robots at Google facilities in New York City and Mountain View, California, where Ren was working as a student research intern. One set of hardware experiments used a tabletop robotic arm tasked with sorting a set of toy food items into two different categories; a setup with a left and right arm added an additional layer of ambiguity.The most complex experiments involved a robotic arm mounted on a wheeled platform and placed in an office kitchen with a microwave and a set of recycling, compost and trash bins. In one example, a human asks the robot to “place the bowl in the microwave,” but there are two bowls on the counter — a metal one and a plastic one.The robot’s LLM-based planner generates four possible actions to carry out based on this instruction, like multiple-choice answers, and each option is assigned a probability. Using a statistical approach called conformal prediction and a user-specified guaranteed success rate, the researchers designed their algorithm to trigger a request for human help when the options meet a certain probability threshold. In this case, the top two options — place the plastic bowl in the microwave or place the metal bowl in the microwave — meet this threshold, and the robot asks the human which bowl to place in the microwave.In another example, a person tells the robot, “There is an apple and a dirty sponge … It is rotten. Can you dispose of it?” This does not trigger a question from the robot, since the action “put the apple in the compost” has a sufficiently higher probability of being correct than any other option.“Using the technique of conformal prediction, which quantifies the language model’s uncertainty in a more rigorous way than prior methods, allows us to get to a higher level of success” while minimizing the frequency of triggering help, said the study’s senior author&nbsp;<a href="https://engineering.princeton.edu/faculty/anirudha-majumdar" target="_blank">Anirudha Majumdar</a>, an assistant professor of mechanical and aerospace engineering at Princeton.Robots’ physical limitations often give designers insights not readily available from abstract systems. Large language models “might talk their way out of a conversation, but they can’t skip gravity,” said coauthor&nbsp;<a href="https://andyzeng.github.io/" target="_blank">Andy Zeng</a>, a research scientist at Google DeepMind. “I’m always keen on seeing what we can do on robots first, because it often sheds light on the core challenges behind building generally intelligent machines.”Ren and Majumdar began collaborating with Zeng after he gave a talk as part of the&nbsp;<a href="https://robo.princeton.edu/seminar" target="_blank">Princeton Robotics Seminar</a> series, said Majumdar. Zeng, who earned a computer science Ph.D. from Princeton in 2019, outlined Google’s efforts in using LLMs for robotics, and brought up some open challenges. Ren’s enthusiasm for the problem of calibrating the level of help a robot should ask for led to his internship and the creation of the new method.“We enjoyed being able to leverage the scale that Google has” in terms of access to large language models and different hardware platforms, said Majumdar.Ren is now extending this work to problems of active perception for robots: For instance, a robot may need to use predictions to determine the location of a television, table or chair within a house, when the robot itself is in a different part of the house. This requires a planner based on a model that combines vision and language information, bringing up a new set of challenges in estimating uncertainty and determining when to trigger help, said Ren.The paper, “<a href="https://openreview.net/forum?id=4ZK8ODNyFXx" target="_blank">Robots That Ask for Help: Uncertainty Alignment for Large Language Model Planners</a>,” was presented Nov. 8 at the Conference on Robot Learning. In addition to Ren, Zeng and Majumdar, coauthors include Anushri Dixit and Alexandra Bodrova of Princeton; and Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu and Dorsa Sadigh of Google DeepMind. The research was supported in part by the U.S. National Science Foundation and the Office of Naval Research.

Engineers at Princeton University and Google have come up with a new way to teach robots to know when they don’t know and ask for clarification from a human. Photo by the researchers

Modern robots know how to sense their environment and respond to language, but what they don’t know is often more important than what they do know. Teaching robots to ask for help is key to making them safer and more efficient.

How do you make a robot smarter? Program it to know what it doesn't know.

Partner:<a href="https://erasystems.ro/" target="_blank">Era Systems</a>Challenge:The customer, an automotive manufacturer, needed an automation solution for the palletization and depalletization of large ABS parts. They decided on a solution where one vision-guided robot operates both processes.Solution:<a href="https://www.photoneo.com/locator-studio/" target="_blank">Locator Studio</a> + <a href="https://www.photoneo.com/phoxi-3d-scanner/" target="_blank">PhoXi 3D Scanner</a>The solution for palletization and depalletization consists of 2 different picking locations with 2 independent production flows:A) Palletization – the ABS parts are taken from an output conveyor and placed on a palletB) Depalletization – the parts are picked from the palletThe FANUC R-2000iC/210F Robot, navigated by Photoneo PhoXi 3D Scanner and Locator Studio, serves both locations. First, it picks the ABS parts from an output conveyor of a thermoformer press and stacks them on a pallet. The pallet is then taken to a cooling area. The height of the full pallet is ~1400-1500 mm from the ground.The cooled parts are then picked from the pallet and fed to a CNC machine for trimming the outer border (the excess material). After trimming, the parts are placed on another pallet.Any of the 2 jobs can be active or inactive at a certain production stage (a few hours or days).The robot can either work in location 1 (the conveyor) or in location 2 (the pallet) or in both locations at the same time. So one robot can serve 2 different robotic cells.Both robotic cells can process 4 part types but only one model during a production cycle.For instance, Heat RT on the conveyor for 7 hours and Heat FT on the pallet for 3 days, then Heat FT – Top/Lid on the conveyor for 2 days and Heat RT on the pallet for 14 hours, and so on.Every time a part model is changed in a location, the operator activates the correct model for that location so the robot knows what to look for.Discussion:This automation solution increased efficiency as one robot can operate two different production flows simultaneously.The robot is navigated by state-of-the-art 3D vision from Photoneo, providing a resolution of up to 3 Million 3D points and an accuracy of 25 – 500 µm across the different models.Would you like to discuss your project with Photoneo?&nbsp;<a href="https://www.photoneo.com/contact-us/" rel="nofollow" target="_blank" id="isPasted">Contact them here</a>.

Palletization and depalletization of large ABS parts

The customer, an automotive manufacturer, needed an automation solution for the palletization and depalletization of large ABS parts. They decided on a solution where one vision-guided robot operates both processes.

In this episode, we talk all about connectomics - the study of animal brains - and how researchers at MIT have started leveraging AI to break through the primary bottleneck: brain image acquisition.

Podcast: Reverse Engineering The Brain With AI

With 3D inkjet printing systems, engineers can fabricate hybrid structures that have soft and rigid components, like robotic grippers that are strong enough to grasp heavy objects but soft enough to interact safely with humans.These multimaterial 3D printing systems utilize thousands of nozzles to deposit tiny droplets of resin, which are smoothed with a scraper or roller and cured with UV light. But the smoothing process could squish or smear resins that cure slowly, limiting the types of materials that can be used.&nbsp;Researchers from MIT, the MIT spinout Inkbit, and ETH Zurich have developed a new 3D inkjet printing system that works with a much wider range of materials. Their printer utilizes computer vision to automatically scan the 3D printing surface and adjust the amount of resin each nozzle deposits in real-time to ensure no areas have too much or too little material.Since it does not require mechanical parts to smooth the resin, this contactless system works with materials that cure more slowly than the acrylates which are traditionally used in 3D printing. Some slower-curing material chemistries can offer improved performance over acrylates, such as greater elasticity, durability, or longevity.In addition, the automatic system makes adjustments without stopping or slowing the printing process, making this production-grade printer about 660 times faster than a comparable 3D inkjet printing system.The researchers used this printer to create complex, robotic devices that combine soft and rigid materials. For example, they made a completely 3D-printed robotic gripper shaped like a human hand and controlled by a set of reinforced, yet flexible, tendons.<iframe width="640" height="360" src="https://www.youtube.com/embed/mw9hYHoD46o?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-gtm-yt-inspected-9="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;&nbsp;</iframe>“Our key insight here was to develop a machine-vision system and completely active feedback loop. This is almost like endowing a printer with a set of eyes and a brain, where the eyes observe what is being printed, and then the brain of the machine directs it as to what should be printed next,” says co-corresponding author Wojciech Matusik, a professor of electrical engineering and computer science at MIT who leads the Computational Design and Fabrication Group within the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL).He is joined on the paper by lead author Thomas Buchner, a doctoral student at ETH Zurich, co-corresponding author Robert Katzschmann PhD ’18, assistant professor of robotics who leads the Soft Robotics Laboratory at ETH Zurich; as well as others at ETH Zurich and Inkbit. The research appears today in Nature.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1700105845585-MIT-VisionJetting-02-press.jpg" style="width: 766px;" class="fr-fic fr-dib">The researchers used their printing system to create complex, robotic devices that combine soft and rigid materials. Since the printer has 16,000 nozzles, the system can control fine details of the device being fabricated. This rendering shows objects that have been halfway built by the printer. Credit: Courtesy of the researchers<h2>Contact free</h2>This paper builds off a low-cost, multimaterial 3D printer known as <a href="https://news.mit.edu/2015/multifab-3-d-print-10-materials-0824" target="_blank">MultiFab</a> that the researchers introduced in 2015. By utilizing thousands of nozzles to deposit tiny droplets of resin that are UV-cured, MultiFab enabled high-resolution 3D printing with up to 10 materials at once.With this new project, the researchers sought a contactless process that would expand the range of materials they could use to fabricate more complex devices.They developed a technique, known as vision-controlled jetting, which utilizes four high-frame-rate cameras and two lasers that rapidly and continuously scan the print surface. The cameras capture images as thousands of nozzles deposit tiny droplets of resin.The computer vision system converts the image into a high-resolution depth map, a computation that takes less than a second to perform. It compares the depth map to the CAD (computer-aided design) model of the part being fabricated, and adjusts the amount of resin being deposited to keep the object on target with the final structure.The automated system can make adjustments to any individual nozzle. Since the printer has 16,000 nozzles, the system can control fine details of the device being fabricated.“Geometrically, it can print almost anything you want made of multiple materials. There are almost no limitations in terms of what you can send to the printer, and what you get is truly functional and long-lasting,” says Katzschmann.The level of control afforded by the system enables it to print very precisely with wax, which is used as a support material to create cavities or intricate networks of channels inside an object. The wax is printed below the structure as the device is fabricated. After it is complete, the object is heated so the wax melts and drains out, leaving open channels throughout the object.Because it can automatically and rapidly adjust the amount of material being deposited by each of the nozzles in real time, the system doesn’t need to drag a mechanical part across the print surface to keep it level. This enables the printer to use materials that cure more gradually, and would be smeared by a scraper.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1700105876641-MIT-VisionJetting-05-press.jpg" style="width: 766px;" class="fr-fic fr-dib">A photograph shows a variety of 3D-printed objects in white, displayed on a tray. The objects are: a robotic hand, cubes made of lattice structures, a biological heart, and a walking robot. Credit: Courtesy of the researchers<h2>Superior materials</h2>The researchers used the system to print with thiol-based materials, which are slower-curing than the traditional acrylic materials used in 3D printing. However, thiol-based materials are more elastic and don’t break as easily as acrylates. They also tend to be more stable over a wider range of temperatures and don’t degrade as quickly when exposed to sunlight.“These are very important properties when you want to fabricate robots or systems that need to interact with a real-world environment,” says Katzschmann.The researchers used thiol-based materials and wax to fabricate several complex devices that would otherwise be nearly impossible to make with existing 3D printing systems. For one, they produced a functional, tendon-driven robotic hand that has 19 independently actuatable tendons, soft fingers with sensor pads, and rigid, load-bearing bones.“We also produced a six-legged walking robot that can sense objects and grasp them, which was possible due to the system’s ability to create airtight interfaces of soft and rigid materials, as well as complex channels inside the structure,” says Buchner.The team also showcased the technology through a heart-like pump with integrated ventricles and artificial heart valves, as well as metamaterials that can be programmed to have non-linear material properties.“This is just the start. There is an amazing number of new types of materials you can add to this technology. This allows us to bring in whole new material families that couldn’t be used in 3D printing before,” Matusik says.The researchers are now looking at using the system to print with hydrogels, which are used in tissue-engineering applications, as well as silicon materials, epoxies, and special types of durable polymers.They also want to explore new application areas, such as printing customizable medical devices, semiconductor polishing pads, and even more complex robots.

This rendering shows a robot being built layer-by-layer using the new process. The black spheres represent the material that the printer uses. The material is then cured by UV light, represented in blue. At the top of the image are the cameras that scan the procedure and adjust accordingly. Image: Moritz Hocher

Computer vision enables contact-free 3D printing, letting engineers print with high-performance materials they couldn’t use before.

This 3D printer can watch itself fabricate objects

This article was discussed in our <a href="https://www.wevolver.com/profile/the.next.byte" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/94de5079-b574-4a3f-8251-3be6800d1ac5?dark=false" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe> The full article will continue below.<hr>Connectomics, the ambitious field of study that seeks to map the intricate network of animal brains, is undergoing a growth spurt. Within the span of a decade, it has journeyed from its nascent stages to a discipline that is poised to (hopefully) unlock the enigmas of cognition and the physical underpinning of neuropathologies such as in Alzheimer’s disease.&nbsp;At its forefront is the use of powerful electron microscopes, which researchers from the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) and the Samuel and Lichtman Labs of Harvard University bestowed with the analytical prowess of machine learning. Unlike traditional electron microscopy, the integrated AI serves as a “brain” that learns a specimen while acquiring the images, and intelligently focuses on the relevant pixels at nanoscale resolution similar to how animals inspect their worlds.&nbsp;“<a href="https://www.biorxiv.org/content/10.1101/2023.10.05.561103v1.abstract" target="_blank">SmartEM</a>” assists connectomics in quickly examining and reconstructing the brain’s complex network of synapses and neurons with nanometer precision. Unlike traditional electron microscopy, its integrated AI opens new doors to understand the brain's intricate architecture.The integration of hardware and software in the process is crucial. The team embedded a GPU into the support computer connected to their microscope. This enabled running machine-learning models on the images, helping the microscope beam be directed to areas deemed interesting by the AI. “This lets the microscope dwell longer in areas that are harder to understand until it captures what it needs,” says MIT professor and CSAIL principal investigator Nir Shavit. “This step helps in mirroring human eye control, enabling rapid understanding of the images.”&nbsp;“When we look at a human face, our eyes swiftly navigate to the focal points that deliver vital cues for effective communication and comprehension,” says the lead architect of SmartEM, Yaron Meirovitch, a visiting scientist at MIT CSAIL who is also a former postdoc and current research associate neuroscientist at Harvard. “When we immerse ourselves in a book, we don't scan all of the empty space; rather, we direct our gaze towards the words and characters with ambiguity relative to our sentence expectations. This phenomenon within the human visual system has paved the way for the birth of the novel microscope concept.”&nbsp;For the task of reconstructing a human brain segment of about 100,000 neurons, achieving this with a conventional microscope would necessitate a decade of continuous imaging and a prohibitive budget. However, with SmartEM, by investing in four of these innovative microscopes at less than $1 million each, the task could be completed in a mere three months.<h2>Nobel Prizes and little worms &nbsp;</h2>Over a century ago, Spanish neuroscientist Santiago Ramón y Cajal was heralded as being the first to characterize the structure of the nervous system. Employing the rudimentary light microscopes of his time, he embarked on leading explorations into neuroscience, laying the foundational understanding of neurons and sketching the initial outlines of this expansive and uncharted realm&nbsp;— a feat that earned him a Nobel Prize. He noted, on the topics of inspiration and discovery, that “As long as our brain is a mystery, the universe, the reflection of the structure of the brain will also be a mystery.”Progressing from these early stages, the field has advanced dramatically, evidenced by efforts in the 1980s, mapping the relatively simpler connectome of <a href="https://www.nature.com/articles/s41586-021-03778-8" target="_blank">C. elegans</a>, small worms, to today’s endeavors probing into more intricate brains of organisms like zebrafish and mice. This evolution reflects not only enormous strides, but also escalating complexities and demands: mapping the mouse brain alone means managing a staggering thousand petabytes of <a href="https://www.cell.com/cell/pdf/S0092-8674(20)31001-1.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">data</a>, a task that vastly eclipses the storage capabilities of any university, the team says.&nbsp;<h2>Testing the waters</h2>For their own work, Meirovitch and others from the research team studied 30-nanometer thick slices of octopus tissue that were mounted on tapes, put on wafers, and finally inserted into the electron microscopes. Each section of an octopus brain, comprising billions of pixels, was imaged, letting the scientists reconstruct the slices into a three-dimensional cube at nanometer resolution. This provided an ultra-detailed view of synapses. The chief aim? To colorize these images, identify each neuron, and understand their interrelationships, thereby creating a detailed map or “connectome” of the brain's circuitry.“SmartEM will cut the imaging time of such projects from two weeks to 1.5 days,” says Meirovitch. “Neuroscience labs that currently can't be engaged with expensive and long EM imaging will be able to do it now,” The method should also allow synapse-level circuit analysis in samples from patients with psychiatric and neurologic disorders.&nbsp;Down the line, the team envisions a future where connectomics is both affordable and accessible. They hope that with tools like SmartEM, a wider spectrum of research institutions could contribute to neuroscience without relying on large partnerships, and that the method will soon be a standard pipeline in cases where biopsies from living patients are available. Additionally, they’re eager to apply the tech to understand pathologies, extending utility beyond just connectomics. “We are now endeavoring to introduce this to hospitals for large biopsies, utilizing electron microscopes, aiming to make pathology studies more efficient,” says Shavit.&nbsp;Two other authors on the paper have MIT CSAIL ties: lead author Lu Mi MCS ’19, PhD ’22, who is now a postdoc at the Allen Institute for Brain Science, and Shashata Sawmya, an MIT graduate student in the lab. The other lead authors are Core Francisco Park and Pavel Potocek, while Harvard professors Jeff Lichtman and Aravi Samuel are additional senior authors. Their research was supported by the NIH BRAIN Initiative and was presented at the 2023 International Conference on Machine Learning (ICML) Workshop on Computational Biology. The work was done in collaboration with scientists from Thermo Fisher Scientific.

MIT researchers invented a technology and software to take electron microscopy to the next level by seamlessly integrating real-time machine learning into the imaging process — “smart microscopy.” Left image: Yaron Meirovitch via the Stable Diffusion XL AI image generator and Alex Shipps via the Midjourney AI image generator. Right image: Daniel Berger and Meirovitch, edited by Alex Shipps/MIT CSAIL

MIT CSAIL researchers combine AI and electron microscopy to expedite detailed brain network mapping, aiming to enhance connectomics research and clinical pathology.

Using AI to optimize for rapid neural imaging

Imagine you’re visiting a friend abroad, and you look inside their fridge to see what would make for a great breakfast. Many of the items initially appear foreign to you, with each one encased in unfamiliar packaging and containers. Despite these visual distinctions, you begin to understand what each one is used for and pick them up as needed.Inspired by humans' ability to handle unfamiliar objects, a group from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) designed Feature Fields for Robotic Manipulation (<a href="https://arxiv.org/abs/2308.07931" target="_blank">F3RM</a>), a system that blends 2D images with foundation model features into 3D scenes to help robots identify and grasp nearby items. F3RM can interpret open-ended language prompts from humans, making the method helpful in real-world environments that contain thousands of objects, like warehouses and households.F3RM offers robots the ability to interpret open-ended text prompts using natural language, helping the machines manipulate objects. As a result, the machines can understand less-specific requests from humans and still complete the desired task. For example, if a user asks the robot to “pick up a tall mug,” the robot can locate and grab the item that best fits that description.“Making robots that can actually generalize in the real world is incredibly hard,” says Ge Yang, postdoc at the National Science Foundation AI Institute for Artificial Intelligence and Fundamental Interactions and MIT CSAIL. “We really want to figure out how to do that, so with this project, we try to push for an aggressive level of generalization, from just three or four objects to anything we find in MIT’s Stata Center. We wanted to learn how to make robots as flexible as ourselves, since we can grasp and place objects even though we've never seen them before.”<h2>Learning “what’s where by looking”</h2>The method could assist robots with picking items in large fulfillment centers with inevitable clutter and unpredictability. In these warehouses, robots are often given a description of the inventory that they're required to identify. The robots must match the text provided to an object, regardless of variations in packaging, so that customers’ orders are shipped correctly. For example, the fulfillment centers of major online retailers can contain millions of items, many of which a robot will have never encountered before. To operate at such a scale, robots need to understand the geometry and semantics of different items, with some being in tight spaces. With F3RM’s advanced spatial and semantic perception abilities, a robot could become more effective at locating an object, placing it in a bin, and then sending it along for packaging. Ultimately, this would help factory workers ship customers’ orders more efficiently.“One thing that often surprises people with F3RM is that the same system also works on a room and building scale, and can be used to build simulation environments for robot learning and large maps,” says Yang. “But before we scale up this work further, we want to first make this system work really fast. This way, we can use this type of representation for more dynamic robotic control tasks, hopefully in real-time, so that robots that handle more dynamic tasks can use it for perception.”The MIT team notes that F3RM’s ability to understand different scenes could make it useful in urban and household environments. For example, the approach could help personalized robots identify and pick up specific items. The system aids robots in grasping their surroundings — both physically and perceptively.“Visual perception was defined by David Marr as the problem of knowing ‘what is where by looking,’” says senior author Phillip Isola, MIT associate professor of electrical engineering and computer science and CSAIL principal investigator. “Recent foundation models have gotten really good at knowing what they are looking at; they can recognize thousands of object categories and provide detailed text descriptions of images. At the same time, radiance fields have gotten really good at representing where stuff is in a scene. The combination of these two approaches can create a representation of what is where in 3D, and what our work shows is that this combination is especially useful for robotic tasks, which require manipulating objects in 3D.”<h2>Creating a “digital twin”</h2>F3RM begins to understand its surroundings by taking pictures on a selfie stick. The mounted camera snaps 50 images at different poses, enabling it to build a <a href="https://www.matthewtancik.com/nerf" target="_blank">neural radiance field</a> (NeRF), a deep learning method that takes 2D images to construct a 3D scene. This collage of RGB photos creates a “digital twin” of its surroundings in the form of a 360-degree representation of what’s nearby.In addition to a highly detailed neural radiance field, F3RM also builds a feature field to augment geometry with semantic information. The system uses&nbsp;<a href="https://openai.com/research/clip" target="_blank">CLIP</a>, a vision foundation model trained on hundreds of millions of images to efficiently learn visual concepts. By reconstructing the 2D CLIP features for the images taken by the selfie stick, F3RM effectively lifts the 2D features into a 3D representation.<h2>Keeping things open-ended</h2>After receiving a few demonstrations, the robot applies what it knows about geometry and semantics to grasp objects it has never encountered before. Once a user submits a text query, the robot searches through the space of possible grasps to identify those most likely to succeed in picking up the object requested by the user. Each potential option is scored based on its relevance to the prompt, similarity to the demonstrations the robot has been trained on, and if it causes any collisions. The highest-scored grasp is then chosen and executed. To demonstrate the system’s ability to interpret open-ended requests from humans, the researchers prompted the robot to pick up Baymax, a character from Disney’s “Big Hero 6.” While F3RM had never been directly trained to pick up a toy of the cartoon superhero, the robot used its spatial awareness and vision-language features from the foundation models to decide which object to grasp and how to pick it up. F3RM also enables users to specify which object they want the robot to handle at different levels of linguistic detail. For example, if there is a metal mug and a glass mug, the user can ask the robot for the “glass mug.” If the bot sees two glass mugs and one of them is filled with coffee and the other with juice, the user can ask for the “glass mug with coffee.” The foundation model features embedded within the feature field enable this level of open-ended understanding.“If I showed a person how to pick up a mug by the lip, they could easily transfer that knowledge to pick up objects with similar geometries such as bowls, measuring beakers, or even rolls of tape. For robots, achieving this level of adaptability has been quite challenging,” says MIT PhD student, CSAIL affiliate, and co-lead author William Shen. “F3RM combines geometric understanding with semantics from foundation models trained on internet-scale data to enable this level of aggressive generalization from just a small number of demonstrations.” Shen and Yang wrote the paper under the supervision of Isola, with MIT professor and CSAIL principal investigator Leslie Pack Kaelbling and undergraduate students Alan Yu and Jansen Wong as co-authors. The team was supported, in part, by Amazon.com Services, the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research’s Multidisciplinary University Initiative, the Army Research Office, the MIT-IBM Watson AI Lab, and the MIT Quest for Intelligence. Their work will be presented at the 2023 Conference on Robot Learning.

Feature Fields for Robotic Manipulation (F3RM) enables robots to interpret open-ended text prompts using natural language, helping the machines manipulate unfamiliar objects. The system’s 3D feature fields could be helpful in environments that contain thousands of objects, such as warehouses. Images courtesy of the researchers.

By blending 2D images with foundation models to build 3D feature fields, a new MIT method helps robots understand and manipulate nearby objects with open-ended language prompts.

Using language to give robots a better grasp of an open-ended world

<h2 dir="ltr" id="isPasted">Event Overview</h2>Hackster's Impact Summit is a free two-day virtual event (14th-15th November, 2023) dedicated to inspiring developers, connecting organizations, and accelerating technological innovation. This year's topic examines how to build resilient and sustainable cities.&nbsp;<h2 dir="ltr">Who Should Attend</h2>The Impact Summit is inviting technologists, engineers, community trailblazers and technology providers. The event offers an opportunity to examine the transformative role of machine learning, advanced data analytics, and IoT in enhancing the management and maintenance of urban infrastructure, public services, and critical facilities.<h2 dir="ltr">Session Highlights</h2>Day 1 commences with Kate Kallot, CEO of Amini, examining the harmony of socioeconomic growth with environmental conservation. Followed by a workshop on predictive maintenance using Nordic Thingy:53, led by David Tischler from Edge Impulse. The day progresses with a demo on SCADA system modernization with Bryan Costanich, Wilderness Labs CEO, and concludes with a panel discussion on urban environment innovation and several workshops.Day 2 opens with a keynote from Tomas Diez, Executive Director of Fab City Foundation, discussing community-driven sustainable urban development. The day continues with a panel on technology resiliency, workshops on sustainable city AI and embedded programming, and concludes with a demo on frugal innovation for urban development by Kevin Schmidt from UNDP.<a href="https://wevlv.co/impactsummit2023" target="_blank" rel="noopener noreferrer" class="button-main-action">Learn more now</a><h2 dir="ltr">Insights</h2>The summit features speakers from leading organizations and companies who are at the forefront of urban technology and development. They will share real-world applications and insights into the impact of technological innovation on city living.<h3 dir="ltr">Technical Deep Dive</h3>Attendees will benefit from a practical approach to smart city challenges, with in-depth technical workshops on IoT, AI at the edge, and predictive maintenance. These sessions will equip participants with knowledge applicable to real-world urban systems.<h3 dir="ltr">Interactive Engagement</h3>Live panel discussions provide a platform for interactive dialogue between technology experts, policymakers, and attendees, focusing on the translation of ideas into tangible urban solutions.<h3 dir="ltr">Learning Outcomes</h3>Participants will leave with actionable strategies for deploying technology in urban infrastructure, insights into sustainable development practices, and a comprehensive understanding of the predictive maintenance landscape.<h3 dir="ltr">Event Accessibility</h3>Sessions are scheduled in Central Time (CST).Post-event access to session recordings will be provided to all registrants.<a href="https://wevlv.co/impact-summit-zoom" target="_blank" rel="noopener noreferrer" class="button-main-action">Register now</a>Would you like to learn more about the Impact Summit? <a href="https://wevlv.co/impactsummit2023" target="_blank" rel="noopener noreferrer">Visit this page.</a>

Accelerating Urban Efficiency through Technology. Explore How AI & IoT Drive the Future of Sustainable Urban Living



Impact Summit 2023 Virtual Event

How neuromorphic computing and BrainChip's Akida solution can benefit AI training


Navigating the Challenges in AI Training

The winners of the <a href="https://www.tinyml.org/event/tinyml-hackathon-2023-pedestrian-detection/" target="_blank" rel="noopener noreferrer">Pedestrian Safety Challenge Hackathon</a> competition have been announced. The hackathon was a collaboration between the TinyML Foundation the City of San Jose and &nbsp;Sony Semiconductor Solutions America (SSSA). This initiative began in May with the aim of reducing pedestrian-involved accidents, aligning with the city's Vision Zero initiatives.&nbsp;The challenge was a response to the increasing number of pedestrian injuries and fatalities due to issues such as distracted driving, distracted walking, and illegal road crossings.<iframe width="640" height="360" src="https://www.youtube.com/embed/ZhBCtfalcOk?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe>The competition saw participation from 29 teams globally, including countries like the United States, Germany, Lebanon, Nigeria, and Saudi Arabia. Teams from Silicon Valley and the San Francisco Bay Area also took part. Mark Hanson, Vice President and Head of Marketing for System Solution Business Development at SSSA, expressed his optimism about the innovative solutions presented by the teams and their potential to benefit communities worldwide.The<a href="https://leopardimaging.com/" target="_blank" rel="noopener noreferrer"> Leopard Imaging </a>team clinched the first place with their solution that utilized SSSA's AiTrios™ platform and IMX500-enabled hardware. The Neurohm team secured the second place, followed by the team from King Abdullah University of Science and Technology (KAUST) in third place. The KAUST team also received a special Edge Impulse award.Leopard Imaging is a global leader in camera design and manufacture, creating state-of-the-art, AI-based cameras for Fortune 500 companies. Their reputation is credited by excellent camera manufacture and camera tuning by experts of the field, and the use of highly precise Active Alignment (AA) machines. Over the years their excellence in building artificially intelligent and robust camera solutions has earned Leopard Imaging many certifications like those for automotive-grade camera manufacture and aviation-grade camera development. Their cameras have spanned all imaginable industries, from autonomous vehicles to cashierless carts, smart street lights, and embedded cameras in drones.Read more about the hackathon in our previous article <a href="https://www.wevolver.com/article/revolutionizing-traffic-safety-the-global-tinyml-traffic-hackathon" target="_blank" rel="noopener noreferrer">here</a>.&nbsp;

 Leopard Imaging wins global challenge. 

Winners of the tinyML Traffic Hackathon demonstrate the future of pedestrian safety

In machine learning, the quality and relevance of data play a pivotal role in determining the success of a project. As the saying goes, "Data is the new oil," but just like oil, raw data needs refining to be useful. Refining, organizing, and ensuring data quality is known as dataset curation. This step is paramount, especially in the machine learning domain, where the accuracy and efficiency of models heavily depend on the data they are trained on.&nbsp;As AI integration across industries accelerates, the significance of data curation becomes even more pronounced. A well-curated dataset ensures optimal model performance and mitigates risks associated with biased or inaccurate predictions; for companies like Edge Impulse, which emphasize the importance of on-device machine learning, the efficiency and reliability of datasets are crucial. This article delves into the best practices in dataset curation, highlighting strategies and considerations that can make or break a machine learning project[1].<h2 dir="ltr">Tailoring Best Practices Based on Data Type</h2>While there are general best practices that apply universally to dataset curation, the nuances of different types of data - sensor, audio, and picture/image - necessitate specific considerations. We will explore whether a one-size-fits-all approach is feasible or if tailored strategies are required for each data type.<h3 dir="ltr">Standard Practices in Dataset Curation</h3>Dataset curation is a meticulous process that ensures data quality, relevance, and usability. The foundational principles that apply across data types include:Versioning and Backing up Files: It's crucial to have a system to track different document or software project versions. Platforms like <a href="https://github.com/" target="_blank">GitHub</a>, <a href="https://osf.io/" target="_blank">Open Science Framework (OSF)</a>, and LabArchives@Brown offer versioning for projects. Moreover, backing up data locally and remotely safeguards against data loss due to unforeseen circumstances.[1]Contextualizing Your Data: Describing data in a standardized manner is essential for comprehension. Data glossaries, dictionaries, README files, and codebooks provide context. This includes column headings, measurement units, value types, and data collection methods.Choosing Formats for Re-Use: Proprietary data from instruments might require licenses for access. It's beneficial to convert such data to open or commonly used formats, ensuring broader accessibility.<h3 dir="ltr">Data-Specific Best Practices</h3>Sensor Data: Time-series data, inherent in sensors, requires careful handling to maintain its sequential integrity. Dealing with noise and ensuring the data's real-world relevance are also paramount.Audio Data: Sound quality assurance is vital, given the variations in audio formats. It's also essential to consider the diverse acoustic environments from which audio data might originate.Image Data: Image quality control is crucial, especially when dealing with different resolutions and formats. Diverse visual contexts, such as lighting and perspective, should also be considered.While there are overarching best practices in dataset curation, the specific nature of the data type in question demands tailored strategies to ensure optimal results.<h2 dir="ltr">Addressing Challenges in Dataset Curation</h2>Dataset curation is integral to the success of machine-learning projects. However, the process has its challenges. Addressing these challenges head-on is crucial for ensuring the quality and reliability of the curated datasets.<h3 dir="ltr">Imbalanced Datasets</h3>One of the most common challenges in dataset curation is dealing with imbalanced datasets. An imbalanced dataset is one where certain data classes are overrepresented while others are underrepresented. This can lead to biased models that perform well on the overrepresented classes but poorly on the underrepresented ones.For instance, in a machine learning project aimed at recognizing different sounds, the model might need help recognizing the latter if the dataset has thousands of urban sounds but only a handful of rural sounds. Addressing this requires techniques like oversampling the minority class, undersampling the majority class, or using synthetic data generation methods.<h3 dir="ltr">Data Privacy and Ethics</h3>Another significant challenge is ensuring data privacy and adhering to ethical standards during dataset curation. With the increasing awareness and regulations around data privacy, such as <a href="https://gdpr-info.eu/" target="_blank">GDPR</a>, it's imperative to handle data carefully, ensuring that personal and sensitive information is anonymized or removed.A recent article from <a href="https://www.nature.com/articles/s43588-020-00015-6" target="_blank">Nature Computational Science</a> highlighted the challenges faced during data curation related to pandemic data.[3] The article emphasized the importance of global data curation and standardization efforts to ensure rapid data integration and dissemination. It also highlighted the ethical, legal, and privacy issues hindering open data sharing. Such challenges are not exclusive to pandemic data but are relevant to any data curation effort.Moreover, ethical considerations extend beyond privacy. For instance, ensuring that the data doesn't reinforce existing biases or lead to discrimination against specific populations based on gender, age, or location.&nbsp;<h2 dir="ltr">Tips and Tricks for Effective Dataset Curation</h2>Dataset curation is a crucial step in the machine learning pipeline. Properly curated datasets ensure the accuracy of models and streamline the training process. Here are some actionable tips to enhance the dataset curation process:1. Plan for Accuracy at the Source: It's more efficient to validate data at its origin than to assess its accuracy later. For instance, asking users to validate their data or using sampling and auditing can help estimate accuracy levels.2. Annotate and Label: Annotating and labeling datasets during curation can simplify data management and troubleshooting. This can include adding timestamps or event locations. However, ensure that metadata is accurate to prevent inaccuracies during data transformation.3. Prioritize Security and Privacy: Large datasets can be vulnerable to breaches. Security practices like encryption, de-identification, and a robust data governance model can mitigate risks. Edge Impulse's emphasis on on-device processing also underscores the importance of data privacy.4. Anticipate Future Needs: Begin the data curation process with a clear vision of the end goal. Track how analytics and machine learning applications utilize datasets and refine the data aggregation process accordingly.5. Balance Data Governance with Agility: Striking a balance between data governance and business agility is crucial. Centralized platforms can help users identify relevant data and processes for their analytics or machine learning projects.6.&nbsp;Identify Business Needs: Ensure the curated data aligns with business objectives. Engage business stakeholders to understand their requirements early and avoid curating irrelevant data.<h3 dir="ltr">Data-Specific Best Practices</h3>1. Sensor Data: Consider the data collection frequency and noise potential for time-series data. Ensure that the data reflects real-world scenarios and is relevant to the problem.2. Audio Data: Ensure sound quality by considering factors like background noise and audio format compatibility. Diverse acoustic environments can introduce variability, so curate data that captures this diversity.3. Image Data: Image quality is paramount. Consider factors like resolution, format, and the context in which images are captured. Diverse visual contexts can provide a more comprehensive dataset for training models.By adhering to these best practices, organizations can curate datasets that are not only high-quality but also tailored to their specific needs, ensuring the success of their machine-learning projects.<h2 dir="ltr">Leveraging Edge Impulse for Dataset Curation</h2>In machine learning, dataset curation is a pivotal step that can make or break the performance of a model. Edge Impulse, a leading development platform for machine learning on edge devices, offers tools that streamline and enhance the dataset curation process.<h3 dir="ltr">Edge Impulse's Data Collection Tools</h3><a href="https://edgeimpulse.com/" target="_blank" rel="noopener noreferrer">Edge Impulse</a> understands the importance of diverse and high-quality data. The platform allows developers to gather data from many sources, be it their sensor hardware, public datasets, or data generated through simulations or synthetic data generation.[5] This flexibility ensures the dataset is comprehensive and relevant to the specific application.<h2 dir="ltr">Data Visualization and Analysis with Edge Impulse</h2>Once data is collected, understanding and analyzing it becomes crucial. Edge Impulse provides advanced data analysis tools that help quickly detect data quality issues, such as visual representation of outliers in a dataset, or mislabeled data. Whether you're working with sensor data, vision, or audio, the platform offers tailored tools that ensure your dataset is of the highest quality, ready to be fed into a machine-learning model.[5]<h2 dir="ltr">Conclusion</h2>Dataset curation is the backbone of any successful machine-learning project. The quality and relevance of the data directly influence the model's performance. With platforms like Edge Impulse, the process of dataset curation becomes more streamlined and efficient. By leveraging the right tools and adhering to best practices, one can ensure that their machine learning models are built on a solid foundation of high-quality data.<h2 dir="ltr">References</h2><ol><li dir="ltr"><a href="https://www.artiba.org/blog/the-essentials-of-machine-learning-data-curation" target="_blank">The Essentials of Machine Learning Data Curation</a></li><li dir="ltr"><a href="https://library.brown.edu/info/data_management_curation/" target="_blank">Brown University Library - Data Curation Best Practices</a></li><li dir="ltr"><a href="https://www.nature.com/articles/s43588-020-00015-6" target="_blank">Data curation during a pandemic and lessons learned from COVID-19 - Nature Computational Science</a></li><li dir="ltr"><a href="https://www.techtarget.com/searchdatamanagement/feature/8-tips-to-improve-the-data-curation-process" target="_blank">8 tips to improve the data curation process | TechTarget</a></li><li dir="ltr"><a href="https://www.edgeimpulse.com/" target="_blank">Edge Impulse</a></li></ol>

Unlocking AI Potential Through Strategic Dataset Curation: A Comprehensive Guide for Enhancing Machine Learning Models


Refining AI's Foundation: Mastering Dataset Curation for Machine Learning

This article originally appeared in&nbsp;<a href="https://www.caltech.edu/about/news/artificial-intelligence-is-key-to-better-climate-models-say-researchers" target="_blank" rel="noopener noreferrer">Caltech</a> and has been republished with permission.<hr>Scientists have become pretty good at predicting the weather, at least up to 10 days in the future. But making accurate, longer-term predictions about how climate is changing remains challenging. The ability to predict climate change is important because it can help guide decisions about insurance coverage in fire- or flood-prone areas; it can also help architects design safer and more comfortable buildings for occupants in a warmer future.Our ability to model climate change has been hampered by the enormous amount of computing power required to simulate all facets of climate, says <a href="https://www.eas.caltech.edu/people/tapio" target="_blank">Tapio Schneider</a>, the Theodore Y. Wu Professor of Environmental Science and Engineering and a senior research scientist at JPL, which Caltech manages for NASA. To make a global climate model accurate, it needs to capture small-scale processes, such as those controlling droplet formation in clouds, over the entire planet. Simulating this huge range of relevant processes is currently impossible.Instead, climate models calculate how temperature, winds, and ocean currents evolve on a mesh with a horizontal resolution around 50–100 kilometers. Smaller-scale processes, such as those in low-lying marine clouds that help shape the climate, are represented by rough approximations called parameterizations. These parameterizations often lack a strong foundation in both scientific principles and data.Recently, a group of climate scientists proposed a massively scaled climate-modeling project as a solution. The project would consist of a series of centers running supercomputers that are staffed by 200–300 scientists each. The goal would be to create climate models with a resolution of 1 kilometer.Schneider says that might prove difficult."Running climate models is enormously computationally expensive," Schneider says. "If you want to make the computational mesh finer by a factor of 10 in the horizontal plane, you need a thousand times the computational power."Schneider says a project of that nature would strain our current computational abilities, and it would not be able to represent the small-scale processes necessary for accurate modeling. It would further concentrate resources among developed nations better able to fund such centers, thereby leaving out developing nations, he says.In <a href="https://www.nature.com/articles/s41558-023-01769-3" target="_blank">a paper</a> recently published in the journal Nature Climate Change, Schneider and 13 other scientists from around the world propose to develop models with a moderately high resolution (tens of kilometers), providing a stronger scientific foundation for parameterizations of smaller-scale processes, using machine learning to fill in the gaps. This, the researchers say, will create robust models without the need for gargantuan computing centers."We cannot possibly resolve the marine clouds over the tropical oceans globally anytime soon because it will require a computer about a hundred billion times faster than the fastest one we have," Schneider says. "What we can do is simulate these clouds in small areas and generate data. Those data can also inform a coarser climate model."Schneider says work on the kinds of AI tools needed to improve the coarser models is happening right now, some of it at Caltech. This technology could do for climate models what better data has done for weather models."Weather forecasts have become quite good over the last few decades. It's a bit of an unsung success story of science," he says. "The primary reason they've become quite good is that they use data much more extensively than they used to. In some ways, we want to achieve the same thing for climate prediction. I think that can be achieved by using AI tools to harness the extensive data about Earth we already have."Andrew Stuart, Bren Professor of Computing and Mathematical Sciences and a co-author of the proposal, has lent his expertise to Schneider's effort, helping build the tools necessary for the project. Stuart says one set of tools aims to make climate models less computationally expensive to run.This can be done by creating parameterizations for coarser models, which Stuart describes as "excellent sketches" of actual physical processes, such as the behavior of individual clouds. In essence, they capture the main features needed to predict the effects that small-scale processes will have on larger-scale processes in a global model simulation. Because they are simplified versions of actual events, they have to be tuned to behave like their real-life counterparts.Even coarser models with parameterizations are expensive to run in terms of computing time and power, and the tuning process means they need to be run repeatedly until the researchers succeed in tuning them to produce reliable and realistic outputs."The primary computational challenge is to perform the tuning with the smallest number of costly model evaluations," Stuart says. "We do that by building on ideas that go back to 1960 and an engineer named Rudolf Kálmán; the ideas have now evolved into what are called ensemble Kalman filters. They allow us to make predictions and calibration with the minimal number of evaluations. They are used in weather forecasting, and we've repurposed them."The second set of tools Stuart has worked on in the climate arena fall under the umbrella of what might be popularly referred to as A.I., and more specifically, machine learning. He says machine learning can help quantify some of the uncertainty inherent in climate modeling and generate new data at scales that are hard to measure."The models have an enormous amount of uncertainty in them," Stuart says. "This is where machine learning enters."The machine learning used by the team has similarities to image generators like OpenAI's DALL-E. In the case of DALL-E, the machine-learning algorithm is trained on a huge set of sample images. So if you ask it, for example, to generate a picture of a man holding an ice cream cone at sunset, it can.The algorithms used in climate science are instead trained to generate pieces—hundreds or thousands of them—that can be plugged into bigger simulations to fill in the missing gaps. Just like DALL-E can conjure up images of apples when asked (because it has learned to identify them), the algorithms used in this research can generate models of processes important to climate modeling, such as droplet formation in clouds.By putting simulated processes of that sort together with real-world data, researchers hope to create realistic predictions about Earth's rapidly changing climate, Schneider says."We can use the physical laws as far as we can, given the limited computational power at our disposal," he says. "What we cannot simulate, we can learn from data."

The butterfly effect—the idea that a butterfly flapping its wings in South America could change the course of a tornado in Texas—has been a popular way to illustrate the difficulty of making predictions about the behavior of chaotic systems like the weather. Because such systems are extremely sensitive to their initial conditions, even the tiniest of changes can send their behavior careening off into unexpected directions.

Artificial Intelligence is Key to Better Climate Models, Say Researchers

This article originally appeared in <a href="https://news.engin.umich.edu/2023/10/choosing-exoskeleton-settings-like-a-pandora-radio-station/" target="_blank" rel="noopener noreferrer">Michigan Engineering</a> and has been republished with permission.<hr>Taking inspiration from music streaming services, a team of engineers at the University of Michigan, Google and Georgia Tech has designed the simplest way for users to program their own exoskeleton assistance settings.Of course, what’s simple for the users is more complex underneath, as a machine learning algorithm repeatedly offers pairs of assistance profiles that are most likely to be comfortable for the wearer. The user then selects one of these two, and the predictor offers another assistance profile that it believes might be better. This approach enables users to set the exoskeleton assistance based on their preferences using a very simple interface, conducive to implementing on a smartwatch or phone.“It’s essentially like Pandora music,” said <a href="https://me.engin.umich.edu/people/faculty/elliott-rouse/" target="_blank" rel="noreferrer noopener">Elliott Rouse</a>, U-M associate professor of robotics and mechanical engineering and corresponding author of the study in Science Robotics. “You give it feedback, a thumbs up or thumbs down, and it curates a radio station based on your feedback. This is a similar idea, but it’s with exoskeleton assistance settings. In both cases, we are creating a model of the user’s preferences and using this model to optimize the user’s experience.”<iframe width="640" height="360" src="https://www.youtube.com/embed/OJ2ApTzZU2w?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe>The team tested the approach with 14 participants, each wearing a pair of ankle exoskeletons as they walked at a steady pace of about 2.3 miles per hour. The volunteers could take as much time as they wanted between choices, although they were limited to 50 choices. Most participants were choosing the same assistance profile repeatedly by the 45th decision.&nbsp;After 50 rounds, the experimental team began testing the users to see whether the final assistance profile was truly the best—pairing it against 10 randomly generated (but plausible) profiles. On average, participants chose the settings suggested by the algorithm about nine out of 10 times, which highlights the accuracy of the proposed approach.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1697684045536-ExoChoice-content1.jpg" style="width: 766px;" class="fr-fic fr-dib">Ellie Wilson, a PhD student in mechanical engineering, demonstrates a new control strategy that customizes assistance provided by a pair of ankle exoskeletons. A machine learning algorithm presents her with two assistance profiles that it has selected based on previous data. She chooses one, and it presents another for comparison. After about 45 choices, most users have found an assistance profile that they choose repeatedly. Photo: Brenda Ahearn/Michigan Engineering“By using clever algorithms and a touch of AI, our system figures out what users want with easy yes-or-no questions,” said Ung Hee Lee, a recent U-M doctoral graduate from mechanical engineering and first author of the study, now at the robotics company Nuro. “I’m excited that this approach will make wearable robots comfortable and easy to use, bringing them closer to becoming a normal part of our day-to-day life.”The control algorithm manages four exoskeleton settings: how much assistance to give (peak torque), how long to go between peaks (timing), and how the exoskeleton both ramps up and reduces the assistance on either side of each peak. This assistance approach is based on how our calf muscle adds force to propel us forward in each step.Rouse reports that few groups are enabling users to set their own exoskeleton settings.“In most cases, controllers are tuned based on biomechanical or physiological results. The researchers are adjusting the settings on their laptops, minimizing the user’s metabolic rate. Right now, that’s the gold standard for exoskeleton assessment and control,” Rouse said.“I think our field overemphasizes testing with metabolic rate. People are actually very insensitive to changes in their own metabolic rate, so we’re developing exoskeletons to do something that people can’t actually perceive.”&nbsp;In contrast, user preference approaches not only focus on what users can perceive but also enable them to prioritize qualities that they feel are valuable.The study builds on the team’s previous effort to enable users to apply their own settings to an ankle exoskeleton. In that study, users had a touchscreen grid that put the level of assistance on one axis and the timing of the assistance on another. Users tried different points on the grid until they found one that worked well for them.&nbsp;Once users had discovered what was comfortable, over the course of a couple of hours, they were then able to find their settings on the grid within a couple of minutes. The new study cuts down that longer period of discovering which settings feel best as well as offering two new parameters: how the assistance ramps up and down.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1697684067237-ExoChoice-content3-1024x683.jpg" style="width: 766px;" class="fr-fic fr-dib">A close-up of the exoskeleton worn by Ellie Wilson. Photo: Brenda Ahearn/Michigan EngineeringThe data from that earlier study were used to feed the machine learning predictor. An evolutionary algorithm produces variations based on the assistance profiles that those earlier users preferred, and then the predictor—a neural network—ranked those assistance profiles. With each choice the users made, new potential assistance profiles were generated, ranked and presented to the user alongside their previous choice.&nbsp;The study was funded by X, Google’s “Moonshot Factory,” Robotics at Google (now Google Deepmind), and the D. Dan and Betty Kahn Foundation. The concept is currently licensed by Alphabet spinoff <a href="https://www.skipwithjoy.com/" target="_blank" rel="noreferrer noopener">Skip Innovations</a>.

Using a simple and convenient touchscreen interface, the algorithm learns the assistance preferences of the wearer.

Choosing exoskeleton settings like a Pandora radio station

Training robust AI models can be challenging when using crowdsourced or publicly available datasets. The most useful neural networks are ones that replace expert knowledge to solve real problems. One example would be a visual inspection model for the factory floor.<h2>Human level performance</h2>Human performance comes into tiers: general or expert. An expert is someone who has spent a minimum amount of time or has received practical training in the field.The accuracy of your AI model is directly correlated to the quality of data used to train it. When using general datasets for training, the model usually surpasses the knowledge level embedded in the training samples, but it remains well below expert-level performance. A computer is more attentive than an operator performing the same inspection task, so the model trained will probably reach an accuracy that is marginally better than the human labelled dataset.If you think it's a good idea to use services like Amazon Mechanical Turk to label your dataset, let me ask you a question. Would you take a person and after reading a quick instruction manual get them to work inside the factory?That does not sound like a great idea. But it turns out that is what every one of us is doing when leveraging services like these. The bulk of your dataset can be generated in this manner, but is it helping your model?<h2>Practical considerations</h2>Tricky samples may not make it to the dataset because someone was not able to understand your labeling instructions, and that sample may get discarded. While human-level performance is achievable with large crowd labeled datasets, the truth is that the easiest samples will be labeled first.I had seen numerous occasions in which turks were simply targeting the lowest hanging fruit by submitting no labels for the empty samples. In that case, it is a first come first rewarded economy. The difficult samples may end up unlabeled.<h2>Model hardening</h2>Synthetic data allows the possibility to make the task at hand harder than it will be in real life.Variance or noise is good for your model, while generating samples programmatically can make the model more resilient to unseen inputs. Human labeled data is often curated and less noisy than the real inputs.This may work out, in the beginning, to quickly get started with your model. But believe me, you don't want a crowd trained model to be put in production across a factory floor without verifying that its performance surpasses expert level or that it gets as closely as possible to those expectations.<h2>Methods for generating data</h2>There are several methods you can leverage when generating synthetic data.<h3>Simulator data</h3>Use a simulator to generate new data. Custom shaders can provide RGB+D and segmentation labels. Microsoft is now focusing on aerospace-grade data generation because it is open source, AirSim 3D simulator project obtained promising results hence a lot of companies were using it for internal purposes.<iframe width="640" height="360" src="https://www.youtube.com/embed/b59sxe-M-bg?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe><h3 id="isPasted">Synthetic augmentation</h3>Simple computer vision enhancements can work as well. Using the OpenCVs inpaint function the price was rewritten as per the examples shown bellow. A mixed approach was implemented here, having the human labelers define the polygon in which the price was contained.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1697620602389-synthetic_inpaint_example-768x376.jpg" style="width: 766px;" class="fr-fic fr-dib"><h2 id="isPasted">Development focus</h2>Validating crowd samples is going to take a similar amount of effort as developing a dataset generator.This is the most essential part of AI and a lot of people get it wrong. If the dataset is poorly curated and not enough time is spent at this stage, your project may end up failing. The dataset should be one of your main priorities, and if you have enough quality data, you can train almost every model to perfection.<h2>Conclusion</h2>The most successful models deployed in production use a mix of the synthetic and real data to achieve great results.<a href="https://www.linkedin.com/company/adapta-robotics" target="_blank">Let us know</a> what other techniques you are using to give your model that extra edge in performance.

Training robust AI models can be challenging when using crowdsourced or publicly available datasets. The most useful neural networks are ones that replace expert knowledge to solve real problems. One example would be a visual inspection model for the factory floor.

Synthetic datasets-Or why you can't train robust models with generic data

This article originally appeared in <a href="https://news.mit.edu/2023/new-technique-helps-robots-pack-objects-tight-space-1017" target="_blank" rel="noopener noreferrer">MIT News</a> and has been republished with permission.<hr>Anyone who has ever tried to pack a family-sized amount of luggage into a sedan-sized trunk knows this is a hard problem. Robots struggle with dense packing tasks, too.For the robot, solving the packing problem involves satisfying many constraints, such as stacking luggage so suitcases don’t topple out of the trunk, heavy objects aren’t placed on top of lighter ones, and collisions between the robotic arm and the car’s bumper are avoided.Some traditional methods tackle this problem sequentially, guessing a partial solution that meets one constraint at a time and then checking to see if any other constraints were violated. With a long sequence of actions to take, and a pile of luggage to pack, this process can be impractically time consuming. &nbsp;&nbsp;MIT researchers used a form of generative AI, called a diffusion model, to solve this problem more efficiently. Their method uses a collection of machine-learning models, each of which is trained to represent one specific type of constraint. These models are combined to generate global solutions to the packing problem, taking into account all constraints at once.<iframe width="640" height="360" src="https://www.youtube.com/embed/mhtYU7IevtE?&amp;wmode=opaque&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true">&nbsp;</iframe>Their method was able to generate effective solutions faster than other techniques, and it produced a greater number of successful solutions in the same amount of time. Importantly, their technique was also able to solve problems with novel combinations of constraints and larger numbers of objects, that the models did not see during training.Due to this generalizability, their technique can be used to teach robots how to understand and meet the overall constraints of packing problems, such as the importance of avoiding collisions or a desire for one object to be next to another object. Robots trained in this way could be applied to a wide array of complex tasks in diverse environments, from order fulfillment in a warehouse to organizing a bookshelf in someone’s home.“My vision is to push robots to do more complicated tasks that have many geometric constraints and more continuous decisions that need to be made — these are the kinds of problems service robots face in our unstructured and diverse human environments. With the powerful tool of compositional diffusion models, we can now solve these more complex problems and get great generalization results,” says Zhutian Yang, an electrical engineering and computer science graduate student and lead author of a <a href="https://arxiv.org/pdf/2309.00966.pdf" target="_blank" data-lf-fd-inspected-kn9eq4roawkarlvp="true">paper on this new machine-learning technique</a>.Her co-authors include MIT graduate students Jiayuan Mao and Yilun Du; Jiajun Wu, an assistant professor of computer science at Stanford University; Joshua B. Tenenbaum, a professor in MIT’s Department of Brain and Cognitive Sciences and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); Tomás Lozano-Pérez, an MIT professor of computer science and engineering and a member of CSAIL; and senior author Leslie Kaelbling, the Panasonic Professor of Computer Science and Engineering at MIT and a member of CSAIL. The research will be presented at the Conference on Robot Learning.<h2>Constraint complications</h2>Continuous constraint satisfaction problems are particularly challenging for robots. These problems appear in multistep robot manipulation tasks, like packing items into a box or setting a dinner table. They often involve achieving a number of constraints, including geometric constraints, such as avoiding collisions between the robot arm and the environment; physical constraints, such as stacking objects so they are stable; and qualitative constraints, such as placing a spoon to the right of a knife.There may be many constraints, and they vary across problems and environments depending on the geometry of objects and human-specified requirements.To solve these problems efficiently, the MIT researchers developed a machine-learning technique called <a href="https://diffusion-ccsp.github.io/" target="_blank">Diffusion-CCSP</a>. Diffusion models learn to generate new data samples that resemble samples in a training dataset by iteratively refining their output.To do this, diffusion models learn a procedure for making small improvements to a potential solution. Then, to solve a problem, they start with a random, very bad solution and then gradually improve it.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1697557883397-ezgif-5-a827fde90a.gif" style="width: 766px;" class="fr-fic fr-dib">Using generative AI models, MIT researchers created a technique that could enable robots to efficiently solve continuous constraint satisfaction problems, such as packing objects into a box while avoiding collisions, as shown in this simulation. Image: Courtesy of the researchersFor example, imagine randomly placing plates and utensils on a simulated table, allowing them to physically overlap. The collision-free constraints between objects will result in them nudging each other away, while qualitative constraints will drag the plate to the center, align the salad fork and dinner fork, etc.Diffusion models are well-suited for this kind of continuous constraint-satisfaction problem because the influences from multiple models on the pose of one object can be composed to encourage the satisfaction of all constraints, Yang explains. By starting from a random initial guess each time, the models can obtain a diverse set of good solutions.<h2>Working together</h2>For Diffusion-CCSP, the researchers wanted to capture the interconnectedness of the constraints. In packing for instance, one constraint might require a certain object to be next to another object, while a second constraint might specify where one of those objects must be located.Diffusion-CCSP learns a family of diffusion models, with one for each type of constraint. The models are trained together, so they share some knowledge, like the geometry of the objects to be packed.The models then work together to find solutions, in this case locations for the objects to be placed, that jointly satisfy the constraints.“We don’t always get to a solution at the first guess. But when you keep refining the solution and some violation happens, it should lead you to a better solution. You get guidance from getting something wrong,” she says.Training individual models for each constraint type and then combining them to make predictions greatly reduces the amount of training data required, compared to other approaches.However, training these models still requires a large amount of data that demonstrate solved problems. Humans would need to solve each problem with traditional slow methods, making the cost to generate such data prohibitive, Yang says.Instead, the researchers reversed the process by coming up with solutions first. They used fast algorithms to generate segmented boxes and fit a diverse set of 3D objects into each segment, ensuring tight packing, stable poses, and collision-free solutions.“With this process, data generation is almost instantaneous in simulation. We can generate tens of thousands of environments where we know the problems are solvable,” she says.Trained using these data, the diffusion models work together to determine locations objects should be placed by the robotic gripper that achieve the packing task while meeting all of the constraints.They conducted feasibility studies, and then demonstrated Diffusion-CCSP with a real robot solving a number of difficult problems, including fitting 2D triangles into a box, packing 2D shapes with spatial relationship constraints, stacking 3D objects with stability constraints, and packing 3D objects with a robotic arm.Their method outperformed other techniques in many experiments, generating a greater number of effective solutions that were both stable and collision-free.In the future, Yang and her collaborators want to test Diffusion-CCSP in more complicated situations, such as with robots that can move around a room. They also want to enable Diffusion-CCSP to tackle problems in different domains without the need to be retrained on new data.“Diffusion-CCSP is a machine-learning solution that builds on existing powerful generative models,” says Danfei Xu, an assistant professor in the School of Interactive Computing at the Georgia Institute of Technology and a Research Scientist at NVIDIA AI, who was not involved with this work. “It can quickly generate solutions that simultaneously satisfy multiple constraints by composing known individual constraint models. Although it’s still in the early phases of development, the ongoing advancements in this approach hold the promise of enabling more efficient, safe, and reliable autonomous systems in various applications.”This research was funded, in part, by the National Science Foundation, the Air Force Office of Scientific Research, the Office of Naval Research, the MIT-IBM Watson AI Lab, the MIT Quest for Intelligence, the Center for Brains, Minds, and Machines, Boston Dynamics Artificial Intelligence Institute, the Stanford Institute for Human-Centered Artificial Intelligence, Analog Devices, JPMorgan Chase and Co., and Salesforce.

MIT researchers are using generative AI models to help robots more efficiently solve complex object manipulation problems, such as packing a box with different objects. Image: Courtesy of the researchers

Researchers coaxed a family of generative AI models to work together to solve multistep robot manipulation problems.

New technique helps robots pack objects into a tight space

Plainsight-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Plainsight-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Plainsight-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid

Edge Impulse-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Mon Oct 30 2023 14:34:39 GMT+0200 (Eastern European Standard Time)

Edge Impulse-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Xsens-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Movella-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:53 GMT+0300 (Eastern European Summer Time)

Movella-Thu Oct 05 2023 18:51:58 GMT+0300 (Eastern European Summer Time)

Movella-Mon Oct 30 2023 14:34:24 GMT+0200 (Eastern European Standard Time)

Movella-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Matroid-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

HepcoMotion-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

HepcoMotion-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

HepcoMotion-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

HepcoMotion-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

HepcoMotion-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

computer vision

Clustering - A Common Unsupervised Learning Technique

Understanding What is Unsupervised Learning, the Mechanisms, Types, and Applications of Various Algorithms and Challenges it presents in Machine Learning

What is Unsupervised Learning: A Comprehensive Guide

Exploring the key concepts related to Unsupervised vs Supervised Learning, understanding the fundamental principles, major algorithms and their real-world applications, and practical distinctions between supervised and unsupervised learning.

Unsupervised vs Supervised Learning: A Comprehensive Comparison

Gain a comprehensive understanding of Edge AI by exploring the fundamental concepts, hardware components, software tools, and real-world applications.

Edge AI: A Comprehensive Guide to its Engineering Principles and Applications

This is the 3rd and final article in a 3 part series featuring the applications of tinyML. The series introduces the concept of tinyML and explains some selected applications of the technology in various fields like sustainability, healthcare, and education.Education is the first step towards innovation. It is an integral part of personal development and equips the learners with the tools necessary to go ahead and solve the challenges faced by humanity. While tinyML is a great tool to develop pedagogical interventions for improving the quality of education in different streams, it is in itself a vast subject that every science and technology student must be familiar with.In our <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future" target="_blank">previous article</a> about the applications of tinyML, we explained how it aids medical research and improves healthcare. This article continues the series by showcasing two more applications of tinyML for education.<h2 dir="ltr">Using tinyML for Neuroscience in K12 education: tinyML4STEM</h2>Anything and everything concerned with the central nervous system is a part of neuroscience. It is a multidisciplinary field that involves a scientific study of how the brain coordinates with the body and makes various body functions possible. The study is not just limited to humans but can also be extended to animals and even plants.Roughly 20% of the world’s population has suffered from a neurological disorder at some point in their lives, and none of the disorders have a cure yet. The only way to make new discoveries in the domain is by promoting research and education.It is also interesting to note that the most fundamental unit of Machine Learning (ML) is called a neuron and the whole concept and modelling of Artificial Intelligence (AI) based systems came from neuroscience. This makes it essential to talk about the specialised subject with the students of computer science to make them understand the underlying principles of the technology.But the problem with neuroscience education has always been the affordability and availability of experiment kits. Unlike many other Science Technology Engineering Management (STEM) fields, the apparatus for performing neuroscience experiments can not be acquired easily by schools and colleges. Graduate students opting for a specialisation in the area are the only ones to get their hands on these experiment kits.Greg Gage and Tim Marzullo realised this problem back in 2008 when they were pursuing PhD at the University of Michigan. The duo tried explaining neuroscience concepts to school children but couldn’t do so without showcasing practicals. There was no way to bring expensive graduate college equipment to schools. To solve this issue and make neuro-science kits affordable and readily available, Greg and Tim founded Backyard Brains.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNDUyMjI0LWltYWdlNy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="EMG-signals-backyard-brains">Illustrations like these can help kids grasp even complex concepts easily. Source: Backyard BrainsThe organisation today makes low-cost neuroscience trainers, DIY kits, tools, software, conducts workshops and a lot more to democratise neuroscience education.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNTI1MDYyLWltYWdlMS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="neuroscience-experiment-kits-backyard-brains">Neuroscience trainer kits by Backyard Brains are great alternatives for schools to introduce the students to the field. Source: Backyard BrainsNeuroscience signals like Electromyogram (EMG), Electrooculogram (EOG), Electrocardiogram (EKG), etc., can be used to get various cues about what is going on inside the body.When neurons fire electrical impulses from the motor cortex, the signals flow through the spinal cord to the parts of the body involved in the action. By putting small electrodes on the skin near the areas where the signals can be picked up, it is possible to observe them via an EMG. The impulses being electrical in nature can be amplified as a single-dimensional signal and visualised easily. These single-dimensional signals are beautiful for ML and are also quite easy to be analysed.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNjI0MDcwLWltYWdlNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="visualising-neuroscience">A snapshot from tinyML talks where Greg demonstrates how EMG signals can be visualised with a low-cost training kit. Source: tinyML YouTubeThe patterns in these signals might not be as apparent to humans as they are to the tinyML algorithms, which can classify and map them with different human behaviours. An implementation of such a technique can be used to develop a brain-machine interface that can help make neuroprosthetics.&nbsp;EOG is another example of a neuro signal that measures the corneo-retinal standing potential of the eye. In other words, it helps understand eye movements. A trained TinyML system can be used to accurately detect eye blinking and movements by feeding the signal to the input.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNjYzNTM5LWltYWdlOC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="EOG-signals-detect-eye-blinking">The spikes that appear on the EOG are due to eye blinking. Source: tinyML YouTubeThen there’s EKG generated by the electrical impulses from the heart. For different people, the patterns of EKG are observed to be similar when looking at the same object. This can be used to predict if the person is telling the truth. In medical applications, signals can be used to check the functioning of the brain. A sample tone is played to the patient, and electric potential signals are observed. When the tone being played changes, the subconscious mind should ideally change the way it reacts, which should be evident in the electric potential signals.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNTU2NjQ2LWltYWdlMy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="hackerhand-copies-gestures-EKG-signals">‘Hackerhand’, a robotic arm mimicking hand movements based on EKG signals. Source: tinyML YouTubeIt is also interesting to note that the signals are generated slightly before someone is about to take action, making it possible for tinyML systems to be able to predict the behaviour in advance. A similar fun project has been developed at Backyard Brains.Readers can participate in many more such projects or share ideas by visiting <a href="https://backyardbrains.com/" target="_blank">Backyard Brains</a>.<h2 dir="ltr">TinyML Open Education Initiative: tinyMLedu</h2>Over the past two decades, ML has been through a journey from a purely academic and theoretical discipline to a widely spread technology in use. It won’t be an understatement to say that technology has touched almost every industry and is now ubiquitous technology. It is expected that the interest in the field will only grow from here, at least in the near foreseeable future. Given its amazing potential in solving some of the greatest challenges faced by humanity, it is important that more people are introduced to technology in a systematic way.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNTk4OTQ2LWltYWdlNS5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="instructors-tinyML-harvard-google-edx">Instructors of the tinyML specialisationUnlike traditional ML, it is possible to run tinyML algorithms on low-cost and low-power embedded systems, which solves a major problem related to the unavailability of powerful hardware for learners. But the shortage of ML educators is still something that needs to be addressed.A joint initiative by Harvard and Google, developed as an academic and industry collaboration, aims to address the challenges by introducing tinyML to the masses with a Massive Open Online Course (MOOC) on edX.&nbsp;The courses that are a part of the tinyML specialisation go beyond teaching the students about the subject by diving deeper into the practical applications and industrial product development lifecycle. The next generation of students needs to understand the end-to-end full-stack development of tinyML projects to be able to gain a complete understanding of the subject.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNjIwMzE4LWltYWdlOS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="tinyML-specialisation-course-structure">A diagram providing an overview of topics explored in tinyML specialisation. Source: [3]The tinyML specialisation, jointly developed by Harvard and Google, is a set of self-paced learning modules that can be taken from anywhere. The specialisation includes four courses:<ol><li dir="ltr">Introduction to tinyML:&nbsp;The first course covers the basics of ML and embedded systems. Everything from the concept to gathering data and training the algorithms is explained.</li><li dir="ltr">Applications of tinyML: The second course builds on the foundation by explaining some parts of the code behind the most widely used tinyML applications. It also introduces the learners to the principles of keyword spotting, visual wake words, anomaly detection, and dataset engineering.</li><li dir="ltr">Deploying tinyML: The third course teaches how code is written, optimised, and deployed for embedded systems. This is where microcontrollers are introduced, and students can purchase a kit for a nominal price (US$49.99) to get hands-on experience.</li><li dir="ltr">Scaling tinyML [Optional]: Once the learners are familiar with the design, development and deployment of tinyML, the final course covers the more advanced aspects related to producing tinyML solutions for the masses.</li></ol>Each course includes several activities such as collabs, hands-on labs, quizzes, readings, assignments, tests, and discussion-forum participation.The key challenges for students taking up the tinyML specialisation can be limited hardware accessibility and the requirement of being familiar with programming. To address the challenges associated with hardware accessibility, open-source emulation platforms like <a href="https://renode.io/" target="_blank">renode.io</a> from Antmicro are being tested to allow learners to try out their code without the hardware. The other challenge can be solved by platforms like <a href="https://www.wevolver.com/profile/edge.impulse" target="_blank">Edge Impulse</a>, which are lowering the barriers to enable people with little to no programming knowledge to implement tinyML solutions.Anyone with interest in tinyML can visit <a href="https://www.edx.org/professional-certificate/harvardx-tiny-machine-learning" target="_blank">tinyML Professional Certificate | edX</a> and enrol for the course for free. The course material is available on <a href="https://github.com/tinyMLx/courseware" target="_blank">GitHub</a>.<h2 dir="ltr" id="isPasted">Conclusion</h2>Though the project kits or MOOCs are not a complete solution to train professional neuroscience researchers or tinyML solution architects, they offer a great place for beginners to start. The kits and courses are designed in such a way that the students can engage in fun activities while they experiment and learn new stuff. This makes the experience memorable and develops an interest in the subject, which could lead more people to choose the specialised fields and make new discoveries.&nbsp;This article concludes our tinyML applications series.About tinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, tinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNjYyNjY4LWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a> explained some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The <a href="https://www.wevolver.com/article/applications-of-tinyml-making-specialised-education-more-accessible" rel="noopener noreferrer" target="_blank">third and final article</a> shall cover the tinyML applications in pedagogy and education.<h2 dir="ltr">References</h2>[1] Marzullo TC, Gage GJ (2012) The SpikerBox: A Low Cost, Open-Source BioAmplifier for Increasing Public Participation in Neuroscience Inquiry. PLOS ONE 7(3): e30837. <a href="https://doi.org/10.1371/journal.pone.0030837" target="_blank">https://doi.org/10.1371/journal.pone.0030837</a>.[2] Gregory J. Gage, The Case for Neuroscience Research in the Classroom, Neuron, Volume 102, Issue 5, 2019, Pages 914-917, ISSN 0896-6273, <a href="https://doi.org/10.1016/j.neuron.2019.04.007" target="_blank">https://doi.org/10.1016/j.neuron.2019.04.007</a>.[3] Vijay J. Reddi et. al., Widening Access to Applied Machine Learning with TinyML, arXiv:2106.04008v2 [cs.LG], Jun 2021, [Online], Available from: <a href="https://arxiv.org/abs/2106.04008" target="_blank">https://arxiv.org/abs/2106.04008</a>.

In this article, we look at two tinyML projects for education. We show how Backyard Brains uses low-cost experiment kits to make neuroscience education more accessible. We also introduce our readers to a specialisation offered by Harvard & Google to help students learn tinyML like never before.

Applications of TinyML making specialised education more accessible

This is the 2nd article in a 3 part series featuring the applications of tinyML. The series introduces the concept of tinyML and explains some selected applications of the technology in various fields like sustainability, healthcare and education.By adopting digital advancements, healthcare providers are updating their services to improve them and make them accessible to more people. 5G networks, Augmented Reality (AR) &amp; Virtual Reality (VR), Wearables, Digital Twins, and many more technologies are enabling this digital transformation.One such technology with great potential in ensuring healthy lives and well-being for all is tinyML. The concept is a crossover between Artificial Intelligence (AI) and embedded systems that involves developing solutions that bring Machine Learning (ML) capabilities to edge devices. Benefits include privacy, affordability, low-power utilisation, the ability to work in remote locations, and a lot more.In our <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">previous article</a> about the applications of tinyML, we explained how it unlocks new possibilities for sustainable development technologies. This article continues the series by showcasing two more applications of tinyML for medical research and improved healthcare.<h2 dir="ltr">MaRTiny: a tinyML solution for extreme heat sensing for urban climate science</h2>As a direct effect of global warming, summers are getting warmer. [1] This disruption of the natural balance poses a risk to all living entities on the planet. Higher temperatures are known to cause a degradation in the air quality and induce severe health-related issues like heat strokes. Looking at the data of weather-related fatalities in America, heat is the single most significant cause of weather-related deaths over the last 30 years, surpassing floods and storms. [2]But it’s not just the ambient temperature that causes discomfort. It’s the Mean Radiant Temperature (MRT) which accounts for the influence of all the surfaces in the surrounding that ultimately determines the occupant’s comfort. Often, the effect of roads, buildings and other entities has an adverse effect on the MRT, creating a complex problem intersecting health and urban infrastructure.To aid the collection of information related to the MRT, a tinyML solution called MaRTiny was developed by researchers from Arizona State University (ASU). It is a $200 low-cost mobile platform for bio-metrological sensing.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3MzYxNzgyLTE2NDI1MTczNjE3ODIucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" width="720" height="297" id="isPasted" class="fr-fic fr-dii" alt="MaRTiny-tech-stack">The tech stack of the MaRTiny platformThe list of sensors embedded on the device include:<ul><li dir="ltr">Ambient temperature sensor</li><li dir="ltr">Globe temperature sensor for sensing the radiant heat</li><li dir="ltr">Humidity sensor</li><li dir="ltr">Ultra Violet (UV) sensor</li><li dir="ltr">Anemometer for measuring wind speed and direction.</li></ul>The platform uses an <a href="https://www.wevolver.com/profile/arduino.pro" target="_blank">Arduino</a> Uno to collect the data and a NodeMCU to provide it with the electronic infrastructure required for wireless data communication. At the core of the system is the Nvidia Jetson Nano development board that provides MaRTiny with the power to run tinyML algorithms.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NDcxMTAyLU5WSURJQV9KZXRzb25fTmFub19EZXZlbG9wZXJfS2l0Xyg0MDY1MDQyNTUwMykuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="nvidia-jetson-nano-development-board">The Nvidia Jetson Nano development board provides MaRTiny with image processing and ML capabilitiesJetson Nano, with its Camera Serial Interface (CSI), can connect with a camera for shadow detection, object detection and people-in-shade detection. Jetson’s ARM Cortex A57 MPcore, 128 CUDA core Maxwell GPU coupled with 4 GB RAM is just enough to offer advanced ML capabilities on edge.The task of MRT estimation was formulated as a supervised learning problem by the inventors that required a lot of true data for training. This true data was collected with the help of a previously developed, larger and more expensive MRT detecting system going by the name of MaRTy. [3]&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NTMzMzM5LTE1MDB4NTAwLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="MaRTiny-MaRTy">MaRTiny aims to offer similar functionality as that of MaRTy (in the image) at just 1% of the price. Image source: @ASUMaRTy, TwitterIt was observed that a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel achieved the highest accuracy with the MaRTiny. The method being computationally light, offered an easy way of deployment on Jetson. To improve the estimations, both SVM and Artificial Neural Network (ANN) models were utilised and trained with data collected via sensors. [4]For privacy, the images taken by MaRTiny are not stored in the cloud or even in the device’s memory. It automatically discards personally identifiable information (PII) before reporting anything back to the servers, which is central to the idea of tinyML and a must-have feature for public deployment.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NTc1MjI0LWltYWdlNy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="MaRTiny-setup">The experimental setup of MaRTinyThe tinyML solution, MaRTiny, can provide crucial insights related to weather conditions and the utilisation of shades and space by the public. Multiple MaRTiny devices can be installed across the city to gather valuable data for architecture and town planning.<h2 dir="ltr">Pneumonia Detection using Embedded Machine Learning</h2>Pneumonia is a respiratory infection caused by viruses, bacterias and fungi. According to the World Health Organisation (WHO) data, it is estimated that as much as 22% of all deaths in children aged from 1 to 5 is due to pneumonia. Bacteria induced pneumonia can be treated with antibiotics and other low-cost and low-tech medical facilities if detected at the right time. [5]Chest X-ray, blood oxygen levels and Complete Blood Count (CBC) are used to detect pneumonia which is time-consuming. Doctors manually review the chest X-ray and decide if the person is infected, which is sometimes inaccurate and requires an automated solution with better accuracy. This is the idea behind the tinyML solution that we are about to discuss in this section.Arijit Das, a 15-year-old tinyML enthusiast, has developed a tinyML model using the <a href="https://www.wevolver.com/profile/edge.impulse" target="_blank">Edge Impulse</a> platform to detect pneumonia from chest X-rays. The model can be deployed on single board development platforms like Raspberry Pi 4, Sony Supersense and a few other devices. With just the main board, a camera shield and the Edge Impulse tinyML model, the tiny setup can detect pneumonia in under a minute which is phenomenal considering the fact that it takes anywhere between 1 to 4 days for detection by manual inspection.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NzA0Nzk2LVJhc3BiZXJyeV9QaV80X01vZGVsX0JfLV9TaWRlLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="Raspberry-Pi-4">Credit card-sized single-board computers like Raspberry Pi come with enough compute power to run optimised tinyML algorithmsThe tinyML model doesn’t require internet connectivity to work and offers an accuracy of over 80%, which is far better compared to manual inspection. This can prove to be quite helpful in largely affected regions like Sub-Saharan Africa and South Asia, where a huge number of examinations need to be conducted related to the same disease.&nbsp;At the time of writing this article, the project is fully functional and has been clinically tested by over 10 doctors in a local survey conducted by the inventor. The project is available on <a href="https://github.com/Pneumonia-Detection-using-EdgeML" target="_blank">GitHub</a> for everyone to try, test and contribute. [6]<h2 dir="ltr">Conclusion</h2>It’s the third year into the pandemic, and the emphasis on healthcare research is more than ever. Introducing technologies that can work with little power and no network connectivity at an affordable price in healthcare can save several lives. Leveraging interdisciplinary technologies like tinyML is the way forward to a safer and healthier future.About tinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, tinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NzY5MDkxLWV5SmlkV05yWlhRaU9pSjNaWFp2YkhabGNpMXdjbTlxWldOMExXbHRZV2RsY3lJc0ltdGxlU0k2SW1aeWIyRnNZUzh4TmpNMk9UZ3lNVEUwT1RRNExWUnBibmxOVEdadmNrZHZiMlFnS0RRd01EQWdlQ0EzTlRBZ2NIZ3BMbXB3WnlJc0ltVmthWFJ6SWpwN0luSmxjMmw2WlNJNmV5SjNhV1IwYUNJNk9UVXdMQ0ptYVhRaU9pSmpiM1psY2lKOWZYMD0ud2VicCIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="tinyML-banner"> The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The&nbsp;<a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a> explained some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" rel="noopener noreferrer" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The third and final article shall cover the tinyML applications in&nbsp;pedagogy&nbsp;and education.<hr><h2>References</h2>[1] Extreme Weather Toolkits, Climate Central, [Online], Available from: &nbsp;<a href="https://medialibrary.climatecentral.org/extreme-weather-toolkits/extreme-heat" target="_blank">https://medialibrary.climatecentral.org/extreme-weather-toolkits/extreme-heat</a>[2] Weather Related Fatality and Injury Statistics, National Weather Service, [Online], Available from:&nbsp;<a href="https://www.weather.gov/hazstat/" target="_blank">https://www.weather.gov/hazstat/</a>[3] Ariane Middel, E. Scott Krayenhoff, ‘Micrometeorological determinants of pedestrian thermal exposure during record-breaking heat in Tempe, Arizona: Introducing the MaRTy observational platform’, Science of The Total Environment, Volume 687, 2019, Pages 137-151, ISSN 0048-9697, Available from:&nbsp;<a href="https://doi.org/10.1016/j.scitotenv.2019.06.085" target="_blank">https://doi.org/10.1016/j.scitotenv.2019.06.085</a>[4] Karthik K. Kulkarni, 2021, ‘Machine Learning and Vision Using Edge Devices for Multimodal Chatbots andBio-meteorological Sensing’, MS Thesis, Arizona State University, United States of America, [Online], Available from:&nbsp;<a href="https://keep.lib.asu.edu/_flysystem/fedora/c7/Kulkarni_asu_0010N_21269.pdf" target="_blank">https://keep.lib.asu.edu/_flysystem/fedora/c7/Kulkarni_asu_0010N_21269.pdf</a>[5] Pneumonia Facts Sheet, World Health Organisation, [Online], Available from:&nbsp;<a href="https://www.who.int/news-room/fact-sheets/detail/pneumonia" target="_blank">https://www.who.int/news-room/fact-sheets/detail/pneumonia</a>[6] Arijit Das, Pneumonia-Detection-using-EdgeML, GitHub, [Online], Available from:&nbsp;<a href="https://github.com/Pneumonia-Detection-using-EdgeML" target="_blank">https://github.com/Pneumonia-Detection-using-EdgeML</a>

In this article, we take a look at two tinyML projects related to healthcare. The first project helps gather Mean Radiant Temperature data outdoors to protect people from extreme heat, and the other one is a solution for affordable, accurate, & rapid detection of pneumonia.

Applications of TinyML in healthcare for a safer future

Changing weather patterns, rising sea levels, wildfires, droughts, depletion of natural resources are just some of the long-term consequences of unsustainable human activities. Indiscriminate exploitation of resources is accelerating global warming, and if preventive countermeasures are not taken immediately, the world may end up at a point of no return.TinyML is a field that aims to bring machine learning (ML) capabilities to low-power embedded systems. Its applications can be found from households to industries and everywhere in between. The idea of sustainability naturally links well with the tinyML as it encourages us to reduce our carbon footprint by executing ML algorithms on low-power devices.In our <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" rel="noopener noreferrer" target="_blank">previous article</a> about tinyML, we introduced our readers to the basics of the technology and how it is helping shape the embedded systems of the future. This article focuses on two interesting applications of tinyML for sustainable development.<h2>Groundwater Wells: The Untapped Edge Use Case</h2>Farming is the oldest and arguably the most crucial industry in the world. It employs around one-fourth of the world’s population and feeds billions of people. [1] Agriculture is also the largest use of fresh water in the world. Regrettably, it involves the largest wastage of water as well, making it incredibly inefficient. Most of the water used here is unregulated and unmonitored.Wastage of water has put 20% of the 39 million groundwater wells at the risk of running dry. [2] Once a well dries up, the only possible option is to dig a deeper one, but that can have a much worse long-term impact on the water table.<a href="https://niolabs.com" target="_blank">Niolabs</a> is a digital transformation and Internet of Things (IoT) solutions organisation that leverages IoT and tinyML in precision farming. Over the past couple of years, the organisation has achieved excellent results with the technology. Precision farming is the future of agriculture involving the collection and utilisation of real-time data to control the various irrigation automation systems. The result is lower consumption of water and energy while saving on operating costs. [3]<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQwMDkyNTAzODY3LXNtYXJ0X2Zhcm1pbmcuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="smart-farming">Precision farming with tinyML takes water conservation to the next level</div>For the implementation of the precision agriculture model, connecting all the devices to the network and passing data to the cloud is not always feasible, especially if a large number of data sources are involved. With thousands of plants being monitored for various environmental variables, the amount of data being generated is enormous and creates a security concern too.Using microcontrollers interfaced with moisture sensors and water control valves, it is possible to implement a simple automatic irrigation system that turns the irrigation system on or off depending on a static value of soil moisture levels. But not all plants require the same amount of moisture. Considering the environmental conditions like temperature, humidity, sunlight, and wind speed that have a direct effect on the water requirements of the plant can also be a challenge on the microcontroller programmed with simple if-else statements.This is where tinyML comes in. By implementing machine learning algorithms on the embedded controllers, it becomes possible to take into account the various environmental variables and dynamically adjust the water supply depending on the plant species and environmental conditions.The precision agriculture solution by Niolabs uses edge computing to enable each node to decide the amount of water required individually. The system uses an Xbee service that tracks the moisture levels near each watering zone and another service that allows users to provide their inputs about the watering requirements.&nbsp;Both the inputs and several other variables are combined and used as a reference to train the irrigation model, which keeps learning and improving with time—proof enough of the fact that combining general knowledge with ML can do wonders.The one thing that makes Niolabs’s implementation of a precision agriculture system unique is its ability to combine millions of seemingly unrelated data points into insightful and straightforward metrics, which can be accessed as a summary or detailed view at the fingertips of the users. The same system that maintains the plantation can also help with predictive maintenance. By using condition monitoring techniques, it can tell when its components might fail.Niolabs aims to come up with a revamped architecture of their application before March 2022. This is when the harvesting season begins, a perfect time to deploy and test the latest solutions. The plan is to create a repeatable tinyML solution that can be transferred to other farms and be scaled for use in a variety of different environments.<h2 dir="ltr">Ribbit Network: The world’s largest crowdsourced network of open-source, low-cost, Carbon Dioxide Gas Detection Sensors</h2>If we take a look at the long-term trend of worldwide greenhouse gas emissions, it is easy to note that the figures have shown constant growth in the past three decades [4]. There is a massive uptrend that continues to get steeper with each passing year. To make informed decisions and new policies regarding climate change, stakeholders and government organisations need accurate data.It is easy to assume that the data that is used presently is sufficient to make critical decisions, but that is not the case. The reference estimations are mostly theoretical calculations or predictions based on indirect economic measures. For example, vehicular emissions are roughly calculated based on the overall kilometers traveled and average emission per vehicle.&nbsp;The measurements that are direct, such as the ones taken by climate tech satellites, prove to be exorbitantly expensive. A single Orbiting Carbon Observatory 2 (OCO2) satellite that makes space-based observations of atmospheric carbon dioxide cost NASA over $467 mn. [5]Ribbit network aims to fill this gap by combining the economics with actual observations to reduce the uncertainties in data collection. Ribbit network takes over the challenge of building the map of emissions and concentration of carbon in the air with the help of tiny and affordable IoT sensors. They call them the ‘Frog’ sensors. The sensors get their name from frogs as these amphibians are hypersensitive to temperature and are the first ones to get affected due to climate change.<blockquote>Our primary interest in TinyML is based on decreasing power consumption. Running as much computing as we can on our IoT devices lets us minimize the carbon footprint that comes with big data center services. - Keenan Johnson</blockquote>The Frog sensors use a Non-Destructive Infra-Red (NDIR) sensor working on the principle of infrared (IR) spectroscopy. A typical NDIR sensor consists of an IR source, sample chamber and an IR detector. For measuring the presence of CO2, the air is allowed to flow inside the sample chamber. Depending on the concentration of CO2, some IR waves of a particular wavelength emitted by the source get absorbed. The difference between the emitted light and the light that makes it through the chamber gives conclusive evidence of the presence and quantitative estimation of the greenhouse gas in the air.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM5ODA3MTYzMTk1LVVudGl0bGVkIGRlc2lnbi5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="ribbit-frog-carbon-dioxide-sensor-tinyml-block-diagram">Block diagram of the Frog sensor. Source: Ribbit network</div>The sensor is interfaced with a Raspberry Pi CM4, which takes care of everything right from powering the sensor to collecting and transmitting the data.The design of the frog sensors is entirely open-source which means, anyone can build their own version of the device by following the <a href="https://github.com/Ribbit-Network" target="_blank">official tutorial</a> from GitHub.&nbsp;<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM5ODA3MzAzMTE3LVVudGl0bGVkIGRlc2lnbiAoMSkuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="ribbit-frog-carbon-dioxide-sensor-tinyml">A closer look at the Frog sensor. Source: Ribbit network</div>Twenty-five alpha versions of frog sensors have been deployed across five countries. Beta versions will be deployed in a much larger number before the final rollout. The team is currently striving hard to solve problems related to determining sensor health, calibrations, the inclusion of wind data, and a lot more. With custom engineering solutions, they wish to reduce the cost of the sensor and power it with solar cells.<h2 dir="ltr">Conclusion</h2>It is not one, two, or three solutions that are going to solve the problem of climate change, but hundreds and thousands of innovative ideas collectively working together that can bring the revolution. That is what tinyML and its community members stand for.&nbsp;Grassroots innovations like the ones we showcased in this article can go long miles in meeting the global sustainable development goals, making the world a better place to live for future generations. Over the next couple of years, we expect tinyML to be leveraged far beyond than presently is.About tinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, tinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM5ODA3ODQ1NDA0LVVudGl0bGVkIGRlc2lnbi5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="tiny-ml-foundation-banner"></div>The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a> explained some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" rel="noopener noreferrer" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The third and final article shall cover the tinyML applications in pedagogy and education.<hr><h2 dir="ltr" id="isPasted">References</h2>[1] The World Bank Data, Employment in agriculture (% of total employment) (modelled ILO estimate), 2021, [Online], Available from: <a href="https://data.worldbank.org/indicator/SL.AGR.EMPL.ZS" target="_blank">https://data.worldbank.org/indicator/SL.AGR.EMPL.ZS</a>[2] Harrison Tasoff, Gauging Groundwater, 23 April 2021, [Online], Available from: &nbsp;<a href="https://www.news.ucsb.edu/2021/020248/gauging-groundwater" target="_blank">https://www.news.ucsb.edu/2021/020248/gauging-groundwater</a>[3] Niolabs, Nio powers the only autonomous vineyard in the world, [Online], Available from: <a href="https://niolabs.com/case-studies/agriculture" target="_blank">https://niolabs.com/case-studies/agriculture</a>[4] United States Environment Protection Agency, Global Greenhouse Gas Emissions Data, [Online], Available from: <a href="https://www.epa.gov/ghgemissions/global-greenhouse-gas-emissions-data" target="_blank">https://www.epa.gov/ghgemissions/global-greenhouse-gas-emissions-data</a>[5] NASA, Orbiting Carbon Observatory-2 Launch Press Kit, July 2014, [Online], Available from: <a href="https://www.jpl.nasa.gov/news/press_kits/oco2-launch-press-kit.pdf" target="_blank">https://www.jpl.nasa.gov/news/press_kits/oco2-launch-press-kit.pdf</a>

In this article, we take a look at two tinyML projects that have the potential to make contributions towards sustainable development goals. While the first project is about revolutionising precision farming, the second one aims to create a network of low-cost sensors for mapping carbon emissions.

computer vision

Technical Guides

TinyML unlocks new possibilities for sustainable development technologies

Applications of TinyML making specialised education more accessible

Applications of TinyML in healthcare for a safer future

Edge AI: A Comprehensive Guide to its Engineering Principles and Applicatio...

Unsupervised vs Supervised Learning: A Comprehensive Comparison

What is Unsupervised Learning: A Comprehensive Guide

Profiles