NVIDIA Jetson AGX Orin Module and Developer Kit

Nvidia Jetson and Edge Impulse facilitate Edge AI development, offering computational power and model training platforms for real-world applications.

Building High-Performance Edge AI Solutions with Nvidia Jetson and Edge Impulse

Audio event detection functions can listen for distinct sounds or patterns embedded within an audio stream. It uses advanced signal processing and machine learning algorithms to accurately identify and classify sound events in real-time, with no internet connectivity required.

Implementing Audio Event Detection on the Edge with Particle Photon 2

<h1 dir="ltr" id="isPasted">Introduction</h1>In the intricate tapestry of modern technology, where devices are becoming increasingly interconnected and intelligent, the ability to accurately interpret the world around us is paramount. This is where the concept of 'sensor fusion' shines brightly, helping to make sense of vast quantities of data. Sensor fusion, in its essence, is the art and science of amalgamating data from multiple sensors to provide a more holistic, accurate, and reliable view of the environment. It's akin to piecing together a jigsaw puzzle, where each sensor provides a fragment of the bigger picture, and when fused together, these fragments reveal a comprehensive image.As we navigate through this article, we will delve deep into the technical intricacies and methodologies underpinning sensor fusion. We'll explore its applications, from autonomous vehicles to healthcare, and highlight the challenges that developers and engineers face in this domain. Furthermore, we'll introduce the transformative potential of integrating sensor fusion with cutting-edge tools like the SensiEDGE CommonSense addon for Sony Spresense and the paradigm of Edge AI. By the end, readers will gain a comprehensive understanding of the current landscape of sensor fusion and the promising capability it brings.<iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> <h1 dir="ltr">Understanding Sensor Fusion</h1>Sensor fusion is a multidisciplinary field that integrates data from multiple sensors to derive a more comprehensive and accurate representation of the environment than any individual sensor can provide. Central to this is the principle of Redundancy and Diversity.&nbsp;Redundant sensors, which are multiple units measuring the same parameter, ensure that the system remains resilient against individual sensor failures or inaccuracies. On the other hand, diverse sensors, each capturing different facets of the environment, ensure a comprehensive view. This dual approach not only bolsters the reliability of measurements but also provides a panoramic perspective of the surroundings.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI0NzkzMTkxLWltYWdlNi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="sensor-fusion-aircrafts">Fig. 1: The idea of combining data from multiple sensors to improve decision-making, accuracy, and reliability is not new. Historically, sectors like defense and aerospace were among the pioneers in harnessing the power of sensor fusion for tasks such as navigation, target tracking, and situational awareness.<h2 dir="ltr" id="isPasted">Methodologies Driving Fusion</h2>Two methodologies particularly stand out in the realm of sensor fusion are Kalman Filtering and Particle Filtering. They are state estimation techniques that are used to combine noisy measurements from multiple sensors to estimate the true state of a system. The Kalman filter, a recursive algorithm, is tailored for linear dynamic systems. It shines in environments with Gaussian distributed noise, predicting future states based on past measurements. Conversely, Particle filters are the go-to for nonlinear systems. They operate by representing a system's posterior distribution using a set of random samples, offering a more flexible approach to state estimation.<h2 dir="ltr">Challenges in Real-world Implementation</h2>While the promise of sensor fusion is undeniable, it's not without its challenges. Temporal Alignment is a significant hurdle. With sensors operating at different sampling rates, ensuring that data is synchronized is crucial for accurate fusion. Another challenge lies in Spatial Calibration. Given that sensors might have varied orientations or fields of view, aligning them to a common spatial frame is essential.&nbsp;Furthermore, as the number of sensors in a system grows, scalability becomes a concern, demanding efficient algorithms that can handle increased data loads without compromising on processing speed.<h2 dir="ltr">Evaluating Fusion Systems</h2>Lastly, the efficacy of a sensor fusion system is often determined by a set of quality metrics. Accuracy, which measures how close the fused data is to the actual state of the environment, is paramount. However, robustness, the system's resilience against anomalies, and computational efficiency, indicating the system's resource utilization, are equally vital.Further reading:&nbsp;<a href="https://www.wevolver.com/article/what-is-sensor-fusion-everything-you-need-to-know" target="_blank">Sensor Fusion: The Ultimate Guide to Combining Data for Enhanced Perception and Decision-Making</a><h1 dir="ltr">Applications of Sensor Fusion</h1>Sensor Fusion facilitates a variety of applications across various domains by amalgamating data from diverse sensors to derive precise and actionable insights. Here are some of the applications where sensor fusion shines:<ul><li dir="ltr">Autonomous Vehicles:&nbsp;One of the most well-known applications of sensor fusion is in self-driving cars. These vehicles use a combination of cameras, LiDAR, radar, ultrasonic sensors, and more to perceive their surroundings, detect obstacles, and navigate safely. By fusing data from these diverse sensors, autonomous vehicles can operate safely in a wide range of conditions, from bright sunlight to nighttime and from clear weather to fog.</li><li dir="ltr">Robotics:&nbsp;Robots, especially those designed for complex tasks in variable environments, use sensor fusion to combine data from cameras, infrared sensors, tactile sensors, and gyroscopes. This allows them to navigate, interact with objects, and perform tasks with higher precision and reliability.</li></ul><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI0ODM3OTc4LWltYWdlNy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="sensor-fusion-wearable">Fig. 2: Sensor fusion is also used in wearable technology, enhancing the capabilities of devices to provide comprehensive insights and improved user experiences.<ul><li dir="ltr">Augmented Reality (AR) and Virtual Reality (VR):&nbsp;AR and VR systems often rely on sensor fusion to track user movements and adjust the displayed visuals accordingly. By combining data from accelerometers, gyroscopes, and cameras, these systems can provide a more immersive and responsive user experience.</li><li dir="ltr">Military and Defense:&nbsp;Advanced defense systems use sensor fusion to combine data from radars, sonars, infrared cameras, and other sensors. This provides a comprehensive situational awareness, enabling quicker and more informed decision-making in complex scenarios.</li><li dir="ltr">Healthcare:&nbsp;Wearable devices that monitor health metrics often use sensor fusion to provide more accurate readings. For instance, a smartwatch might combine data from a heart rate sensor, accelerometer, and gyroscope to better understand a user's activity level and overall health.</li></ul>Other notable applications include environmental monitoring, where sensor fusion aids in forest fire detection and prediction; smart cities, which utilize the technique for traffic management and public safety; agriculture, where precision farming is enhanced through fused data; industrial automation for monitoring equipment health and product quality; and navigation and mapping, where devices like smartphones and drones benefit from improved location accuracy.<h1 dir="ltr">Sensor Fusion with CommonSense addon for Sony Spresense Microcontroller and Edge Impulse</h1>In the rapidly evolving world of IoT and edge computing, the synergy between robust hardware and intelligent software becomes crucial. The <a href="https://www.sensiedge.com/commonsense" target="_blank">SensiEDGE CommonSense</a> add-on, tailored for the <a href="https://developer.sony.com/spresense/" target="_blank">Sony Spresense</a> board, epitomizes this synergy, setting new standards in sensor fusion.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI0ODY5OTExLWltYWdlNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="commonsense-addon-module-for-spreesense">Fig. 3: CommonSense module mounted over a Sony Spresense board. Source:&nbsp;<a href="https://edgeimpulse.com/blog/access-augmented-sensor-capabilities-for-sony-spresense-with-our-new-commonsense-integration" target="_blank" rel="noopener noreferrer">Introducing CommonSense Integration - Edge Impulse</a>The CommonSense add-on is a testament to the advancements in sensor technology within the IoT landscape. Despite its compact and unassuming design, it boasts a plethora of capabilities. Central to its operation is an array of 10 unique sensors that allow the add-on to monitor, measure, and perceive the environment when attached to a Sony Spresense development board.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI0OTI4MzkxLWltYWdlMy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="sensiedge-features">Fig. 4: Infographic showcasing the features of SensiEDGE CommonSense boardIt's adept at capturing a myriad of environmental parameters, from air quality and temperature to humidity. Additionally, its motion tracking capabilities, encompassing an accelerometer, gyroscope, and magnetometer, allow for a broad spectrum of data collection. Augmented by features like audio detection through its microphone and versatile connectivity options via Bluetooth and Wi-Fi, it emerges as an invaluable asset for developers aspiring to craft comprehensive IoT solutions.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI0OTUzNDQ4LWltYWdlNS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="sensiedge-commonsense-applications">Fig. 5: &nbsp;CommonSense add-on by SensiEDGE is a compact yet powerful module designed for diverse applicationsThe power of the CommonSense add-on is further magnified when its data is fused, offering a holistic view of the monitored environment or subject. Such consolidated data, when interpreted through machine learning models, paves the way for precise predictions and actionable insights.&nbsp;This is where <a href="https://edgeimpulse.com/" target="_blank">Edge Impulse</a>, a platform renowned for its expertise in embedded machine learning, comes into play. The amalgamation of the CommonSense add-on and Sony's Spresense board with Edge Impulse's software expertise results in a blend of sensor fusion and edge AI. This empowers developers to design, refine, and implement machine learning models right on the device, leveraging AI with no reliance on cloud connectivity.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI0OTg4Mzg3LWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 302px;" class="fr-fic fr-dib" alt="edge-impulse-stufio-platform">Fig. 6: Edge Impulse is a developer-first integrated platform for designing, training, testing and deploying edge AI applications.For a deeper dive into this integration and its potential, consider exploring this webinar that accentuates the potential of sensor fusion on the Sony Spresense:<h2><iframe width="640" height="360" src="https://www.youtube.com/embed/5HyGcLK0bQM?&amp;ab_channel=EdgeImpulse&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>Conclusion</h2>The integration of SensiEDGE CommonSense add-on with Sony Spresense and Edge Impulse makes for a powerful combination that enhances the reliability, accuracy, and efficiency of data interpretation in various applications, all on the edge.&nbsp;Leveraging advanced hardware and intelligent software promises to drive innovation, offering a more holistic and nuanced understanding of our environment. In essence, this collaboration is set to redefine the boundaries of sensor fusion, paving the way for groundbreaking advancements in the interconnected world of modern technology.<h1 dir="ltr">About the sponsor: Edge Impulse</h1><a href="https://www.edgeimpulse.com/" target="_blank">Edge Impulse</a> is the leading development platform for embedded machine learning, used by over 1,000 enterprises across 200,000 ML projects worldwide. We are on a mission to enable the ultimate development experience for machine learning on embedded devices for sensors, audio, and computer vision, at scale.&nbsp;From getting started in under five minutes to MLOps in production, we enable highly optimized ML deployable to a wide range of hardware from MCUs to CPUs, to custom AI accelerators. With Edge Impulse, developers, engineers, and domain experts solve real problems using machine learning in embedded solutions, speeding up development time from years to weeks. We specialize in industrial and professional applications including predictive maintenance, anomaly detection, human health, wearables, and more.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjk3NzI1MTI5ODY4LWltYWdlMS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="edge-impulse"><h2 dir="ltr"><iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> </h2><h2 dir="ltr">References</h2>[1] Mitchell, H. (2007). Multi-Sensor Data Fusion. Springer, Berlin, Heidelberg. <a href="https://doi.org/10.1007/978-3-540-71559-7_1" target="_blank">https://doi.org/10.1007/978-3-540-71559-7_1</a>[2] Chui, C.K., Chen, G. (2017). Kalman Filtering. Springer, Cham.&nbsp;<a href="https://doi.org/10.1007/978-3-319-47612-4_1" target="_blank">https://doi.org/10.1007/978-3-319-47612-4_1</a>[3] ElProCus. What is Sensor Fusion: Working &amp; Its Applications. ElProCus. [Online]. Available:&nbsp;<a href="https://www.elprocus.com/sensor-fusion/" target="_blank">https://www.elprocus.com/sensor-fusion/</a>[4] Sony, Spresense 6-core microcontroller board with ultra-low power consumption [Internet]. Available from:&nbsp;<a href="https://developer.sony.com/spresense/" target="_blank">https://developer.sony.com/spresense/</a>[5] SensiEDGE, CommonSense: Sensor AI Board [Internet]. Available from:&nbsp;<a href="https://www.sensiedge.com/commonsense" target="_blank">https://www.sensiedge.com/commonsense</a>

Sensor fusion enables the seamless integration of data from multiple sensors, paving the way for advanced Edge AI implementations that optimize real-time processing, enhance decision-making, and boost system responsiveness in dynamic environments.

Redefining Sensor Fusion with CommonSense addon for Sony Spresense and Edge Impulse

Article #5 of Spotlight on Innovations in Edge Computing and Machine Learning: A computer vision project that monitors vehicle traffic in real-time using video inferencing performed on the Brainchip Akida Development Kit.

Real-Time Traffic Monitoring with Neuromorphic Computing

Article #2 of Spotlight on Innovations in Edge Computing and Machine Learning: Edge AI techniques such as Keyword Spotting can turn an ordinary device into a smart appliance.

Implementing Smart Voice Control in Appliances: A Comprehensive Guide

The recent <a href="https://www.tinyml.org/event/summit-2023/" target="_blank">TinyML Summit</a> in March 2023 clearly exposed the advancements in TinyML, as well as the challenges of actually deploying the technology. The market is clearly in an early stage with thousands of projects being developed while very few have made it to production as yet. This article addresses the challenges of deploying TinyML solutions and discusses how the LwM2M standard may solve some of these issues.<h3 dir="ltr">Introduction to TinyML</h3>While MCUs are becoming more powerful, machine learning models can be designed to utilize less resources. This enables the implementation of TinyML; deep learning models which can run on resource-constrained IoT devices. TinyML can be used to analyze raw sensor data locally which reduces or removes the need to send data to the cloud, lowers battery consumption and preserves data privacy.Nevertheless, implementing intelligent IoT solutions does not solely rely on the ML model itself. It involves challenges such as implementing continual learning, enabling low-power wireless communication, managing devices remotely, ensuring secure communication, updating firmware (over the air) and enabling device interoperability.&nbsp;<h3 dir="ltr">TinyML &amp; LwM2M complementing each other</h3>LwM2M is an application-layer communication standard which simplifies messaging and device management for IoT devices. The protocol dictates a data format and defines device management mechanisms and standardized processes for firmware-over-the-air updates (FOTA). The protocol is well-suited for LPWAN standards such as NB-IoT and LTE-M.While TinyML provides the device intelligence (using tooling such as Edge Impulse or Cartesiam.ai), the LwM2M protocol provides connectivity, standardized communication and device management. When combined, they create a holistic solution for smart IoT devices.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjgxMjE4NjAxMTE2LUVsZW1lbnRzIG9mIGFuIEludGVsbGluZ2VudCBEZXZpY2UuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 766px;" class="fr-fic fr-dib">TinyML is only one part of a smart solution<h3 dir="ltr" id="isPasted">Retrofitting existing devices with smart sensors</h3>Cloud-based analysis of a device&rsquo;s raw sensor data is inefficient due to the volume of data the device needs to transmit. A more efficient way is to process data from sensors directly on the device using TinyML. For example, analyzing the X, Y and Z values of the accelerometer can detect complex movements or vibrations which could give valuable insights, enabling use cases such as predictive maintenance, monitoring the utilization of valuable goods or classifying movements of people or animals.These days, more and more smart sensors are being developed. In addition to their sensing capabilities, smart sensors come with an embedded MCU which runs the TinyML model and communicates only the detected patterns to the main MCU of the device. These TinyML integrated sensors are referred to as the Sensor 2.0 Paradigm by Prof. Vijay Janapa Reddi (Harvard University) during his <a href="https://www.youtube.com/watch?v=pbFYiR2sTlY&list=PLeisuBi-nfBMYMZ1keA04xUA8sVb7uxR2&index=10&t=447s" target="_blank">recent lecture</a> at the TinyML Summit. Smart sensors simplify the implementation of TinyML as it allows for retrofitting existing devices with TinyML capabilities without having to redesign the embedded firmware. In addition, it can solve privacy issues due to the true isolation of raw sensor data. Smart sensors can analyze voices or camera images while ensuring people&rsquo;s privacy as data does not leak out to the main MCU of the device.<h3 dir="ltr" id="isPasted">Continual Learning</h3>One of today&rsquo;s key challenges is to keep the TinyML model reliable post-deployment. Oftentimes, datasets used for training purposes differ from real-world data, leading to inaccurate models. In addition, the environmental context may change over time (e.g. due to decalibration of industrial machines, or changing climate conditions) leading to deterioration of the model quality.Continual learning refers to the ability of TinyML models to adapt over time. This can be accomplished by learning from new data sets without the need to retrain the model from scratch. Although the techniques to realize continual learning are well-known, the practical implementations are often missing when running the models on resource-constrained devices. Often, this has to do with a missing device-management layer which takes care of ML model lifecycle management.LwM2M may be the missing link to realize continual learning as it comes with native support for remote configurations and firmware updates. Using proven methods, ML models can be updated continuously without having to physically access each device to update its firmware.<h3 dir="ltr">Conclusions</h3>TinyML solutions for resource-constrained IoT devices are a catalyst for the development of intelligent solutions. To move beyond the current POC phase and start deploying actual TinyML applications, it is necessary to provide mechanisms for efficient communication, device and firmware management and secure connectivity. The use of the LwM2M protocol to manage TinyML models and provide connectivity is a path towards standardizing TinyML and enriching the ecosystem of intelligent IoT solutions.<hr><h2 dir="ltr">Case Study - TinyML Pattern Detector</h2>A TinyML solution has been designed using the LwM2M standard. The Thingy:91, a development device which uses the nRF9160 SiP from Nordic Semiconductor. It runs the Zephyr OS operating system and was trained to detect three different motion patterns, as shown in the visual below. <img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjgxMjE4NzIzMDY1LVBhdHRlcm5zLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 766px;" class="fr-fic fr-dib">Motion patterns recognized by the IoT deviceThe steps to implement the presented concept are as follows:<ol><li dir="ltr">Data collection and labeling</li><li dir="ltr">Design and train the classifier</li><li dir="ltr">Deploy and test the classifier on the device</li><li dir="ltr">Provide network connectivity and device management by implementing a LwM2M client</li><li dir="ltr">Manage devices and data using a LwM2M server</li></ol>The example uses the <a href="https://www.edgeimpulse.com/" target="_blank">Edge Impulse</a> platform to collect the data from the Thingy:91 accelerometer, train the ML classifier and generate standalone libraries for C++. The library containing the TinyML model can run on the device and control the LED signaling pattern detection. The <a href="https://github.com/AVSystem/Anjay" target="_blank">Anjay LwM2M client</a> provides a connectivity layer for the solution, handling low-level technology issues related to communication and secure data transmission.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjgxMjE5MDYxMzIzLUVkZ2UgSW1wdWxzZSB0cmFpbmluZy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 766px;" class="fr-fic fr-dib">Training the TinyML model by classifying the movements using Edge ImpulseFinally, the device is registered in the LwM2M server where the data is stored and visualized. Using the <a href="https://www.avsystem.com/products/coiote-iot-device-management-platform/" id="isPasted" style="background-color: transparent;" target="_blank">Coiote IoT Device Management</a> portal from <a href="https://www.avsystem.com/" style="background-color: transparent;" target="_blank">AVSystem</a>, the device can be instructed to notify the server every time a specific pattern is detected, or to reduce the frequency to once every several seconds, minutes or hours and send a counter indicating the number of times a pattern has been detected.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjgxMjE4OTI2NzQ0LW5ldyBvYmplY3QgNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 766px;" class="fr-fic fr-dib">Visualization of the movement patterns in Coiote. Object ID /33650 represents the Pattern Detector <h3 dir="ltr" id="isPasted">To try out this Pattern Detector yourself, visit the <a href="https://iotdevzone.avsystem.com/docs/Demo_Projects/Edge_Impulse_demo/" target="_blank">AVSystem DevZone</a>.</h3>

LwM2M may be the missing link to implement continual learning in TinyML projects.

TinyML - Continual Learning with LwM2M

As a machine learning technique, Tiny Machine Learning (TinyML) combines reduced and optimized machine learning applications that require hardware, system, and software components. These operate at the edge of the cloud in real-time while converting data with minimal energy and cost.The size requirements and computation capabilities of TinyML make it ideally suited for environmental sensors, but use cases are wide and varied. At the edge, where the sensing takes place, or in the cloud, where more powerful computations happen, sensed data is processed only after transmission. To handle the massive data volumes produced by this process, edge and cloud processing needs to be intelligently partitioned. This is where TinyML shines.<h3>What is TinyML?</h3>Machine learning applications usually use large power-hungry processors without considering memory requirements or transmission rates. Rather than focusing on efficiency, the goal is to process data as quickly as possible to decrease latency. However, decreasing latency can result in higher power consumption and bandwidth demands.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1678395343950-1678395343949.png" alt="Tiny ml market growth" class="fr-fic fr-dii">Machine learning is also usually customized for unique targeted application requirements. As a result, sharing algorithms between platforms can become challenging since their architectures are likely to differ. TinyML, however, is specifically designed to perform a wide range of tasks.The field of TinyML is a broad, fast-growing field of machine learning technologies and applications that include hardware, algorithms, and software. These developments converge to allow the capability of performing on-device sensor data analytics that consumes very low power. Often, TinyML can integrate into PCBA design using hardware that is already familiar.Through TinyML, you can develop applications that are always on, especially for battery-operated sensors. With embedded artificial learning available on hardware that has a relatively small form factor, it ushers in a new era of development.<h3>Advantages of TinyML</h3>Several architectural improvements result from moving the initial processing closer to the source of the data. Each of these contributes to the value of an IoT solution, and one cannot exist without the other:<ul><li>Latency&ndash;Closer computations minimize feedback latency to the smallest extent possible.</li><li>Connection independent&ndash;By eliminating the need for a fixed connector, a wide range of new options are offered.</li><li>Energy savings&ndash;TinyML offers targeted solutions with the lowest power consumption possible. Microcontrollers with minimal capabilities offer the horsepower necessary for computations while consuming very little power.</li><li>Data security&ndash;By keeping data and processing local, there&rsquo;s a higher barrier to spatial hacking at the sensor level.</li><li>Size&ndash;Many of the hardware implementations available are the size of a small stick of gum, measuring 45x18mm.</li></ul><h3>The Role of TensorFlow Lite</h3>Development tools for TinyML include not only the hardware but software platforms that allow algorithms from common building blocks. TensorFlow Lite is an example of a tool for developing code to be used in the TinyML environment.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1678395344271-1678395344271.png" alt="Role tensorflow lite" class="fr-fic fr-dii">TensorFlow Lite is an embedded machine-learning framework that has a subcategory specifically designed for microcontrollers. The library allows developers to deploy models on mobile edge devices that use microcontrollers. Existing models from the library can be used and retrained with new edge data.In embedded systems, TensorFlow Lite is an excellent option for systems that run on batteries and must adhere to strict energy budgets. Among the things this library offers are pre-trained machine-learning models for<ul><li>Smart responses&ndash;the system generates intelligent responses similar to those generated by an A.I. chatbot</li><li>Object detection&ndash;identifies multiple elements in an image, accommodating 80 different objects.</li><li>Recommendations&ndash;the recommendation system provides personalized recommendations based on the user&rsquo;s behavior.</li></ul>Alternatively, developers can use new models. The chosen TensorFlow model can compress into a flat buffer with the TensorFlow Lite conversion process loaded into the edge device. The processor efficiency can then get optimized by selecting the most appropriate floating-point number or integer math for the required data.<h3>Growth expectations for TinyML</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1678395344189-1678395344189.png" alt="Growth expectation tiny ml" class="fr-fic fr-dii">With the wide acceptance of the TinyML development platform, the expectations for growth are phenomenal. ABI Research forecasts TinyML unit volumes will explode from 15 million units in 2020 to 2.5 billion units in 2030. This is an incredible compound annual growth rate of 167% for 10 years.However, even if those projections are accurate, TinyML will still represent less than 10% of all IoT devices despite the rapid growth rate. Nevertheless, TinyML&rsquo;s widespread use will likely lead to its use in many future projects. To improve your design skills now,&nbsp;<a href="https://www.tinyml.org/" target="_blank">discover how TinyML can be integrated with familiar devices</a>.<h3>Conclusion</h3><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1678395344051-1678395344051.png" alt="Tiny ml market growth" class="fr-fic fr-dii">TinyML market growth will be inevitable. Future projects will likely be impacted by TinyML IoT and other peripheral markets in the future. Start your TinyML development with manufacturing in mind to maximize the success of your project.The right TinyML manufacturing solution starts with your CM partner helping you to understand the requirements and solutions. Developing a plan for manufacturing at the beginning of the development process can mitigate release-to-production concerns.<h4>You have a project plan. We have ways to achieve it.</h4><a href="https://macrofab.com/demo" target="_blank">SEE HOW IT WORKS</a>

ABI Research forecasts that TinyML unit volumes will explode from 15 million units in 2020 to 2.5 billion units in 2030.  Find out about this machine-learning technique now. 

TinyML: What is It and Why Does It Matter?

This article was discussed in our <a data-saferedirecturl="https://www.google.com/url?q=https://www.wevolver.com/profile/the.next.byte&source=gmail&ust=1647995729616000&usg=AOvVaw0aQXASXSRqe8X0H1Q367Vs" href="https://www.wevolver.com/profile/the.next.byte" id="isPasted" rel="noopener noreferrer" target="_blank">Next Byte podcast</a>.&nbsp;<iframe height="200px" width="100%" frameborder="no" scrolling="no" seamless="" src="https://player.simplecast.com/85e3afbd-7cfe-4610-9f0b-a640df7c26c2?dark=false" class="fr-draggable" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true"></iframe>The full article will continue below.<hr>Energy efficiency and machine learning don&rsquo;t always go hand-in-hand. While low-power sensor networks can make great use of deep-learning systems for processing data, they typically do so by transmitting said data to remote servers equipped with powerful processors and plenty of memory.On-device machine learning at the microprocessor scale &mdash; known as &ldquo;tinyML&rdquo; &mdash; looks to shift the work from data centers into the field, and a team of researchers at Imperial College London and the Massachusetts Institute of Technology (MIT) are hoping to go still further: The development and deployment of a machine learning system so efficient it can perform inference on energy-harvesting sensors deployed under the sea, operating entirely without batteries.<h1>Fundamental challenges</h1>The team, including first author Yuchen Zhao, aimed to deliver an answer to a simple question: Whether or not it is possible to design and build battery-free devices capable of machine learning and on-device inference, operable in underwater environments.Building on earlier works on battery-free sensing and energy-efficient machine learning systems, the team built a prototype: A device, built around an off-the-shelf microcontroller, which could harvest energy, capture sounds, process them through a lightweight deep neural network to identify particular marine species, and transmit the results back to a receiver unit &mdash; and all without needing a battery.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ3MzgyMTIxMzUwLXVuZGVyd2F0ZXJtbDIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="An overview diagram of an undersea sensor system which harvests sound energy and uses backscatter communication to transmit the results of on-board inference based on animal sounds captured by a hydrophone.">Despite having mere microwatts of power to play with, an energy-harvesting sensor system has been proven capable of on-device inference &mdash; even underwater.The idea of an ultra-low-power sensor, underwater or otherwise, which is powered by harvested energy and communicates via energy-efficient backscatter isn&rsquo;t new, but the team points out the majority of previous work has been focused on land rather than sea operation, and that most simply power their sensors but rely on battery- or mains-powered devices with more computational capabilities to process captured data.Performing computation on-device, meanwhile, would not only increase the flexibility of the deployed sensor systems but also work around significant issues with current approaches &mdash; including the low-bandwidth nature of underwater communications introducing large delays to the process, and the need to store captured data in-memory until a receiver is available.&ldquo;To overcome these challenges,&rdquo; the team writes, &ldquo;our node performs on-board inference. This results in a lower-power, time-and-resource efficient, and more scalable system design.&rdquo;<h1>Three-part hardware harmony</h1>The team&rsquo;s prototype boasts very modest specifications. At its heart is a Texas Instruments MSP430FR5994 microcontroller development board, connected to an omnidirectional hydrophone for capturing undersea sounds. The same sound energy as monitored by the sensor node also serves as its power source, being converted by a transducer into electrical energy.Despite the harvested energy being minimal at best, and the device lacking an integrated battery for energy accumulation over time, the prototype delivers on the team&rsquo;s goal: As energy is harvested, the microcontroller powers up, samples audio from the sensor, performs inference on-device, and then transmits the resulting classification via backscatter uplink.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ3MzgyMjg1ODY4LXVuZGVyd2F0ZXJtbDQuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="A picture of a Texas Instruments MSP430FR4994 microcontroller development board, next to a white and blue device described by its creators as an energy harvesting backscatter system for underwater use.">The team&#39;s prototype is based on a Texas Instruments MSP430 mcirocontroller, chosen for its low power demands, and a potted backscatter system designed two years earlier.Impressively, everything is handled in just 5.4mJ of power &mdash; with the inference stage taking three seconds to complete at a draw of 1.3mW. While by far the highest draw of the process, the researchers claim it represents a considerable saving over the typical approach of simply transmitting the raw data for later analysis &mdash; estimating that backscatter transmission of audio data would consume over 30 per cent more power than performing the inference on-device and simply transmitting the result, while cutting the time taken considerably.&ldquo;This result,&rdquo; the team notes, &ldquo;demonstrates that battery-free edge inference is feasible, and that it makes on-board inference both more efficient and faster, even with this preliminary inference model. We envision that the gains can be significantly improved as the research evolves.&rdquo;<h1>Training day</h1>The secret to the project&rsquo;s success is a highly-efficient machine learning model, trained off-device on a more powerful system using the Watkins Marine Mammal Sounds Database. The audio data was resampled to match the sample rate of the microcontroller&rsquo;s analog to digital converter (ADC) before being used to train high-level Keras models via TensorFlow.To generate smaller microcontroller-compatible models from these high-level Keras models, the team turned to Keras2C. &ldquo;This library generates smaller DNN models than TensorFlow Lite does,&rdquo; the researchers explain, &ldquo;and requires fewer third-party libraries when generating models.&rdquo;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ3MzgyMjk4Mjk0LXVuZGVyd2F0ZXJtbDMuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="A diagram showing the training and operation flow of the proposed underwater audio classification system, showing a centralised training system which occurs offline to produce a high-level model which is then processed for use on-device.">The team trained the model off-device, then processed it for use in the resource-constrained environment of the energy-harvesting microcontroller.In simulated testing, based on samples held-back from the Watkins dataset, performance of the resulting microcontroller-deployed on-device inference model showed promise: The team reports an 84 per cent success rate for offline single-mammal classification, though switching to eight-mammal classification dropped accuracy down to 76 per cent while increasing the model size beyond that which could be deployed on the prototype.Online accuracy, using data captured by the prototype&rsquo;s analog to digital converter based on recordings of four marine mammals, proved lower: 63 per cent, a drop the team attributes to the relatively low number of testing samples and a loss of data resolution during the sampling process.&ldquo;Nonetheless,&rdquo; the team notes, &ldquo;this result demonstrates that our prototype was successful in on-board classification of four marine animals with a reasonable accuracy at extremely low power, and this accuracy may be improved as the research evolves and with more comprehensive online evaluation.&rdquo;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ3MzgyMzA5NDcxLXVuZGVyd2F0ZXJtbDUuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="A boxplot and kernel density chart, showing the offline test accuracy of various machine learning models: The highest accuracy is seen with models of N-32 size.">The highest accuracy came from models with an input layer size of 32 (N-32), but this proved too large for deployment on the microcontroller even after processing.The researchers suggest a range of directions for future study, including on-device model personalization to boost accuracy, a model-optimized hardware/software co-design approach to boost performance and efficiency, and the investigation of methods to allow training, rather than just inference, to occur on-device.The team&rsquo;s work has been made available under open-access terms <a href="https://arxiv.org/abs/2202.08174" target="_blank">on Cornell&rsquo;s arXiv preprint server</a>. The researchers also made their underwater backscatter system available under the permissive Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license upon its development two years ago, with code and design files <a href="https://github.com/saadafzal24/Underwater-Backscatter" target="_blank">available on GitHub</a>.<h1>Reference</h1>Yuchen Zhao, Sayed Saad Afzal, Waleed Akbar, Osvy Rodriguez, Fan Mo, David Boyle, Fadel Adib, and Hamed Haddadi: <cite>Towards Battery-Free Machine Learning and Inference in Underwater Environments</cite>. DOI <a href="https://doi.org/10.48550/arXiv.2202.08174" target="_blank">arXiv:2202.08174 [cs.LG]</a>.

While most machine learning takes place in high-power data centers, these researchers are trying to deploy the technology beneath the waves.

Designed to bring machine learning to the Earth's last great frontier, a prototype smart sensor platform users on-device inference to classify marine mammal calls — powering itself exclusively from the sounds it captures.

Ultra-efficient machine learning offers on-device inference powered solely by sound - under the sea

The modern airwaves are thicker than ever: From smartphones and smartwatches to smart meters, smart homes, smart cars, and smart cities, there are more devices trying to communicate than ever before — and all trying to share an ever-crowded spectrum.One possible solution is to make the communications systems smarter, monitoring the airwaves and allowing better management of scarce spectrum and improving how devices can coexist on a growing radio access network — but these integrated sensing and communication (ISAC) systems can be vulnerable to attack.It’s this problem that a team from the Universities of Zurich and Murcia and the Swiss Department of Defense, Civil Protection, and Sports (DDPS) is aiming to solve with a machine-learning framework dubbed CyberSpec, designed to allow resource-constrained crowd-sensing spectrum sensors to detect — and hopefully thwart — a range of attacks.<h1>Real-world validation</h1>Where a number of academic projects are born in the lab, CyberSpec stands out: The team developed and deployed the system to a commercial crowd-sensing radio-frequency monitoring platform, ElectroSense. Designed specifically to address the growing burden placed on wireless communication networks, ElectroSense monitors the current occupancy of a given portion of spectrum and uses cognitive radio network (CRN) technology to shuffle devices off to currently under-utilized bands.In the wild, rather than in the lab, ElectroSense and similar devices need to be robustly protected against attack — which is where CyberSpec comes in.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644432799907-cyberspec5.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A Raspberry Pi 4 Model B 8GB single-board computer, viewed from above with the Ethernet and USB ports to the right.">ElectroSense is designed for deployment on Raspberry Pi single-board computers, so a lightweight solutions was required to detect attacks. “CyberSpec [is] an ML/DL [Machine Learning/Deep Learning]-oriented framework using device behavioral fingerprinting to detect anomalies produced by SSDF [Spectrum Sensing Data Falsification] attacks affecting resource-constrained spectrum sensors,” the team writes of the concept.The key: “Resource-constrained.” The ElectroSense platform is designed to run on commercial off-the-shelf Raspberry Pi single-board computers, which lack the heavy computational power often associated with deep learning approaches to behavioral fingerprinting. CyberSpec, the team claims, can work within these constraints — using machine learning techniques without harming the performance of the ElectroSense system itself, despite the relatively modest specifications of the hardware they share.The CyberSpec framework, as tested, has two aspects. The first is its behavioral fingerprint capabilities, wherein it monitors the internal behavior of the sensor on which it is deployed and creates fingerprints to which future behavior can be compared.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644432656734-cyberspec4.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A table showing the device metrics available for monitoring on an ElectroSense node; the metrics chosen are highlighted in blue.">The team ran through a range of metrics for fingerprinting; selected metrics are highlighted in blue. The fingerprints are generated based on monitoring of a range of internal metrics: Network traffic, virtual memory allocations, file system operations, the system scheduler, system calls, CPU utilization, device driver loading and unloading, and random number generation systems — all things which can be monitored in real-time without straining the host system.The second aspect is the detection of cyber-attacks, in which the fingerprints are used as training data for machine and deep learning algorithms in order to produce a system capable of detecting anomalies.<h1>Defining and detecting the problem</h1>First, though, there was the problem of definition. As the researchers point out in their paper, there is currently no universally-accepted formal definition of the behavior and impact of SSDF attacks — a problem when it comes to detecting the fingerprints of said behavior.To overcome this, the team defined seven novel SSDF attacks: Repeat, Mimic, Confusion, Noise, Spoof, Freeze, and Delay. The seven cover three main attack types: The simulation of non-existent communications; hiding illegal transmissions or otherwise unwanted data; and performing both actions simultaneously.<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644432585981-cyberspec3.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="A flowchart showing one of the attack types defined in the paper: The Mimic attack.">The team defined seven attack types across three families; the flow of the Mimic attack is pictured. With the attacks defined and ready for deployment, the team was able to test out the CyberSpec system on its ElectroSense crowd-sensing spectrum sensor network — and the results it delivered impress.With the fingerprinting module running directly on each ElectroSense node, and the attack detection module running on a trusted central server, the CyberSpec framework was able to generate a baseline fingerprint of normal operations based on eight days’ operation then successfully and quickly detect the Noise, Spoof, Confusion, and Mimic attacks in all tested configurations.Repeat and Delay, however, proved more challenging: “They are not adequately detected,” the researchers note, “until the affected bandwidth reaches 80 and 160MHz,” and even then only when using using one specific model — Autoencoder — of the five tested. Interestingly, the detection of these attacks is indirect: “When the affected bandwidth is sufficient,” the team writes, “the attack impacts the rest of the monitored events, and these variations are slightly detected.”Freeze, in turn, was found to evade detection — but the researchers believe they know why, and how to resolve it: “CyberSpec does not monitor read file operations,” they write, “and the behavior of these attacks relies on that.”<img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1644432621342-cyberspec2.jpg" style="width: 300px;" class="fr-fic fr-dib" alt="Charts showing the performance of each tested machine learning model against all seven attack types.">The CyberSpec system proved able to detect five of the seven attack types within 60 seconds. For the attacks it can detect, though, CyberSpec proved impressively capable: The framework can, the researchers found, determine the difference between normal operations and an active attack in under 60 seconds, and with a very low computational cost to the host device.The team has indicated its future work will focus on scalability to larger numbers of monitored devices, add privacy-preserving mechanisms like federated learning to prevent private data from being transmitted off-device, the use of supervised algorithms for attack classification, and adding new events to the monitoring system to resolve the issues in detecting Repeat and Freeze attacks.The paper on CyberSpec is available under open-access terms <a href="https://arxiv.org/abs/2201.05410v1" target="_blank">on Cornell’s arXiv.org preprint server</a>.<h1>Reference</h1>Alberto Huertas Celdrán, Pedro Miguel Sánchez Sánchez, Gérôme Bovet, Gregorio Martínez Pérez, Burkhard Stiller: <cite>CyberSpec: Intelligent Behavioral Fingerprinting to Detect Attacks on Crowdsensing Spectrum Sensors</cite>. DOI <a href="https://arxiv.org/abs/2201.05410" target="_blank">arXiv:2201.05410 [cs.CR]</a>

Designed to run on a Raspberry Pi-based spectrum sensor platform, CyberSpec can find an attack in 60 seconds.

The CyberSpec framework is designed to detect anomalous behavior linked to cyber-attacks against crowd-sensing spectrum sensors, even when said sensors are running on lightweight resource-constrained hardware like a Raspberry Pi.

CyberSpec turns machine learning onto the problem of spectrum sensor attacks, without overloading the host

This is the 3rd and final article in a 3 part series featuring the applications of tinyML. The series introduces the concept of tinyML and explains some selected applications of the technology in various fields like sustainability, healthcare, and education.Education is the first step towards innovation. It is an integral part of personal development and equips the learners with the tools necessary to go ahead and solve the challenges faced by humanity. While tinyML is a great tool to develop pedagogical interventions for improving the quality of education in different streams, it is in itself a vast subject that every science and technology student must be familiar with.In our <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future" target="_blank">previous article</a> about the applications of tinyML, we explained how it aids medical research and improves healthcare. This article continues the series by showcasing two more applications of tinyML for education.<h2 dir="ltr">Using tinyML for Neuroscience in K12 education: tinyML4STEM</h2>Anything and everything concerned with the central nervous system is a part of neuroscience. It is a multidisciplinary field that involves a scientific study of how the brain coordinates with the body and makes various body functions possible. The study is not just limited to humans but can also be extended to animals and even plants.Roughly 20% of the world’s population has suffered from a neurological disorder at some point in their lives, and none of the disorders have a cure yet. The only way to make new discoveries in the domain is by promoting research and education.It is also interesting to note that the most fundamental unit of Machine Learning (ML) is called a neuron and the whole concept and modelling of Artificial Intelligence (AI) based systems came from neuroscience. This makes it essential to talk about the specialised subject with the students of computer science to make them understand the underlying principles of the technology.But the problem with neuroscience education has always been the affordability and availability of experiment kits. Unlike many other Science Technology Engineering Management (STEM) fields, the apparatus for performing neuroscience experiments can not be acquired easily by schools and colleges. Graduate students opting for a specialisation in the area are the only ones to get their hands on these experiment kits.Greg Gage and Tim Marzullo realised this problem back in 2008 when they were pursuing PhD at the University of Michigan. The duo tried explaining neuroscience concepts to school children but couldn’t do so without showcasing practicals. There was no way to bring expensive graduate college equipment to schools. To solve this issue and make neuro-science kits affordable and readily available, Greg and Tim founded Backyard Brains.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNDUyMjI0LWltYWdlNy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="EMG-signals-backyard-brains">Illustrations like these can help kids grasp even complex concepts easily. Source: Backyard BrainsThe organisation today makes low-cost neuroscience trainers, DIY kits, tools, software, conducts workshops and a lot more to democratise neuroscience education.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNTI1MDYyLWltYWdlMS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="neuroscience-experiment-kits-backyard-brains">Neuroscience trainer kits by Backyard Brains are great alternatives for schools to introduce the students to the field. Source: Backyard BrainsNeuroscience signals like Electromyogram (EMG), Electrooculogram (EOG), Electrocardiogram (EKG), etc., can be used to get various cues about what is going on inside the body.When neurons fire electrical impulses from the motor cortex, the signals flow through the spinal cord to the parts of the body involved in the action. By putting small electrodes on the skin near the areas where the signals can be picked up, it is possible to observe them via an EMG. The impulses being electrical in nature can be amplified as a single-dimensional signal and visualised easily. These single-dimensional signals are beautiful for ML and are also quite easy to be analysed.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNjI0MDcwLWltYWdlNC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="visualising-neuroscience">A snapshot from tinyML talks where Greg demonstrates how EMG signals can be visualised with a low-cost training kit. Source: tinyML YouTubeThe patterns in these signals might not be as apparent to humans as they are to the tinyML algorithms, which can classify and map them with different human behaviours. An implementation of such a technique can be used to develop a brain-machine interface that can help make neuroprosthetics.&nbsp;EOG is another example of a neuro signal that measures the corneo-retinal standing potential of the eye. In other words, it helps understand eye movements. A trained TinyML system can be used to accurately detect eye blinking and movements by feeding the signal to the input.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkwNjYzNTM5LWltYWdlOC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="EOG-signals-detect-eye-blinking">The spikes that appear on the EOG are due to eye blinking. Source: tinyML YouTubeThen there’s EKG generated by the electrical impulses from the heart. For different people, the patterns of EKG are observed to be similar when looking at the same object. This can be used to predict if the person is telling the truth. In medical applications, signals can be used to check the functioning of the brain. A sample tone is played to the patient, and electric potential signals are observed. When the tone being played changes, the subconscious mind should ideally change the way it reacts, which should be evident in the electric potential signals.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNTU2NjQ2LWltYWdlMy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="hackerhand-copies-gestures-EKG-signals">‘Hackerhand’, a robotic arm mimicking hand movements based on EKG signals. Source: tinyML YouTubeIt is also interesting to note that the signals are generated slightly before someone is about to take action, making it possible for tinyML systems to be able to predict the behaviour in advance. A similar fun project has been developed at Backyard Brains.Readers can participate in many more such projects or share ideas by visiting <a href="https://backyardbrains.com/" target="_blank">Backyard Brains</a>.<h2 dir="ltr">TinyML Open Education Initiative: tinyMLedu</h2>Over the past two decades, ML has been through a journey from a purely academic and theoretical discipline to a widely spread technology in use. It won’t be an understatement to say that technology has touched almost every industry and is now ubiquitous technology. It is expected that the interest in the field will only grow from here, at least in the near foreseeable future. Given its amazing potential in solving some of the greatest challenges faced by humanity, it is important that more people are introduced to technology in a systematic way.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNTk4OTQ2LWltYWdlNS5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="instructors-tinyML-harvard-google-edx">Instructors of the tinyML specialisationUnlike traditional ML, it is possible to run tinyML algorithms on low-cost and low-power embedded systems, which solves a major problem related to the unavailability of powerful hardware for learners. But the shortage of ML educators is still something that needs to be addressed.A joint initiative by Harvard and Google, developed as an academic and industry collaboration, aims to address the challenges by introducing tinyML to the masses with a Massive Open Online Course (MOOC) on edX.&nbsp;The courses that are a part of the tinyML specialisation go beyond teaching the students about the subject by diving deeper into the practical applications and industrial product development lifecycle. The next generation of students needs to understand the end-to-end full-stack development of tinyML projects to be able to gain a complete understanding of the subject.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNjIwMzE4LWltYWdlOS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="tinyML-specialisation-course-structure">A diagram providing an overview of topics explored in tinyML specialisation. Source: [3]The tinyML specialisation, jointly developed by Harvard and Google, is a set of self-paced learning modules that can be taken from anywhere. The specialisation includes four courses:<ol><li dir="ltr">Introduction to tinyML:&nbsp;The first course covers the basics of ML and embedded systems. Everything from the concept to gathering data and training the algorithms is explained.</li><li dir="ltr">Applications of tinyML: The second course builds on the foundation by explaining some parts of the code behind the most widely used tinyML applications. It also introduces the learners to the principles of keyword spotting, visual wake words, anomaly detection, and dataset engineering.</li><li dir="ltr">Deploying tinyML: The third course teaches how code is written, optimised, and deployed for embedded systems. This is where microcontrollers are introduced, and students can purchase a kit for a nominal price (US$49.99) to get hands-on experience.</li><li dir="ltr">Scaling tinyML [Optional]: Once the learners are familiar with the design, development and deployment of tinyML, the final course covers the more advanced aspects related to producing tinyML solutions for the masses.</li></ol>Each course includes several activities such as collabs, hands-on labs, quizzes, readings, assignments, tests, and discussion-forum participation.The key challenges for students taking up the tinyML specialisation can be limited hardware accessibility and the requirement of being familiar with programming. To address the challenges associated with hardware accessibility, open-source emulation platforms like <a href="https://renode.io/" target="_blank">renode.io</a> from Antmicro are being tested to allow learners to try out their code without the hardware. The other challenge can be solved by platforms like <a href="https://www.wevolver.com/profile/edge.impulse" target="_blank">Edge Impulse</a>, which are lowering the barriers to enable people with little to no programming knowledge to implement tinyML solutions.Anyone with interest in tinyML can visit <a href="https://www.edx.org/professional-certificate/harvardx-tiny-machine-learning" target="_blank">tinyML Professional Certificate | edX</a> and enrol for the course for free. The course material is available on <a href="https://github.com/tinyMLx/courseware" target="_blank">GitHub</a>.<h2 dir="ltr" id="isPasted">Conclusion</h2>Though the project kits or MOOCs are not a complete solution to train professional neuroscience researchers or tinyML solution architects, they offer a great place for beginners to start. The kits and courses are designed in such a way that the students can engage in fun activities while they experiment and learn new stuff. This makes the experience memorable and develops an interest in the subject, which could lead more people to choose the specialised fields and make new discoveries.&nbsp;This article concludes our tinyML applications series.About tinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, tinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMjkxNjYyNjY4LWltYWdlMi5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib">The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a> explained some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The <a href="https://www.wevolver.com/article/applications-of-tinyml-making-specialised-education-more-accessible" rel="noopener noreferrer" target="_blank">third and final article</a> shall cover the tinyML applications in pedagogy and education.<h2 dir="ltr">References</h2>[1] Marzullo TC, Gage GJ (2012) The SpikerBox: A Low Cost, Open-Source BioAmplifier for Increasing Public Participation in Neuroscience Inquiry. PLOS ONE 7(3): e30837. <a href="https://doi.org/10.1371/journal.pone.0030837" target="_blank">https://doi.org/10.1371/journal.pone.0030837</a>.[2] Gregory J. Gage, The Case for Neuroscience Research in the Classroom, Neuron, Volume 102, Issue 5, 2019, Pages 914-917, ISSN 0896-6273, <a href="https://doi.org/10.1016/j.neuron.2019.04.007" target="_blank">https://doi.org/10.1016/j.neuron.2019.04.007</a>.[3] Vijay J. Reddi et. al., Widening Access to Applied Machine Learning with TinyML, arXiv:2106.04008v2 [cs.LG], Jun 2021, [Online], Available from: <a href="https://arxiv.org/abs/2106.04008" target="_blank">https://arxiv.org/abs/2106.04008</a>.

In this article, we look at two tinyML projects for education. We show how Backyard Brains uses low-cost experiment kits to make neuroscience education more accessible. We also introduce our readers to a specialisation offered by Harvard & Google to help students learn tinyML like never before.

Applications of TinyML making specialised education more accessible

The recent breakthroughs in AI research came due to the growing demand for improved architecture, offering high performance and less complexity. Self-supervised learning has been the key to several AI challenges in speech, NLP and computer vision data-intensive workloads. These challenges refer to the changing environment and different input parameters requiring the AI model to learn by directly observing the surroundings. However, to keep up with the technological advancements, self-supervised learning differs at various levels of the algorithm when applied for a single modality. Even though the idea behind the implementation remains the same, there are performance issues when using the same algorithm across.To make it easy for the researchers and AI developers, Meta AI, previously known as Facebook AI, has introduced a novel methodology, data2vec, through a generalized framework of self-supervised learning for speech, vision and language processing. Rather than predicting modality-specific features such as words, tokens or units in speech processing, the proposed idea of the generalized framework is to predict the latent representation of the input data. Continuing, Meta AI’s data2vec claims to be the first high-performance, self-supervised AI algorithm that works efficiently and effectively across multiple modalities.&nbsp;Executing the data2vec general framework separately on different modalities like speech, image and text, the new method outperformed the existing single-purpose algorithms. This shows holistic self-supervised learning that improves the performance of multiple modalities rather than a single task by not relying on contrastive learning. The Meta AI’s data2vec has brought machines closer to actual self-learning by analyzing different aspects in the surrounding. The adaptability of AI algorithms is the future for several real-life applications with exceptional results like never before.&nbsp;“In an effort to get closer to machines that learn in more general ways about the environment, we designed data2vec, a framework for general self-supervised learning that works for images, speech and text where the learning objective is identical in each modality,” researchers explain. “We hope that a single algorithm will make future multi-modal learning simpler, more effective and lead to models that understand the world better through multiple modalities.”<h2 dir="ltr" id="isPasted">Inside the Meta AI’s Data2vec Framework</h2>The current research focused more on improving the performance for a single modality which changed the approach towards building a robust framework for other modalities. In data2vec, the training models are made to predict the input data through its own representation irrespective of the modality. Instead of predicting a specific set of features, a single algorithm can work with different types of inputs, like speech, vision, and text.&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMzAwNDU1Nzc5LVRlYWNoZXIgU3R1ZGVudCBNb2RlbC5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 800px;" class="fr-fic fr-dib" alt="Teacher Student Neural Network Model">The proposed model of teacher and student network makes it easy to understand such algorithm training. Primarily, the teacher network goes through the computation for target representation of image, text or speech. Once this is performed, the same input is masked and the process is repeated in the student network. The latter predicts the latent representation of the teacher network. Even though the input data is not clearly visible to the training model, the student network has to predict the representation of the full input data. The weights of the teacher network are the exponentially decaying average of the student. With different modalities in the picture, the framework uses modality-specific feature encoders and masking strategies.&nbsp;The framework uses the standard transformer architecture that comes with the modality-specific encoding of the input data. Researchers decided to use ViT-strategy for computer vision, a multi-layer 1-D convolutional neural network for speech data, and text input is pre-processed to obtain sub-word units.<h2 dir="ltr" id="isPasted">Did Data2vec Framework Outperform Existing Methods?</h2>These are the results and the experimental setup employed in the new generalized framework for self-supervised machine learning in speech, vision and text processing.&nbsp;<ol><li>For computer vision: The framework was pre-trained on the images of the ImageNet-1K training set. The Data2vec performance for computer vision on the popular ImageNet benchmark for ViT-B models compared with other recent methods seems to significantly outperform with an accuracy improvement of 1.5% over <a href="https://arxiv.org/abs/2104.14294" rel="noopener noreferrer" target="_blank">DINO</a>.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMzAwNDcwODM0LURhdGEydmVjIGZvciBjb21wdXRlciB2aXNpb24ucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 800px;" class="fr-fic fr-dib" alt="Data2vec for computer vision"></li><li dir="ltr" id="isPasted">For speech processing: For the evaluation, the speech processing was done through pre-training of Data2vec on 960 hours of speech audio input data from <a href="https://www.danielpovey.com/files/2015_icassp_librispeech.pdf" target="_blank">LS-960</a>. Due to its clean speech audio and standard used in the industry, the results are easier to compare. The performance of the base model on the LibriSpeech benchmark with 10 hours of labeled data compared with other methods seems to show improved performance as the WER goes down.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMzAwNDc3NDU3LURhdGEydmVjIGZvciBzcGVlY2ggcHJvY2Vzc2luZy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 800px;" class="fr-fic fr-dib" alt="Data2vec for speech processing"></li><li dir="ltr" id="isPasted">For NLP: For language processing, <a href="https://arxiv.org/abs/1804.07461" id="isPasted" target="_blank">General Language Understanding Evaluation (GLUE) benchmark</a> has been used that includes tasks for natural language inference, sentence similarity, grammaticality, and sentiment analysis. The performance for the base model has once again outperformed (but with a slight margin) the <a href="https://arxiv.org/abs/1907.11692" target="_blank">RoBERTa baseline</a> when retrained with the original BERT setting.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQzMzAwNDgyMzY5LURhdGEydmVjIGZvciB0ZXh0IHByZWRpY3Rpb24ucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 800px;" class="fr-fic fr-dib" alt="Data2vec for text prediction"></li></ol><h2>Summary of Data2vec Framework for Self-Supervised Learning</h2>Summarizing the innovation brought in through the generalized framework for Self-Supervised learning for speech, vision and language processing:<ol><li>Uniform model architecture for multiple modalities without reducing the performance.</li><li>Data2vec framework outperforms existing self-supervised algorithms on ImageNet-1K for ViT-B and ViT-L models.</li><li>Improvement on speech recognition for low-resource setup of Libri-light and NLP’s competitive performance against RoBERTa on GLUE. &nbsp;</li></ol>For more details on the methodology of the research, check out the <a href="https://ai.facebook.com/research/data2vec-a-general-framework-for-self-supervised-learning-in-speech-vision-and-language" rel="noopener noreferrer" target="_blank">open-sourced research paper</a>.

A holistic open-source framework to improve the performance of multiple modalities and outperform the existing single-modality algorithms.

Meta AI Introduces Data2vec Open-Source Framework for Self-Supervised Learning in NLP, Speech and Vision Processing

This is the 2nd article in a 3 part series featuring the applications of tinyML. The series introduces the concept of tinyML and explains some selected applications of the technology in various fields like sustainability, healthcare and education.By adopting digital advancements, healthcare providers are updating their services to improve them and make them accessible to more people. 5G networks, Augmented Reality (AR) &amp; Virtual Reality (VR), Wearables, Digital Twins, and many more technologies are enabling this digital transformation.One such technology with great potential in ensuring healthy lives and well-being for all is tinyML. The concept is a crossover between Artificial Intelligence (AI) and embedded systems that involves developing solutions that bring Machine Learning (ML) capabilities to edge devices. Benefits include privacy, affordability, low-power utilisation, the ability to work in remote locations, and a lot more.In our <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">previous article</a> about the applications of tinyML, we explained how it unlocks new possibilities for sustainable development technologies. This article continues the series by showcasing two more applications of tinyML for medical research and improved healthcare.<h2 dir="ltr">MaRTiny: a tinyML solution for extreme heat sensing for urban climate science</h2>As a direct effect of global warming, summers are getting warmer. [1] This disruption of the natural balance poses a risk to all living entities on the planet. Higher temperatures are known to cause a degradation in the air quality and induce severe health-related issues like heat strokes. Looking at the data of weather-related fatalities in America, heat is the single most significant cause of weather-related deaths over the last 30 years, surpassing floods and storms. [2]But it’s not just the ambient temperature that causes discomfort. It’s the Mean Radiant Temperature (MRT) which accounts for the influence of all the surfaces in the surrounding that ultimately determines the occupant’s comfort. Often, the effect of roads, buildings and other entities has an adverse effect on the MRT, creating a complex problem intersecting health and urban infrastructure.To aid the collection of information related to the MRT, a tinyML solution called MaRTiny was developed by researchers from Arizona State University (ASU). It is a $200 low-cost mobile platform for bio-metrological sensing.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3MzYxNzgyLTE2NDI1MTczNjE3ODIucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" width="720" height="297" id="isPasted" class="fr-fic fr-dii" alt="MaRTiny-tech-stack">The tech stack of the MaRTiny platformThe list of sensors embedded on the device include:<ul><li dir="ltr">Ambient temperature sensor</li><li dir="ltr">Globe temperature sensor for sensing the radiant heat</li><li dir="ltr">Humidity sensor</li><li dir="ltr">Ultra Violet (UV) sensor</li><li dir="ltr">Anemometer for measuring wind speed and direction.</li></ul>The platform uses an <a href="https://www.wevolver.com/profile/arduino.pro" target="_blank">Arduino</a> Uno to collect the data and a NodeMCU to provide it with the electronic infrastructure required for wireless data communication. At the core of the system is the Nvidia Jetson Nano development board that provides MaRTiny with the power to run tinyML algorithms.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NDcxMTAyLU5WSURJQV9KZXRzb25fTmFub19EZXZlbG9wZXJfS2l0Xyg0MDY1MDQyNTUwMykuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="nvidia-jetson-nano-development-board">The Nvidia Jetson Nano development board provides MaRTiny with image processing and ML capabilitiesJetson Nano, with its Camera Serial Interface (CSI), can connect with a camera for shadow detection, object detection and people-in-shade detection. Jetson’s ARM Cortex A57 MPcore, 128 CUDA core Maxwell GPU coupled with 4 GB RAM is just enough to offer advanced ML capabilities on edge.The task of MRT estimation was formulated as a supervised learning problem by the inventors that required a lot of true data for training. This true data was collected with the help of a previously developed, larger and more expensive MRT detecting system going by the name of MaRTy. [3]&nbsp;<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NTMzMzM5LTE1MDB4NTAwLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="MaRTiny-MaRTy">MaRTiny aims to offer similar functionality as that of MaRTy (in the image) at just 1% of the price. Image source: @ASUMaRTy, TwitterIt was observed that a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel achieved the highest accuracy with the MaRTiny. The method being computationally light, offered an easy way of deployment on Jetson. To improve the estimations, both SVM and Artificial Neural Network (ANN) models were utilised and trained with data collected via sensors. [4]For privacy, the images taken by MaRTiny are not stored in the cloud or even in the device’s memory. It automatically discards personally identifiable information (PII) before reporting anything back to the servers, which is central to the idea of tinyML and a must-have feature for public deployment.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NTc1MjI0LWltYWdlNy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="MaRTiny-setup">The experimental setup of MaRTinyThe tinyML solution, MaRTiny, can provide crucial insights related to weather conditions and the utilisation of shades and space by the public. Multiple MaRTiny devices can be installed across the city to gather valuable data for architecture and town planning.<h2 dir="ltr">Pneumonia Detection using Embedded Machine Learning</h2>Pneumonia is a respiratory infection caused by viruses, bacterias and fungi. According to the World Health Organisation (WHO) data, it is estimated that as much as 22% of all deaths in children aged from 1 to 5 is due to pneumonia. Bacteria induced pneumonia can be treated with antibiotics and other low-cost and low-tech medical facilities if detected at the right time. [5]Chest X-ray, blood oxygen levels and Complete Blood Count (CBC) are used to detect pneumonia which is time-consuming. Doctors manually review the chest X-ray and decide if the person is infected, which is sometimes inaccurate and requires an automated solution with better accuracy. This is the idea behind the tinyML solution that we are about to discuss in this section.Arijit Das, a 15-year-old tinyML enthusiast, has developed a tinyML model using the <a href="https://www.wevolver.com/profile/edge.impulse" target="_blank">Edge Impulse</a> platform to detect pneumonia from chest X-rays. The model can be deployed on single board development platforms like Raspberry Pi 4, Sony Supersense and a few other devices. With just the main board, a camera shield and the Edge Impulse tinyML model, the tiny setup can detect pneumonia in under a minute which is phenomenal considering the fact that it takes anywhere between 1 to 4 days for detection by manual inspection.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NzA0Nzk2LVJhc3BiZXJyeV9QaV80X01vZGVsX0JfLV9TaWRlLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="Raspberry-Pi-4">Credit card-sized single-board computers like Raspberry Pi come with enough compute power to run optimised tinyML algorithmsThe tinyML model doesn’t require internet connectivity to work and offers an accuracy of over 80%, which is far better compared to manual inspection. This can prove to be quite helpful in largely affected regions like Sub-Saharan Africa and South Asia, where a huge number of examinations need to be conducted related to the same disease.&nbsp;At the time of writing this article, the project is fully functional and has been clinically tested by over 10 doctors in a local survey conducted by the inventor. The project is available on <a href="https://github.com/Pneumonia-Detection-using-EdgeML" target="_blank">GitHub</a> for everyone to try, test and contribute. [6]<h2 dir="ltr">Conclusion</h2>It’s the third year into the pandemic, and the emphasis on healthcare research is more than ever. Introducing technologies that can work with little power and no network connectivity at an affordable price in healthcare can save several lives. Leveraging interdisciplinary technologies like tinyML is the way forward to a safer and healthier future.About tinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, tinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQyNTE3NzY5MDkxLWV5SmlkV05yWlhRaU9pSjNaWFp2YkhabGNpMXdjbTlxWldOMExXbHRZV2RsY3lJc0ltdGxlU0k2SW1aeWIyRnNZUzh4TmpNMk9UZ3lNVEUwT1RRNExWUnBibmxOVEdadmNrZHZiMlFnS0RRd01EQWdlQ0EzTlRBZ2NIZ3BMbXB3WnlJc0ltVmthWFJ6SWpwN0luSmxjMmw2WlNJNmV5SjNhV1IwYUNJNk9UVXdMQ0ptYVhRaU9pSmpiM1psY2lKOWZYMD0ud2VicCIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="tinyML-banner"> The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The&nbsp;<a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a> explained some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" rel="noopener noreferrer" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The third and final article shall cover the tinyML applications in&nbsp;pedagogy&nbsp;and education.<hr><h2>References</h2>[1] Extreme Weather Toolkits, Climate Central, [Online], Available from: &nbsp;<a href="https://medialibrary.climatecentral.org/extreme-weather-toolkits/extreme-heat" target="_blank">https://medialibrary.climatecentral.org/extreme-weather-toolkits/extreme-heat</a>[2] Weather Related Fatality and Injury Statistics, National Weather Service, [Online], Available from:&nbsp;<a href="https://www.weather.gov/hazstat/" target="_blank">https://www.weather.gov/hazstat/</a>[3] Ariane Middel, E. Scott Krayenhoff, ‘Micrometeorological determinants of pedestrian thermal exposure during record-breaking heat in Tempe, Arizona: Introducing the MaRTy observational platform’, Science of The Total Environment, Volume 687, 2019, Pages 137-151, ISSN 0048-9697, Available from:&nbsp;<a href="https://doi.org/10.1016/j.scitotenv.2019.06.085" target="_blank">https://doi.org/10.1016/j.scitotenv.2019.06.085</a>[4] Karthik K. Kulkarni, 2021, ‘Machine Learning and Vision Using Edge Devices for Multimodal Chatbots andBio-meteorological Sensing’, MS Thesis, Arizona State University, United States of America, [Online], Available from:&nbsp;<a href="https://keep.lib.asu.edu/_flysystem/fedora/c7/Kulkarni_asu_0010N_21269.pdf" target="_blank">https://keep.lib.asu.edu/_flysystem/fedora/c7/Kulkarni_asu_0010N_21269.pdf</a>[5] Pneumonia Facts Sheet, World Health Organisation, [Online], Available from:&nbsp;<a href="https://www.who.int/news-room/fact-sheets/detail/pneumonia" target="_blank">https://www.who.int/news-room/fact-sheets/detail/pneumonia</a>[6] Arijit Das, Pneumonia-Detection-using-EdgeML, GitHub, [Online], Available from:&nbsp;<a href="https://github.com/Pneumonia-Detection-using-EdgeML" target="_blank">https://github.com/Pneumonia-Detection-using-EdgeML</a>

In this article, we take a look at two tinyML projects related to healthcare. The first project helps gather Mean Radiant Temperature data outdoors to protect people from extreme heat, and the other one is a solution for affordable, accurate, & rapid detection of pneumonia.

Applications of TinyML in healthcare for a safer future

Changing weather patterns, rising sea levels, wildfires, droughts, depletion of natural resources are just some of the long-term consequences of unsustainable human activities. Indiscriminate exploitation of resources is accelerating global warming, and if preventive countermeasures are not taken immediately, the world may end up at a point of no return.TinyML is a field that aims to bring machine learning (ML) capabilities to low-power embedded systems. Its applications can be found from households to industries and everywhere in between. The idea of sustainability naturally links well with the tinyML as it encourages us to reduce our carbon footprint by executing ML algorithms on low-power devices.In our <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" rel="noopener noreferrer" target="_blank">previous article</a> about tinyML, we introduced our readers to the basics of the technology and how it is helping shape the embedded systems of the future. This article focuses on two interesting applications of tinyML for sustainable development.<h2>Groundwater Wells: The Untapped Edge Use Case</h2>Farming is the oldest and arguably the most crucial industry in the world. It employs around one-fourth of the world’s population and feeds billions of people. [1] Agriculture is also the largest use of fresh water in the world. Regrettably, it involves the largest wastage of water as well, making it incredibly inefficient. Most of the water used here is unregulated and unmonitored.Wastage of water has put 20% of the 39 million groundwater wells at the risk of running dry. [2] Once a well dries up, the only possible option is to dig a deeper one, but that can have a much worse long-term impact on the water table.<a href="https://niolabs.com" target="_blank">Niolabs</a> is a digital transformation and Internet of Things (IoT) solutions organisation that leverages IoT and tinyML in precision farming. Over the past couple of years, the organisation has achieved excellent results with the technology. Precision farming is the future of agriculture involving the collection and utilisation of real-time data to control the various irrigation automation systems. The result is lower consumption of water and energy while saving on operating costs. [3]<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQwMDkyNTAzODY3LXNtYXJ0X2Zhcm1pbmcuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="smart-farming">Precision farming with tinyML takes water conservation to the next level</div>For the implementation of the precision agriculture model, connecting all the devices to the network and passing data to the cloud is not always feasible, especially if a large number of data sources are involved. With thousands of plants being monitored for various environmental variables, the amount of data being generated is enormous and creates a security concern too.Using microcontrollers interfaced with moisture sensors and water control valves, it is possible to implement a simple automatic irrigation system that turns the irrigation system on or off depending on a static value of soil moisture levels. But not all plants require the same amount of moisture. Considering the environmental conditions like temperature, humidity, sunlight, and wind speed that have a direct effect on the water requirements of the plant can also be a challenge on the microcontroller programmed with simple if-else statements.This is where tinyML comes in. By implementing machine learning algorithms on the embedded controllers, it becomes possible to take into account the various environmental variables and dynamically adjust the water supply depending on the plant species and environmental conditions.The precision agriculture solution by Niolabs uses edge computing to enable each node to decide the amount of water required individually. The system uses an Xbee service that tracks the moisture levels near each watering zone and another service that allows users to provide their inputs about the watering requirements.&nbsp;Both the inputs and several other variables are combined and used as a reference to train the irrigation model, which keeps learning and improving with time—proof enough of the fact that combining general knowledge with ML can do wonders.The one thing that makes Niolabs’s implementation of a precision agriculture system unique is its ability to combine millions of seemingly unrelated data points into insightful and straightforward metrics, which can be accessed as a summary or detailed view at the fingertips of the users. The same system that maintains the plantation can also help with predictive maintenance. By using condition monitoring techniques, it can tell when its components might fail.Niolabs aims to come up with a revamped architecture of their application before March 2022. This is when the harvesting season begins, a perfect time to deploy and test the latest solutions. The plan is to create a repeatable tinyML solution that can be transferred to other farms and be scaled for use in a variety of different environments.<h2 dir="ltr">Ribbit Network: The world’s largest crowdsourced network of open-source, low-cost, Carbon Dioxide Gas Detection Sensors</h2>If we take a look at the long-term trend of worldwide greenhouse gas emissions, it is easy to note that the figures have shown constant growth in the past three decades [4]. There is a massive uptrend that continues to get steeper with each passing year. To make informed decisions and new policies regarding climate change, stakeholders and government organisations need accurate data.It is easy to assume that the data that is used presently is sufficient to make critical decisions, but that is not the case. The reference estimations are mostly theoretical calculations or predictions based on indirect economic measures. For example, vehicular emissions are roughly calculated based on the overall kilometers traveled and average emission per vehicle.&nbsp;The measurements that are direct, such as the ones taken by climate tech satellites, prove to be exorbitantly expensive. A single Orbiting Carbon Observatory 2 (OCO2) satellite that makes space-based observations of atmospheric carbon dioxide cost NASA over $467 mn. [5]Ribbit network aims to fill this gap by combining the economics with actual observations to reduce the uncertainties in data collection. Ribbit network takes over the challenge of building the map of emissions and concentration of carbon in the air with the help of tiny and affordable IoT sensors. They call them the ‘Frog’ sensors. The sensors get their name from frogs as these amphibians are hypersensitive to temperature and are the first ones to get affected due to climate change.<blockquote>Our primary interest in TinyML is based on decreasing power consumption. Running as much computing as we can on our IoT devices lets us minimize the carbon footprint that comes with big data center services. - Keenan Johnson</blockquote>The Frog sensors use a Non-Destructive Infra-Red (NDIR) sensor working on the principle of infrared (IR) spectroscopy. A typical NDIR sensor consists of an IR source, sample chamber and an IR detector. For measuring the presence of CO2, the air is allowed to flow inside the sample chamber. Depending on the concentration of CO2, some IR waves of a particular wavelength emitted by the source get absorbed. The difference between the emitted light and the light that makes it through the chamber gives conclusive evidence of the presence and quantitative estimation of the greenhouse gas in the air.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM5ODA3MTYzMTk1LVVudGl0bGVkIGRlc2lnbi5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="ribbit-frog-carbon-dioxide-sensor-tinyml-block-diagram">Block diagram of the Frog sensor. Source: Ribbit network</div>The sensor is interfaced with a Raspberry Pi CM4, which takes care of everything right from powering the sensor to collecting and transmitting the data.The design of the frog sensors is entirely open-source which means, anyone can build their own version of the device by following the <a href="https://github.com/Ribbit-Network" target="_blank">official tutorial</a> from GitHub.&nbsp;<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM5ODA3MzAzMTE3LVVudGl0bGVkIGRlc2lnbiAoMSkuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="ribbit-frog-carbon-dioxide-sensor-tinyml">A closer look at the Frog sensor. Source: Ribbit network</div>Twenty-five alpha versions of frog sensors have been deployed across five countries. Beta versions will be deployed in a much larger number before the final rollout. The team is currently striving hard to solve problems related to determining sensor health, calibrations, the inclusion of wind data, and a lot more. With custom engineering solutions, they wish to reduce the cost of the sensor and power it with solar cells.<h2 dir="ltr">Conclusion</h2>It is not one, two, or three solutions that are going to solve the problem of climate change, but hundreds and thousands of innovative ideas collectively working together that can bring the revolution. That is what tinyML and its community members stand for.&nbsp;Grassroots innovations like the ones we showcased in this article can go long miles in meeting the global sustainable development goals, making the world a better place to live for future generations. Over the next couple of years, we expect tinyML to be leveraged far beyond than presently is.About tinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, tinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM5ODA3ODQ1NDA0LVVudGl0bGVkIGRlc2lnbi5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="tiny-ml-foundation-banner"></div>The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a> explained some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" rel="noopener noreferrer" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The third and final article shall cover the tinyML applications in pedagogy and education.<hr><h2 dir="ltr" id="isPasted">References</h2>[1] The World Bank Data, Employment in agriculture (% of total employment) (modelled ILO estimate), 2021, [Online], Available from: <a href="https://data.worldbank.org/indicator/SL.AGR.EMPL.ZS" target="_blank">https://data.worldbank.org/indicator/SL.AGR.EMPL.ZS</a>[2] Harrison Tasoff, Gauging Groundwater, 23 April 2021, [Online], Available from: &nbsp;<a href="https://www.news.ucsb.edu/2021/020248/gauging-groundwater" target="_blank">https://www.news.ucsb.edu/2021/020248/gauging-groundwater</a>[3] Niolabs, Nio powers the only autonomous vineyard in the world, [Online], Available from: <a href="https://niolabs.com/case-studies/agriculture" target="_blank">https://niolabs.com/case-studies/agriculture</a>[4] United States Environment Protection Agency, Global Greenhouse Gas Emissions Data, [Online], Available from: <a href="https://www.epa.gov/ghgemissions/global-greenhouse-gas-emissions-data" target="_blank">https://www.epa.gov/ghgemissions/global-greenhouse-gas-emissions-data</a>[5] NASA, Orbiting Carbon Observatory-2 Launch Press Kit, July 2014, [Online], Available from: <a href="https://www.jpl.nasa.gov/news/press_kits/oco2-launch-press-kit.pdf" target="_blank">https://www.jpl.nasa.gov/news/press_kits/oco2-launch-press-kit.pdf</a>

In this article, we take a look at two tinyML projects that have the potential to make contributions towards sustainable development goals. While the first project is about revolutionising precision farming, the second one aims to create a network of low-cost sensors for mapping carbon emissions.

TinyML unlocks new possibilities for sustainable development technologies

Edge Impulse recently announced official support for the <a href="https://www.syntiant.com/ndp101" target="_blank">Syntiant NDP101</a> processor. The NDP101 (Neural Decision Processor™) is an always-on sensor and speech recognition processor, with ultra-low power consumption of less than 140 uW while recognizing words. It features a Dense Neural Network Architecture of four layers with 256 inner nodes and runs real-time audio inferencing up to 100 times per second. The NDP101 is a solution for wake word detection, speech classification or any audio-related application.<iframe width="540" height="600" src="https://c8056756.sibforms.com/serve/MUIFAMyA6BGksOUPEokHuBnEmZRHhYgJ4RXb4BV0gvzN98ymNIvQdTcBFQnRVfviT6yZSFW_R2g1hENBu7p7Afnyv9Kt0unAAbfFU5mMQhBwI-8hQnKf2r1mmRKhUgDLmtk8bpVSZ7kZGpleCPqtYRv8_kkvODL7F4neGLNGgdpFDm9pPzoYpwbQkc0BU5GVOFoq8CSmD0DDvDjG" frameborder="0" scrolling="auto" allowfullscreen="" style="display: block;margin-left: auto;margin-right: auto;max-width: 100%;" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe> Equipped with the ultra-low-power Syntiant <a href="https://www.globenewswire.com/Tracker?data=hVCGZ6kMB-cDkZNKwE61sOEIRHpn97IpKQBQdH_1nVFus9Ct-Q9YnTOq2yFHiWA3lgDtI1M213rZ8aS7v3sIaQ==" target="_blank">NDP101</a> Neural Decision Processor, the new TinyML Board can enable speech and sensor applications to run at under 140 and 100 microwatts, respectively, delivering 20x more throughput and 200x efficiency improvement compared to traditional MCU-based systems.&nbsp;Sized at 24 mm x 28 mm, the Syntiant TinyML Board is a small, self-contained system that allows trained models to be easily downloaded via Edge Impulse through a micro-USB connection without the need for any specialized hardware. The board also is fully compatible with Arduino’s open source platform.The TinyML Board’s onboard digital microphone and motion sensor enable easy configuration for any speech, event detection or six-axis motion- and vibration-related application. Shipping with a built-in “Alexa play music” model, key product features include:<ul><li dir="ltr">Syntiant NDP101 Neural Decision Processor™</li><li dir="ltr">Syntiant Core 1™ neural network</li><li dir="ltr">Supports up to 560k parameters and 64 output classifications</li><li dir="ltr">Host processor: Arm Cortex-M0+ 32-bit microcontroller, including:<ul><li dir="ltr">256KB flash memory</li><li dir="ltr">32KB host processor SRAM</li></ul></li><li dir="ltr">5 digital I/Os compatible with Arduino MKR series boards</li><li dir="ltr">2MB on-board serial flash</li><li dir="ltr">48MHz system clock</li><li dir="ltr">6-axis motion sensor</li><li dir="ltr">Digital microphone</li></ul>To make it easy to build and deploy audio classification models on ultra-low power silicon, Edge Impulse has partnered with <a data-saferedirecturl="https://www.google.com/url?q=https://www.wevolver.com/profile/syntiant&source=gmail&ust=1638971436649000&usg=AOvVaw1Wadj23Bavue5Sm1Qe1Ea9" href="https://www.wevolver.com/profile/syntiant" target="_blank">Syntiant</a> to bring support for the NDP101 processor to Edge Impulse Studio - giving developers a quick and efficient way to collect data, build models, and then deploy to the <a data-saferedirecturl="https://www.google.com/url?q=https://www.syntiant.com/tinyml&source=gmail&ust=1638971436649000&usg=AOvVaw2CJOwWnOHkiLeqNFjAEFQe" href="https://www.syntiant.com/tinyml" rel="nofollow" target="_blank">Syntiant TinyML Board</a>.Best of all, you can collect audio data directly through the Edge Impulse Studio. The TinyML Board supports the Arduino environment to make it easier for developers to get started.<iframe width="640" height="360" src="https://www.youtube.com/embed/uK3v8yXynhY?&amp;wmode=opaque&amp;rel=0&amp;enablejsapi=1&amp;origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe>

Edge Impulse recently announced official support for the Syntiant NDP101 processor, an always-on sensor and speech recognition processor, with ultra-low power consumption of less than 140 uW while recognizing words. 

Build Speech, Audio and Sensor Machine Learning Applications with Syntiant's TinyML Platform and Edge Impulse

<h2 dir="ltr" id="isPasted">Introduction</h2>Artificial Intelligence (AI) refers to the ability of a system to learn new things. There are several approaches to do so, but Machine Learning (ML) is the one that is the centre of attraction these days. It involves training a neural network with a lot of data to make it understand a situation and make some decisions.Traditionally, an AI-powered system acquires analog signals with the help of a transducer. This can be acoustic, visual, chemical, thermal or any signal available in nature the transducer is developed for. In the next step, the data is conditioned by the microprocessor and sent to the cloud servers with the help of a wireless transmission link. Cloud servers powered by multiple interconnected CPUs and GPUs process this data and share the results back.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM2OTg1MzE4NjUxLXNlcnZlcl9yb29tLTIuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 300px;" class="fr-fic fr-dib" alt="servers-data-centres-tinyml">Conventional ML requires servers and data centres with high compute power</div>While most ML and training used to happen on powerful cloud servers, over the last couple of years, the experiments have led us to a point where some part of the ML can be performed on the edge, right where the data source is.In this article, we are going to discuss the fundamentals, key advantages and applications of TinyML. Through this article, we have tried to give readers a basic idea of the TinyML project development flow and how the technology holds the potential to revolutionise the way data is processed by electronic systems.But what exactly led us to a point where we started trying to fit machine learning algorithms running on clouds into edge computing devices, bringing together such isolated fields? Let’s find out.<h2 dir="ltr">What is TinyML good for?</h2>By taking machine learning to the edge, TinyML aims to improve the products on four fronts. They are:<ul><li dir="ltr">Privacy:&nbsp;Consider, for example, smart home devices that always listen to the conversation you have at home while they wait patiently to listen to the hot word that activates them. One wouldn’t prefer to share their private conversations that take place at home/office with third-party entities. By running machine learning programs on the edge, it is possible to eliminate the risk of such a privacy incident.</li><li dir="ltr">Power:&nbsp;Transmission of data in Internet of Things (IoT) devices is probably the operation that consumes most of the energy. &nbsp;Processing data on the device cuts off a large portion of the energy consumed in the data communication layer.</li><li dir="ltr">Cost: By not transmitting data, the cost of setting up servers and the on-device radio is cut, which greatly brings down the overall price of the device.</li><li dir="ltr">Reliability:&nbsp;Not sending data anywhere means that the response time is significantly reduced. Processing on the edge also means that the least amount of data is lost during transmission. The problem of devices not being able to work during server maintenance or outrage is also solved.</li></ul>The case with embedded systems (especially the ones interacting with us) is that we expect these devices to respond to our actions instantly. Any delay in the response has a direct effect on user experience. Embedded systems are almost always powered by batteries. This means that the device should be able to function in an extremely energy-constrained environment. Embedded Systems also have very limited memory and a lot of sensors.So how can the issue of designing a system that can process so much data in such a constrained environment be solved? The answer is TinyML.<h2 dir="ltr">What is TinyML?</h2>TinyML stands for Tiny Machine Learning, a field of study lying at an intersection of Embedded Systems and AI that involves developing systems that run ML models on ultra-low-power microcontrollers. The idea is to push the implementation to where the information source is.There was a time when microcontrollers used proprietary programming languages and tools, which made it too intimidating for beginners and hobbyists to even start with the technology. Things have changed a lot since then. With the development of the Arduino platform, microcontrollers have become more accessible than ever. The introduction of ARM’s Cortex M series chips is also considered a prominent event since it made 32-bit microcontrollers mainstream.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM2OTg1Mzk0NTgzLWFyZHVpbm9fbWljcm9jb250cm9sbGVyLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="arduino-microcontroller-tinyml">Rapid microcontroller prototyping platforms like Arduino reduce the steep learning curve for getting started with embedded systems, IoT and TinyML</div>Collaborations between companies like Google, Arduino, ARM, Qualcomm, <a href="https://www.wevolver.com/profile/edge.impulse" target="_blank">Edge Impulse</a>, etc, is making it increasingly simpler with each passing day to get started with TinyML devices.<h2 dir="ltr">Overview of the development flow of TinML projects</h2>The initial stage of development involves designing and training a machine learning model in frameworks like TensorFlow. Google has introduced a special version of the tool called TensorFlow Lite, which comes with a binary footprint of less than 16 KB and doesn’t have any library dependencies.The generated model from TensorFlow Lite is combined with the application code with the help of Integrated Development Environments (IDEs) like Arduino IDE, Keil MDK and Mbed studio. Next up, the software compiles the code into a binary file before deploying it on the appropriate hardware. Many development boards like Arduino Nano 33 BLE, Sparkfun Edge, STM32F746NG MCU discovery kit, ESP32-DevKitC, Sony Spresense etc., can be used to develop TinyML projects.Don’t forget to check out the video links in the references section to find out more about getting started with TinyML projects.<h2 dir="ltr">Applications of TinyML</h2>TinyML can be used wherever conventional ML is required. But there are some specific areas where there’s no alternative to TinyML. Real-time applications, applications involving extremely personal data or applications where a continuous power source is not available are especially suited for TinyML. Here is a list of the most common applications of TinyML in domestic and industrial environments:Computer Vision: TinyML can be used for face recognition, text recognition and various applications where cameras are involved. It is possible to make TinyML devices that can respond to you when you look at them or wave at them. Appliances that detect the presence of someone, smartphones that use face unlock and all other devices which take a decision based on visual cues can be built with TinyML.Audio recognition &amp; Natural Language Processing: Small neural networks of size in the order of a few KBs are trained to detect wake words. These algorithms are deployed on a dedicated DSP chip and are used to process audio continuously, looking for the wake words.<div class="fr-img-space-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM2OTg1NDczNDI3LWVkZ2VfZGV2aWNlcy5qcGciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib" alt="google-home-amazon-echo-tinyml">Smart home appliances and smartphones come with a dedicated chip for identifying wake words locally</div>The chip requires just a few milliWatts (mWs) of power which is far less compared to what a typical smartphone processor uses. And since the data doesn’t leave the device, privacy concerns are largely addressed. Google Assistant’s implementation of “OK Google” wake word detection remains one of the most iconic TinyML applications ever.Time-Frequency Signal analysis:&nbsp;ML, in general, performs well at finding patterns in messy data. By training models with proper data sets, it is possible to monitor the condition of electrical machines.&nbsp;Motors and Drives can be fitted with TinyML devices that monitor the vibrations and other electromechanical parameters to detect an anomaly in operation for intelligent and predictive maintenance.TinyML applications can potentially be applied to all the sectors in some way or the other. Energy, retail, financial services, travel, tourism, hospitality, manufacturing, healthcare, life sciences are just some fields where the technology can bring massive improvements.<h2 dir="ltr">TinyML for good</h2>TinyML for good is an online conference showcasing some inspirational thinkers that used TinyML to develop innovative solutions and products in the field of education, healthcare and the environment sectors.&nbsp;Visit the event page of <a href="https://www.tinyml.org/event/tinyml-for-good" target="_blank">TinyML for good</a> to know more about it.&nbsp;Alternatively, you can directly head over to the <a href="https://us02web.zoom.us/webinar/register/6716342408482/WN_cquU_7lRRzOOKhzvVX2X-Q" target="_blank">registration page</a> to get yourself enrolled for the event. Dive into the world of TinyML and learn about the possibilities of the technology on 17th November 2021. No prior technical experience is required.<img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjM2OTgyMTE0OTQ4LVRpbnlNTGZvckdvb2QgKDQwMDAgeCA3NTAgcHgpLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 300px;" class="fr-fic fr-dib" alt="TinyML-for-good-event-logo">About TinyML FoundationWith its workshops, webinars, expert talks, research conferences, competitions, and a plethora of other events, TinyML Foundation enables students, researchers, and industry personnel to come together and share their experiences about the technology. It is the incredibly collaborative nature of the community that allows it to advance rapidly.<h2 dir="ltr">Conclusion</h2>We can arrive at a consensus that the research on TinyML devices and algorithms is fueled by the growing applications of the technology. It wouldn’t be an understatement to say that in the next couple of years, we can see this technology going to every portable electronic device there is. It is really exciting to witness how things change in unique and unexpected ways.The <a href="https://www.wevolver.com/article/tinyml-for-good-where-machine-learning-meets-edge-computing" target="_blank">introductory article</a> familiarised the readers with the concept of tinyML.The <a href="https://www.wevolver.com/article/tinyml-unlocks-new-possibilities-for-sustainable-development-technologies" target="_blank">first article</a>&nbsp;explains some possibilities of tinyML applications in sustainable technologies.The <a href="https://www.wevolver.com/article/applications-of-tinyml-in-healthcare-for-a-safer-future/" rel="noopener noreferrer" target="_blank">second article</a> is about tinyML applications aiding healthcare and medical research.The third and final article shall cover the tinyML applications in pedagogy and education.<hr><h2 dir="ltr">References</h2>[1] M. Shafique, T. Theocharides, V. J. Reddy and B. Murmann, "TinyML: Current Progress, Research Challenges, and Future Roadmap," 2021 58th ACM/IEEE Design Automation Conference (DAC), 2021, pp. 1303-1306, DOI: 10.1109/DAC18074.2021.9586232.[2] Banbury et al. (2020). Benchmarking TinyML Systems: Challenges and Direction.[3] TensorFlow Lite team. (2021). On-device training in TensorFlow Lite. Available from: <a href="https://blog.tensorflow.org/2021/11/on-device-training-in-tensorflow-lite.html" target="_blank">https://blog.tensorflow.org/2021/11/on-device-training-in-tensorflow-lite.html</a>[4] "Get Started with TinyML Webinar". &nbsp;2021. YouTube. Uploaded by: Hackster.io. <a href="https://www.youtube.com/watch?v=7OCIm_gh55s" target="_blank">https://www.youtube.com/watch?v=7OCIm_gh55s</a>

The article covers the basics of TinyML, a technology that improves the privacy, energy efficiency, affordability and reliability of devices utilising artificial intelligence. Be a part of TinyML for good, an online event and showcase of ideas that aims at exploring the topic in more detail.

TinyML for Good: where machine learning meets edge computing

Edge Impulse-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Edge Impulse-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Edge Impulse-Mon Oct 30 2023 14:34:39 GMT+0200 (Eastern European Standard Time)

Edge Impulse-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Plainsight

APSX LLC

Photoneo-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Photoneo-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Photoneo-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Photoneo-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Photoneo-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Photoneo-Wed Sep 20 2023 08:53:41 GMT-0400 (Eastern Daylight Time)

Photoneo-Wed Sep 20 2023 08:54:57 GMT-0400 (Eastern Daylight Time)

Photoneo-Wed Sep 20 2023 08:55:20 GMT-0400 (Eastern Daylight Time)

Photoneo-Wed Sep 20 2023 09:08:47 GMT-0400 (Eastern Daylight Time)

Photoneo-Tue Oct 31 2023 23:59:59 GMT+0200 (Eastern European Standard Time)

Photoneo-Mon Oct 30 2023 08:42:19 GMT-0400 (Eastern Daylight Time)

Matroid-Fri Jun 30 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Mon Jul 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

Matroid-Thu Aug 31 2023 23:59:59 GMT+0200 (Central European Summer Time)

Matroid-Sat Sep 30 2023 23:59:59 GMT+0300 (Eastern European Summer Time)

embedded machine learning

embedded machine learning

Profiles