A short guide to understanding the edge neural network design. 


Neural Network Architectures at the Edge: Modelling for Energy Efficiency and Machine Learning Performance

<p><em id="isPasted">This article is based on two blogs by Rouzbeh Shirvani and Sean McGregor, with additions and editing by John Soldatos.&nbsp;</em>&nbsp;</p><style type="text/css" id="isPasted">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; text-align: justify; font: 11.0px Arial; color: #000000} p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; text-align: justify; font: 11.0px Arial; color: #000000; min-height: 12.0px} p.p3 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px Arial; color: #000000; min-height: 16.0px} p.p4 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Arial; color: #000000; min-height: 14.0px} p.p5 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Arial; color: #000000} p.p6 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Arial; color: #000000; min-height: 12.0px} li.li1 {margin: 0.0px 0.0px 0.0px 0.0px; text-align: justify; font: 11.0px Arial; color: #000000} span.s1 {font: 11.0px Symbol} span.s2 {text-decoration: underline ; color: #0000ff} span.s3 {text-decoration: underline} span.s4 {font: 7.3px Arial} ul.ul1 {list-style-type: disc}
</style><p>In recent years, there is a surge of interest in Machine Learning (ML) and Artificial Intelligence (AI) solutions. This interest is driven by advances in parallel hardware, scalable software systems, advanced machine learning frameworks, as well as cloud/edge computing. Furthermore, the explosion of digital data collected makes it possible to develop more effective and accurate ML systems based on deep neural networks. This is because deep learning (DL) outperforms classical ML when large volumes of training data are available.&nbsp;</p><p>State-of-the-art ML/DL systems span both cloud computing and edge computing infrastructures, and can have varying characteristics, such as ML model performance (e.g., classification accuracy), training speed, and energy efficiency. These characteristics determine whether a system can meet specific application requirements such as latency and environmental performance. In most cases it is not possible to optimize all these parameters simultaneously. Therefore ML/DL developers often have to determine the relevant trade-offs and optimize the performance according to their specific application needs.&nbsp;</p><p>The choice of processors is key in setting the performance characteristics of a model. Larger, more power-hungry processors may appear to be the higher performing option, but specifically designed neural processors, or accelerators, often provide the better power/performance tradeoff. Complicating matters, deep learning systems are statistical and non-deterministic, which means that there is rarely a single optimal system configuration that yields the best performance and accuracy. Rather, it is possible to deploy alternative configurations of hardware and algorithms with comparable performance and efficiency.</p><h2><strong>Introducing ML/DL and Neural Architectures Benchmarks</strong></h2><p>The variety of available neural network architectures, configurations, and performance-related parameters makes it challenging to compare alternative ML/DL products and services. To facilitate this, engineers often turn to proven benchmarks for compute intense and data intense systems. For example:</p><ul><li>The <a href="http://www.spec.org/spec/faq/#01SPEC.General.10whatis" target="_blank">Standard Performance Evaluation Corporation (SPEC</a>) family of benchmarks focuses on computational workloads over different computing architectures including cloud-based systems.</li><li>The <a href="https://netlib.org/utk/people/JackDongarra/faq-linpack.html" target="_blank">LINPACK Benchmarks</a> provide measures of a computer&rsquo;s floating-point rate of execution i.e., they focus on floating-point computing power.&nbsp;</li><li>The <a href="https://www.tpc.org/" target="_blank">TPC consortium</a> includes industry leaders in computing systems and datasets, which have provided a suite of benchmarks for stress testing the transaction processing and data warehousing capabilities of data intensive systems.</li><li>The <a href="https://hpcg-benchmark.org/" target="_blank">High Performance Conjugate Gradients (HPCG)</a> benchmark provides novel metrics that enable the evaluation and ranking of High Performance Computing (HPC) systems.</li></ul><p>These benchmarks aid in evaluating different components of ML/DL systems, which are typically deployed over computationally intensive platforms. However, they are overly focused on conventional computational workloads and hence not sufficient for training and executing deep learning tasks that are typically more complex. &nbsp;</p><p>To provide proper benchmarks for ML systems an end-to-end approach is needed. It must consider all the different steps that are involved in the development and execution of an ML/DL pipeline i.e., from the training of an accurate model to the execution of different implementation and computation options towards optimizing system performance (e.g., execution speed).&nbsp;</p><p>Independent benchmarks provide a credible and impartial evaluation of alternative solutions, providing stakeholders with invaluable insights on the available market offerings. Moreover, they provide guidance for the co-design of ML/DL solutions by ML modelers, data scientists and embedded systems engineers.&nbsp;</p><p>Specifically, the benchmarks provide these teams with tangible metrics that can help them evaluate alternative system designs for a variety of popular use cases. Furthermore, ML benchmarking suites enable market players to compete against the same set of metrics, fostering innovation and healthy competition.&nbsp;</p><h2><strong>The MLPerf Benchmarking Suite</strong></h2><p><a href="https://mlcommons.org/en/" target="_blank">MLPerf is one of the most popular benchmarking suites</a> for ML/DL systems. It is comprised of a rich set of system tests that span ML models, software, hardware, and the power efficiency characteristics of an ML/DL system. MLPerf was established in 2018 by several organizations including Google, Harvard University, and Stanford University. Since then, it has greatly boosted innovation in the ML space and has helped vendors deliver high-performance and energy efficient ML/DL systems.</p><p>The suite includes several benchmarks for different ML tasks. Each benchmark comprises the following characteristics:</p><ul><li><strong>The name of the task</strong> (e.g., Image Classification), which indicates the benchmarking context. For instance, the Image Classification task aims at selecting the class that best describes the content of an image.</li><li><strong>The dataset</strong>, which is typically a popular dataset for the task at hand. For instance, the <a href="https://www.image-net.org/" target="_blank">ImageNet database</a> is commonly used for benchmarking Image Classification tasks.</li><li><strong>The model to be used</strong>, which identifies the ML/DL model that will be used to gauge the computational performance of the ML hardware and software system. As an example, several variations of the Residual Network (ResNet) are commonly used to benchmark hardware performance in computer vision tasks like Image Classification.</li><li><strong>The Quality Threshold</strong>, which denotes a level of acceptable quality for the systems that are benchmarked. For instance, a 74.9% classification accuracy can be used as the threshold for placing an ML/DL system in the Top-1 accuracy.</li></ul><p>Along with the above-listed parameters, MLPerf standardizes and specifies other aspects of the benchmarking process such as the initialization of parameters, the optimizer schedule to be used, and data augmentation processes. In this way, it ensures that the benchmarked systems are comparable and can be ranked in an unbiased way.&nbsp;</p><p>The results of the MLPerf benchmarking process are published in a transparent way. This makes it easy for market stakeholders to understand how different systems perform in the various tasks and how they compare to each other. The publication of the results for each system includes information about the system&rsquo;s benchmark scores, its market availability, the models used (e.g., <a href="https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md" target="_blank">MobileNet-v1</a> for mobile phones and <a href="https://en.wikipedia.org/wiki/Residual_neural_network" target="_blank">ResNet-50</a> v1.5 for bigger accelerators), as well as the scenarios and configurations of the benchmarking process.&nbsp;</p><p>The scenarios are destined to cover different deployment configurations, including inference at the edge and inference at cloud data centers. Moreover, there is information about the number of accelerators used by the systems under comparison, as the MLPerf benchmark is designed to rank entire systems and not individual components.</p><p>In practice, the process of running the MLPerf benchmarking is quite straightforward. It starts with cloning the MLPerf repository and setting up the configuration parameters. Accordingly, the system under benchmarking is added to the list of available system and a proper Docker image for the tests is built. The latter enables data preprocessing, as well as execution of validation models and tests. &nbsp;</p><h2><strong>Syntiant&rsquo;s Performance in the MLPerf Benchmarking</strong></h2><p><a href="https://www.eetasia.com/syntiant-leads-tinyml-benchmark-results/" target="_blank">Syntiant is strong leader in tinyML keyword spotting</a> according to the respective MLPerf benchmarking process.&nbsp;&nbsp;The company&rsquo;s performance in the benchmark is summarized as follows:</p><ul><li><strong>Achieved Performance</strong>: Syntiant&rsquo;s NDP120 ran the <strong>tinyML keyword spotting benchmark</strong> in 1.80 ms. This is a remarkably fast inference, making the company the clear winner in this category.&nbsp;</li><li><strong>Energy Usage</strong>: The benchmarked system used 49.59 &mu;J of energy at 1.1V/100 MHz. Energy efficiency was further improved by setting supply voltage down to 0.9 V and reducing clock frequency to 30 MHz, resulting in only 35.29 &mu;J of energy being consumed. Of course, reducing the clock frequency increases the latency to 4.30 ms, but this highlights both the energy and latency floor of the device, leaving it to the system designer to optimize power and performnce based on their specific application needs.&nbsp;</li></ul><p>The NDP120 is based on Syntiant&rsquo;s second generation AI accelerator core, Syntiant Core 2<sup>&reg;</sup>, alongside a Tensilica HiFi DSP (Digital Signal Processor) for feature extraction and an Arm Cortex-M0 core for system management. Syntiant did not enter any other benchmark scores for the NDP120, as the chip was architected for time-series data inferences, such as speech and audio events.&nbsp;</p><p>Syntiant Core 2<sup>&reg;</sup>, is a highly flexible, ultra-low-power deep neural network inference engine with a highly configurable audio front-end interface. Thanks to this front-end interface, NDP120 is very easily programmable. The ease of deployment is further aided by its native support for all major neural network architectures, allowing for the direct execution of neural network layers without intermediary processing steps.&nbsp;</p><p>NDP120 can run multiple models, such as a keyword spotting model and an audio event detection model, concurrently and supports flexible feature extraction and acoustic far-field processing with the included DSP. Using multiple neural network layers, the system provides accurate high-performance inference and classification, which supports use cases in a variety of application. For instance, it can accurately capture voice commands (e.g., Alexa prompts) in very noise environments.</p><p>The device is ideal for ultra-low-power, always-on, edge applications, where an inference based on time series audio or sensor data is required. The deployed solutions using the NDP120 span market &nbsp;verticals in consumer and industrial, such as security devices detecting acoustic events, mobile phones, tablets, smart speakers, smart displays, as well as smart home appliances and security devices. As with all Syntiant Core 2 family of solutions, it provides optimized memory access and at-memory compute, ensuring high efficiency and extremely low power consumption. The latter is clearly evident in the benchmarking of energy consumption across its peers.</p><p><strong>For more information about Syntiant NDP120 visit: <a href="https://www.syntiant.com/ndp120" rel="noopener noreferrer" target="_blank">https://www.syntiant.com/ndp120</a></strong></p><h2 id="isPasted"><strong><em>About the sponsor: Syntiant</em></strong></h2><p><em>Syntiant Corp. is a leader in delivering end-to-end deep learning solutions for always-on applications by combining purpose-built silicon with an edge-optimized data platform and training pipeline. The company&rsquo;s advanced chip solutions merge deep learning with semiconductor design to produce ultra-low-power, high-performance, deep neural network processors for edge AI applications across a wide range of consumer and industrial use cases, from earbuds to automobiles. Syntiant&rsquo;s Neural Decision Processors&trade; typically offer more than 100x efficiency improvement, while providing a greater than 10x increase in throughput over current low-power MCU-based solutions, and subsequently, enabling larger networks at significantly lower power.</em></p><p><em><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjYzMzU5OTg2NjI5LVVudGl0bGVkIGRlc2lnbi02ICgxKS5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 300px;" class="fr-fic fr-dib"></em><br></p>

Proven benchmarks provide a structured method for comparing ML/DL products and services. 


The Art and Science of Benchmarking Neural Network Architectures

<p dir="ltr" id="isPasted">Over the last decade, deep neural networks have emerged as the solution to several AI complex applications from speech recognition and object detection to autonomous vehicular systems. In essence, deep learning isn’t a single algorithm, but a family of algorithms with many layers that lack transparency, making it hard to understand the error, especially a downside for areas where these errors have a high cost. Most of the complexity in these deep neural networks comes from the input dataset which is used to make the training and inference faster and more efficient. Measuring input simplicity has been one of the lines of research that have caught the attention of AI architects to devise a framework that not only improves performance but also focuses on interpretability.&nbsp;</p><p dir="ltr">The removal of certain parameters from the input data was an essential part of the research work to understand the correlation of a particular input region or frequency on the output performance. However, most of the existing work depended on random removal or exploiting the domain knowledge that made the inputs simpler on which network performance was analyzed. A team of researchers from the University Medical Center Freiburg and Google Brain [1] proposed a technique for the model to synthesize the inputs that retrain prediction-relevant information. Through this, without relying on prior knowledge, the model can achieve a state of intuition for simplifying the input. Some of the findings reported [2,3] that in density-based anomaly detection, the generative image models usually assign higher probability densities and lower bits for visually simpler inputs.</p><h2 dir="ltr">What does simplification of input mean?</h2><p dir="ltr">In the paper, “When less is more: Simplifying inputs aids neural network understanding,” the team designed SimpleBits, a framework that synthesizes the inputs to contain less information without affecting the output performance. The approach measures simplicity with the encoding bit size that is given to the pre-trained generative model and reduces the bit size to simplify the inputs. Additionally, the framework can be used to analyze the trade-off between the information content and the task performance. After training neural networks, SimpleBits can act as an analysis tool to understand what information was used by the model to make an optimized informed decision.&nbsp;</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 766px;"><span class="fr-img-wrap"><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1645493691498-Overview+of+Simplification+of+Inputs+for+Deep+Neural+Networks.png" style="width: 800px;" class="fr-fic fr-dib" alt="Overview of Simplification of Inputs for Deep Neural Networks"><span class="fr-inner" spellcheck="false">Overview of Simplification of Inputs for Deep Neural Networks [Image Credit: Research Paper]</span></span></span></p><div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 477.5px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'><br></div><p><br></p><p dir="ltr">The evaluation of simplification of the inputs was performed in different settings: conventional training, dataset condensation and post-hoc analysis. The image above depicts the tasks covered in the paper, starting with SimpleBits acting as a per-instance simplifier during the training, where each image in the dataset is simplified but the total number of images remains unchanged. In simplifying the input, the framework removes superfluous information for tasks with injected distractors, while on a natural image dataset, the framework highlights task-relevant information. The chosen dataset contained both useful and redundant information for task learning, helping the researchers to evaluate the model accuracy over different datasets. By increasing the simplification process, the researchers noticed an accuracy drop reporting of the trade-off between input simplification and task performance for different setups.&nbsp;</p><p dir="ltr">For dataset simplifications with condensation, the dataset is compressed to a few samples per class which are simplified at the same time. In this setting, the training data are converted into a smaller set of synthetic images which are further simplified to reduce the encoding size without affecting the task performance. Finally, for the post-training setting, SimpleBits can act as a post-hoc analysis tool used for auditing processes. “Our exploration of SimpleBits guided audits suggests it can provide intuition into model behavior in individual examples, including finding features that may contribute to misclassifications,” the team notes.&nbsp;</p><h2 dir="ltr">Summary of simplified inputs that aid neural network understanding&nbsp;</h2><ol><li dir="ltr"><p dir="ltr">An information-based method to simplify the inputs while maintaining the output performance and neural network accuracy. This can be done without any prior knowledge or expertise.</p></li><li dir="ltr"><p dir="ltr">The framework is able to remove injected distractors, retaining accuracy while reducing the complexity of the input dataset images.</p></li><li dir="ltr"><p dir="ltr">It can be used as an analysis tool to monitor the trade-off between task performance and input simplifications to check if any essential information is not removed.&nbsp;</p></li></ol><h2 dir="ltr">References</h2><p dir="ltr">[1] Robin Tibor Schirrmeister, Rosanne Liu, Sara Hooker, Tonio Ball: When less is more: Simplifying inputs aids neural network understanding. DOI <a href="https://arxiv.org/abs/2201.05610" target="_blank">arXiv.2201.05610</a> [cs.LG]</p><p dir="ltr">[2] Kirichenko, P., Izmailov, P., and Wilson, A. G. Why normalizing flows fail to detect out-of-distribution data. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.&nbsp;</p><p dir="ltr">[3] Schirrmeister, R., Zhou, Y., Ball, T., and Zhang, D. Understanding anomaly detection with deep invertible networks through hierarchies of distributions and features. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.&nbsp;</p>

SimpleBits framework for Simplifying of Inputs for Deep Neural Networks

Over the last decade, deep neural networks have emerged as the solution to several AI complex applications from speech recognition and object detection to autonomous vehicular systems.

How to simplify the inputs for better neural network performance?

<p dir="ltr" id="isPasted">We have already covered a detailed article on <a href="https://www.wevolver.com/article/smart-face-mask-platform-in-response-to-covid-19-pandemic" target="_blank">the smart face mask platform</a> integrating software and hardware to offer a user-friendly platform– that displays physiological signals [1]. With Covid-19 rising in several countries, face masks have become a part of day-to-day life– prolonged use of these face masks has adverse effects, primarily caused due to CO2 rebreathing. There have been reports concluding a significant drop in blood oxygen saturation and increase in heart and respiratory rate due to the increased CO2 concentration in the dead space volume (DSV). DSV is defined as the volume of ventilated air in which there is no significant gas exchange of oxygen and carbon dioxide.&nbsp;</p><p dir="ltr">The established mechanism to monitor the concentration or partial pressure of carbon dioxide in respiratory gases– capnography and other portable hardware to measure the gas inside the mask DSV have created complex issues for the required components and its bulky structure. Even though the proposed solutions are reliable with high resolution and accuracy, they are limited to laboratory experiments due to their size and power requirements. To solve the challenges in the space of wearability and conformability, utilizing the recent technological advancements and exploiting the developed research reports, a team of researchers, affiliated with the University of Granada, Spain, have proposed a system with wearable hardware that interacts with the custom smartphone application via a near-field communication link (NFC) [2].</p><h2 dir="ltr">NFC tag for Wireless CO2&nbsp;readout</h2><p dir="ltr">In the paper, “Smart facemask for wireless CO2 monitoring” the team has developed a system that consists of an opto-chemical sensor combined with a battery-free NFC tag with a resolution of 103 ppm for measuring the gas concentration in the DSV of the facemask. The sensor circuitry is fabricated in a tiny 60x45 mm2 sized polymeric substrate with the NFC link generated for the interaction with the custom-developed software application. The smartphone application is responsible for data processing, alert management, result displaying and sharing purposes. “Through performance tests during daily activity and exercise monitoring, we demonstrate its utility for non-invasive, wearable health assessment and its potential applicability for preclinical research and diagnostics,” the team explains in the research article.</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 766px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ1MzM2MjExNDIyLUNpcmN1aXQgYmxvY2sgZGlhZ3JhbSBvZiB0aGUgTkZDIHRhZy5wbmciLCJlZGl0cyI6eyJyZXNpemUiOnsid2lkdGgiOjk1MCwiZml0IjoiY292ZXIifX19" style="width: 800px;" class="fr-fic fr-dib" alt="Circuit block diagram of the NFC tag"><span class="fr-inner" spellcheck="false">Circuit block diagram of the NFC tag [Image Credit: Research Paper]</span></span></span></p><div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 477.5px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'><br></div><p>While designing the portable hardware, the general design criteria were based on the IoT design flow– sensing, connectivity, compactness, integration, scalability, low-power consumption and affordability. Keeping all this into consideration, the proposed system with an NFC tag printed on a transparent flexible substrate is fixed on the inner part of the FFP2-type facemask. The goal is to read out the CO2 concentration through the deposited chemical sensor with the fluorescent response of which results will be wirelessly transferred onto the software applications. Based on stable inorganic phosphor, the opto-chemical CO2 sensor is the most essential hardware component in the whole hardware circuitry. Being an NFC-based system, the monitoring is completely passive with no requirement of battery– whereas the power supply is obtained from the near electromagnetic field induced by the external NFC reader (in this case the smartphone).&nbsp;</p><p dir="ltr">In the process for CO2 monitoring, the NFC tag has an attached ultraviolet light-emitting diode, which when powered on, the sensing substrate is active. The luminescence intensity is measured to determine the CO2 concentration through the red coordinate of the RGB color space. The incoming luminescence intensity is collected by the digital color detector then sent to the microcontroller unit. The final results will be sent to the mobile application for further processing.&nbsp;</p><h2 dir="ltr">Did performance prove its utility?</h2><p dir="ltr">For evaluation purposes, the smart facemask was tested on short-term and long-term usage in static conditions. For the initial test case, the CO2 concentration was monitored every 30 seconds through the proposed smart face mask as well as a reference instrument such as MultiRAE Lite, RAE Systems. By comparing the results, the curves show to be in good agreement with close results. In the early stage of the monitoring, the face smart curve is slightly in a delayed fashion in comparison to the reference standard– this is because of the time required to fill the mask’s DSV. Three apnoeas can be seen when the holding of breath during the test was performed to see the varying results and accuracy.</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 766px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ1MzM2NDI0MDA0LVNtYXJ0IGZhY2VtYXNrIHBlcmZvcm1hbmNlLnBuZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 800px;" class="fr-fic fr-dib" alt="Smart Face Mask Performance"><span class="fr-inner" spellcheck="false">Smart Face Mask Performance: Short Term (left) and Long Term (right) [Image Credit: Research Paper]</span></span></span></p><div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 286.5px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'><br></div><p>For the long-term test, the face mask was worn for over 2.5 hours and the measurements were taken every 15 minutes. The monitoring is performed with temperature values into consideration to check the stability of the opto-chemical sensor. Two apnoeas at the stamp time of 80 and 110 minutes show a sharp drop in CO2 concentration. Overall, the CO2 accumulation for common conditions ranged from 1.5% to up to 3.5% which is well beyond the atmospheric air of approximately 0.04%. “Our results are in line with the reviewed literature, with CO2 values around 2% during low work-rate activities and peak values up to ~5% during high-intensity exercise,” the team notes.&nbsp;</p><p dir="ltr">In summary, the developed platform for CO2 monitoring with batteryless hardware structure enabling NFC-based wireless communication and scalable hardware fitted inside the FFP2 face mask makes the system versatile and robust for real-life deployment. For future work, the researchers expect to combine several optical chemical sensors that can be integrated into a single flexible passive NFC tag to monitor several gases' concentrations.&nbsp;</p><h2 dir="ltr">References</h2><p dir="ltr">[1] Alexander Curtiss, Blaine Rothrock, Abu Bakar, Nivedita Arora, Jason Huang, Zachary Englhardt, Aaron-Patrick Empedrado, Chixiang Wang, Saad Ahmed, Yang Zhang, Nabil Alshurafa, and Josiah Hester. 2022. FaceBit: Smart Face Masks Platform. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4, Article 151 (Dec 2021), 44 pages. DOI: <a href="https://doi.org/10.1145/3494991" target="_blank">https://doi.org/10.1145/3494991</a></p><p dir="ltr">[2] P. Escobedo, M. D. Fernández-Ramos, N. López-Ruiz, O. Moyano-Rodríguez, A. Martínez-Olmos, I. M. Pérez de Vargas-Sansalvador, M. A. Carvajal, L. F. Capitán-Vallvey, and A. J. Palma. Smart facemask for wireless CO2 monitoring. Nature Communication 13, Article 72 (Jan 2022). DOI: <a href="https://doi.org/10.1038/s41467-021-27733-3" target="_blank">https://doi.org/10.1038/s41467-021-27733-3</a>&nbsp;</p>

Smart Face Mask for CO2 monitoring through an opto-chemical sensor; results displayed on Smartphone application

A system with an opto-chemical sensor combined with a battery-free NFC tag of resolution 103 ppm for measuring the CO2 gas concentration in the dead space volume of the facemask

Wearable and Intelligent Face Mask for CO2 monitoring

<p dir="ltr" id="isPasted">Mutations in the virus have given birth to multiple research streams for the treatment of viral infection. COVID-19 caused because of the SARS-CoV-2 virus, has changed the daily routine of human life– making it more conscious to protect itself from the global pandemic. Most of the day-to-day safety protocols in place are in the form of passive devices; limit the scope of improvement and cannot alert the user for potential risk. Wearing masks has been the most used protection against the SARS-CoV-2 virus infection– but lacks the technology to signal the user for mask degradation of improper fit. The innovation towards building smart devices that are scalable, lightweight and energy-efficient has been on the rise after realizing the need for active devices.</p><p dir="ltr">In the space of the maker community, there have been efforts in designing hardware platforms that can extract physiological signals from facial sensing points. Smart glasses [2] for measuring the heart rate to smart filter [3] for PPE to monitor the air filter quality in the detection of COVID-19 have played a significant role in wearable physiological sensing. Most of these solutions lacked scalability and on-device computing capabilities– in response to the COVID-19 spread. All these resource-constrained computing platforms and software/hardware ecosystem helped the researchers to develop an efficient and user-friendly smart device.</p><p dir="ltr">A team of researchers from Northwestern University, Georgia Institute of Technology, and the University of California at Los Angeles have proposed an open-source hardware platform for smart face mask applications. The increasing new highs of COVID-19 positive cases due to new infectious variants, and analyzing the slow global vaccination rate along– make this smart hardware platform a way to keep monitoring health parameters. Thanks to edge computing and AI inference advancements, the smart face masks platform creates a new research area for developing sustainable and convenient forms of health management, even for the COVID-19 pandemic.</p><h2 dir="ltr">What is FaceBit?&nbsp;</h2><p dir="ltr">In the paper, “FaceBit: Smart Face Masks Platform”, the team proposes the design of a hardware and software ecosystem consisting of FaceBit circuit board along with a custom-built mobile application that can provide data collection, visualization and alert. Along with alertness, the hardware interacts with physiological changes– monitoring heart rate (through FaceBeat algorithm) without skin contact via ballistocardiography and respiration rate through temperature changes. Additionally, the hardware also checks the mask fitting and wear time from pressure signals; leveraging the always on-device with an energy-efficient system. All the data collected from the onboard sensors are wirelessly transmitted to the custom phone application for analysis.&nbsp;</p><p dir="ltr">Scaling such a product involves several factors that are to be considered– FaceBit aims to be low cost, energy-efficient and easy-to-integrate hardware with rich sensing functionalities. The hardware offers harvesting energy from breathing, motion and sunlight to supplement the tiny primary cell battery that alone delivers a battery lifetime of 11 days or more. With the help of a small magnet, the tiny form factor device fits on top of your existing masks without any modification. &nbsp;Moreover, the inclusion of a flexible interface has made it more convenient for the users to monitor the health parameters and keep track of the COVID-19 infection risk.</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 766px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ0MzExMDA5ODg2LU92ZXJ2aWV3IG9mIEZhY2VCaXQucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 800px;" class="fr-fic fr-dib" alt="Overview of Facebit Smart Face Mask Platform"><span class="fr-inner" spellcheck="false">Overview of Facebit Smart Face Mask Platform [Image Credit: Research Paper]</span></span></span></p><div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 253.1px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'><br></div><p>The FaceBit block diagram shows the essential parts that make up the whole ecosystem consisting of three energy sources– solar, thermal and battery, while three wireless sensors (temperature sensor, IMU and pressure sensor) for monitoring physiological signals. All the incoming data from the remote sensors is then digitally processed and categorized under respiratory rate, heart rate and mask fitting. Wirelessly sent to user-controlled mobile applications to complete the design flow of the hardware/software smart mask platform.</p><p dir="ltr">On the other side of the hardware platform is the FaceBit Companion application for phones and desktops that interacts with the FaceBit circuit board via Bluetooth Low Energy using a custom GATT profile. Built on iOS and macOS using Swift Programming Language, the user interface is written within the SwiftUI framework and uses Mac Catalyst that supports iOS applications deployed on macOS. “The application handles both high-frequency time-series data streams for debugging, as well as low frequency computed metrics from the sensor board for actual deployment,” the researchers explain. “The vision is that a user will be able to customize the home screen with desired widgets to receive timely information on the signals of interest.”</p><p dir="ltr"><span class="fr-img-caption fr-fic fr-dib" style="width: 766px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjQ0MzExMTIwNTM0LUZhY2VCaXQgUGhvbmUgQXBwbGljYXRpb24ucG5nIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo5NTAsImZpdCI6ImNvdmVyIn19fQ==" style="width: 800px;" class="fr-fic fr-dib" alt="FaceBit Phone Application"><span class="fr-inner" spellcheck="false">FaceBit Phone Application [Image Credit: Research Paper]</span></span></span></p><div style='pointer-events: none; overflow: hidden; padding-bottom: 2px; z-index: auto; position: absolute; left: 0px; top: 467.15px; height: 39px; width: 766px; font-style: normal; font-variant: normal; font-weight: 400; font-stretch: 100%; font-size: 14px; font-family: "PT Serif", serif; text-align: center; text-transform: none; letter-spacing: normal; word-spacing: 0px; tab-size: 8;'><br></div><p>For future studies, the developers aim to optimize the signal pipeline to tune the FaceBeat algorithm to reduce the number of falsely detected heart rates. A battery-free system is something that can improve the sustainability of the device by integrating low-power modules like Ambiq Apollo3. As an open-source contribution, the innovators encourage the community to modify the hardware and software to bring more functionalities– available at <a href="https://facebit.health/" target="_blank">facebit.health</a>.&nbsp;</p><h2 dir="ltr">Reference</h2><p dir="ltr">[1] Alexander Curtiss, Blaine Rothrock, Abu Bakar, Nivedita Arora, Jason Huang, Zachary Englhardt, Aaron-Patrick Empedrado, Chixiang Wang, Saad Ahmed, Yang Zhang, Nabil Alshurafa, and Josiah Hester. 2022. FaceBit: Smart Face Masks Platform. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4, Article 151 (Dec 2021), 44 pages. DOI: <a href="https://doi.org/10.1145/3494991" target="_blank">https://doi.org/10.1145/3494991</a></p><p dir="ltr">[2] Hernandez, Javier et al. "Wearable Motion-Based Heart Rate at Rest: A Workplace Evaluation." IEEE Journal of Biomedical and Health Informatics 23, 5 (September 2019): 1920 - 1927</p><p dir="ltr">[3] Lut, J.-J. &amp; Goddijn, M. Smart Personal Protective Equipment: Sensing and Control: The Future of Face Masks (Delft University of Technology, Delft, 2020).</p>

FaceBit aims to be low-cost, energy-efficient, and easy-to-integrate hardware with rich sensing functionalities– monitors physiological signals

Smart Face Mask Platform in Response to COVID-19 Pandemic

<p dir="ltr" id="isPasted"><em>The <a href="https://www.linleygroup.com/events/agenda.php?num=52&day=1" target="_blank">Linley Processor Conference</a> is an annual event providing an overview of recent achievements and forecasting trends and challenges. The Linley Group hosted its Fall Processor Conference in person on October 20-21. &nbsp;Its virtual event featured live-streamed presentations on October 27-29 and November 3-5. The full conference can be accessed on-demand&nbsp;</em><a href="https://gateway.on24.com/wcc/eh/3377544/linley-fall-processor-conference-2021" rel="noopener noreferrer" target="_blank"><em>here</em></a><em>.</em></p><p>The recent conference featured a full program of talks and panel discussions covering a broad range of topics such as:</p><ul><li dir="ltr"><p dir="ltr">High-Performance Processors</p></li><li dir="ltr"><p dir="ltr">Server Acceleration</p></li><li dir="ltr"><p dir="ltr">SoC Design</p></li><li dir="ltr"><p dir="ltr">Edge-AI Processing &amp; Software</p></li><li dir="ltr"><p dir="ltr">Edge IP</p></li><li dir="ltr"><p dir="ltr">Low-Power AI</p></li><li dir="ltr"><p dir="ltr">DSP &amp; IoT</p></li><li dir="ltr"><p dir="ltr">FPGAs for AI</p></li></ul><p>In this article, we summarize a few of the featured keynote presentations.&nbsp;</p><h2>AI Moves from the cloud to the edge</h2><p dir="ltr">The conference kicked off with a keynote from Linley Gwennap, principal analyst for The Linley Group. He spoke about the latest trends in AI silicon as the technology moves from the cloud to the edge. AI has firmly evolved from a product to a feature, with AI acceleration appearing in server processors, PC processors, FPGAs, and a vast array of edge devices. These chips are making an impact across the sector. In the context of data centers, new chips focus on improving performance per watt. Architecture innovation is particularly rampant at the edge, where power limitations are stringent, and software stacks are smaller. Edge chips target a range of applications from battery-powered sensors to powerful camera-based systems. So-called ‘Conscious cameras’ use edge processing to make decisions based on object detection and facial recognition. These cameras can learn what they need to be seeing and send triggers when something is unrecognized or deemed a threat. Applications range from smart home monitoring to industrial quality control.&nbsp;</p><p><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1636049063870-Triton-Edge-SoC-all-sides1.png" style="width: 788px;" class="fr-fic fr-dib"></p><p dir="ltr"><em>Image 1: Edge processing in cameras will have a massive impact on security and industry. Image credit: Triton. &nbsp;</em></p><h2>Core Edge AI Processors: From In-Memory to At-Memory</h2><p dir="ltr"><a href="https://www.wevolver.com/profile/syntiant" target="_blank">Syntiant</a> co-founder and chief technology officer Stephen Bailey, Ph.D., presented a talk as part of the conference’s Low-Power Sensing and AI” session. Dr. Bailey demonstrated how Syntiant’s custom at-memory, deep learning processor architecture provides the required cost, performance, and efficiency necessary for deploying ultra-low-power, always-on applications.&nbsp;</p><p dir="ltr">“Syntiant was founded to deliver edge AI solutions with chips based on in-memory, analog deep-learning processing technology,” Bailey explains. &nbsp;“We discovered early in our journey that demand for these solutions was more immediate and diverse than anticipated, and that a custom at-memory, digital deep-learning processor architecture would provide the required cost, performance, and efficiency while providing chip-process portability.” His lecture described the Syntiant journey so far and shared architectural insights and performance comparisons of the Syntiant Core 2 processor found in their Neural Decision Processor (NDP) chips.&nbsp;</p><p dir="ltr"><img data-fr-image-pasted="true" src="https://lh5.googleusercontent.com/eJSpociG9qyBCOguEoEmTux_dQ7JYI4zCrMGKKxBIsVICZoUXaabh2UffgRws4Eij7xuiTqXlMZNdJTqAtNxY0k6mdY95Iuh2MZM2t8AmDAWDLP2C16k6aPp3j_QyhhHiN8-pOa2" width="602" class="fr-fic fr-dii" style="width: 756px; height: 707.104px;"></p><p dir="ltr" id="isPasted">Image 2: The tiny and powerful Syntiant chips. Image credit: Syntiant.&nbsp;</p><h2 dir="ltr">The SystemReady Standards Program: Enabling Systems Where Software ‘Just Works’</h2><p><a href="https://www.arm.com/#:~:text=Arm%20is%20the%20leading%20technology,%2C%20Neoverse%2C%20Ethos%20and%20SecurCore." target="_blank">Arm</a> aims to bring together software and hardware to accelerate IoT design and fuel a new IoT economy. Arm enables specialized processing built on the economics, design freedom, and accessibility of general-purpose compute. Arm lead standards architect and fellow, Dong Wei presented a talk titled “SystemReady Standards Program: Enabling Systems Where Software ‘Just Works’.” The lecture outlines the standards-based approach of Arm System Ready, which provides requirements that can be applied to a broad range of devices from cloud data centers to edge and IoT. It reduces or eliminates the cost of customization for the system firmware and kernel build.&nbsp;</p><h2>Achieving Real-Time High-Resolution Video Inference at Ultra-Low Power</h2><p><a href="https://roviero.com/" rel="noopener noreferrer" target="_blank">Roviero</a> is a deep-tech startup founded in early 2019 that has developed the CortiOne™ NPE (Neural Processing Engine) and integrated CortiSoft™ compiler technology. The company has a vision to enable AI in all edge devices through a hybrid hardware-software design. Despite massive leaps, AI implementation in edge devices is still very limited due to the power consumed by the compute and associated memory.&nbsp;</p><p>Models with good accuracy are still large in size and contribute to high memory and power consumption. &nbsp;Ravi Setty, SVP of Engineering and Cofounder of Roviero, discussed their solution to this problem; Roviero’s CortiOne NPE and compiler. The CortiSoft compiler optimizes memory access such that the model memory is in a sleep state 99% of the time, thereby dramatically reducing power consumption while maintaining full accuracy. Watch a demo of the compiler below.&nbsp;</p><p><span class="fr-video fr-deletable fr-fvc fr-dvb fr-draggable" contenteditable="false" draggable="true"><iframe src="https://www.youtube.com/embed/8QOA_3T_JjY?&amp;wmode=opaque&amp;rel=0" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 740px; height: 401px;"></iframe></span><br></p><h2>Summary</h2><p>The Linley Fall Processor Conference 2021 provided a robust look at the current state of processors. The significant focus on edge processing and AI demonstrates that the next few years will see significant growth and change in this area. The full conference is available to watch on-demand <a href="https://gateway.on24.com/wcc/eh/3377544/linley-fall-processor-conference-2021" rel="noopener noreferrer" target="_blank">here.</a></p><h2><em>About the sponsor: &nbsp;The Linley Group</em></h2><p dir="ltr"><em>The Linley Group is the industry's leading source for independent technology analysis of semiconductors for a broad range of applications including AI, networking, communications, data center, and embedded. The company provides strategic consulting services, in-depth analytical reports, and conferences focused on advanced technologies for chip and system design.&nbsp;</em></p><p><img src="https://s3-us-west-1.amazonaws.com/wevolver-project-images/froala%2F1636051457879-linley+dl.jpg" style="width: 300px;" class="fr-fic fr-dib"></p>

A rich line-up of speakers discuss the trends and challenges of the industry. 


From AI and beyond: Highlights of the Linley Fall Processor Conference 2021

Utilizing audio edge processors for efficient execution of  Machine Learning Tasks.

An engineer's guide to deploying machine learning in smart devices using audio edge processors

A closer look at the recent market trends and audio technologies that enhance the functionalities of TWS devices.

TWS Evolution: What is next in True Wireless Audio?

<p dir="ltr"><em>Wevolver and&nbsp;<a href="https://www.wevolver.com/profile/syntiant" target="_blank">Syntiant</a> are creating a series that explore the work of innovators and the future of pervasive AI. Syntiant is developing ultra-low-power AI processors. Because they believe in the importance of innovation, Syntiant is engaging in these fireside chats with engineers and designers who are on the cutting edge of their field.</em></p><p dir="ltr">The fourth conversation in our Voice Of Innovation series was held between Syntiant CEO Kurt Busch and Andreas Urschitz, Division President Power &amp; Sensor Systems at Infineon. Infineon is one of the largest semiconductor companies in the world, they have partnered with Syntiant for many years with the shared goal of innovating artificial intelligence at the edge. In this wide-ranging conversation, Kurt and Andreas discuss the potential of deep learning, strategies to improve efficiency, and the importance of system innovation.&nbsp;</p><h1 dir="ltr">Building the five senses into semiconductors</h1><blockquote><p dir="ltr">“Infineon is pursuing its vision of emulating the five human senses, eventually even the sixth human sense, into each and every kind of electronics.” - Andreas Urschitz.</p></blockquote><p dir="ltr">To make electronics understand in a meaningful way and without latency, such as understanding voice through a microphone and interpreting a specific command or making electronics understand a particular gas combination in the environment or make equipment smell, requires artificial intelligence at the edge through machine learning.&nbsp;</p><p dir="ltr">We have always tried to move the real world into the visual into the virtual world in some way, says Kurt, and with the advent of deep learning, we were able to do things very very quickly. For example, in a very short period of time speech recognition was developed, and within a couple of years, devices such as Amazon Echoes have appeared. This is the sort of thing that deep learning can bring to the table - both rapid innovation and entirely new types of interfaces.&nbsp;</p><h1 dir="ltr">Prioritizing sustainability&nbsp;</h1><p dir="ltr">With processing at the edge, no matter how good a device is there is going to be a demand for more, continues Kurt. What we are creating is a never-ending cycle of increased intelligence inside these devices and it's actually truly exciting because a solution that you have today that works well, you know that next year you are going to have 10 times the performance and that will generate seamless user interfaces.&nbsp;</p><p dir="ltr">Of course, this increase in compute has power requirements. According to Kurt and Andreas, both Syntiant and Infineon are dedicated to prioritizing ways to increase the efficiency of their solutions in order to strive for better environmental credentials and clear sustainability. Syntiant has ramped up its research and investment into power flow solutions, but just energy saving is not enough. What’s crucial is the data that is collected through the millions of sensors we are now surrounded by is analyzed and pre-filtered so that the actual amount of data that needs to be managed in the data center is reduced to the bare minimum. The solution to this is edge processing.</p><h1 dir="ltr">The importance of system innovation</h1><p dir="ltr">Digitalization is changing so many things in our environment, says Andreas. One of the things we have observed over the last couple of years is that formerly we are confronted with a world with so much information and technology available but often quite spread out over the planet.&nbsp;</p><p dir="ltr">Andreas: “We must ask ourselves how could you combine the different technologies - &nbsp;software, &nbsp;hardware - to create meaningful use cases that are helping humanity? We see systemic proceedings such as bringing partners together who are owners of different bits and pieces of technologies to come up with things that either solve customers’ problems or create opportunities as an essential way to make meaning of these technological advances. I believe there are a lot of chances for different companies to team up jointly to strive for system innovation to truly make an impact that goes beyond what is possible alone.”&nbsp;</p><hr><p>This video is part of a series of fireside chats in which Wevolver and Syntiant partner to engage with global innovators. In <a href="https://www.wevolver.com/tag/fireside-chat" rel="noopener noreferrer" target="_blank">previous discussions </a>we spoke with Arduino CEO Fabio Violante, Star Wars Animatronic Designer Gustav Hoegen and Fashiontech Designer Anouk Wipprecht. &nbsp;I encourage you to stay tuned and to follow <a href="https://www.wevolver.com/profile/syntiant" target="_blank">Syntiant's profile</a> for more of these conversations.</p>

#4 of our 'Voice of Innovation' fireside chat series, in which Syntiant CEO Kurt Busch talks with Andreas Urschitz Division President Power & Sensor Systems at Infineon.

Fireside Chat With Andreas Urschitz from Infineon

<p dir="ltr"><em>This article was co-authored by Raj Senguttuvan and Vikram Shrivastava.&nbsp;</em>&nbsp;</p><h1 dir="ltr">Introduction</h1><p dir="ltr">The last decade, but particularly 2020, saw a massive increase in video calling and engagement with voice-enabled devices primarily through the work-from-home trend created by the pandemic. However, our use of video calls isn&rsquo;t restricted to just work Zoom meetings. We are now using video calling for teaching, exercise classes, to experience live music, as a tool for interacting at conferences, and <a href="https://www.eatwith.com/" target="_blank">even to go to restaurants.</a> This virtual engagement occurs via laptops, smartphones, tablets, and other IoT devices such as Amazon Echo Show, Facebook Portal, Peloton, <a href="https://imp.i277339.net/c/1141873/823006/11505?u=https%3A%2F%2Ftempo.fit%2Fpdp%2Ftempo-studio&subId2=[]st[p]cjv2u8o050017o5yeyn1o93c5[i]H2ac18[t]w[r]google.com[d]D[z]m" target="_blank">Tempo Studio</a>, etc. We have also embraced the use of voice-controlled devices such as home assistants and wireless earphones, and other smart home applications.&nbsp;</p><p dir="ltr">The core point that limits enjoyable and engaging interactive audio, video call, or home assistant experiences is consistent sound quality in the presence of noise and other distractors. The ability of your device to intelligently manage sound is what makes or breaks your ability to communicate.&nbsp;</p><h2 dir="ltr">Increasing audio expectations</h2><p dir="ltr">Dubbed &lsquo;smart sound&rsquo; by product makers, audio intelligence is the ability of a device to process sound to deliver the best user experience. As the increase in voice-led device use for communication, entertainment, and health management increases, so does the demand for a seamless, low-barrier experience with increased functionality. Users now expect devices to understand more than simple wake or keywords (such as Alexa) and seek the ability to move between devices and applications with superior sound quality to enable an immersive and seamless experience, whether for professional conferencing or personal entertainment.&nbsp;</p><p dir="ltr">Devices should be able to combine your voice and/or voice command with your personal preferences and environmental data to adapt sound processing to your specific context. This is known as contextual awareness.&nbsp;</p><h1 dir="ltr">What is Contextual Awareness?</h1><p dir="ltr">Contextual-aware devices combine user-specific information, such as location, preferences, and ambient sensor data, to better understand what the user is asking for and more accurately execute functions in response to a certain command or trigger.&nbsp;</p><p dir="ltr">Always-listening devices use signal processing technology combined with <a data-lf-fd-inspected-kn9eq4roawkarlvp="true" href="https://mediatum.ub.tum.de/doc/1463108/document.pdf" rel="noopener noreferrer" target="_blank">machine learning</a> to differentiate between sound types such as natural sounds, voices, background interference, etc. These sounds are often divided into &ldquo;scenes&#39;&#39; and &ldquo;events&rdquo;. A scene is the users setting such as a noisy airport terminal or a quiet working space. The events are someone speaking, a glass breaking, or a dog barking. A context-aware device can process these groups of sounds to ensure the intent of the action, whether it is a video call or a voice command, and is processed for optimal experience.&nbsp;</p><p dir="ltr"><br></p><div class="fr-img-space-wrap"><span class="fr-img-caption fr-fic fr-dib" style="width: 788px;"><span class="fr-img-wrap"><img src="https://images.wevolver.com/eyJidWNrZXQiOiJ3ZXZvbHZlci1wcm9qZWN0LWltYWdlcyIsImtleSI6ImZyb2FsYS8xNjE0NzkxMjg1ODE2LUlNQUdFIDQ2LTAxLmpwZyIsImVkaXRzIjp7InJlc2l6ZSI6eyJ3aWR0aCI6OTUwLCJmaXQiOiJjb3ZlciJ9fX0=" style="width: 788px;" class="fr-fic fr-dib"><span class="fr-inner" spellcheck="false">Figure 1: Context-aware system schematic.</span></span></span><p class="fr-img-space-wrap2">&nbsp;Dedicated audio edge processors with a focus on audio fidelity and with machine learning optimized cores are the key to contextually aware and high-quality audio communication devices. These processors can deliver enough compute power to process complex audio using just a small percentage of the energy of a generic Digital Signal Processing (DSP) implementation. These audio edge processors use ML algorithms for superior sound processing in addition to providing a &nbsp;more contextual audio experience.&nbsp;</p></div><p><br></p><h2 dir="ltr">&ldquo;I think you are on mute&rdquo;</h2><p dir="ltr">Video conference calls have been a focal point in 2020 and continue to dominate work and social lives in 2021. The gaps in technology are revealed quickly when devices, people, and environments fail to keep up with the demands of the calls - to the point where the phrase &lsquo;I think you&#39;re on mute&rsquo; was dubbed the<a href="https://www.humanresourcesonline.net/the-most-used-business-phrases-of-2020-i-think-you-re-on-mute" rel="noopener noreferrer" target="_blank">&nbsp;most used business term</a> in 2020. Dedicated audio processors with ML is the necessary approach to solve some of the common video conferencing pain points.&nbsp;</p><p dir="ltr">ML on dedicated audio processors are already addressing some common video conference pain points:</p><ul><li dir="ltr"><strong>Selective noise canceling</strong>. By understanding a range of scenes and action noises, certain background elements such as a fan, heater, or street noise can be removed, thereby, improving overall call quality and the ability to work in a wider variety of environments.&nbsp;</li><li dir="ltr"><strong>Hi-fi audio reconstruction.</strong> Analogous to the way budget smartphones use smart software to increase the image quality from low-quality cameras, ML in real-time audio processing can improve the audio quality from low-quality microphones.&nbsp;</li><li dir="ltr"><strong>Spatial audio for improved user experience.</strong> Already available on advanced gaming headphones and some TWS devices, advanced digital sound optimization techniques provide users a spatial audio experience. Spatial audio can create a natural listening experience for the user, which can help mitigate &ldquo;<a href="https://news.stanford.edu/2021/02/23/four-causes-zoom-fatigue-solutions/" target="_blank">Zoom Fatigue</a>&rdquo;.</li></ul><p dir="ltr">In the future, ML on audio processors will have even more advanced capabilities such as real-time speech processing that would enable automatic language translations. Additionally, ML can solve the &lsquo;on mute&rsquo; problem by becoming more aware of the user&rsquo;s context and offer prompts to unmute.</p><h1 dir="ltr">A look at specialized audio edge processors</h1><p dir="ltr">There are several areas audio edge processors get right to deliver exceptional audio and voice command experiences.&nbsp;</p><h2 dir="ltr">Noise and Distance</h2><p dir="ltr">Beamforming is the process where the device can engage its ability to focus the listening or output sound in a particular direction. Noise suppression makes conversations in everyday scenarios possible. A listening device determines the direction(s) of the voice source(s) and that of noise sources. ML classification techniques are then used to determine which beams have voice or noise in them. &nbsp;The DSP then focuses on beams with voice content only for further voice UI processing. For example, in a conferencing system, the device must identify the direction of arrival of sound, and multiple speakers must be tracked at all times in a 360-degree fashion. Noise sources can also be classified further for audio event detection like glass break, fire alarms, and gunshots, etc for further expanding the role into audio aware smart home systems.&nbsp;</p><p dir="ltr">Proximity detection is also essential for dynamic listening and speaking experience. &nbsp;The device detects user proximity to the microphone and adjusts the microphone&rsquo;s gain. This feature enables active video conferencing for presentations, workouts, and learning environments. These features are central to the design of advanced video conferencing devices such as the new Amazon Show which features a screen that rotates to where it senses the user.&nbsp;</p><p><span class="fr-video fr-deletable fr-fvc fr-dvb fr-draggable" contenteditable="false" draggable="true"><iframe src="https://www.youtube.com/embed/fxUSSgrHL-o?&wmode=opaque&rel=0&enablejsapi=1&origin=https://www.wevolver.com" frameborder="0" allowfullscreen="" class="fr-draggable" style="width: 789px; height: 504px;" data-gtm-yt-inspected-8="true" data-lf-form-tracking-inspected-kn9eq4roawkarlvp="true" data-lf-yt-playback-inspected-kn9eq4roawkarlvp="true" data-lf-vimeo-playback-inspected-kn9eq4roawkarlvp="true"></iframe></span></p><h2 dir="ltr">Latency</h2><p dir="ltr">Humans can <a href="https://towardsdatascience.com/real-time-noise-suppression-using-deep-learning-38719819e051" rel="noopener noreferrer" target="_blank">generally tolerate</a> up to 200ms of end-to-end latency on a call before we begin to talk over each other. Low latency processing in edge processors is, therefore, a critical requirement for ensuring high-quality voice communication.</p><h2 dir="ltr">Power consumption</h2><p dir="ltr">Audio edge processors designed with proprietary architectures, hardware accelerators, and special instruction sets can optimally run audio and ML algorithms. These optimizations help with reducing power consumption in audio-heavy use cases such as video conferencing.</p><h2 dir="ltr">Integration</h2><p dir="ltr">Audio edge processors that open up their architectures and development environments accelerate innovation by providing audio application developers the tools and support to create new devices and applications. Future audio devices will be a collaborative effort.</p><h2 dir="ltr">Security</h2><p dir="ltr">Edge processing can minimize the need for cloud processing providing many benefits, including increased data security. For example, most consumers are not comfortable with data from personal smart home devices being transferred continuously to the cloud for processing. These fears have proven true with several <a href="https://www.theguardian.com/technology/2020/dec/23/amazon-ring-camera-hack-lawsuit-threats" target="_blank">major breaches</a> by leading device manufacturers in the last years. Peace of mind can be achieved when personal data is processed for analysis or inference at the device. An excellent example of this is a smart home security device that has been trained to hear certain event sounds, such as glass breaking, that act as a trigger to alert the homeowner. As the processing of the sound and the alert occurs on the device, it does not need to be in continuous connection to the cloud, which enhances the security of the system.</p><h1 dir="ltr">Delivering Audio Intelligence at the Edge</h1><p dir="ltr"><a href="https://www.knowles.com/products/audio-processors" target="_blank">Knowles AISonic&trade; IA8201: Dual Core</a> is an audio edge processor specifically designed for advanced audio and machine learning applications, enabling power-efficient intelligence and privacy at the edge. It offers robust voice activation and multi-microphone audio processing with sufficient compute power to perform advanced voice processing and audio output, context awareness, and gesture control for today&rsquo;s most advanced consumer electronics. Processing voice and audio voice with an audio edge processor like IA8201 can keep the power-hungry host processor off as long as possible to extend battery life. &nbsp;</p><p dir="ltr">In addition, it also enables new use cases for enhanced user experience by having the capability to run ML algorithms on multiple sensor inputs to deliver a more natural and contextual user experience.</p><p dir="ltr">Audio edge processors like IA8201 have a place in a variety of audio and voice-enabled applications including smart speakers, video conferencing devices, tablets, Bluetooth headphones, and smart home devices.</p><p dir="ltr"><strong id="isPasted">Recommended reading:&nbsp;</strong><span style="background-color: transparent;"><a href="https://www.wevolver.com/article/enhancing-the-utility-of-audio-ai-in-everyday-devices-using-audio-source-classification" rel="noopener noreferrer" target="_blank">Enhancing the utility of Audio AI in everyday devices using Audio Source Classification</a></span></p><h1 dir="ltr">Conclusion</h1><p dir="ltr">Dedicated audio edge processors will define the next generation of audio and voice-enabled devices creating more context-aware, immersive, and seamless audio communication experiences. Their ability to enable efficient processing for low power and low latency voice communications, noise reduction, context awareness, and accelerated ML inferencing of sensor inputs opens the possibilities to an explosion of new user experiences for the human-machine interfaces.&nbsp;</p><hr><h1><em>References and further reading:</em><a href="https://blog.adobe.com/en/publish/2019/04/09/contextual-ai-the-next-frontier-of-artificial-intelligence.html#gs.tuw2rt" target="_blank"></a></h1><p><a href="https://blog.adobe.com/en/publish/2019/04/09/contextual-ai-the-next-frontier-of-artificial-intelligence.html#gs.tuw2rt" target="_blank">https://blog.adobe.com/en/publish/2019/04/09/contextual-ai-the-next-frontier-of-artificial-intelligence.html#gs.tuw2rt</a></p><p dir="ltr">Antonini, Mattia &amp; Vecchio, Massimo &amp; Antonelli, Fabio &amp; Ducange, Pietro &amp; Perera, Charith. (2018). Smart Audio Sensors in the Internet of Things Edge for Anomaly Detection. IEEE Access. PP. 1-1. 10.1109/ACCESS.2018.2877523.&nbsp;</p><p dir="ltr">Do, Ha &amp; Sheng, Weihua &amp; Liu, Meiqin &amp; Zhang, Senlin. (2016). Context-aware sound event recognition for home service robots. 739-744. 10.1109/COASE.2016.7743476.&nbsp;</p><p dir="ltr"><a data-lf-fd-inspected-kn9eq4roawkarlvp="true" href="https://www.knowles.com/docs/default-source/default-document-library/knowles-ia8201-product-brief-final1b6a11c731dff6ddbb37cff0000940c19.pdf" target="_blank">https://www.knowles.com/docs/default-source/default-document-library/knowles-ia8201-product-brief-final1b6a11c731dff6ddbb37cff0000940c19.pdf</a></p><p dir="ltr"><a href="https://voicebot.ai/2019/06/01/dedicated-audio-processors-at-the-edge-are-the-future-here-are-the-reasons-why/" target="_blank">https://voicebot.ai/2019/06/01/dedicated-audio-processors-at-the-edge-are-the-future-here-are-the-reasons-why/</a></p><p dir="ltr"><a href="https://www.networkworld.com/article/3262105/how-edge-computing-makes-voice-assistants-faster-and-more-powerful.html" target="_blank">https://www.networkworld.com/article/3262105/how-edge-computing-makes-voice-assistants-faster-and-more-powerful.html</a></p>

edge processors

ORGANIZATIONS.

SHAPING THE INDUSTRY.

Relay2

Knowles Corporation

Axelera AI

The Linley Group

ADLINK Technology

Latest Posts