Embedded Machine Learning Challenges Explained

Before diving into the details of Embedded Machine Learning it is essential to address - the obvious. Often new terms and technologies are approached with a must-have attitude, however, the technical application or product where Embedded Machine Learning makes sense needs to be established in the first place. In some cases, a simple threshold approach is just fine. Once this is cleared then the insightful journey can begin.

Embedded machine learning applications consist of machine learning pipelines that are deployed on embedded devices. The development of such pipelines in computationally constrained environments presents most of the challenges of developing any machine learning application.^[1]  However, the challenges are intensified due to the need for capturing data from embedded devices and deploying machine learning models on them. Specifically, the challenges of embedded machine learning span the following areas:

• Diversity and Peculiarities of Embedded Devices – Embedded Programming Challenges: There are hundreds of different types of embedded devices that vary in their core characteristics such as their small form factor, their power consumption requirements, their operating system, and the connectivity protocols that they support. Hence, the development of embedded machine learning models and applications must consider the peculiarities of the embedded device ^[2], as well as the safety, security and reliability challenges that stem from its nature. While it is possible to group machine learning development requirements for different types of devices (e.g., based on their operating system), the implementation of embedded machine learning applications is far from being a “one-size-fits-all” process i.e., like developing a conventional machine learning application for a MacBook or a Windows PC (Personal Computer). Developers and deployers of enterprise-scale machine learning applications must ultimately address challenges stemming from the peculiarities of the target embedded environment, which increases development effort and costs.  This makes them hardly accessible to domain experts, which are not always familiar with embedded systems’ challenges. 
• Data Collection and Data Availability: The quality and accuracy of any machine learning model depend on the amount and quality of data that are used for its training. Specifically, the availability of well-structured, properly labeled, statistically unbiased, representative, and rich datasets for the problem at hand is a foundation for the development of an effective model that works in real-life conditions i.e., beyond the training data ^[3]. Nowadays, the machine learning community acknowledges the lack of proper data as one of the main setbacks against the development of machine learning systems, which is the reason why the transition to a data economy is a prerequisite for the accelerated adoption of machine learning. The data availability challenges appear intensified in the case of embedded machine learning and TinyML applications. Most enterprises do not typically collect data from their embedded devices, which means that data collection must take place prior to the development of embedded machine learning models. The data collection process can sometimes be tedious and time-consuming, as it must deal with embedded programming environments and the low-level network interfaces of the embedded devices.
• Data Pre-processing and Preparation: Data scientists spend significant effort on preparing their data for model training. The preparation entails for example the tasks of handling missing values and performing imputations, dealing with outliers, and confronting statistical imbalances in the input datasets. These processes are tedious and time-consuming. In the case of embedded systems, they can be even more challenging, as there is a need for understanding the statistical properties of datasets from embedded devices. Likewise, the peculiarities of the models that are supported by the embedded devices (e.g., deep neural networks) must be considered as well.
• Model Development and Deployment: With proper training datasets at hand, embedded machine learning focuses on the development and evaluation of proper models (e.g., deep neural networks) for the task at hand. Following the evaluation process, effective models are deployed on the embedded devices. This process requires the use of new libraries and tools, which shrink models in ways that enable them to fit within the embedded devices. To this end, there is a need for specialized machine learning libraries that are tailored to mobile and embedded devices. Much as these libraries are derivatives of legacy libraries (e.g., TensorFlow), their use entails an additional learning curve which makes embedded machine learning more challenging.
• Lack of a Rich set of Visual Development Tools: The development and deployment of embedded machine learning models can greatly benefit from visual programming environments and related machine learning tools. Nowadays, there is a very rich set of such environments for conventional computers, yet much fewer tools for embedded development. Therefore, developers are not provided with many tools that can facilitate and accelerate the process of retraining, rebuilding, and revaluating embedded machine learning models for different types of embedded devices and their operating systems. In several cases, this can increase the time and effort needed to implement, test, fine-tune, and ultimately deploy an embedded machine learning model. Embedded machine learning developers need easy, fast, and flexible ways for experimenting with different model parameters (e.g., the number of layers and neurons of a deep learning network), also given the lack of accumulated experience and best practices regarding model development and optimization.
• AutoML for Embedded Machine Learning: In recent years, the machine learning community takes advantage of AutoML (Automated Machine Learning) tools and techniques, which streamline and automate all the steps of the machine learning process.^[4]  Based on AutoML tools, machine learning developers automate and accelerate the process of rebuilding and redeploying entire machine learning pipelines, including data pre-processing, data preparation, model training, model evaluation, and model deployment steps. Furthermore, certain types of AutoML, like NAS (Neural Architecture Search) ^[5], ^[6], automate the selection of the best possible neural network architecture for a given task, which decouples the machine learning engineer from several tedious and complex tasks like feature engineering, feature selection, and model optimization. In this way, AutoML separates machine learning from low-level model development tasks and gradually brings its power to domain experts without in-depth statistical data analysis and machine learning knowledge. Nevertheless, these merits of AutoML are not widely accessible to embedded machine learning developers and users, which makes it more difficult for domain experts to develop and deploy applications without involving embedded systems developers and data science experts. This is a setback to the democratization of embedded machine learning and its wider use in industrial applications. 

Figure 1: Embedded machine learning will play a huge role in the future of manufacturing. 

Solution Guidelines for Domain Experts

Fortunately, IoT and data science experts of the embedded machine learning community are intensively working on solutions that alleviate these challenges and bring the merits of this new evolutionary paradigm closer to industry practitioners. In principle, the solution lies in the provision of tools, techniques, and blueprints that simplify the development and deployment of embedded machine learning applications. In several cases, the automation of development and deployment steps can also accelerate the implementation process, while hiding the complexity from industrial end-users. Moreover, the creation of a vibrant community of embedded machine learning developers and experts is a key to the proliferation and spreading of knowledge within relevant industrial sectors.

To address the complexity of embedded programming and the diversity of the various embedded systems platforms, the embedded machine learning community is gradually providing ready-to-use examples and blueprints on how to develop applications for specific platforms. Building on past knowledge and sample applications for specific platforms, it is possible to significantly reduce development times and costs. Beyond providing blueprints, the embedded machine learning community is continually contributing knowledge on how to select the best possible embedded device for a task at hand. This selection is critical for the success of machine learning applications and must be carefully planned.

Embedded machine learning development can benefit from datasets collected for certain devices and application contexts. Such datasets could lower the data collection effort needed in the scope of embedded machine learning projects. They can support different stages of a project’s implementation, including for example the initial experimentation and prototyping phase. In several cases, data collected for specific embedded devices and applications can be repurposed and reused in other contexts. This is possible thanks to machine learning techniques like transfer learning ^[7], but also thanks to community efforts to repurpose these datasets through their appropriate processing. In several cases, industry practitioners and domain experts can also benefit from pre-trained machine learning models, which can greatly lower data collection and preparation efforts. The embedded machine learning community is already offering pre-trained models for several devices. 

To lower the application development and deployment effort for machine learning applications, industry practitioners must use visual tools for IoT and Machine Learning applications. Such tools provide access to popular features for programmers, such as flow programming, reusable libraries, access to one-click-samples, and various “drag-and-drop” functionalities.  In this direction, embedded machine learning developers assemble end-to-end tooling solutions based on the combination of mainstream IoT (e.g., Arduino IDE) and Machine Learning (e.g., Jupyter Notebooks) tools. The ultimate vision of the growing embedded machine learning community is to provide AutoML functionalities to domain experts, which will help them develop end-to-end solutions in a single integrated environment, without a need for deep knowledge of embedded systems and data science processes.

Most importantly, domain experts and industrial end-users must be provided with easy access to large and well-structured knowledge about embedded machine learning applications, including knowledge about the business benefits and the practical use cases of embedded machine learning, as well as access to resources that facilitate application development.

The Edge Impulse Ecosystem

The above-listed solutions are not in the inception stage. Rather they are already available to thousands of developers and many hundreds of enterprises that are riding the wave of embedded machine learning solutions. As a prominent example, Edge Impulse, one of the leading platforms for embedded machine learning, has already a vibrant ecosystem and an active community of many thousands of developers and experts. Edge Impulse is a one-stop shop for embedded machine learning resources. It provides a single access point to fully integrated and best-in-class resources for developing and deploying real-life embedded machine learning applications. Edge Impulse alleviates the lack of visual tools for embedded machine learning applications, which are a key to facilitating development and deployment of industrial applications.

By joining the Edge Impulse ecosystem, business enterprises can instantly gain access to a pool of cutting-edge machine learning resources and solutions, in these areas:

1. Data Management for consistency

a) Access to data sets for model training: Within the Edge Impulse ecosystem developers find datasets that can be used to train machine learning models for embedded devices. These datasets are used for training and experimentation. However, they can be also useful as a starting point for production-ready, enterprise-scale deployments, following some fair effort for their enhancement and repurposing. Furthermore, within the Edge Impulse ecosystem, users can discover publicly available projects and related datasets that accelerate their embedded machine learning projects.  Edge Impulse provides insights on how such public datasets can be enhanced, repurposed, and used in conjunction with embedded devices. 

b) MLOps for Embedded Devices: Edge Impulse supports modern MLOps practices for collaboration and communication between data scientists, machine learning engineers, and ML operations experts. Edge Impulse is currently offering a complete platform for organizations to cross collaborate and manage their MLOps pipelines. It is the sole platform that offers MLOps support in the embedded devices space. Documenting data and processing steps along the way for compliance and tech consistency (data lineage).

2. Feature Extraction for Machine Learning at the Edge

a) Access to Cloud-based API: The functionalities of the Edge Impulse Studio are accessible through the Edge Impulse API. The latter can be used to build AutoML pipelines, which find the best parameters for machine learning models (e.g., feature extraction using various digital signalling processing blocks), and enable domain experts to optimize their embedded machine learning applications without in-depth IoT and data science knowledge. Edge Impulse provides access to documentation and other knowledge resources about using the Edge Impulse Studio as a cloud service.

b) Delivery of Best-in-Class Embedded Models for Industrial Use Cases: Edge Impulse enables the delivery of best-in-class optimized embedded models. The Edge Impulse tools go beyond enabling embedded machine learning developments to the delivery of optimal models that deliver the best possible performance and accuracy. This is one of the main benefits offered by Edge Impulse to industrial enterprises, which have very demanding performance requirements. 

3. Operationalization of Machine Learning at the EDGE

a) Edge Impulse Studio for Embedded Machine Learning as a Service: The Edge Impulse platform offer access to the Edge Impulse Studio, an integrated visual environment that enables the development of end-to-end embedded machine learning pipelines. Using the Studio, it is possible to control embedded devices and to experiment with data ingestion functions. Moreover, the Studio provides AutoML functionalities which facilitates and speed up machine learning development. Furthermore, the Edge Impulse Studio is accessible as a cloud service, which enables an Embedded Machine Learning as a Service paradigm.

b) Examples and Tutorials for Embedded Machine Learning on various devices: Edge Impulse provides readily available examples and blueprints on how to build solutions for a rich set of embedded devices, such as  ST B-L475E-IOT01A, Arduino Nano 33 BLE Sense, Eta Compute ECM3532 AI Sensor, Himax WE-I Plus, Nordic Semiconductor nRF52840 DK, Nordic Semiconductor nRF5340 DK, and Silicon Labs Thunderboard Sense 2. Moreover, there are resources for users that would like to build embedded machine learning in any smartphone. These resources facilitate enterprises that use the above-listed boards to kick-start their developments and to lower relevant learning curves.  They are extremely useful even for those that plan to use other devices, as they provide tangible examples of how embedded machine learning works in practice.

4. Knowledge creation and sharing

a) Tutorials and Resources on Machine Learning Modules (including deep learning): In the scope of the Edge Impulse platform, it is possible to access tutorials about how to leverage different machine learning models. There are also resources about building modular applications, where different processing blocks are plugged in applications. For instance, there are resources about plugging in signal processing blocks and building neural networks based on popular deep learning frameworks like Keras.

b) Embedded Machine Learning Knowledge Base: Edge Impulse provides many knowledge assets about embedded machine learning, including tutorials, examples, blueprints, as well as documentation. Moreover, the company has recently sponsored the development of the course, “Introduction to Embedded Machine Learning” on the popular Coursera platform. The course fosters the development of embedded machine learning skills for many students, researchers, and practitioners worldwide.

Many of the above-listed functionalities are offered freely to thousands of developers that active members of the Edge Impulse ecosystem. However, Edge Impulse offers a richer set of resources, including datasets and cloud-based tools, to enterprises that aim at using embedded machine learning to improving their business results. While the free services of Edge Impulse facilitate developers to get started with embedded machine learning and to improve their skills, the enterprise version is a vehicle for improving business competitiveness and productivity. Specifically, as earlier outlined, Edge Impulse delivers optimized, best-in-class embedded machine learning models, which make the Edge Impulse ecosystem the number one choice for industrial applications with stringent performance requirements. Moreover, Edge Impulse offers industrial enterprises access to AutoML and MLOps tools, which improve collaboration and reduce development costs.

In conclusion, embedded machine learning and TinyML are currently revolutionizing the IoT and AI landscape. AI is no longer reserved for high-end computing infrastructures that comprise GPUs and TPUs. Rather, it can also be deployed on IoT devices and microcontrollers, in order to deliver tangible performance, latency, privacy, cost, and sustainability benefits. Industrial enterprises need not wait to capitalize on these benefits. They must already explore how they can leverage embedded machine learning to harness unused data and to improve their business processes. By joining the Edge Impulse ecosystem today, they can gain access to a wealth of resources that can help them adopt and fully leverage embedded machine learning, in ways that could set them apart from their competitors. This is what makes it easy for an embedded engineer to deploy Machine Learning - at all - in the field and "hyper-fast" compared to the time before Edge Impulse.

About the sponsor: Edge Impulse

Edge Impulse is the leading development platform for embedded machine learning, used by over 1,000 enterprises across 10,000 ML projects worldwide. We are on a mission to enable the ultimate development experience for machine learning on embedded devices for sensors, audio, and computer vision, at scale. 

From getting started in under five minutes to MLOps in production, we enable highly optimized ML deployable to a wide range of hardware from MCUs to CPUs. With Edge Impulse, developers, engineers, and domain experts solve real problems using machine learning in embedded solutions, speeding up development time from years to weeks. We specialize in industrial and professional applications including predictive maintenance, asset tracking, and monitoring, and human and animal sensing. 

References

1. G. Berry, "Challenges and potential solutions for complex embedded systems," 2011 Proceedings of the Ninth ACM International Conference on Embedded Software (EMSOFT), Taipei, 2011, pp. 1-1, doi: 10.1145/2038642.2038644.

2. Wirth, R. & Hipp, Jochen. (2000). CRISP-DM: Towards a standard process model for data mining. Proceedings of the 4th International Conference on the Practical Applications of Knowledge Discovery and Data Mining.

3. Y. Roh, G. Heo and S. E. Whang, "A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective," in IEEE Transactions on Knowledge and Data Engineering, doi: 10.1109/TKDE.2019.2946162.

4. Xin He, Kaiyong Zhao, Xiaowen Chu, “AutoML: A survey of the state-of-the-art”, Knowledge-Based Systems, Volume 212, 2021, ISSN 0950-7051, https://doi.org/10.1016/j.knosys.2020.106622.

5. H. Jin, Q. Song and X. Hu, "Auto-Keras: An efficient neural architecture search system", Proc. Int. Conf. Knowl. Discovery Data Mining, pp. 1946-1956, 2019.

6. X. Zhang, Z. Huang and N. Wang, "You only search once: Single shot neural architecture search via direct sparse optimization", Proc. ICLR, pp. 1-11, 2019.

7. F. Zhuang et al., "A Comprehensive Survey on Transfer Learning," in Proceedings of the IEEE, vol. 109, no. 1, pp. 43-76, Jan. 2021, doi: 10.1109/JPROC.2020.3004555.