Edge-Based Machine Learning Application Development is Getting a Whole Lot Easier

Machine learning (ML) development has become dramatically more accessible thanks to a growing set of ML development tools, particularly those from leading cloud service providers like Amazon Web Services (AWS), Google, and Microsoft. These tools make ML development accessible, but they don’t necessarily make it easier to implement edge-based applications for the Internet of Things (IoT), smart homes, industrial automation systems, and other embedded areas. However, a rapidly growing set of hardware and software resources are beginning to address the unique requirements of developing ML applications for embedded systems.

Much of the focus of “Big ML” is on very large models and enterprise-level applications running on massively scalable farms of graphics processing units (GPUs) and servers. In contrast, “tinyML” for edge applications presents an entirely different set of problems stemming, in part, from the resource limitations of microcontrollers (MCUs) and in part from the application’s real-time requirements.

In the past, MCUs were designed to meet relatively simple requirements for memory and peripherals. Performance was often not a huge concern for most embedded applications that specified cycle times measured in milliseconds at best. Now, requirements for more robust resources, connectivity, and performance have increased in more conventional embedded applications. In addition, the growing desire to execute ML algorithms with minimal latency in embedded applications has dramatically changed what’s needed and expected from advanced MCUs. As such, semiconductor manufacturers have responded with advanced MCUs like NXP’s i.MX RT1170 Crossover MCUs that offer features and capabilities that not too long ago were considered server-class characteristics.

A capable hardware foundation is essential for implementing an ML-based embedded application, but delivering such an application ultimately hinges on the ability to successfully develop the relatively large complement of software required for ML-based edge solutions. Besides implementing the data acquisition software at the heart of nearly all embedded systems, developers also need to build a suitable machine learning model trained on representative data and deploy the corresponding inference model on the MCU (Figure 1).

Figure 1: Implementation of an ML-based application for the edge adds ML-specific project phases that extend development beyond the more familiar tasks needed to build sensor-based smart products. (Image source: NXP Semiconductors)

For embedded developers implementing ML-based applications at the edge, the additional work required for ML model creation effort boils down to four key tasks:

• Define the data set: In this critical initial step, developers decide on the data sources and required characteristics of interest for their application.
• Define the model: Here, developers determine the model topology and hyperparameters needed to tune it.
• Train the model: This compute-intensive step is typically performed on robust servers and GPU farms.
• Generate the inference model: Prior to actual deployment, developers convert the training model to an optimized inference model by removing model features that are needed only during training or that contribute little or nothing to inference.

Taken individually, these steps are well within the capabilities of an embedded developer. In implementing the sensors and data acquisition system, the developer knows enough about the data to handle the data scientist role typically required in enterprise-level ML applications. Beyond that initial step, however, developers have been largely left on their own to puzzle out model definition, training, and conversion.

Early adopters have quickly taken advantage of the bits and pieces of technology that quickly emerged for creating ML models suitable for the edge. Libraries like TensorFlow Lite bring much of the power of Google’s open-source TensorFlow ecosystem to resource-constrained mobile and edge systems. Development libraries, like Arm’s Common Microcontroller Software Interface Standard - Neural Network (CMSIS-NN) and its neural network (NN) software development kit (SDK), leverage hardware capabilities of Arm GPUs and central processing units (CPUs). Tools like Facebook’s Glow ML compiler and Microsoft’s ONNX Runtime ML optimizer aid inference model generation. For deployment, specialized software like Au-Zone Technologies’ DeepViewRT inference engine provides a production-ready inference engine designed to simplify model conversion.

The software I mentioned here barely scratches the surface of the available software libraries and tools designed for (or adaptable to) MCU-based ML application development. One of the more interesting trends in edge ML has been the emergence of ML development solutions directly from MCU semiconductor manufacturers.

Although ML researchers might be better equipped to create exceptional ML algorithms, the MCU vendor is uniquely qualified to solve the engineering problem of creating an effective ML development environment for resource-constrained MCU-based systems. Designed specifically for this problem, tools like NXP’s eIQ Toolkit help speed the development of ML models intended to run on MCUs (Figure 2.).

Figure 2: The availability of tools like NXP’s eIQ Toolkit has dramatically simplified the development of MCU-based ML applications for the edge. (Image source: NXP Semiconductors)

Conclusion

Because of the unique requirements of embedded devices, edge ML application development might not reach the kind of turnkey deployment that’s being pursued for enterprise-level ML applications, but it’s already gaining the kind of hardware and software support that’s enabled so many smart devices.

I’ll be diving into edge ML development and NXP’s ML ecosystem in an upcoming article that shows how much easier it has become to deploy ML applications at the edge. It may not be fully turnkey yet, but edge ML is becoming a lot more accessible and easily deployed.