Tiny Machine Learning

Mahesh Chowdhary, Swapnil Sayan Saha

STMicroelectronics Inc. (AMS MEMS Division)

ST Restricted

Real-Time Inference

2

Several applications need to make “complex inferences” for “time-critical” and “remote” applications from “unstructured data”.

ST Restricted

Conventional AI Deployment

3

Conventionally, such inferences have been made using machine learning models running on edge servers, continuously trained using new data on the cloud.

ST Restricted

What is Tiny Machine Learning?

4

Hardware and software suites that enable always-on, ultra-low power, and on-device data analytics.

Primary memory: 10⁰ - 10² kB

Secondary memory: 10³ kB

Power consumption:

µW to mW regime

Architecture: Single-core

ST Restricted

What is Tiny Machine Learning?

5

Hardware and software suites that enable always-on, ultra-low power, and on-device data analytics.

Code translation and generation

Operator optimizations

Inference engine optimizations

Model compression

ST Restricted

Tiny Machine Learning Workflow

6

ST Restricted

Agenda

7

1

Feature Projection

2

Lightweight Model Backbones

3

Neural Architecture Search

4

Compiler Suites and Model Compression

ST Restricted

Feature Projection

ST Restricted

Feature Extraction and Selection

9

ANOVA

AdaBoost

Recursive

Sequential

Random Forest

Statistical

Temporal

Spectral

Recursive

Windowing

Feature Extraction

Feature Selection

Model Training

Projected Data

Windowed Data

Final Model

Model Input (Selected Features)

Model Performance

ST Restricted

Feature Selection Techniques

• ANOVA: Computes the F-statistic by using analysis of variance on all input features w.r.t the target labels. ANOVA ranks the significance of the target label by estimating the degree of linear dependency between random variables. Features with a higher f-statistic and low p-values rank highest.

​

• AdaBoost: Uses a DT to fit the original dataset, and then fits additional copies of the classifier to the same datasets with weights applied to the incorrectly classified instances. The ranking of features is calculated by traversing each subsequent classifier and calculating the Gini importance across each tree.

​

• Random Forest: Fits several DT classifiers on various sub-samples of the dataset, uses Adaboost ranking and averaging to improve the predictive accuracy and control over-fitting.

10

ST Restricted

Feature Selection Techniques

• Sequential Feature Selection: Adds or removes features to form a feature subset greedily. At each stage, the estimator chooses the best feature to add or remove based on the cross-validation score of a DT. The ranking is calculated by assigning each feature a binary value to indicate whether it is an important feature or not.

​

• Recursive Feature Elimination: Recursively considers smaller and smaller sets of features. First, a DT is trained on the initial set of features and the importance of each feature is obtained either through any specific attribute or callable. Then, the least important features are pruned from the current set of features. The procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached

11

ST Restricted

Feature Transformation: Matrix Factorization

12

ST Restricted

Lightweight Model Backbones

ST Restricted

Sparse Low-Dimensional Projection

14

Sparsely projecting data onto a low-dimensional manifold (prototypes) yields lightweight linear classifiers such as decision tree or kNN.

ST Restricted

Low-Rank, Sparse, and Quantized Recurrent Blocks

15

FastRNN and FastGRNN combines the lightweightness of vanilla recurrent networks (e.g., RNN and GRU) with stability of long short-term memory networks (e.g., LSTM)

ST Restricted

Temporal Convolution

16

ST Restricted

Lightweight Spatial Convolution

17

ST Restricted

Neural Architecture Search

ST Restricted

What is Neural Architecture Search?

19

Automatically find the most performant neural network architecture from a hyperparameter space within some constraints.

Optimization Function

Search Space

Search Algorithm

Candidate

model

Score

Most performant model

*

ST Restricted

Hardware Constraint Profiling

20

Slowest but most accurate

Fastest but least accurate

ST Restricted

Optimization Function

21

NAS is a non-linear program with constraints.

Search space Ω contains neurosymbolic hyperparameters, trainable weights, neural operators, and symbolic program atoms.

Goal: construct a fault-free AI program such that latency and error are minimized, while the memory usage is maximized within device memory limits.

ST Restricted

Search Algorithms

22

ST Restricted

Evolutionary NAS

23

ST Restricted

NAS as Bayesian Optimization

24

Two components: Surrogate function and acquisition function

A surrogate function approximates an optimization function, e.g., Gaussian process.

An acquisition function selects the next promising set of points to sample.

Surrogate posterior mean

True objective plane

Sample

Uncertainty

Observed

points

Current

sample

Credits: BayesO

ST Restricted

Compiler Suites and Model Compression

ST Restricted

Compressing a Neural Network

26

ST Restricted

TinyML Compiler Optimizations

27

ST Restricted

Notable TinyML Compiler Suites

28

ST Restricted

TensorFlow Lite Micro

29

ST Restricted

© STMicroelectronics - All rights reserved.�ST logo is a trademark or a registered trademark of STMicroelectronics International NV or its affiliates in the EU and/or other countries. For additional information about ST trademarks, please refer to www.st.com/trademarks. �All other product or service names are the property of their respective owners.

ST Restricted