1 of 16

On-sensor online learning and classification under 8 KB memory

​

Mahesh Chowdhary 1, Swapnil Sayan Saha 1, 2 �1 STMicroelectronics� �2 University of California - Los Angeles

1

2 of 16

On-board inertial motion recognition

2

Advancements in machine learning have paved the way for performantly realizing aforemention motion recognition applications

Tiny machine learning (tinyML) enables motion recognition on microcontrollers, field programmable gate arrays, and application-specific integrated circuits

Gesture recognition

Pose estimation

Workout tracking

Gait analysis

Activity detection

3 of 16

Challenges of deploying motion classification

3

Integrated processors have tight resource constraints (8 kB memory)

On-sensor domain adaptation using constrained learning theories

Diversity in motion primitives prevent the adoption of static classifiers

Need efficient segmentation algorithm to detect activity during training

4 of 16

Challenges of deploying motion classification

4

Integrated processors have tight resource constraints (8 kB memory)

On-sensor domain adaptation using constrained learning theories

Diversity in motion primitives prevent the adoption of static classifiers

Need efficient segmentation algorithm to detect activity during training

We showcase an ultra-lightweight, application-agnostic, and on-sensor learning and inference algorithm for on-sensor motion recognition

5 of 16

Hardware: Intelligent Sensor Processing Unit (ISPU)

5

ISPU core: 8-kilogate Harvard RISC STRED processor running at 5 MHz or 10 MHz

Supports standard C code and models models from popular machine learning frameworks

Can execute 30 separate algorithms running concurrently using interrupts

6 of 16

Learning and inference pipeline

6

Automatic Segmentation (running mean and variance)

Template Creation (quantized gravity vector images)

Template Storage

Template Retrieval

Template Matching (universal image quality index)

Raw accelerometer data

Inference

Convolutional Blurring

Training

Inference

7 of 16

Training: segmentation algorithm

7

User remains static for certain time period and then performs the event of interest

Rolling mean and variance is monitored to identify start and end of event of interest

Smoothing parameters tuned via Bayesian optimization on the training set

-1

8 of 16

Training: template creation and storage

8

Extract the gravity vector swing from roll and pitch calculated from accelerometer samples

Select the two swing axes with maximal variance and create a quantized image template; image template provides an efficient and spatial view of the motion primitive

Store the axes names with maximal variance, image template, and motion counter as an array of structures

9 of 16

Example templates for exercise activities

9

Bicep Curl

Lateral

Jack

10 of 16

Inference: blurring and matching

10

The universal image quality index (UQI) is used to match stored templates with current activity template (if axis variance information cannot do a classification)

Convolutional blurring is performed before UQI calculation to make the matching robust to small domain variances

11 of 16

Repetition counting

11

An adaptive rep counter counts the repetitions of detected motion primitives in each motion template

A peak detector identifies a point as a peak if it has the maximal value and was preceded to the left by a value lower by ∇

Operates on accelerometer axis with maximal variance

12 of 16

Workout tracking: performance

12

96.7% accuracy for 6-class activity recognition with only 10 second templates

13 of 16

Workout tracking: resource usage

13

2.5x lower memory usage than microcontroller implementation; 5x lower current consumption; 1000-2000x lower memory usage than existing online learning frameworks

14 of 16

Workout tracking: rep counting

14

Rep counting error: 0.44 reps across 6 different motion primitives (half the tolerable error of 0.8)

15 of 16

Summary

    • Very low resource usage: presented algorithm (including training) can be executed on ISPU sensor (and low resource uControllers) with <10 kB SRAM
    • Personalization: algorithm can modify information about existing motion primitives according to user motion characteristics and is robust to domain shifts across users
    • Unique similarity metric: universal Image Quality Metric used with convolution blurring to achieve robust performance in template matching and classification
    • New motion artifacts / classes: it is possible to add new action primitives using the algorithm

​

15

16 of 16

Thank You!

16

© STMicroelectronics - All rights reserved.�ST logo is a trademark or a registered trademark of STMicroelectronics International NV or its affiliates in the EU and/or other countries. For additional information about ST trademarks, please refer to www.st.com/trademarks. �All other product or service names are the property of their respective owners.