1

Inertial Navigation on Extremely Resource-Constrained Platforms: Methods, Opportunities and Challenges

Swapnil Sayan Saha ^{1, 2}, Yayun Du ³, Sandeep Singh Sandha ⁴, Luis Antonio Garcia ⁵,

Mohammad Khalid Jawed ¹, Mani Srivastava¹

​

¹ University of California, Los Angeles, ² STMicroelectronics, ³ Northwestern University,

⁴ Abacus.AI, ⁵ University of Southern California

Swapnil Sayan Saha, Yayun Du, Sandeep Singh Sandha, Luis Garcia, Mohammad Khalid Jawed, and Mani Srivastava. "Inertial Navigation on Extremely Resource-Constrained Platforms: Methods, Opportunities and Challenges", in 2023 IEEE/ION Position, Location and Navigation Symposium (PLANS), IEEE, 2023.

1

Inertial odometry - viable solution in GPS or network-denied localization applications demanding small footprint, low-access delay, and low-power pathway.

2

Search and Rescue

Animal Tracking

UUV, Underwater Robots

Indoor UAV, Indoor Robots

Indoor Navigation

Underground Robots

Picosatellite localization

Inertial Odometry – Locating Objects with IMU

[1] Saha, Swapnil Sayan, et al. "Tinyodom: Hardware-aware efficient neural inertial navigation." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6.2 (2022): 1-32.

The Curse of Drift in Inertial Odometry

3

• MEMS inertial sensors suffer from the curse of drift due to angular random walk, bias instability and noise.

​

• Naive double integration of accelerometer readings: cubic explosion of error.

[1] Saha, Swapnil Sayan, et al. "Tinyodom: Hardware-aware efficient neural inertial navigation." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6.2 (2022): 1-32.

Classical Inertial Navigation - Lightweight but Approximate

4

​

• This approach is lightweight but not robust across domains or deviations from the assumed system model.

[1] Harle, Robert. "A survey of indoor inertial positioning systems for pedestrians." IEEE Communications Surveys & Tutorials 15.3 (2013): 1281-1293.

• Application-specific heuristics and human-engineered system models have been used for decades to mitigate odometric error explosion (e.g., INS or SHS).

Neural Inertial Navigation – Robust but Expensive

5

​

[1] Brossard, Martin, Axel Barrau, and Silvère Bonnabel. "AI-IMU dead-reckoning." IEEE Transactions on Intelligent Vehicles 5.4 (2020): 585-595.

[2] Wagstaff, Brandon, and Jonathan Kelly. "LSTM-based zero-velocity detection for robust inertial navigation." 2018 International Conference on Indoor Positioning and Indoor Navigation (IPIN). IEEE, 2018.

[3] Herath, Sachini, Hang Yan, and Yasutaka Furukawa. "RoNIN: Robust Neural Inertial Navigation in the Wild: Benchmark, Evaluations, & New Methods." 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.

[4] Chen, Changhao, et al. "Ionet: Learning to cure the curse of drift in inertial odometry." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

• Neural inertial navigation provides superior long-term resolution over classical inertial navigation techniques.

• This approach is resource hungry, data hungry, and not physics-aware.

Proposed Solutions

6

[1] Saha, Swapnil Sayan, et al. "Tinyodom: Hardware-aware efficient neural inertial navigation." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6.2 (2022): 1-32.

[2] Du, Yayun, et al. “Neural-Kalman GNSS/INS Navigation for Precision Agriculture”, in 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2023.

[3] Saha, Swapnil Sayan, et al. “TinyNS: Platform-Aware Neurosymbolic Auto Tiny Machine Learning.” in ACM Transactions on Embedded Computing Systems (2023). (under review after revision)

• Platform and physics-aware inertial-sequence learning pipeline for microcontrollers for accurate yet lightweight inertial odometry to track humans, animals, cars, drones etc.

​

​

​

• A neural-Kalman filter to combine neural-inertial navigation with classical physics-aware inertial navigation models for precise and accurate localization.

​

​

​

• Data-efficient transfer learning to improve utility of pre-trained neural-inertial navigation models.

​

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Inertial Sequence Learning

7

[1] Chen, Changhao, et al. "Ionet: Learning to cure the curse of drift in inertial odometry." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

• Regress displacement and heading rate in polar coordinates from gyroscope and accelerometer chunks using a neural network.

​

• Three issues:
• Suffers from gravity pollution, heading-rate singularities and varying inertial artefacts.
• Cannot distinguish rotational artefacts from translational artefacts.
• Not expandable for 3D trajectories due to dependence on gravity anchor.
• Resource-hungry, not suitable for real-time deployment on edge devices.

Robust 3D Inertial Sequence Learning

8

❶ Velocity and magneto-centric DNN regresses velocities and uses magnetic North as an additional anchor point.

❷ A physics metadata module supplies latent information about whether valid translational movements have occurred.

❸ A barometric g-h filter immune to inertial and environmental perturbations to regress altitude from pressure sensors.

❶

❷

❸

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Robust 3D Inertial Sequence Learning

9

❶ Velocity and magneto-centric DNN regresses velocities and uses magnetic North as an additional anchor point.

❷ A physics metadata module supplies latent information about whether valid translational movements have occurred.

❸ A barometric g-h filter immune to inertial and environmental perturbations to regress altitude from pressure sensors.

❶

❷

❸

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

The formulation will eventually allow extremely small models to achieve the accuracy of large models.

Picking a Lightweight Neural Network

10

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

[1] Lea, Colin, et al. "Temporal convolutional networks: A unified approach to action segmentation." European Conference on Computer Vision. Springer, Cham, 2016.

[2] van den Oord, et al. "WaveNet: A Generative Model for Raw Audio." in 9th ISCA Speech Synthesis Workshop (pp. 125-125), 2016.

• Temporal convolutional network (TCN) - discovers global context in long sequences, maintains input resolution and coverage.

​

• No explosion of parameter, memory footprint, layer count or overfitting.

​

• Secret sauce: Causal convolution, dilations and gated residuals.

Optimizing the Neural Network for Deployability

11

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

[1] Sandha, S. S., Aggarwal, M., Saha, S. S., & Srivastava, M. (2021, December). Enabling hyperparameter tuning of machine learning classifiers in production. In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) (pp. 262-271). IEEE.

​

• Platform-aware neural architecture search over a TCN hyperparameter space Ω.

• Optimization problem with competing objectives:

• Maximize odometric resolution.

​

• Minimize inference latency.

​

• Fit the neural network within the target hardware memory constraints.

• Gradient-free Bayesian optimization [1] to solve the optimization problem.
• Surrogate: Gaussian process.
• Acquisition function: Monte Carlo sampling with upper confidence bound (exploration and exploitation)

2D Navigation Performance

12

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

2D Navigation Performance

13

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Height/Depth Estimation Resolution: ±0.1 m

Sample Trajectories

14

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Ablation Study – Physics-Aware Formulation

15

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Physics, velocity, and magnetometer-centric neural inertial navigation outperforms variants not including the three aspects.

Architectural Adaptation and Device Exploitation

16

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Our Bayesian NAS performs intelligent architectural adaptations to fully exploit target hardware capabilities in order to improve error metric.

Proposed Solutions

17

[1] Saha, Swapnil Sayan, et al. "Tinyodom: Hardware-aware efficient neural inertial navigation." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6.2 (2022): 1-32.

[2] Du, Yayun, et al. “Neural-Kalman GNSS/INS Navigation for Precision Agriculture”, in 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2023.

[3] Saha, Swapnil Sayan, et al. “TinyNS: Platform-Aware Neurosymbolic Auto Tiny Machine Learning.” in ACM Transactions on Embedded Computing Systems (2023). (under review after revision)

• Platform and physics-aware inertial-sequence learning pipeline for microcontrollers for accurate yet lightweight inertial odometry to track humans, animals, cars, drones etc.

​

​

​

• A neural-Kalman filter to combine neural-inertial navigation with classical physics-aware inertial navigation models for precise and accurate localization.

​

​

​

• Data-efficient transfer learning to improve utility of pre-trained neural-inertial navigation models.

​

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Combining Neural Networks with Symbolic Programs

18

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

• Kalman filter: combines a noisy process model with noisy measurement updates to provide optimal state estimates.
• Two steps: Propagate and Update.

​

• Use a neural system model with symbolic measurement updates (or vice versa).
• Extended Kalman Filter (EKF): Works with non-linear process and observation models.

Primer on Extended Kalman Filter

19

• Covariance update:

• Measurement update:

• Kalman gain:

​

• System evolution:

​

• Covariance estimate:

• Discrete-time EKF propagate step:

Predicted State

State evolution model

Control inputs

Process noise

Process noise

covariance

State transition

Process noise

transition

Predicted Covariance

Near-optimal Kalman gain

Measurement noise

covariance

Updated state

Observation model

Measurement noise

Noisy measurements

Measurement transition

Updated Covariance

• Discrete-time EKF update:

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Neural Extended Kalman Filter (NEKF)

20

• Separate neural and non-neural parts in Kalman propagate. Neural network provides a black box mapping between sensor readings and states.

Linear state evolution

(known)

Neural network

Non-linear state evolution

(known)

• Use the linearized Jacobian of the neural network w.r.t the past state and inputs in the Lyapunov function.

Sensor Allan parameters

Jacobian term

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Application: Neural-Kalman GNSS/INS Fusion

21

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Robot

Ground Truth

Reference landmarks

• Application: Tracking agricultural robots using neural inertial navigation and GPS.

​

• Neural network provides a model-free evolution of the robot dynamics.

Localization Performance for Agricultural Robots

22

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

• Combines smoothness and short-term accuracy of neural networks with long-term precision of noisy GPS/GNSS updates under 1 MB of memory.

​

• Constrains the error to 2.75 m even with 20 minutes of GPS outage.

Proposed Solutions

23

[1] Saha, Swapnil Sayan, et al. "Tinyodom: Hardware-aware efficient neural inertial navigation." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6.2 (2022): 1-32.

[2] Du, Yayun, et al. “Neural-Kalman GNSS/INS Navigation for Precision Agriculture”, in 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2023.

[3] Saha, Swapnil Sayan, et al. “TinyNS: Platform-Aware Neurosymbolic Auto Tiny Machine Learning.” in ACM Transactions on Embedded Computing Systems (2023). (under review after revision)

• Platform and physics-aware inertial-sequence learning pipeline for microcontrollers for accurate yet lightweight inertial odometry to track humans, animals, cars, drones etc.

​

​

​

• A neural-Kalman filter to combine neural-inertial navigation with classical physics-aware inertial navigation models for precise and accurate localization.

​

​

​

• Data-efficient transfer learning to improve utility of pre-trained neural-inertial navigation models.

​

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

Transfer Learning

24

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

• Transfer learning: Train a few layers of a pre-trained neural network on a small labeled dataset in the target domain, allowing the network to adapt to the deployment conditions without losing previous knowledge.

​

• Example: Neural inertial navigation models struggle to generalize across different datasets or applications (1.6x - 13.6x performance loss).

​

​

Data-Efficient Transfer Learning

25

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

• Fine-tuning is 20x data-efficient over training from scratch.

​

• 1 minute of fine-tuning outperforms training from scratch by 3x - 14x.

​

Collecting Labeled Inertial Odometry Data

26

Vegetable picking

Insecticide spraying

Raw Video Frame

RGB to Gray

Extended-Maxima Transform

Morphological Opening

Video Pre-processing

Object Tracking (Kanade-Lucas-Tomasi)

Frame 1

Frame i

Frames

Pixel to Position Transformation

Bounding box to centroid extraction

Wind drift correction

Missing data interpolation

Smoothing

Scaling

Bounding

Boxes

x and y robot position in global coordinates

Robot

Landmark

Robot

Landmark, i

Landmark, j

Landmark, k

Landmark, l

h

v

Platform and physics-aware navigation

Neural-Kalman filtering

Data-efficient transfer learning

​

• Collecting high resolution labeled data in the wild is challenging.

​

• Solution: Automated labeled data generator using quadrotor video feeds.

​

​

Open Research Directions

27

On-device domain adaptation

Context-Aware Embeddings

Uncertainty-Aware Inertial Navigation

THANK YOU

Contact: swapnilsayan@g.ucla.edu

​

Code: https://github.com/nesl/tinyodom

https://github.com/nesl/agrobot

28