Deep Convolutional Bidirectional LSTM for Complex Activity Recognition with Missing Data

Swapnil Sayan Saha*, Sandeep Singh Sandha* and Mani Srivastava

​

Networked and Embedded Systems Laboratory

University of California, Los Angeles

​

IoT - Large-Scale Multimodal Sensing

Embedded and

Wearable Computing

Mobile Computing

Cloud Services

Planetary-scale Networks

Credit: Mani Srivastava, UCLA ECE M202A, 2019

Challenge: Imperfect Data in the Wild

Sampling Rate Jitter

Timestamp Uncertainty

Missing Data

X X X X X X X X X X X

X X X X

X X X X X X X

w_T = k

f_s ≠ c

Physical Layer

Kernel Space

Application Layer

Data

Timestamp

Jitter in timestamping

Existing Methods

Limitations

• Human-engineered features – loss in high-dimensional spatial and temporal context for temporal sequences.
• Traditional interpolation techniques require implicit modelling assumptions.
• Deep learning methods outperform classical methods [1][2][3] but are unable to handle window and timing jitter.
• Existing methods fail to handle sampling rate invariability and timestamp uncertainty in fast temporal streams with missing data.

Feature Extraction

Interpolation, Imputation, Matrix Completion

[1]. Che, Zhengping, et al. "Recurrent neural networks for multivariate time series with missing values." Scientific reports 8.1 (2018): 1-12.

[2]. Yoon, Jinsung, William R. Zame, and Mihaela van der Schaar. "Estimating missing data in temporal data streams using multi-directional recurrent neural networks." IEEE Transactions on Biomedical Engineering 66.5 (2018): 1477-1490.

[3]. J. Yoon, J. Jordon and M. van der Schaar. “GAIN: Missing Data Imputation using Generative Adversarial Nets.” Proceedings of the 35th International Conference on Machine Learning, in PMLR 80:5689-5698. 2018.

​

​

Deep-Learning Methods

Proposed Pipeline

Controlled artificial time shifts in multimodal data

Metadata channels to quantify missing data and pop-ahead samples

Timing jitter in window length

Recurrent-convolutional architecture with conditional logic

Sampling Rate Jitter

[1]. Stisen, Allan, et al. "Smart devices are different: Assessing and mitigating mobile sensing heterogeneities for activity recognition." Proceedings of the 13th ACM conference on embedded networked sensor systems (SenSys). 2015.

[2]. Sandha, Sandeep Singh, et al. "Exploiting smartphone peripherals for precise time synchronization." 2019 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control, and Communication (ISPCS). IEEE, 2019. Code: https://github.com/nesl/Time-Sync-Across-Smartphones

Scheduling delay in presence of CPU load

Heavy multitasking and I/O loads

Variation in

OS timestamping delay

Timing stack delay

Sampling Rate Variability

Inferences

• Sampling interval of smartphone sensors vary widely [1].
• e.g. Inertial sensors: 40 – 100 Hz for 100 Hz accelerometer [1].
• e.g. Microphone: 189 KHz – 195 KHz for 192 KHz/24 bit [2].
• Our solution: Expose neural architectures to artificial jitter in window length.

​

Data Timestamp Uncertainty

[1]. Sandha, Sandeep Singh, et al. "Time Awareness in Deep Learning-Based Multimodal Fusion Across Smartphone Platforms." 2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI). IEEE, 2020.

Inferences

• System clock across sensors and devices is imperfect [1].
• Android NTP timing stack error: ~ 5000 mS.
• 6% drop in multimodal fusion classification accuracy with 1000 mS error.
• Poor management of timing stack and software stack delays inhibits direct usage of data timestamps.
• Our solution: Add controlled artificial shifts to the data

​

Artificial “mis”alignment to mitigate timestamping errors

1-sec error results in ~6% accuracy drop

No time error 96.1% accuracy

Missing Data Samples

[1]. Hossain, Tahera, et al. "A Method for Sensor-Based Activity Recognition in Missing Data Scenario." Sensors 20.14 (2020): 3811.

[2]. Yoon, Jinsung, William R. Zame, and Mihaela van der Schaar. "Estimating missing data in temporal data streams using multi-directional recurrent neural networks." IEEE Transactions on Biomedical Engineering 66.5 (2018): 1477-1490.

[3]. Che, Zhengping, et al. "Recurrent neural networks for multivariate time series with missing values." Scientific reports 8.1 (2018): 1-12.

[4]. S. S. Saha, S. S. Sandha and M. Srivastava, "Deep Convolutional Bidirectional LSTM for Complex Activity Recognition with Missing Data", Human Activity Recognition Challenge - Smart Innovations, Systems and Technologies, Springer (2020).

Inferences

• Missing data hurts deep learning performance (task dependent).
• e.g. Medical data imputation: 18 – 65% degradation [2].
• e.g. Medical data classification: 2 – 5% degradation [3].
• e.g. Complex activity recognition: 11 – 24% degradation [4].
• Our solutions:
• Independent mask metadata channel to characterize missing data [3] (affected by window and timestamping jitter due to presumptive sample alignment).
• Window alignment with contained samples popped ahead (actual data maybe 0).

​

Communication outage

Sensor malfunction

Power outage

Limited memory

Missing data

Sampling rate

jitter

Timing

errors

[1][2]

Neural Architecture for Making Complex Inferences

Inferences

• Time evolving spatial context in complex event primitives [1]:
• Handcrafted features require careful trade-off between differentiability and generalizability.
• Traditional classifiers offer unstable deployment performance [2] void of contextual dynamics.
• Deep learning methods adept at handling abnormal data in the wild over classical methods [3][4].
• Our solution:
• Ensemble of multilabel bidirectional LSTM-CNN networks (10X1) with majority decision fusion for coarse activity recognition.
• Conditional activation of binary LSTM-CNN networks (10X2) for granular activation recognition.

​

[1]. Xing, Tianwei, et al. "Deepcep: Deep complex event processing using distributed multimodal information." 2019 IEEE International Conference on Smart Computing (SMARTCOMP). IEEE, 2019.

[2]. Wang, Lin, et al. "Summary of the Sussex-Huawei locomotion-transportation recognition challenge 2019." Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 2019.

[3]. Che, Zhengping, et al. "Recurrent neural networks for multivariate time series with missing values." Scientific reports 8.1 (2018): 1-12.

[4]. Yoon, Jinsung, William R. Zame, and Mihaela van der Schaar. "Estimating missing data in temporal data streams using multi-directional recurrent neural networks." IEEE Transactions on Biomedical Engineering 66.5 (2018): 1477-1490.

​

​

Evaluation on Multimodal Fusion

Timing Error (ms)

Accuracy

No Augmentation: Accuracy drop by 3% with �600ms time errors

No Augmentation: Accuracy drop by 6%�with 1000ms time error�

1000ms Augmentation can handle ~600ms time errors

Idea: Add controlled �artificial shifts to the data

Settings: Up to 1000ms shift between modalities

No artificial time errors were introduced

[1]. Sandha, Sandeep Singh, et al. "Time Awareness in Deep Learning-Based Multimodal Fusion Across Smartphone Platforms." 2020 IEEE/ACM Fifth International Conference on Internet-of-Things Design and Implementation (IoTDI). IEEE, 2020.

Evaluation on Cooking Activity Dataset

Results

• 2 – 17% improvement in performance (w-w) from LSTM-CNN over pure CNN/LSTM.

​

• Time shift and window jitter augmentation with pop-ahead samples:
• 6% improvement in macro-activity classification over vanilla approach.
• 24% improvement in micro-activity classification over vanilla approach.

​

• Additional mask metadata channel:
• Further 5% improvement in macro-activity classification.

​

• Existing methods degrade classifier performance or show insignificant effects.

[1]. S. S. Saha, S. S. Sandha and M. Srivastava, "Deep Convolutional Bidirectional LSTM for Complex Activity Recognition with Missing Data", Human Activity Recognition Challenge - Smart Innovations, Systems and Technologies, Springer (2020).

Conclusion

• Data in the wild has sensing uncertainties.
• Multimodal data-processing architectures require time-awareness.
• Injection of runtime sensing uncertainty metadata into training pipeline.
• Learning pipeline robust to presence of missing data without losing information-rich high-dimensional context.

​

​

• Future Work:
• Does synthetic data generation yield enhanced granular inference recognition?
• Benchmark the learning pipeline on other applications.
• Task, space and time agnostic deep-learning architectures for complex event processing.
• Regression and estimation problems.

​