1 of 18

Gesture Recognition Using mmWave Sensors

第15組

朱彥慈、黃朝旺

2 of 18

Introduction: Background knowledge of Gesture Recognition

1/3/2022

2

3 of 18

Introduction: Vision/Goal�Non-Contact Human Machine Interface

1/3/2022

3

4 of 18

9 Gestures

1/3/2022

4

  • Swipe Up
  • Swipe Down
  • Swipe Right
  • Swipe Left
  • Spin CW
  • Spin CCW
  • Letter Z
  • Letter S
  • Letter X

5 of 18

Dataset

1/3/2022

5

  • 5 Features
  • 9 Classes
  • 7402 Samples
  • Train:Eval:Test = 6:2:2

6 of 18

Original features used by the collector of the dataset

1/3/2022

6

  • x/65535
  • y/65535
  • Range_idx/65535
  • Peak_value/65535
  • Doppler_idx/65535

7 of 18

Features Engineering

1/3/2022

7

  • x, y coordinates
  • Distances
  • Angles
  • Velocities

8 of 18

New features used by us

1/3/2022

8

  • x, y coordinates, -1 < x < 1, 0 < y < 1

if (x, y) > 32767

[(x, y) – 65536] / 1024

else

(x, y) / 1024

  • Distances, 0 < d < 1.414
    • d = sqrt(x**2 + y**2)
  • Angles, -π/2 < θ < π/2
    • θ = arctan(x/y)
  • Velocities = doppler_idx × V_res

If doppler_idx > 32767

doppler_idx - 65536

9 of 18

Task Flow

1/3/2022

9

Dataset Analysis

Feature Engineering

Model Training

Predicting

Performance Analysis

Original features

New features

CNN

LSTM

Transformer

10 of 18

12/29/2021

10

Model

  • CNN
    • 3 layer 1D-CNN with kernel size is 3
    • full connect layer with 128 neurons
    • drop out rate: 0.5 ; activate function: Relu ; maxpooling
  • LSTM
    • 1 layer LSTM
    • full connect layer with 128 neurons
    • drop out rate: 0.5 ; activate function: Relu ; global average pooling
  • Transformer
    • multi-heads: 8
    • feed forward: 1D-CNN
    • drop out rate: 0.5 ; activate function: Relu ; global average pooling

11 of 18

Experiment

  • Hyper-parameter
    • Epoch: 100
    • Batch size: 128
    • Loss: cross entropy
    • Optimizer: adam
    • Early stopping: True

Model

Parameter

Running time (for each epoch)

CNN

318,089

1s

LSTM

630,153

4s

Transformer

1,518,612

10s

12 of 18

Result

Model

Original features (val)

Original features (test)

New feature (val)

New feature (test)

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

Accuracy

Macro F1

CNN

97.77

97.74

98.31

98.55

99.73

99.67

99.59

99.56

LSTM

98.04

98.11

97.37

97.64

99.59

99.47

99.59

99.62

Transformer

92.23

90.64

91.22

89.68

98.85

98.78

98.51

98.43

13 of 18

  • Original feature

​

​

​

​

​

​

  • New feature

Transformer - overfitting

14 of 18

Conclusion

  • The new feature can greatly improve the performance.
  • Both LSTM and CNN are capable of achieving great performance.
  • Overfitting occurs when training transformers.

15 of 18

Q&A

16 of 18

Backup- CNN

17 of 18

Backup- LSTM

18 of 18

Backup- Transformer