1 of 37

DGMD S-17:

Hand Signal Detection

2 of 37

Project Overview

3 of 37

Project Objective

  • Implement system whereby robot recognizes various hand gestures
  • Robot responds to gestures accordingly
  • Beneficial when dealing with traffic police/manual drivers

3

4 of 37

Team Members & Roles

  • Andrew Rodriguez - Design & Development
  • Longchang Cui - Design & Development
  • Jaya Johnson - Proof of Concept & Testing
  • Sarah Elsharkawi - Product Analysis & Presentation

4

5 of 37

Project Goals

5

6 of 37

Develop a recognition system for static gestures: stop, turn right, and turn left

6

1.

7 of 37

Guide robot movement based on the recognized static hand gesture

7

2.

8 of 37

Develop a recognition system for streaming gestures: move forward and move backward + guide movement based on streaming gestures

8

3.

9 of 37

Stretch: Test the final module on all 3 robot models

9

4.

10 of 37

Our Process

10

​

Phase II - Design & Development

Creating the algorithm needed to achieve goals

​

2

​

Phase I - Initial Setup

Setting up the required hardware and software

​

1

​

Phase III - Integration

Putting it all together + live demo

​

3

11 of 37

Phase I:

Initial Setup

12 of 37

Hardware Setup

WAVE = move forward

12

We have 3 different robot models:

Waveshare Jetbot (x2)

Yahboom Jetbot

Sparkfun Jetbot

13 of 37

Hardware: Setbacks/Challenges

  • Purchase of additional supplies (i.e. batteries, chargers ssd disks) caused delays in project start time
  • Setting up of Jetbot software, as each kit came with its own version of Jetson libraries (re-imaging/installing resolved this)
  • One Jetbot’s batteries would not power up OLED/ interact with motors despite powering up Nano (unresolved, had to switch to different model)
  • One Jetbot arrived late due to issues with customs clearance

13

14 of 37

Software Setup

  • OS: Linux
  • Language: Python, GIT, Jupyter notebooks
  • Libraries:
    • Jetbot (camera, basic motion)
    • trt_pose
    • jetcam
    • torch2trt

14

15 of 37

Software: Setbacks/Challenges

  • Initially installed “Openpose”: powerful framework for human pose tracking from videos/images/cameras
    • Jetson nano was not powerful enough to run framework
    • Frame rate of live camera feed = 1 ~ 2
    • Too slow to track gestures in real-time
  • Found “trt_pose": lightweight library that tracks human pose (skeleton)
    • Frame rate = 21 ~ 22
    • Fast enough to track pose/recognize gestures in real-time

15

16 of 37

Determining Hand Gestures

WAVE = move forward

16

The robot was trained to recognize and respond to the following hand gestures...

17 of 37

WAVE = move forward // BOTH ARMS MOVING BACK = move backward

17

18 of 37

LEFT ARM STRAIGHT OUT = turn left // RIGHT ARM STRAIGHT OUT = turn right

​

18

19 of 37

BOTH ARMS STRAIGHT OUT = stop

19

20 of 37

Phase II:

Design & Development

21 of 37

1. Use a hash table to track the existence of left/right wrist, elbow, shoulder, and neck joints on each frame.

21

Algorithm Design

​

22 of 37

2. Use Queue to store all the relative joints information for continuous pose tracking.

​

​

22

Algorithm Design

​

23 of 37

3. Check the relative wrist position from neck

  • Neck’s position = origin point
  • Wrists’ x and y coordinates used to calculate hand’s position/direction

​

​

​

23

Left Wrist = [Horizontal, Left]

Right Wrist = [Down, Left]

Left Wrist = [Down, Left]

Right Wrist = [Down, Right]

X = Left

Right

Y = Up

Down

Horizontal

Algorithm Design

​

24 of 37

4. Convert joints X, Y coordinates to vectors, and calculate the angles of 2D vectors.

  • Convert X, Y Coordinates to vectors

​

​

24

elbow_to_wrist_vector = (wrist.x - neck.x, wrist.y - neck.y)

elbow_to_shoulder_vector = (shoulder.x - neck.x, shoulder.y - neck.y)

​

  • Calculate the angle ꆪ between two vectors using the inverse cosine.

Algorithm Design

​

25 of 37

5. Analyze the human pose information and detect the gestures every two seconds.

  • With enough data from camera feeds, we can perform steps 3, 4 to get vector/angle information on each frame for 2 seconds.
  • Example: If we detect that wrist direction is upward, and angles of wrist-elbow-shoulder are between 40 to 120 degrees that last at least 8 frames. Then, we send gesture information to the robot to operate.

​

  • We can use similar logic to add various gestures

​

​

25

Algorithm Design

​

26 of 37

6. Depending on the gesture detected, operate the robot forward, backward, left, or right.

26

Inputs = [Wrists Up,

Angles between 40 ~ 120 degrees for 8 frames]

Robot Move Forward

Algorithm Design

​

27 of 37

27

Track wrist/elbow/ shoulder/ neck joints per frame

Store joint info. for cont. pose tracking

Check relative wrist position from neck

Convert (x, y) to vectors & calculate angles

Analyze info. & detect gestures every 2 sec

Operate robot forward/ backward/left/right

Algorithm: Summary

​

28 of 37

Phase III:

Integration

29 of 37

Learn More:

Our Project Github Repository

https://code.harvard.edu/loc630/DGMD-S-17-Project-Autonomous-Car-and-Hand-Signal-Recognition

29

30 of 37

Video:

Live Demo

31 of 37

31

32 of 37

Final Thoughts

33 of 37

Conclusion

  • Trt_pose worked very well with hardware/imaging/software setup
  • Library was easy to install/test with all different robots
  • Successfully implemented recognition system and achieved goals

33

34 of 37

Goals Achieved

34

Develop a recognition system for static hand gestures: stop, turn right, and turn left

​

Guide robot movement based on the recognized static hand gesture

​

Develop a recognition system for streaming hand gestures: move forward and backward + guide movement

​

Test the final module on all 3 robot models

​

35 of 37

Limitations

  • FPS is too low on Jetson Nano
  • 2 second interval may be too long (Accuracy VS Efficiency)
  • Distance is a big factor
  • Analysis of multiple human poses

​

35

36 of 37

Future Areas of Improvement/Exploration

  • Work with 3D coordinates instead of 2D ones
  • Add more complex signals (i.e. gestures with batons/colored flags)
  • Integrate signs/gestures into a single package
  • Extend to different signs in different geolocations - determine the right gesture for the right system
  • Improve latency with separate threads

36

37 of 37

Thanks!

Any questions?

You can find us at:

37