1 of 10

Gesture Recognition

Ananya Rao, Amanda Steiner

2 of 10

Problem Statement

Our focus is to enable a user to command Cozmo using hand gestures. In particular, Cozmo will be able to turn to an area the user is pointing towards.

3 of 10

Approach

Using transfer learning, we taught a neural net how to distinguish pictures with hands from those without.

Once Cozmo recognizes that the hand gesture is a point, he identifies where the fingertip is, and based on this, decides where to turn.

To find the fingertip, we filter out the pink post-it that the user is wearing and calculate the centroid to turn in that direction.

4 of 10

Demo

5 of 10

Filtering Out the Centroid of Hot Pink Post-It

6 of 10

Cozmo In Action

7 of 10

Cozmo In Action

8 of 10

Conclusion

9 of 10

Results

Cozmo can recognize when there is a hand, and when the hand is pointing.

He is a little inconsistent, but this could be solved by having a larger, more comprehensive dataset.

We discovered that hot pink is the easiest to filter out by color on Cozmo’s camera and works extremely well as an indicator.

10 of 10

Future Work

Future work might include making Cozmo decide which object the user is pointing at when there are two or more objects in frame.

Another possible extension would be to integrate this feature into cozmo-tools.