1 of 8

Learning Hand Gestures using Synergies in a Humanoid Robot

04/26/2024

Parthan Olikkal1, Dingyi Pei1, Bharat Kashyap Karri2, Ashwin Satyanarayana3, Nayan Kakoty4, Ramana Vinjamuri1

1University of Maryland Baltimore County, USA

2BITS, Pilani, India

3City Tech at CUNY, USA

4Tezpur University, India

​

1

2 of 8

Introduction

  • A simple kinematic model of human hand has > 20 DoFs making it extremely challenging to be replicated
  • Hand gestures offers a natural communication methods – vital for human-robot interaction
  • How does the human brain choose which combination of muscles and joints to be recruited to perform any task?
    • Modularity Hypothesis by Bernstein : Concept of Synergies
  • Any movement can be decomposed as – Reaching, Grasping and Release phases
  • Similar trend can be observed for any joint

Start (Initial Position) → Target → Return (Initial Position)

  • These implies that a Gaussian curve can well present this trend under some constraint

​

04/26/2024

2

3 of 8

Methods and Analysis

  • MediaPipe Framework
    • Using single RGB camera, real-time hand gesture can be recognized
    • Outputs 21 landmarks on the 3D hand-knuckle skeletal image on the hand
  • Experiment
    • Hand gestures were shown to the RGB camera of a humanoid robot
    • Landmarks were identified based on Euclidean distance
    • 10 active hand gestures considered: Okay, open palm, three, peace, up, rock on, hang loose, four, pinch middle, pinch ring
  • Joint Angular Velocities
    • End postures of each gestures were converted to angular joint velocities using a gaussian function → 11 Joint angular velocities (MCPs, PIPs, IP, CMC)
    • Dataset split as : training (7) and testing (3) sets

​

04/26/2024

3

4 of 8

Methods and Analysis

4

04/26/2024

  • 11 joint angular velocity profiles were generated through the gaussian function
  • Each pie represents one hand gesture, and each shaded sector represents the presence of active gaussian function

5 of 8

Results

  • Derivation of Synergies
    • From the 11 synthetic joint angular velocities of 7 training hand gestures, through dimensionality reduction of PCA
    • 3 kinematic synergies contributed to 90% of variance
    • These kinematic synergies were used to reconstruct the hand gestures in the testing set. 10-fold cross validation was performed as well
  • Translating Hand Gestures to Humanoid
    • The kinematic synergies extracted, and the reconstructed pattern of gestures were translated into a humanoid using moving average and scaling coefficient

04/26/2024

4

6 of 8

Discussion

04/26/2024

5

  • One of the first attempts to extract synthetic kinematic synergies from human inspired gaussian function generated angular joint velocities.
  • Though a simple mapping of the 21 landmarks can perform the hand gestures, it is demonstrated here that using only a subset of the 21 landmarks, the selected hand gestures can be executed
  • No need of multiple demonstration
  • Single RGB camera

7 of 8

Conclusion

  • Limitations
    • Current scaling coefficient and mapping functions are linear
    • Uses an offline model, and static wrist movement
    • Limited number of hand gestures in the dataset

​

  • Future Works
    • Non-linear mapping function can be developed to improve the continuous hand movement gestures for each joint
    • Extending the results to an online model that imitates the hand gestures in real-time
    • Increasing the scope of the dataset to include American Sign Language (ASL) along with flexion of wrist movements.

​

04/26/2024

6

8 of 8

References

  1. P. Olikkal, D. Pei, B. K. Karri, A. Satyanarayana, N. M. Kakoty and R. Vinjamuri, "Learning Hand Gestures using Synergies in a Humanoid Robot," 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO), Koh Samui, Thailand, 2023.
  2. N. Bernstein, "The Co-ordination and Regulation of Movement", 1967.
  3. R. Rastgoo, K. Kiani and S. Escalera, "Hand sign language recognition using multi-view hand skeleton", Expert Syst. Appl, vol. 150, Jul. 2020.
  4. F. Zhang et al., "MediaPipe Hands: On-device Real-time Hand Tracking", 2020, [online] Available: http://arxiv.org/abs/2006.10214.
  5. R. Vinjamuri et al., "Dimensionality Reduction in Control and Coordination of the Human Hand", IEEE Trans. Biomed. Eng, vol. 57, no. 2, pp. 284-295, 2010.
  6. P. Olikkal, D. Pei, T. Adali, N. Banerjee and R. Vinjamuri, "Musculoskeletal Synergies in the Grasping Hand; Musculoskeletal Synergies in the Grasping Hand", 2022 44th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc, 2022.
  7. P. Olikkal, D. Pei, T. Adali, N. Banerjee and R. Vinjamuri, "Data Fusion-Based Musculoskeletal Synergies in the Grasping Hand", Sensors (Basel), vol. 22, no. 19, Oct. 2022.
  8. V. Patel, M. Burns, Z. H. Mao, N. E. Crone and R. Vinjamuri, "Linear and Nonlinear Kinematic Synergies in the Grasping Hand", J Bioeng. Biomed. Sci, vol. 5, no. 2, pp. 163, 2015.

8

04/26/2024