1 of 55

Virtual Piano

Tan Lai Chian Alan

2 of 55

Outline

  1. Background
  2. Problems
  3. Objectives
  4. Implementation
  5. Demo

3 of 55

Background

4 of 55

Piano

5 of 55

Barriers

  1. High Cost
  2. Medium-large space
  3. Regular maintenance
  4. Not portable

6 of 55

Video

https://www.youtube.com/watch?v=H22CjCq0CTM

https://www.youtube.com/watch?v=GeWIQJsbEf0

https://www.youtube.com/watch?v=yM1Y4gUZhpI

​

7 of 55

Smartphones

8 of 55

Smartphone Apps

Two types:

  1. On screen piano apps
    • Good performance
    • Small key sizes
    • Lack of realism

​

  1. Paper piano/Augmented Piano
    • Real-time image processing
    • Bigger key sizes
    • Encompasses its own set of problems

​

9 of 55

Problems

10 of 55

Problem

  • Detection inaccuracies
  • Piano layout constraints
  • Keypress region constraint
  • Latency
  • Still far from realistic playing of piano
    • Proper fingering, dynamics and articulations, and more…

​

11 of 55

Example of existing problems

https://www.youtube.com/watch?v=4cq4V9LElpg

12 of 55

Google Play Store

13 of 55

Virtual Piano

  1. Android version of Paper Piano
  2. Image processing based
  3. No sensors involved
  4. OpenCV

14 of 55

Objectives

15 of 55

Objectives

  1. Propose a set of methodologies in piano, fingers, touch, keystroke detection
  2. Functional smartphone application
  3. Exploration
    1. Multiple Fingers
    2. Multiple Hands
    3. Dynamic Touch

16 of 55

Implementation

17 of 55

Setup

18 of 55

Camera Perspective

Top View

Front View

19 of 55

Piano Layout

Hand Drawn

Printed

20 of 55

Performance Issue

  • Image processing on smartphones is very computationally intensive
  • Methodologies/algorithms were kept simple
  • Resolution: 325px by 288px
  • Frame rate: 15-25 fps

21 of 55

Main Components

  1. Piano Detection
  2. Finger Detection
  3. Keypress Detection
  4. Playing Sound

22 of 55

Piano Detection

  1. Detection of white keys
  2. Computation of piano mask
  3. Detection of black keys

23 of 55

How to detect the white keys?

24 of 55

Color Segmentation

25 of 55

What makes up a digital image?

26 of 55

Color Spaces

  1. RGB
  2. Grayscale
  3. HSV
  4. YCbCR
  5. etc...

27 of 55

Detection of White Keys

  1. RGB to Grayscale
  2. Binary Thresholding
  3. Contour detection
  4. Reduce contour points using Douglas-Peucker algorithm
  5. Eliminate white contours by area and number of points
  6. Sort keys using merge sort algorithm

28 of 55

RGB

Grayscale

(Value: 0 - 255)

​

Binary thresholding

(White: 150 - 255, Black: 0 - 149)

Find contours using

Border-Following algorithm

Reduced contour points

using Douglas-Peucker algorithm

After elimination

by area and contour points

29 of 55

Piano Mask

  1. Convex Hull of white keys
  2. Obtain piano mask

30 of 55

Convex Hull

31 of 55

How to detect the black keys?

32 of 55

Detection of Black Keys

  1. Morphological dilation to remove unwanted piano lines
  2. Invert binary image
  3. Apply piano mask and apply dilation
  4. Apply same method used for detection white keys

33 of 55

RGB

Grayscale

(Value: 0 - 255)

​

Binary threshold

(White: 150 - 255, Black: 0 - 149)

Apply morphological dilation

Invert image

Piano mask

34 of 55

After applying piano mask

Piano Keys

35 of 55

Result

(Video on piano detection)

36 of 55

Result

(Video on hand drawn piano detection)

37 of 55

Fingertip Detection

  1. RGB to HSV color space
  2. HSV thresholding
    1. Hue: 3-30
    2. Saturation: 50-255
    3. Value: 50-255
  3. Find contour with largest area
  4. Find lowest point of contour

38 of 55

RGB

HSV

​

HSV thresholding

Find contours using

Border-Following algorithm

Get contour with largest area

Find lowest point of contour

39 of 55

Result

(Video on fingertip detection)

40 of 55

Touch/Keystroke Detection

  1. Point-In-Polygon algorithm
  2. Motion analysis
    1. Comparing current frame with previous frame
    2. Difference of fingertip y-coordinate points

41 of 55

Sound

  • Android media playback options
    • SoundPool
      • Good for short sound clips
    • AudioManager
    • MediaPlayer
  • Sound clip size
    • Size: ~32kb

42 of 55

Result

(Video)

43 of 55

Accidental Keystrokes

  1. Fluctuation of fingertip points
    • Solution: KeyPressedIndex variable
  2. Perspective error in 2D view
    • Takeaway: Avoid elevating the camera as much as possible (Increases difficulty in detecting piano keys though)
    • Partial solution:
      • Reduce size of white key
      • Limit white key keypress region to lower half of piano

44 of 55

Before correction

(Video)

45 of 55

After correction

(Video)

46 of 55

Advance Functions

  1. Multiple Fingers
  2. Multiple Hands
  3. Dynamic Touch

47 of 55

Multiple Fingers

(Video)

48 of 55

Multiple Hands

  1. Find second largest hand contour
  2. Same process to find fingertips

49 of 55

Dynamic Touch

Relative loudness determined by value of y-coordinate difference between 2 consecutive frames

piano: 3-10 pixels

mezzo-piano: 11-20 pixels

forte: 21-30 pixels

fortissimo: > 30 pixels

50 of 55

Dynamic Touch

51 of 55

Achieved

  1. Flexibility
    1. Can cater up to 2 octave of keys (14 white keys, 10 black keys)
    2. Or less
    3. Wider coverage of keypress region
  2. 4 octaves of sounds (2 octaves each time)
  3. Multiple Hands, Multiple Fingers
  4. Dynamic touch

52 of 55

Future Improvements

  1. More robust detection methods
    1. Piano detection
    2. Finger detection
      1. Multiple fingers detection
    3. Keypress detection
  2. Better performance
  3. More features such as articulation

53 of 55

Demo

54 of 55

Questions?

55 of 55

Questions?

How does your approach/results compare to those in your literature review?

  1. Pros
    1. Able to cater to different skin tones without sampling
    2. Wider coverage on key press regions
    3. Not constrained to a fixed piano layout
    4. Motion analysis for touch detection allows implementation of dynamics, articulation, etc..
  2. Cons
    • Accidental keystrokes
    • Hard to detect multiple fingers (Using fingertip detection)