1 of 72

Harshvardhan

Takawale

Nirupam

Roy

Yang

Bai

Irtaza

Shahid

Scribe: Simultaneous Voice and Handwriting Interface

2 of 72

Motivation

System Design

Evaluation

Application

3 of 72

Motivation

System Design

Evaluation

Application

4 of 72

Motivation

System Design

Evaluation

Application

5 of 72

Motivation

System Design

Two-Factor Authentication (2FA)

Evaluation

Application

6 of 72

Motivation

System Design

Two-Factor Authentication (2FA)

Something you

ARE

Voice

Biometric

Evaluation

Application

7 of 72

Motivation

System Design

Two-Factor Authentication (2FA)

Something you

ARE

Something you

KNOW

Voice

Biometric

Signatures

+

Evaluation

Application

8 of 72

Motivation

System Design

Two-Factor Authentication (2FA)

Something you

ARE

Something you

KNOW

Secure

Access

Voice

Biometric

Signatures

=

+

Evaluation

Application

9 of 72

Motivation

System Design

Evaluation

Application

[1] Z. Wang et al.: Hand Gesture Recognition Based on Active Ultrasonic Sensing of Smartphone: A Survey

10 of 72

Motivation

System Design

Evaluation

Application

11 of 72

Motivation

System Design

Evaluation

Application

12 of 72

How do people prefer to write? Especially in-Air!

13 of 72

  • “I think it’s the most natural version and the one that I am probably most accustomed to.” [P1]

​

  • “Pen-like device will be much better than finger. Because we are used to that.” [P2]

​

  • “Drawing freeform would be easier with just my finger, but for actual writing, I would prefer to use a stylus, so that I can more precisely control exactly where the device is reading from.” [P6]

14 of 72

Motivation

System Design

Evaluation

Application

Can voice interfaces detect hand signatures?

15 of 72

Can voice interfaces detect hand signatures?

Motivation

System Design

Evaluation

Application

16 of 72

Challenges

Motivation

System Design

Limited frequency band for tracking

High precision tracking

Multipath interference

Evaluation

Application

17 of 72

Challenges

Motivation

System Design

Limited frequency band for tracking

High precision tracking

Multipath interference

Evaluation

Application

18 of 72

Frequency band is limited resource

10k

Frequency

20k

30k

40k

Amplitude

Inaudible

Audible

Motivation

System Design

Evaluation

Application

19 of 72

Frequency band is limited resource

10k

Frequency

20k

30k

40k

Amplitude

Inaudible

Audible

Microphone

filter

Microphone’s recordable bandwidth

Motivation

System Design

Evaluation

Application

20 of 72

Frequency band is limited resource

Frequency band of human voice

Motivation

System Design

Applications

Conclusion

21 of 72

Frequency band is limited resource

Amazon Alexa and Amazon Echo’s recordable bandwidth

10k

Frequency

20k

30k

40k

Amplitude

Higher frequencies

Human voice

Microphone

filter

Human voice’s bandwidth

Motivation

System Design

Evaluation

Application

22 of 72

Frequency band is limited resource

Amazon Alexa and Amazon Echo’s recordable bandwidth

10k

Frequency

20k

30k

40k

Amplitude

Higher frequencies

Human voice

Microphone

filter

Human voice’s bandwidth

Frequency band of human voice

Frequency band for sensing

Motivation

System Design

Applications

Conclusion

23 of 72

Amazon Alexa and Amazon Echo’s recordable bandwidth

10k

Frequency

20k

30k

40k

Amplitude

Higher frequencies

Human voice

Human voice’s bandwidth

Frequency band of human voice

Frequency band for sensing

The solution: pure-tone based ranging

Occupy a single frequency, minimizing frequency resource needed for ranging!

Motivation

System Design

Applications

Conclusion

24 of 72

Amazon Alexa and Amazon Echo’s recordable bandwidth

10k

Frequency

20k

30k

40k

Amplitude

Higher frequencies

Human voice

Human voice’s bandwidth

Motivation

System Design

Evaluation

Application

Occupy a single frequency, minimizing frequency resource needed for ranging!

Frequency band for sensing

The solution: pure-tone based ranging

25 of 72

Amazon Alexa and Amazon Echo’s recordable bandwidth

10k

Frequency

20k

30k

40k

Amplitude

Higher frequencies

Human voice

Human voice’s bandwidth

Frequency band for sensing

The solution: pure-tone based ranging

Occupy a single frequency, minimizing frequency resource needed for ranging!

Easily interfered by multipath!

Audible to users!

Motivation

System Design

Evaluation

Application

26 of 72

Amazon Alexa and Amazon Echo’s recordable bandwidth

10k

Frequency

20k

30k

40k

Amplitude

Higher frequencies

Human voice

Human voice’s bandwidth

Frequency band of human voice

Frequency band for sensing

The solution: pure-tone based ranging

Occupy a single frequency, minimizing frequency resource needed for ranging!

Easily interfered by multipath!

Audible to users!

Motivation

System Design

Applications

Conclusion

27 of 72

Multipath: Ranging error by tracking phase

Ground-truth phase rotation

 

 

Error of phase

28 of 72

Multipath: Ranging error by tracking phase

Ground-truth phase rotation

Vector of NLOS path

 

 

 

Error of phase

29 of 72

Multipath: Ranging error by tracking phase

Ground-truth phase rotation

Vector of NLOS path

Captured phase rotation

 

 

 

 

Error of phase

30 of 72

Multipath: Ranging error by tracking phase

Ground-truth phase rotation

Vector of NLOS path

Captured phase rotation

 

 

 

 

Error of phase

(b) Low-frequency signal

t = t1

t = t2

31 of 72

Multipath: Ranging error by tracking phase

Ground-truth phase rotation

Vector of NLOS path

Captured phase rotation

 

 

 

 

Error of phase

(b) Low-frequency signal

t = t1

t = t2

High-frequency signal

t = t1

t = t2

Tracking using high-frequency signal induces less distance error under multipath

32 of 72

High-frequency signal leads to smaller distance offset

Easily interfered by multipath!

Audible to users!

Motivation

System Design

Distance offset is proportional to the inverse of frequency

Error of phase

 

Evaluation

Application

33 of 72

Challenges

Motivation

System Design

Limited frequency band for tracking

High precision tracking

Multipath interference

Evaluation

Application

34 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Motivation

System Design

Evaluation

Application

35 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Whole process is inaudible for human!

Motivation

System Design

Evaluation

Application

36 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

F2

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Secondary

signal

Motivation

System Design

Evaluation

Application

37 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

F2

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Secondary

signal

Whole process is inaudible for human!

Motivation

System Design

Evaluation

Application

38 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

F2

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Secondary

signal

Implicit frequency shifting

Motivation

System Design

Evaluation

Application

39 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

F2

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Secondary

signal

Implicit frequency shifting

Whole process is inaudible for human!

Motivation

System Design

Evaluation

Application

40 of 72

Cross-Frequency Continuous Wave (CFCW) Design: frequency shifting

F2

10k

Frequency

20k

30k

40k

50k

60k

70k

80k

90k

100k

Amplitude

Inaudible

Audible

Microphone

filter

F1

Primary signal

Secondary

signal

Implicit frequency shifting

Whole process is inaudible for human!

Motivation

System Design

Evaluation

Application

41 of 72

Cross-Frequency Continuous Wave (CFCW) Design: phase mapping

Motivation

System Design

Evaluation

Application

42 of 72

Cross-Frequency Continuous Wave (CFCW) Design: phase mapping

Motivation

System Design

Evaluation

Application

43 of 72

Challenges

Motivation

System Design

Limited frequency band for tracking

High precision tracking

Multipath interference

Evaluation

Application

44 of 72

Cross-Frequency Continuous Wave (CFCW) Design: multipath elimination

LOS

NLOS

Environmental reflections

Motivation

System Design

Evaluation

Application

45 of 72

Cross-Frequency Continuous Wave (CFCW) Design: multipath elimination

LOS

NLOS

Environmental reflections

Time

LOS

NLOS

WIN-LOS

WIN-NLOS

Received signal

Motivation

System Design

Evaluation

Application

46 of 72

Freq. hop

Multipath noise

Primary

signal

Frequency

Time

47 of 72

Freq. hop

Multipath noise

Primary

signal

Frequency

Time

48 of 72

Freq. hop

Freq. hop

Multipath noise

Secondary

signal

Primary

signal

Frequency

Time

49 of 72

Frequency hopping

is transparent to target signal

Freq. hop

Freq. hop

Multipath noise

Produces

difference

signal

Implicit

mixing

Secondary

signal

Primary

signal

Frequency

Time

X

50 of 72

Frequency hopping

is transparent to target signal

Freq. hop

Freq. hop

Multipath noise

Produces

difference

signal

Implicit

mixing

Secondary

signal

Primary

signal

Frequency

Time

X

Multipath produces

out-of-band noise

51 of 72

Application-specific design to convert 3D in-air handwritings into 2D “on-paper” handwritings.

Motivation

System Design

Applications

Conclusion

52 of 72

Application-specific design

Motivation

System Design

3D location optimization

Pen-lift removal

Flatten writing surface

Alexa

 

 

 

Alexa

0 degree

60

degree

90

degree

Evaluation

Application

53 of 72

Application-specific design

Motivation

System Design

3D location optimization

Pen-lift removal

Flatten writing surface

 

 

 

0 degree

60

degree

90

degree

Evaluation

Application

54 of 72

Application-specific design: 3D location optimization

Motivation

System Design

Applications

Conclusion

Alexa

 

 

 

55 of 72

Application-specific design: remove pen-lifts from virtual surface

Motivation

System Design

Applications

Conclusion

56 of 72

Application-specific design: remove pen-lifts from virtual surface

Motivation

System Design

Applications

Conclusion

57 of 72

Application-specific design: flatten writing surface

Alexa

0 degree

60

degree

90

degree

Motivation

System Design

Applications

Conclusion

58 of 72

Motivation

System Design

Quantitative Evaluation

Speaker

Authentication

Professional Signature Verification

Evaluation

Application

1D ranging error: 73 μm

3D tracking error: 1.4 mm

Increase in speaker authentication error: 2%

Signature verification accuracy: 90%

59 of 72

Application 1: �Capture physical signatures on Alexa

Motivation

System Design

Evaluation

Application

60 of 72

Trajectory of stylus

Recovered trajectory

Motivation

System Design

Evaluation

Application

Application 1 : Capture physical signatures on Alexa

61 of 72

Application 2:

Handwriting tracking co-exist with voice interface

Motivation

System Design

Evaluation

Application

62 of 72

Trajectory of stylus

63 of 72

Trajectory of stylus

Recovered trajectory

64 of 72

Speech-to-text:

“Hi, I am drawing a star”

Trajectory of stylus

Recovered trajectory

Motivation

System Design

Evaluation

Application

ASR

Application 2 : Handwriting tracking co-exist with voice interface

65 of 72

Application3:

Bring Alexa to speech disabilities

Motivation

System Design

Evaluation

Application

66 of 72

Trajectory of stylus

Recovered trajectory

Postprocessed handwriting

Motivation

System Design

Evaluation

Application

Application 3 : Bring Alexa to speech disabilities

67 of 72

Thank you!

Harshvardhan Takawale

Nirupam Roy

Yang Bai

Irtaza Shahid

Check out

our paper:

Group website:

http://icosmos.cs.umd.edu

68 of 72

Recovered trajectory

69 of 72

Evaluation

Ranging accuracy

70 of 72

Ranging accuracy

Localization accuracy

1-D ranging error of 73 micro-meter and below 1.4 millimeters of median error in 3D trajectory tracking

Motivation

System Design

Applications

Conclusion

71 of 72

72 of 72

Performance under multipath