1 of 25

AccessWear: Making Smartphone Applications Accessible to Blind Users

MobiCom 2023, Madrid Spain

2nd October 2023

​

Prerna Khanna, Shirin Feiz, Jian Xu, IV Ramakrishnan, Shubham Jain, Xiaojun Bi, Aruna Balasubramanian

�

1

2 of 25

Introduction

2

Globally there are

43 million

blind users.

​

Are smartphone interactions accessible?

2

3 of 25

3

1 Ali Abdolrahmani, Ravi Kuber, and Amy Hurst. “An empirical investigation of the situationally induced impairments experienced by blind mobile device users”. In: Proceedings of the 13th International Web for All Conference. 2016, pp. 1–8.

2 Jian Xu, Syed Masum Billah, Roy Shilkrot, and Aruna Balasubramanian. “DarkReader: bridging the gap between perception and reality of power consumption in smartphones for blind users”. In: The 21st International ACM SIGACCESS Conference on Computers and Accessibility. 2019, pp. 96–104.

3 Barbara Leporini, Maria Claudia Buzzi, and Marina Buzzi. “Interacting with mobile devices via VoiceOver: usability and accessibility issues”. In: Proceedings of the 24th Australian Computer-Human Interaction Conference. 2012, pp. 339–348.

​

​

​

  • One-handed interactions are challenging1.

​

  • Susceptible to shoulder-surfing attacks2.

​

  • Gestures are overloaded3.

​

In this work, we focus on using alternate interactions techniques for blind users.

Are Smartphone Interactions Accessible?

3

4 of 25

4

Wearable Devices Can Serve as Alternate Modes of Interaction

4

5 of 25

5

AccessWear: Making Smartphone Applications Accessible to Blind Users

5

6 of 25

6

​

IRB-approved study with 9 blind users and 16 sighted users

Survey: Is there a need for alternate interaction?

Comparative experiment: Compare gestures of blind & sighted users.

Semi-structed interviews: What gestures do blind users prefer?

Exploratory Study

6

7 of 25

“In public, I don’t feel comfortable with the phone, I don’t want to draw much attention, swipes and all gestures are less noticeable.”

​

“Gives me the confidence to not take out the phone, everyone carries a watch, if I wave no one will know I am using a phone.”

​

“More security, keep phone inside the pocket.”

​

“Convenient, easy access.”

​

​

​

​

​

​

7

Implication 1: Participants prefer an alternate input modality.

Need for Alternate Gestures

7

8 of 25

​

​

​

​

​

​

8

1: strongly do not prefer

7: strongly prefer

Implication 2: Gesture should be easily customizable.

Median

score: 7 5 7 3

Forearm gestures

Shape gestures

Finger gestures

Wrist rotation gestures

Varied Gesture Preferences

8

9 of 25

​

​

​

​

​

​

9

Sighted Users

Blind Users

Template

User’s gesture

Inter-user variance

Blind users: 65.2

Sighted users: 29.8

Implication 3: Existing gesture recognition techniques using training data from sighted users are unlikely to work for blind users.

High Inter-user Variance

9

10 of 25

​

​

​

​

​

​

10

​

  • A robust gesture recognition system that works well with commodity sensors for blind users.

​

​

  • Near-zero-effort gesture replacement that allows a user to flexibly use any smartwatch gesture to interact with unmodified applications.

​

AccessWear Contributions

10

11 of 25

1Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. “Closing the gaps in inertial motion tracking”. In: Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 2018, pp. 429–444.

2Michael Xuelin Huang, Yang Li, Nazneen Nazneen, Alexander Chao, and Shumin Zhai. “Tap-Net: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN or Off-Screen Mobile Input”. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021, pp. 1–11.

3Hongyi Wen, Julian Ramos Rojas, and Anind K Dey. “Serendipity: Finger gesture recognition using an off-the-shelf smartwatch”. In: Proceedings of the 2016 CHI conference on human factors in computing systems. 2016, pp. 3847–3851.

​

​

11

MUSE (MobiCom’ 18)1

Tap-Net (CHI’ 21)2

Serendipity (CHI’ 16)3

Does not run on mobile

Requires lot of

training data

Requires Per-user training

​

Top 20 SOTA gesture recognition works do not meet these requirements.

​

Gesture Recognition Challenges

11

12 of 25

  • Accelerometer data is noisy for blind users
  • Gyroscope drift can be a problem; but we focus on short gestures

​

​

​

​

​

12

Accelerometer

data

Gyroscope

data

Insight 1: Using Gyroscope Data Only

12

13 of 25

13

Pre-stroke

Post-stroke

Nucleus

Gesture Phases

13

14 of 25

​

​

​

​

​

​

14

​

The nucleus is the core of the gesture and is consistent across all users.

​

Insight 2: Identifying User-Invariant Micro movements “Nucleus”

14

15 of 25

15

Pre-stroke

Post-stroke

Nucleus

Jitters

Forearm gesture

Wrist gesture

 

Nucleus Detection

15

16 of 25

  • Application logic and the interaction are coupled.
  • Integrating new gesture requires rewriting the application.

​

​

​

​

16

Gesture Replacement Challenges

16

17 of 25

17

Gesture Replacement Insights

17

18 of 25

18

Virtual Sensor Service

Sensor Service

User

Layer

Framework

Layer

Hardware

Layer

Smartphone

Phone App

Window Manager Service

Sensors

Virtual Touch

Sensor

Record Touch Gesture

Swipe

Replay Swipe

[t1, x1, y1]

[t2, x2, y2]

……

Record and Replay

Input Virtualization: Decouple Application Logic from Input

18

19 of 25

19

GUI

Next-Item

Forearm-right

Metaprogram

An alternate gesture can replace a touchscreen gesture for any application with zero changes to the application

Metaprogram: Flexible Gesture Mapping

19

20 of 25

20

Gyroscope Sensor

AcessWear

Watch Proxy

Smart Watch

Gesture Recognition

Mapping

Meta-

Program

Touch Screen Hardware

AccessWear

Phone Service

Smart Phone

App

Touch

Replayed

Run-time Implementation

20

21 of 25

21

​

  • Real world user study (IRB approved) with 8 blind users.
    • 5 male and 3 female
    • 34-61 years of age

​

  • Designed a task: required a series of gestures.

​

  • Collect traces of 5 gestures performed 3 times for offline analysis.

​

Real-World User Study

21

22 of 25

22

AccessWear works near real time.

220 msec reaction time of users1

​

1Henry Hamburger. Donald RJ Laming. Information, theory of choice-reaction times. New York: Academic Press, 1968. 1969.

Latency for Gesture Recognition and Replay�

22

23 of 25

23

Gesture Recognition Accuracy

AccessWear is 92% accurate and outperforms other baselines for blind users even with limited data.

23

24 of 25

24

We recruited 5 sighted users and added 3 more gestures.

.

​

​

Interacting with unmodified applications using alternate gestures: Spotify, YouTube, Gallery, and Browser

​

93% gesture recognition

accuracy

Extending AccessWear

24

25 of 25

  • Developed AccessWear, a real-time smartwatch-based gesture recognition system for blind users without any training data.

​

  • Developed novel input virtualization mechanism that eliminates the need for each app to integrate alternate gestures.

​

  • Demonstrated with a real-world user study with blind users that AccessWear is 92% accurate.

​

​

​

25

Summary

https://github.com/SBUNetSys/AccessWear

25