Privacy Leakage Study and Protection for Virtual Reality Devices

​

Dirk Catpo Risco, Brody Vallier, Emily Yao

​

7 August 2024

​

Project Advisor: Prof. Yingying (Jennifer) Chen

PhD Students as Mentors: Changming Li, Honglu Li, Tianfang Zhang

Introducing the Team Members

Students:

​

2

Dirk Catpo Risco

RU ECE MS

Changming Li

RU ECE PhD

​

​

Prof. Yingying (Jennifer) Chen

​

​

Honglu Li

RU ECE PhD

​

​

Tianfang Zhang

RU ECE PhD

​

​

Brody Vallier

RU ECE UG

Emily Yao

HTHS HS

​

​

Advisor:

Mentors:

Motivation

• AR/VR devices have attracted millions of users and facilitate a broad array of emerging AR/VR applications
• As a key component for motion tracking, Inertial Measurement Unit (IMU) consists of an accelerometer for measuring acceleration and a gyroscope for detecting rotations
• Both sensors are present in each controller and the Head Mounted Display (HMD)

​

​

​

​

3

Objectives

​

• Data from zero-permission motion sensors encodes various types of the user’s private information, such as activity information and preferences

​

• This project aims to study the sensor data management in commercial AR/VR headsets and analyze the potential of private information leakage

4

Methodologies

​

• Investigate privacy leakage in Augmented Reality (AR)/Virtual Reality (VR) devices

​

• Extract data from the IMU on AR/VR headset and controllers for Human Activity Recognition (HAR)

​

• Use Support Vector Machine (SVM) and Large Language Model (LLM) to show how IMU data maliciously exposes activities of victim users

5

Attack Illustration

• Utilize SVM as a baseline model to identify effective statistical features (e.g., mean, max, etc.) from motion data to recognize human activity

​

• Design LLM prompts based on the effective statistical features

​

• If LLM achieves comparable accuracy to SVM on motion prediction, it validates that adversaries could expose victim’s motion status without requiring data from victims

6

Motion Data Preprocessing

• Denoise and smooth data to generate accurate waveforms
• Compute 3D trajectories to visualize the motions

7

Example: Side Raise

Feature Extraction for SVM

8

Front Raise #2

IQR = 0.21

Mean = -0.11

Peak-to-peak = 8.0

Front Raise #1

IQR = 0.16

Front Raise #3

IQR = 0.21

Time (s)

Peak-to-peak = 7.3

Mean = 0.03

Peak-to-peak = 7.4

Mean = 0.04

Feature Extraction for SVM

9

Y-Axis Gyroscope

Ang (deg/s)

Ang (deg/s)

Ang (deg/s)

Time (s)

Y-Axis Gyroscope

Mean = 0.35

Peak-to-peak = 21.4

Head Left

IQR = 1.09

Mean = 0.15

Head Right

IQR = 0.39

Peak-to-peak = 12.2

Head Down

IQR = 0.32

Mean = 0.16

Peak-to-peak = 2.1

Support Vector Machine (SVM)

• An effective machine learning algorithm to find a hyperplane that separates classified data points

​

• Works well on accurately classifying motion sensor data

​

• Adversaries may require a huge amount of data from victim users during model training for accurate prediction

​

10

Large Language Model (LLM)

• Works well on recognizing human language and other complex tasks

​

• Can understand data and reproduce required outputs with designated prompts

​

• Pre-trained on vast amounts of data, adversaries may require no training data from victim users to accurately expose human motions

11

Experimental Setup

• Using Android Studio, we develop an application to extract data from the IMU sensors on Head-Mounted Display (HMD) and controllers of Meta Quest

​

​

12

Experimental Setup

• We designed 6 activities for evaluation, including two hand-related activities and four head-related activities

13

Front Raise

Head Down

Activity Inference Using SVM

• 3 statistic features (mean, peak-to-peak, and interquartile range) are extracted from the motion sensor data

​

• The overall accuracy of exposing 6 types of activities using SVM achieves 99.33%

14

Activity Inference Using LLM

15

• Developed a prompt for Gemini Advanced to understand the motion data
• Explained the goal of the task and data types to be received
• Asked LLM to extract features from the data and provided specific knowledge about how to utilize the features
• Provided a response structure for results

​

Example prompt for specifying accelerometer readings

Activity Inference Using LLM

• Using our prompt with Gemini Advanced, we achieve 90.6% accuracy

​

16

*H = Head L = Left Hand R = Right Hand*

Conclusion and Future Work

• With designated prompt, LLM achieves an accuracy similar to SVM, indicating the potential activity information leakage without training effort using LLM

​

• With further prompt fine-tuning, the adversaries could realize stronger activity exposure attack using LLM

17

Thank You for Your Time

Questions?

Scan for

Project Website → → →