1 of 13

Privacy Leakage Study and Protection for Virtual Reality Devices

Dirk Catpo Risco, Brody Vallier, and Emily Yao

25 July 2024

Project Advisor: Dr. Chen

Mentors: Changming Li, Honglu Li, and Tianfang Zhang

2 of 13

Introducing the Team

Team members:

Mentors:

Advisor:

2

Dirk Catpo Risco

RU ECE MS

Changming Li

RU ECE PhD

Dr. Yingying (Jennifer) Chen

Honglu Li

RU ECE PhD

Tianfang Zhang

RU ECE PhD

Brody Vallier

RU ECE UG

Emily Yao

HTHS HS

3 of 13

Project Overview

Augmented reality (AR)/virtual reality (VR) devices are becoming more popular
Used in many applications (e.g. healthcare, communication, tourism)
Privacy concerns arise due to zero-permission sensors
We study activity privacy leakage in AR/VR devices
Our study focuses on activity recognition based on motion sensors

3

4 of 13

Week 8 Recap

Built a 1-dimensional (1D) convolution neural network (CNN) for activity recognition

Improving the prompt design to get more accurate predictions results from LLM

Setting CNN as baseline to adjust the LLM’s performance

4

5 of 13

Week 9 Progress

Built threat models for this AR/VR human activity recognition project (HAR)

Conducted feature extraction methods and used a support vector machine (SVM) model to select effective features

Improved LLM fixed prompt from a previous 78.18% to a 90.6%

5

6 of 13

Potential Security Application: Example 1

Typically, an attacker would need to acquire a large dataset to train an activity recognition model to understand user action
With an LLM, the attacker can use small groups of data collected via zero-permission motion sensors to infer benign user activities without model training
Attacker tracks user activities and acquire privacy-sensitive information:

Preferences for VR application usage
VR PIN pad patterns for unlocking VR devices.

6

7 of 13

Potential Security Application: Example 2

For AR/VR activity recognition, attackers can get access to AR/VR motion sensor
Attacker can inject designated noise to the received motion sensor data to induce model generating wrong prediction labels
LLM can perform robust against this noise and generate correct prediction when using manipulated data as input

Defend the recognition system

7

8 of 13

Feature Extraction for SVM

Statistical Methods:

Mean: average value of data
Std: variation of data
Min: smallest value in data
Max: largest value in data
Median: middle value of data
Interquartile range: spread of middle half of data
Peak to Peak: largest - smallest value in data
Entropy: randomness in data

8

9 of 13

SVM Model Result

6 motions
250 samples per motion
70/30 train-test split
Kernel is linear
Most effective features were mean, peak to peak, and interquartile range resulting in 99.33% test accuracy

9

10 of 13

LLM Prompt Adjustments

Changed threshold ranges on determining whether or not there is movement based on data
Updated example response by emphasizing what type of output response we want
Provided additional expert knowledge based on number of points above/below threshold

10

11 of 13

LLM Previous Statistics

11

Week 8: Gemini Advanced Accuracy
Trial #	Front Raise	Side Raise	Head Left	Head Right	Head Up	Head Down
1
2
3
4
5
6
7
8
9
10
Accuracy (%)	100	57	93	87	67	67
Key	Accurate (3/3)	Partial (2/3)	Inaccurate (1/3)	None (0/3)	Total (%)	78.33

12 of 13

LLM New Statistics

12

Week 9: Gemini Advanced Accuracy
Trial #	Front Raise	Side Raise	Head Left	Head Right	Head Up	Head Down
1
2
3
4
5
6
7
8
9
10
Accuracy (%)	100	100	90	76.7	76.7	100
Key	Accurate (3 /3)	Partial (2/3)	Inaccurate (1/3)	None (0/3)	Total (%)	90.6

13 of 13

Week 10 Goals

Improve LLM fixed prompt to get better results than 90.6% by adding statistical features into the prompt derived from SVM results (99.33%)

13