Privacy Leakage Study and Protection for Virtual Reality Devices

​

Dirk Catpo Risco and Brandon (Jinu) Son

​

6 June 2024

​

Project Advisor: Dr. Chen

Mentors: Changming Li, Honglu Li, and Tianfang Zhang

1

Introducing the Team

Week 1 Recap

• Benefits and applications of augmented reality (AR)/virtual reality (VR) devices

​

• Privacy concerns of zero-permission sensors

​

• Research paper [1] regarding eavesdropping attack Face-Mic

[1] Shi, C., Xu, X., Zhang, T., Walker, P., Wu, Y., Liu, J., Saxena, N., Chen, Y. and Yu, J., 2021, October. Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (pp. 478-490).

Week 2 Progress

• Read research paper [2] regarding large language models (LLMs) comprehending the physical world

​

• Build a connection between research paper and also privacy concerns of AR/VR devices

​

[2] Xu, H., Han, L., Yang, Q., Li, M. and Srivastava, M., 2024, February. Penetrative ai: Making llms comprehend the physical world. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications (pp. 1-7).

LLM Paper Motivation

• Using different smartphone sensor information, an LLM understands the physical world

​

• If success was shown using smartphone sensors, AR/VR sensors should as well

​

System Overview

• Three different smartphone sensors being used
• Accelerometer
• Satellite
• WiFi

​

• Input sensor info to LLM with fixed prompt

​

• LLM outputs user motion and environment conditions

Experiments

• Step count: if the user is walking
• Satellites detected: if the user is outside
• WiFi scanned: location of user
• Research paper examples:

Example 1

Example 2

Week 3 Goals

• Get familiar with AR/VR device:
• Meta Quest
• How to use device
• Configure settings on host computer

• Extract motion data from inertial measurement unit (IMU) sensors
• Connecting motion sensor application program interface (API) to access data
• Data processing method

Project Goal

• Extracting motion data from IMU sensors, input this data to an LLM to predict what the user is doing