AI for Live NASA Rover Data Exploration in AR
Lance Santana
Lance.santana@berkeley.edu
NASA Glenn Research Center - Advanced Computing & Visualization Branch
Mentor: Herb Schilling
Mission Directorate: Space Technology
Abstract
�This project uses conversational AI to help NASA scientists explore Simulated Lunar Operations Laboratory (SLOPE) data in augmented reality (AR). A Python backend handles speech, analysis, and chart generation while Unity focuses on smooth immersive performance. The prototype uses prerecorded datasets and simulated telemetry, with live data integration planned.
Introduction
NASA’s SLOPE facility studies rover mobility across challenging planetary terrain, producing data on wheel slip, traction, torque, sinkage, stability, and motion.
This project uses conversational AI and AR to make that data easier to access and interpret.
Problem
Traditional analysis may require scientists to:
These steps slow the path from data to insight.
Objective
Develop a voice-driven system that can:
System Architecture
The system separates the Unity AR application from a Python-based AI backend. Unity handles headset interaction, voice capture, conversation history, and display, while the backend performs speech recognition, dataset selection, data analysis, chart generation, and speech synthesis. This division keeps computationally intensive processing outside Unity so the headset can prioritize smooth rendering and user comfort.
Current Capabilities
The prototype supports end-to-end conversational exploration of SLOPE rover datasets in VR. Users can ask questions by voice or text and receive answers, charts, and spoken feedback without leaving the headset.
Limitations
The system is currently a research prototype rather than a production deployment. Its main data sources are prerecorded experiments and simulated telemetry.
Figure 2. Voice or text queries are processed by a Python AI backend, which analyzes SLOPE data and returns charts and spoken feedback to the headset.
This poster is property of the United States Government
Technical Approach
Speech Processing: Local Whisper and Kokoro models enable fully offline voice input and spoken responses.
AI Query Routing: LangChain identifies question intent, required data, and visualization type, selects the relevant dataset subset, and generates a summary of each analysis to support contextual follow-up questions.
Data Analysis: Pandas filters, processes, and analyzes the selected data based on the user’s query.
Dynamic Visualization: Matplotlib generates the appropriate chart from the analyzed data and returns it to the Unity headset.
AR/VR Integration: Unity handles user interaction, conversation context, and display while computationally intensive AI and data processing run on the Python backend.
Conclusion
This project demonstrates how conversational AI can simplify data exploration in AR/VR, turning spoken questions into automated analysis, charts, and verbal responses.
Thank you to Herb Schilling for making this project possible!
Figure 3. Unity VR prototype showing the rover terrain environment, controller-based interaction, and an response panel displaying the system's results.
Figure 1. Two test vehicles driving on GRC-1 lunar simulant in the SLOPE lab, one on a flat surface and one on the adjustable tilt-bed.