Acknowledgements:
Special thanks to Professor Yogananda Isukapalli, Alex Lai, Brian Li, Eric Hsieh, and our mentors Jessica Marquez and John Karasinski
P.E.T.E: Procedure Execution Tracking Engine
Sophie Guan, Anoushka Sawant, Aaron Sin, Spencer Tang, Frank Yao
Background
Astronauts onboard the International Space Station (ISS) follow intricate step-by-step instructions and often encounter maintenance challenges when completing procedures. We propose an automated solution: a procedure-tracking system that uses a computer vision model and sensors to validate user progress.
Software Flow
Hardware Diagram
GUI
NVIDIA Jetson Orin Nano
Econ E-Cam81_CUNX Camera
ESP-32S Microcontroller
MPU- 6050 Acc/Gyro
Computer Vision
Each step in the procedure involves real-time image processing and identification of all elements involved, with specific substeps that must be satisfied before proceeding to the next step.
Camera Capture:
Transforms raw camera data into a workable live video feed
Sensor Capture:
Transforms raw gyro/acc/temp data to usable metrics
Detection Phase:
Computes bounding boxes and angular displacement
Validation Phase:
Validates detection against ground truth
User Display and Input:
Shows bounding boxes and allows for manual override
Model and Performance:
Runs YOLOv7-tiny, a real-time object detection model built for edge devices. This smaller model provides faster inference speed and lower system
cost due to reduced parameters (6M).
Dataset and Training Enhancements:
Training dataset were manually collected and labeled and augmented with CutMix augmentation. Training employed Transfer Learning on top of MS COCO dataset.