FVD - Encore
Date: Wednesday, Nov 30 2022
Team H - TouRI
Prakhar Pradeep
Jigar Patel
Shruti Gangopadhyay
Jashkumar Diyora
Shivani Sivakumar
Software and�Interface Lead
Hardware and �Sensors Lead
Autonomous Navigation�Lead
Perception�Lead
Autonomous Manipulation Lead
TouRI Advisors/Sponsors
Zackory Erikson
Yonatan Bisk
RCHI Lab, RI, CMU
CLAW Lab, LTI, CMU
Assistant Professor
Assistant Professor
TouRI - Overview
Use Case
Use Case
Use Case
TouRI
Use Case
PAN-TILT DISPLAY
To provide human interaction
MANIPULATOR
To facilitate interaction with the environment
MOBILE BASE
To facilitate traversal
Use Case
Use Case
Use Case
Use Case
Use Case
Use Case
Autonomous Pick and Place
System Design
Receive user input from interface with a latency less than 5 seconds
System Design
System Design
System Design
System Design
System Design
Interface subsystem - Performance
Receive user input from interface with a latency less than 5 seconds
System & Demo Overview
Software Architecture - Autonomous Picking
Software Architecture - Autonomous Picking
PERCEPTION
MANIPULATION
Functional Architecture - Autonomous Placing
PERCEPTION
MANIPULATION
Automated data labelling pipeline
Dectectron2 framework
Perception - 2D Pipeline
"0" : 'cmu_tartan_bottle',
"1" : 'tennis_ball_toy',
"2" : 'cmu_cup',
"3" : 'cmu_bottle',
"4" : 'monkey_keychain',
"5" : 'transparent_bottle',
"6" : 'all_star_dogs_belt',
"7" : 'dog_collar',
"8" : 'cow_keychain',
"9" : 'beanie',
"10" : 'unicorn',
"11" : 'airpods_case'
Labels
Souvenir picking 3D pipeline (C++)
3D Pipeline - Souvenir centroid estimation
Point Cloud Cropping
Centroid estimation
Souvenir Detection
Souvenir picking 3D pipeline (C++)
3D Pipeline - Shipping box centroid estimation
Point Cloud Cropping
Centroid estimation
Shipping box Detection
Plane detection
Autonomous pick and place
Performance Validation
Requirements Satisfied:
Subsystem Performance Results:
Perception subsystem - Performance
Manipulation subsystem - Performance
Average: 57.62 mins
Average: 58.87 mins
Manipulation subsystem - Performance
Requirement Metrics | Achieved Metrics |
Detect objects with a precision of 70% and recall of 60% | Objects detected with a precision of 81% and recall of 94% |
Estimate centroid objects with a precision of 65% | Centroid of objects estimated with a precision of 81% |
Plan manipulator motion to grasp object within 3 minutes | Manipulator motion to grasp object planned in 12 seconds |
Grasp or place object within 5 minutes, with 67% success rate (2 successful trails out of 3) | Object grasped/dropped in 52 seconds, with 80% success rate (4 successful trials out of 5) |
Manipulation and Perception Sub Systems - Performance
Natural Language Understanding and Cognition
Natural Language Understanding
Hey Touri, pick up a gift for my dog
WAKE WORD
SPEECH RECOGNITION
COGNITION
Natural Language Understanding
Multilingual speech recognition as well as speech translation and language identification
WAKE WORD
SPEECH RECOGNITION
COGNITION
Cognition
Fine-tuned GPT-3 (text-davinci-003)
OBJECT ID
Knowledge-base
Visual feedback
Command
Cognition
Fine-tuned GPT-3 (text-davinci-003)
OBJECT ID
Knowledge-base
Visual feedback
Command: Buy a gift for my dog
Object: Dog belt
Cognition
Fine-tuned GPT-3 (text-davinci-003)
OBJECT ID
Knowledge-base
Visual feedback
Command: My friend loves music. Pick a gift for him
Object: No object
Cognition
Command: Pick up the unicorn
Object: Unicorn
Command: Buy a gift for my dog
Object: Dog belt
Command: My friend loves music. Pick a gift for him
Object: Airpod case
Mechatronics Subsystem
and Interface
What you will see - Mechatronics/Interface
Provide upto 1080*720 resolution video call interface for user to interact with surroundings with a lag less than 2 seconds
What you will see - Mechatronics/Interface
Provide traversal feedback to user every 5 seconds
What you will see - Mechatronics/Interface
Provide gimbal motion of 60 degrees in pitch and 120 degree in yaw for the display device
What you will see - Mechatronics/Interface
Receive user input from interface with a latency less than 5 seconds
What you will see - Mechatronics/Interface
Receive user input from interface with a latency less than 5 seconds
Navigation Subsystem
Hard Case Test- Dynamic Environment
System & Demo Overview
2 modes of the navigation system- Autonomous mode and Teleoperation mode
Teleoperated Navigation
Autonomous Navigation
Tour location received from interface
Robot localization and path planning
Static and dynamic obstacle avoidance
Robot teleoperation through interface
Obstacle detection and user alerts through interface
Functional Architecture
Sensors
Functional Architecture
Robot Localization
Functional Architecture
Path planning & Obstacle Avoidance
Functional Architecture
Teleoperated control
Navigation subsystem - Functional Architecture
Navigation subsystem - Performance
Requirement Metrics | Achieved Metrics |
Traverse on hard, flat indoor floors reliably at 0.4 m/s during teleoperation | Traversed on hard, flat indoor floors reliably at 0.4 m/s during teleoperation |
Reach the desired location 50 m away within 30 minutes | Reached the location 60.96 m away within 5 minutes 34 seconds |
Plan global path to the desired location within 3 minutes | Planned global path to the desired location within 4.23 seconds |
Detect and avoid obstacles with mAP of 80% during autonomous navigation | Detected and avoided 90% of the obstacles |
Detect obstacles with mAP of 80% during teleoperation | Detected 100% of the obstacles |
Navigation subsystem - Performance
60 m autonomous traversal time
Time for path planning
Obstacle avoidance success
Average: 5.34 mins
Average: 4.23 secs
90% success rate
Thank You Everyone�
Special Thanks
Prof Zackory, Yonatan, John, Dimi and TAs
AI Makerspace - Greg
�
Time for Souvenir Giveaway!
Get your voucher from Jash and wait for next instructions
�
�