System Development Review

Monday Oct 18th

Calen Robinson

Gitaek Lee

Jinkun Liu

Thomas Xu

Yukun Xia

​

Predictive Avoidance for Industrial Mobile Robots

Context:

• OMRON deploys fleets of industrial robots
• Obstacle avoidance can be improved to increase productivity

Main goals:

• Develop algorithm to detect and classify obstacles
• Avoid obstacles more efficiently by predicting their future motion
• Increase overall productivity for a fleet of robots over time

Sponsor: OMRON

2

Real-Life System (RLS)

• TurtleBot
• Functionalities:
• Obstacle Classification
• Obstacle Localization

Main Systems

Virtual Robot System (VRS)

• Gazebo
• Functionalities:
• Path Planning/Following
• Trajectory Prediction
• Obstacle Avoidance

​

3

Use Case

• The robot moves autonomously toward a commanded waypoint location
• The robot detects and localizes a moving obstacle in its path.
• The robot classifies it as a forklift
• The robot predicts the future trajectory of the forklift
• The robot evades the forklift based on its predicted trajectory
• The robot continues on the path to its goal location.

Use Case

5

• The robot moves autonomously toward a commanded waypoint location
• The robot detects and localizes a moving obstacle in its path.
• The robot classifies it as a forklift
• The robot predicts the future trajectory of the forklift
• The robot evades the forklift based on its predicted trajectory
• The robot continues on the path to its goal location.

Use Case

6

Requirements

Performance Requirements - Virtual System

From sponsor

specs and

environment

Overall performance metric

For real-time

operation

8

The System will:

V1. Autonomously move with max speed of 1.8 m/s

V2. Autonomously move with max rotation speed of 60 deg/s

V3. Autonomously move with min cruise speed of 0.5 m/s

V4. Increase robot productivity¹ by > 5% using avoidance strategies based on object classification, compared to nominal avoidance²

V5. Operate in real-time with planning time within 0.1 s

^[1] Productivity: The number of finished delivery tasks per unit time (e.g. day)

^[2] Nominal avoidance: Local A*. Currently employed by OMRON robots

(without any classification/prediction, treats everything as static)

​

​

Performance Requirements - Real-life System

Updated

Updated

The System will:

R1. Classify obstacles with precision of 70%

R2. Classify obstacles with recall of 80%

R3. Detect positions of obstacles of interest¹ within 0.1 m accuracy

R4. Detect forklifts within a range of 2 m

R5. Detect pedestrians within a range of 1 m

R6. Output results of positioning and classification within 100 ms per

frame

​

^[1] Obstacles considered: Forklifts, pedestrians

​

Non-functional Requirements

10

NF1. Mobility subsystem should be non-holonomic

(Restriction based on sponsor’s robots)

NF2. Modular¹ avoidance subsystem

(Allow comparison against nominal case, local A*)

NF3. Ability to operate in a non-empty² environment

​

^[1] Modular: Able to operate with either only local A* (nominal case) or predictive avoidance subsystem

^[2] Non-empty: The environment for validation is not only free space, but contains static walls/corridors/objects so that it resembles an actual industrial environment more closely

Real-life System Status

Real-life System:

Hardware (Physical Robot)

12

LIPO Battery

RPLidar A2

Point Grey

Camera

Power Distribution PCB

NVIDIA Jetson Xavier AGX

OpenCR

Motors + Wheels

Cooling Fan

Wi-Fi Adapter

Camera Lens

April Tag

LIPO Battery

RPLidar A2

Point Grey

Camera

Power Distribution PCB

NVIDIA Jetson Xavier AGX

OpenCR

Motors + Wheels

Cooling Fan

Wi-Fi Adapter

Camera Lens

April Tag

TurtleBot Burger

Components:

• Computing ✓
• Mobility ✓
• Power Distribution ✓
• Sensing (LiDAR + Camera) ✓

​

Progress since CDR:

• PCB Cooling Fan Added
• Camera + Lens Upgraded (FLIR Firefly S)
• April Tag Added
• New Cosmetic Feature

​

Physical Robot:

Current Status

Faced:

• PCB gets hot
• Cooling fan mounted on top with 3D-printed parts

​

• Difficulty with 3D printing quality
• Saved by new machines in B506

​

Remaining:

• Constant Wi-Fi disconnection
• New Wi-Fi router just arrived

​

• Battery connection frying/instability
• New connector WIP, will continue monitoring

​

• Camera overheating after heavy use
• Still investigating

​

Physical Robot:

Challenges

Real-life System:

Ground Truth Subsystem

16

Ground Truth Camera

Objects

World April Tags

Studio Lights

Ground Truth Camera

Objects

World April Tags

Studio Lights

Components:

• Ground Truth Camera & Calibration ✓

​

• April Tags on Objects ✓

​

• April Tags Frame Transforms (Everything in Ground Frame) ✓

​

• Automatic Validation of Perception Performance (Precision, Recall, Localization Accuracy) X

Ground Truth Subsystem

Current Status

18

Ground Truth Subsystem

Analysis & Testing

19

(1.0031, 1.0006)

(1.6064, 1.6063)

(2.0138, 0.3952)

(-0.0260, 1.0009)

Best case error < 1 cm (center of the camera)

Worst case error < 3 cm (edge of the frame)

(0.0000, 1.0000)

(1.0000, 1.0000)

(1.6000, 1.6000)

(2.0000, 0.4000)

Faced:

• April Tag detection unreliable
• Increase lighting & April tag sizes

​

Remaining:

• 3D April tag localization accuracy (depth) issues
• Calibrate the world April tags offline
• Ignore depth dimension, project objects onto ground

​

​

Ground Truth Subsystem

Challenges

Real-life System:

Software (Perception)

21

Sensor

Fusion

LiDAR Detection

Camera Detection

22

Sensor

Fusion

LiDAR Detection

Camera Detection

Components:

• Camera Intrinsics/Extrinsics Calibrations ✓

​

• YOLO ✓
• Deployed to Jetson at 10 FPS
• More data needed for forklifts

​

• Sensor Fusion X
• LiDAR Extrinsics Calibration WIP ✓
• Forklift Centerpoint Estimation X

Perception:

Current Status

23

Components:

• Camera Intrinsics/Extrinsics Calibrations ✓

​

• YOLO ✓
• Deployed to Jetson at 10 FPS
• More data needed for forklifts

​

• Sensor Fusion X
• LiDAR Extrinsics Calibration WIP ✓
• Forklift Centerpoint Estimation X

Perception:

Current Status

24

Intrinsics Calibration

Extrinsics Calibration

Perception:

Analysis & Testing - YOLO

Common forklift detection failure cases:

• Head-on

​

​

​

​

​

• Overlap w/ other forklift

​

​

​

​

​

​

• Against cluttered backgrounds (chairs, legs)

Perception:

Analysis & Testing - LiDAR detection

26

Faced:

• YOLOv5 not written in Darknet, cannot integrate with ROS directly with existing packages
• Deploy YOLOv5 with TensorRT

​

• Cannot detect police woman pedestrian (black pants)
• Wrapped legs in Velcro

​

Remaining:

• Forklift recall not satisfying requirements
• More data being gathered for forklifts in lab

​

• YOLO detection susceptible to background changes (human legs, chairs)
• Fence being added to testing environment with printed factory backgrounds

​

Perception:

Challenges

Virtual System Status

Overall System Depiction - Virtual

Simulation Environment

Obstacles

Robot + Avoidance, Local A*

Obstacle Filtering

Trajectory Dictionary

Minor Changes

Major Updates

Subsystem:

Simulation Environment

Full-Scale Factory Environment (136 m x 64 m)

​

• All models and factory environment: Done
• Add pedestrians simulation: Done
• Expand factory environment: Done

Forklift task area

​

​

​

​

​

​

​

​

​

​

​

​

​

Pedestrian task area

​

​

​

​

​

​

​

​

​

​

​

​

​

Expanded area

Robot

Unloading Zone

​

Long Recordings

Subsystem:

Trajectory Dictionary

Processed Small Segments

• Main Implementation: Done
• Add pedestrian dictionary: Done
• More data for forklifts and pedestrians trajectories: Possible To-do

Subsystem:

Predictive Avoidance

• Main Implementation: Done
• Avoiding Forklifts: Done
• Avoiding Pedestrians: Done, Tuning in Progress

Subsystem:

Local A*

• Main Implementation: Done, Needs Tuning + Bug Fixes
• Interface with Software Stack: Refactoring in Progress

Subsystem:

Obstacle Filtering

(Not to scale. Configuration WIP)

• Simulated Cameras: Done
• Object Occlusions: Done
• Simulated Noise: Done, Tuning TODO
• Position, Heading, Detection %

Forward

Virtual Robot System - Obstacle Filtering:

Testing

(Not to scale. Configuration WIP)

35

Forward

Virtual Robot System:

Challenges

Faced:

• Technical Debt
• Lots of disorganized functions and hard-coding
• Refactoring with new classes and packages

​

Remaining:

• Pedestrian simulation package quirks
• No velocity information, no collision
• Might need manual velocity calculation & collision checks

​

• Collision behavior in long-running simulation
• Current plan: reset robot near collision + time penalty

​

36

Project Management

Schedule

Budget

39

• Total Budget: $5,000
• Spent 66%
• Big ticket items:
• NVidia Jetson
• Turtlebot
• RPLidar
• ANSI standard

Milestones

40

Fall Validation - Virtual Robot System

41

1. In live simulation, have a single robot complete a small set of scenarios.
1. Short scenario where robot has to dodge an obstacle(s)
2. Obstacles are forklifts and pedestrians which follow predetermined paths
3. Robot uses predictive avoidance
2. Repeat 1. with the robot using local A* instead of predictive avoidance

Scenario: Robot has one waypoint which is close by. One obstacle with a nearby trajectory

Fall Validation - Virtual Robot System

42

Long Simulation: Robot continuously cycles through many waypoints. Multiple objects continuously cycle through trajectories

• Show a previously captured video of predictive avoidance running for a long period of time at ~20x speed
• Show analysis of predictive avoidance vs local A* for long running simulations.
• Success Criteria: predictive avoidance is atleast 5% more productive than local A*

Fall Validation - Real Life System

Process

​

43

• Place robot and obstacles interesting bed
• Detect obstacles with perception system on turtlebot and ground truth system
• Move robot and obstacles around
• Compare perception results with ground truth results

Fall Validation - Real Life System

Success Criteria

​

44

1. Forklifts within 2 m and pedestrians within 1 m will be detected
2. Detection distance error should be less than 0.1 m
3. YOLO classifies obstacles with precision of 70%
4. YOLO classifies obstacles with recall of 80%
5. Output detection and localization results within 100ms per frame

Risk Management

45

Risk Management

46

Risk Management

47

Risk Management

48

Questions

https://cmu.zoom.us/j/98918577222?pwd=aWU2YTI1dGtRdzBrU3o3QnpSZ2tndz09