1 of 63

Critical Design Review

Team S.T.A.R.S

2 of 63

Project Description

Simulating realistic traffic behavior and interactions by learning from the real world.

3 of 63

Use Case

For developers of autonomous vehicle software like Mr. Stark.
Mr. Stark wants to test higher level algorithms but doesn’t want to go through the hassles of real world testing.
He decides to use simulators but existing simulator behavior models are not very realistic.

4 of 63

Use Case

Mr. Stark then sees the S.T.A.R.S product and decides to add the S.T.A.R.S plugin to his simulator system.
He loads his Self Driving software stack into the simulator and starts running a few scenarios.
Like in real-world, these agents do break rules occasionally.

5 of 63

Use Case

Also Mr. Stark can tweak certain parameters to ensure his software handles edge cases as well.
To create some specific type of scenarios, he retrains the S.T.A.R.S behavior model using some previously collected data.

6 of 63

Use Case

With the help of the S.T.A.R.S system, Mr. Stark can update his software quickly, and gain the benefit of iterative development and “disengagement-free” testing

7 of 63

Mandatory Performance Requirements

System Requirements

7

Capture video stream at a speed of at least 30 FPS
Capture 100% view of intersection
Have image preprocessing detection recall of more than 75%
Have image preprocessing detection precision of more than 75%
Have MOTA and MOTP values on our tracking pipeline to be at least 40%
Have a simulation speed of minimum 10 FPS supporting at least 3 simultaneous agents
Have x% MSE between predicted trajectory from the behavioral model and ground truth trajectory in a given scenario

The system will ...

8 of 63

Mandatory Non-functional requirements

System Requirements

8

Have a modular design for components
Have a compact and portable sensor suite
Have well documented code and APIs
Allow for tuning aggression parameters of models

The system shall ...

9 of 63

Desirable Performance Requirements

System Requirements

9

Have detection recall of 90% or more
Have detection precision of 90% or more
Have a simulation speed of 30 FPS or more
Support 5 or more simultaneous actors with minimum 15 fps

The system will ...

10 of 63

11 of 63

12 of 63

13 of 63

14 of 63

15 of 63

16 of 63

17 of 63

18 of 63

19 of 63

20 of 63

System status

21 of 63

Subsystems

Data Capture

Data Processing

Behavior Modelling

Simulation

Runner

22 of 63

Data Capture

23 of 63

Targeted Requirements

Capture video stream at a speed of at least 30 FPS
Capture 100% view of intersection

24 of 63

Subsystem Description - Overview

Traffic Camera

Open Datasets

Steering Wheel System

25 of 63

Locations where we have correspondences

26 of 63

Data Capture - Videos from Traffic Camera

Automated capture of video data from a virtual camera mounted at an intersection in Carla.

Mount a camera at a given traffic light,
Variable parameters - duration, framerate, weather and agent locations
Automated capture process for images, ground truth and video

Modeling; analysis; testing

Experimented with different locations
Different formats(image by image vs continuous video storage)

Strong/weak points

Traffic camera is currently in the Carla environment
In the real world, density and weather are not controllable, hence might have to account for delays

27 of 63

28 of 63

Data Capture - Steering Wheel System

Simulator data - accurate information available
Steering Wheel System - allows ‘realism’

Convert simulator data to requisite format
Ego vehicle is manually driven
Automated data collection and storage

Careful calibration of steering wheel required for ‘realism’

29 of 63

Data Capture - NuScenes

Large, pre-annotated dataset available
Real world data
High accuracy - multiple sensors

Convert data to requisite format
Get traffic light states using front cam image

Accurate traffic light states - difficult

Data extraction from NuScenes dataset

30 of 63

Data Processing

31 of 63

Targeted Requirements

M.P. 3 Have image preprocessing detection recall of more than 75%

M.P. 4 Have image preprocessing detection precision of more than 75%

M.P. 5 Have MOTA and MOTP values on our tracking pipeline to be at least 40%

D.P.1 Have detection recall of 90% or more

D.P.2 Have detection precision of 90% or more

32 of 63

Subsystem Depiction/Description

Detection

Extracted training data from Carla to train an object detection model for vehicles

detectron2 based on FasterRCNN model for vehicles
~2600 training images and ~670 test images

Trajectory Processing

Detection output

Detection out

Used SORT for preliminary tracking of the bounding boxes in camera view.
Converting detections from the videos to a birds-eye-view perspective
Tracking the trajectories of each agent in bird’s eye view

Detection output

Bird’s eye view tracking

Tracking by SORT

Homography transform

33 of 63

Data Processing - Image Based Processing

Modeling; analysis; testing

Training strategy for attaining metrics

Methods

increasing training data, training for more iterations, prototyping with different model architectures

PR curve for picking confidence scores for our applications
Multiple Object Tracking metrics - MOTA and MOTP .

Strong/weak points

Works great for current dataset and videos.
Usage in real world will need one more round of training and data sampling.
Besides few configuration parameters, the data processing pipeline for processing remains the same
To compute homography, the pipeline requires an image of the intersection in the bird’s eye view beforehand alongside manually hand picked correspondences.

34 of 63

Data Processing - Detection

ConfThreshold @0.5IoU	Precision	Recall
10.01%	72.26%	95.80%
20.00%	81.71%	95.24%
30.04%	86.38%	94.55%
40.06%	89.92%	94.06%
50.13%	92.74%	93.34%
60.02%	94.38%	92.65%
70.00%	95.98%	91.66%
80.08%	97.54%	90.32%
90.06%	98.70%	88.18%
95.05%	99.25%	85.50%
98.01%	99.59%	81.21%
99.00%	99.88%	76.09%
99.90%	100.00%	37.74%

35 of 63

Data Processing - Tracking and Trajectory Processing

Right: For generating the final trajectories, the bottom center is transformed to bird’s eye view and is tracked in pixel space.

Left: Output from the detector and preliminary tracked results using Simple and Online Realtime Tracker (SORT).

Traffic Light ID	MOTA	MOTP	Traffic Density
54	75.59	73.32	Low
55	86.43	54.11	Low
61	98.17	80.38	Low
62	98.44	62.41	Low
65	72.79	60.26	Low
65	52.7	51.08	Moderate
62	89.02	61.19	Moderate
61	94.38	76.05	Moderate
55	78.85	76.05	Moderate
54	60.94	72.59	Moderate
65	65.78	61.66	Dense
55	91.19	80.69	Dense
Average	80.36	67.48

36 of 63

Behavior Modelling

37 of 63

Behavior Modelling

Targeted Requirement - M.P. 7 Have x% MSE between predicted trajectory from the behavioral model and ground truth trajectory in a given scenario.
Implemented Learning-by-Cheating (LBC) based baseline behavioral model which takes input as world state and computes control inputs for each agent.

38 of 63

Behavior Modelling

Current Status

Incorporated baseline behavior model capable of demonstrating general traffic behavior.(Conditional Imitation Learning based)

Responding to traffic lights
Driving within lane boundary
Following lead vehicle
Collision Avoidance

39 of 63

Behavior Modelling

Future Work

Fine-tune CIL behavioral model with real data.
Adding probabilistic rule-based model to accommodate realizing of certain scenarios more often.
Using Behavior Trees to control when to use input from CIL based model and when to rely on rule based model.

40 of 63

Simulation

41 of 63

Simulation Sub-system

The Simulation Subsystem takes care of simulating the realistic environment given model from the Modelling subsystem.

Its major function are running a specific scenario and supporting multiple agents simultaneously.

We are currently using CARLA software as the simulation platform.

42 of 63

Simulation Sub-system

Targeted Requirements:

M.P. 6 Have a simulation speed of minimum 10 FPS supporting at least 3 simultaneous agents.

Current Status:

Baseline behavioural model working with 3 agents within range of 9-11 server FPS.
Support to run specific scenarios using baseline behavioral model

Future work:

To optimize the performance of simulation with S.T.A.R.S. behaviour model,

Client side development by introducing a simulator bridge
Server side development by adding code for running the model in the simulator itself.

43 of 63

Demo Video for Simulation Sub-system

44 of 63

45 of 63

Modeling, analysis and testing

45

Number of Vehicles	FPS
0	27.28
1	16.66
2	11.32
3	10.61
4	8.43
5	6.38
6	5.66
7	4.92
8	4.36
9	3.82
10	3.64

46 of 63

Simulation Sub-system

Strong/Weak Points:

Current system performs as per the requirements. However, it is at the border of the fps values required. This provides almost no room to accomodate any new development to improve the model.
On the bright side, the analysis made on the codebase has provided some insights regarding how this can further be improved. As we have already developed a framework which is working, we can confidently spend time improving the performance (fps).
To support different CARLA maps retraining is required.

47 of 63

Project Management

48 of 63

Work Breakdown Structure

49 of 63

Schedule Status

Spring Milestones Achieved:

Documents for software best practices ready (Feb 19)
Image based processing complete. (Feb 19)
Finalized baseline learning based algorithm for Behavior Modelling. (Mar 04)
Development for baseline Behavior Modelling complete. (Mar 04)
Trajectory based processing complete. (Mar 25)
Design for traffic simulation sub-system. (Mar 25)
Visualization tool for Image and Trajectory Processing. (Apr 08)
Sub-system level integration for Image and Trajectory Processing Ready. (Apr 08)
Image and Trajectory Processing Complete with evaluation metrics. (Apr 20)
Development for Traffic Simulation sub-system complete. (Apr 20)
System Integration Complete. (Apr 20)

50 of 63

Schedule Status

Fall Milestones:

Modelling

Research and prototype different imitation learning based models.
Collect data using Steering wheel system.
Collect data using sensor suite in real-world.
Train behavioural model using own data sources.
Design and develop probabilistic rule-based model.

Simulation

Optimize Simulator code to accommodate higher FPS.

System level integration.
CI Pipeline (Jenkins) and Docker Setup ready. (Moved from Spring)

We are currently on track with respect to the schedule.

51 of 63

Test Plan

Deadline	Test Plan/Deliverables
PR 7: Early September	Data collection using Steering wheel system complete.
PR 8: Mid-September	Modelling subsystem modifications/implementation and testing Research and prototyping different imitation learning based models complete.
PR 9: Early October	Data collection using sensor suite in real-world complete. Train behavioural model using own data sources.
PR 10: Mid-October	Simulation subsystem implementation and testing done.
PR 11: Mid-November	CI Pipeline (Jenkins) and Docker Setup ready. System level integration done.
PR 12: Late November	End-to-end integrated test complete.

Modelling

Simulation

Integration

52 of 63

Fall Validation Demonstrations

Location: NSH B512

Equipment: Desktop system running Carla and the S.T.A.R.S. pipeline, Steering wheel system such as Logitech G920, Data Capture Unit (DCU) designed and fabricated by S.T.A.R.S.

53 of 63

Fall Validation Demonstration 1

53

Experiment Procedure	Requirement Satisfied	Validation Criteria
Objective: To demonstrate various data capturing methods and pre-processing pipeline Procedure Overview Process pre-recorded real world videos. Capture data from steering wheel system.	MP 1	Data Capture Unit successfully mounted on the frame, collects data ≥ 30 frames per second
	MP 2	Data Capture Unit enables 100% view of intersection.
	MP 3	The system detects 75% of actors (cars and pedestrians) seen in input video.
	MP 4	Have image preprocessing detection precision of more than 75%.
	MP 5	Have MOTA and MOTP values on our tracking pipeline to be at least 40%.

54 of 63

Fall Validation Demonstration 2

54

Experiment Procedure	Requirement Satisfied	Validation Criteria
Objective: To demonstrate the final traffic behavior model developed by S.T.A.R.S. Procedure Overview Replicate the scenario captured from Demo1 inside Simulator. Run 3-5 agents simultaneously.	MP 6	The simulation speed in CARLA ≥ 10 frames per second supporting ≥ 3 actors simultaneously.
	MP 7	Will have x% Mean square error between the predicted trajectory from the behavioral model and ground truth trajectory in a given scenario.

55 of 63

Fall Validation Demonstration 3

55

Experiment Procedure	Requirement Satisfied	Validation Criteria
Objective: To demonstrate that the system can be tuned for aggression. Procedure Overview Given a S.T.A.R.S. system, various parameters will be changed as part of aggression tuning. Many of these models will be started with such configuration inside Simulator.	M.N.4	Allow for tuning aggression parameters of models.

56 of 63

Budget Status

System	Cost	Units	Total
Purchased
Computer	$2,000	1	$2,000
Computer Peripherals(Monitor, IO, Cabling)	$180	1	$180
Camera(GoPro)	$400	1	$400
			$2,580
Plan to Purchase
Camera(GoPro)	$400	2	$800
Steering Wheel Controllers	$250	2	$500
Mounting	$300	1	$300
Mechanical Enclosure	$200	1	$200
Connectors/Interfacing	$50	1	$50
External Storage	$120	1	$120
Cloud APIs for Images(Annotation)	$250	1	$250
			$2,220
		Total	$4,800

57 of 63

Risk Management

57

L* = Likelihood C* = Consequence T = Technical M = Management S = Scheduling C = Cost

Likelihood	5		5
	4		3,9		1
	3		7,8	2,4
	2				6,10
	1
TeamH		1	2	3	4	5
TeamH		Consequence

58 of 63

Risk Analysis

58

S. No.	Description	Requirement	Type	L*	C*	Risk reduction plan
1	Slow model execution during simulation	M.P.6	T	2->4	4	Software design: Decouple the model execution and simulation using a bridge-like interface
2	Performance of learning�model is not adequate	M.P.7	T	3	4	Review multiple algorithms during literature survey and prototype stage Fall back on rule based probabilistic approach
3	Unable to capture data due to COVID-19 Pandemic	All	S, M	4	3->2	Fall back on open data sets
4	Inability to formulate�the modelling problem	M.P.7	T	3	2	Talk to professors and research scholars at CMU working on behavioral modelling of agents

L* = Likelihood C* = Consequence T = Technical M = Management S = Scheduling C = Cost

59 of 63

Risk Analysis

59

S. No.	Description	Requirement	Type	L*	C*	Risk reduction plan
5	Missing project milestones	All	S, M	3->2	5	Daily standup for the team Allocate buffer time to each task
6	Lack of large�annotated datasets	M.P.7	T, R	2	4	Create synthetic data captured from a simulator environment
7	Stakeholder disengagement	All	M	3	2	Define requirements clearly Regular sync-up meetings to ensure we are on track with the requirements
8	Hardware component failure	M.N.2	T, C, S	3	2	Spare hardware for critical components Add buffer time to handle these failures
9	Unable to procure�video capture license	M.P.1, M.P.2	M, S	4	2	Fall back on open data sources or�synthetic datasets
10	Inadvertent failure during demos	None	S, M	2	4	Following the integration & testing schedule Version control and unit testing

L* = Likelihood C* = Consequence T = Technical M = Management S = Scheduling C = Cost

60 of 63

Risk Analysis

60

S. No.	Description	Requirement	Type	L*	C*	Risk reduction plan
11	Detection and tracking�algorithms don’t work in different weather conditions	M.P.3, M.P.4, M.P.5	T, S	4->3	2->1	Finetune detection/ tracking models with datasets that include weather variation Scheduling most of final data capture for spring/summer
12	Lead time of hardware�causes delays in�development/impacts�the schedule	M.P.3, M.P.4, M.P.5, M.P.6, M.P.7	S	3->1	3->1	Research readily available components within budget, anticipate requirements and place orders early

L* = Likelihood C* = Consequence T = Technical M = Management S = Scheduling C = Cost

Eliminated/Scoped out

61 of 63

Lessons Learned - Spring 2020

Risk Management is critical! - Including Risk management in the project plan itself was helpful
Lean development/Agile - MVP first, then figure out integrations and then continue to iterate
Focus on seeking and adopting best practices for the code you’re writing. Software Design and Engineering principles
Collaborating on big tasks helps - Having atleast 2 people aware of each major subsystem is better
Divide and conquer admin tasks

Assigning owners to each major course deliverable(PRs, Group tasks) helped people focus on their work

In terms of lessons learned,

we realised risk management is critical. And ensuring to review the risks and make necessary changes to priorities is helped us.
Lean development, is probably what many of us are doing as well.

- for each subsystem -> build MVP first then integrate subsystems and then iterate
has helped us move faster

Adopting the best practices of SWEngg and design have helped as well
Thiss helped collaboration on big tasks, which as a result, helps us have a good variance of ideas for that task

Risk Management is critical! - Including Risk management in the project plan itself was helpful
Always be on the outlook - papers published in Dec 2019 were extremely relevant to our project
Be bold - Talk to people! Ask!

Spoke to the creator of CARLA early on and got extremely valuable insights into the project
We got a Workstation at a great deal that helped speed up our development process, part academic discount, part sponsorship

Lean development - For each subsystem, focus on MVP first, figure out integrations and then continue to iterate while maintaining the interfaces.
Focus on seeking and adopting best practices for the languages you’re using. Also look into design of software systems.
Divide and conquer admin tasks

Assigning owners to each major course deliverable(PRs, Group tasks) helped people focus on their work

Collaborating on big tasks helps
Having atleast 2 people aware of each major subsystem is better

Risk Management is critical! - Including Risk management in the project plan itself was helpful
Always be on the outlook for relevant work - papers and libraries published in Dec 2019 became extremely crucial to our project
Be bold - Talk to people! Ask!

Spoke to the creator of CARLA early on and got valuable insights into the project
We got a Workstation at a great deal that helped speed up our development process, part academic discount, part sponsorship

Lean development - MVP first, then figure out integrations and then continue to iterate
Focus on seeking and adopting best practices for the code you’re writing. Software Design and Engineering principles
Divide and conquer admin tasks

Assigning owners to each major course deliverable(PRs, Group tasks) helped people focus on their work

Collaborating on big tasks helps
Having atleast 2 people aware of each major subsystem is better

62 of 63

Key Activities - Fall 2020

Double down on behavior modelling

Training and tweaking conditional imitation learning models
Adding behavior trees for finer control over behavior

Capturing the behaviors

Steering Wheel data capture
More Open Datasets

Simulator Runner improvements

Simulator Bridge for Clients

Incremental updates to Data Processing and Simulation

Improving Birds Eye View tracking model
Incorporating Modelling related requirements into data processing system

End to end integration

63 of 63

Questions?