Autoware Mini Lecture 8:�Validation

Tambet Matiisen

October 29th, 2024

Validation methods

Bag scenario

OpenSCENARIO

CARLA route scenario

01 Bag scenario

02 OpenSCENARIO

03 CARLA route scenario

Bag scenario�a.k.a. replay log testing

01

Bag scenario

$ rosbag info ~/autoware_mini_ws/src/autoware_mini/data/bag_scenarios/tartu_demo/raekoda.bag�…�topics: /detection/detected_objects 300 msgs : autoware_msgs/DetectedObjectArray � /detection/traffic_light_status 154 msgs : autoware_msgs/TrafficLightResultArray � /initialpose 1 msg : geometry_msgs/PoseWithCovarianceStamped� /initialvelocity 1 msg : geometry_msgs/TwistStamped � /move_base_simple/goal 1 msg : geometry_msgs/PoseStamped � /tf 1524 msgs : tf2_msgs/TFMessage

Launching

roslaunch autoware_mini start_sim.launch scenario_name:=raekoda

roslaunch autoware_mini start_sim.launch scenario_name:=kaubamaja

roslaunch autoware_mini start_sim.launch scenario_name:=bot_garden

roslaunch autoware_mini start_sim.launch scenario_name:=bot_garden2

roslaunch autoware_mini start_sim.launch scenario_name:=statoil

​

Closed-loop

Green bicycle -

simulated car

Lexus model -

real car

Pros and cons

Pros:

• Simple and light-weight
• Relatively easy to create
• Real-world situations

Cons:

• Gets stuck if does not closely match the original car behavior
• Cannot easily change or modify the simulation
• How to measure success automatically?

OpenSCENARIO�a.k.a. hand-crafted scenarios

02

OpenSCENARIO

• ASAM standard
• V1 based on XML
• V2 with Python-like syntax

OpenSCENARIO tools

• MATLAB RoadRunner
• web.auto scenario editor
• Truevision.ai designer
• QGIS OpenSCENARIO Generation Plugin
• Blender Driving Scenario Creator add-on

Source: https://se.mathworks.com/products/roadrunner-scenario.html

Examples

In one terminal:�~/CARLA_0.9.13/CarlaUE4.sh -RenderOffScreen -prefernvidia

In another terminal:�roslaunch autoware_mini start_carla.launch map_name:=Town01 use_scenario_runner:=true

1. Under Simulation enable Carla image view.
2. Choose 3EmergencyBreak from Scenario Execution list and click Execute.
3. Click on 2D Nav Goal and put destination somewhere ahead.
4. Try also scenarios 4ObstacleTurn and 7RunningRedLight.
5. Take a look at ~/autoware_mini_ws/src/autoware_mini/data/scenarios/Town01/3EmergencyBreak.xosc

Pros and cons

Pros:

• Expressive language
• Can model rare/dangerous scenarios
• Many ways to check the success condition

Cons:

• Complicated format
• No good GUI tools
• Need to set goal manually

CARLA route scenario

03

CARLA route scenario

Consists of two components:

1. Route descriptions ~/autoware_mini_ws/src/autoware_mini/data/routes/routes_devtest.xml
2. Scenarios applied at various points of the route�~/autoware_mini_ws/src/autoware_mini/data/routes/all_towns_traffic_scenarios_public.json

Fixes the problem of manually setting the goal.

Does not use OpenSCENARIO files! Instead uses built-in Python scenarios!

CARLA built-in scenarios�selected from NHTSA pre-crash typology

Source: https://leaderboard.carla.org/scenarios/

Control loss� 01 - Control loss without previous action.

Traffic negotiation� 02 - Unprotected left turn at intersection with oncoming traffic.� 03 - Right turn at an intersection with crossing traffic.� 04 - Crossing negotiation at an unsignalized intersection.� 05 - Crossing traffic running a red light at an intersection.� 06 - Crossing with oncoming bicycles.

Highway� 07 - Highway merge from on-ramp.� 08 - Highway cut-in from on-ramp.� 09 - Static cut-in.� 10 - Highway exit.� 11 - Yield to emergency vehicle.

Obstacle avoidance� 12 - Obstacle in lane.� 12a - Obstacle in lane.� 13 - Door obstacle.� 14 - Slow moving hazard at lane edge.� 14a - Slow moving hazard at lane edge.� 15 - Vehicle invading lane on bend.

Braking and lane changing� 16 - Longitudinal control after leading vehicle’s brake.� 17 - Obstacle avoidance without prior action.� 18 - Pedestrian emerging from behind parked vehicle.� 19 - Obstacle avoidance with prior action - pedestrian or bicycle.� 19a - Obstacle avoidance with prior action - vehicle.� 20 - Parking Cut-in

Parking� 21 - Parking Exit.

CARLA built-in scenarios

CARLA built-in scenarios

Running

In one terminal:�~/CARLA_0.9.13/CarlaUE4.sh -RenderOffScreen -prefernvidia

In another terminal:�roslaunch autoware_mini start_carla.launch map_name:=Town01 use_scenario_runner:=true route_id:=1

• Under Simulation enable Carla image view.
• Enjoy the ride.

CARLA route scenario report

╒════════════════════════╤═════════════════════════════════════╤═════════╤══════════════╤════════════════╕�│ Actor │ Criterion │ Result │ Actual Value │ Expected Value │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ RouteCompletionTest (Req.) │ SUCCESS │ 100 │ 100 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ CollisionTest (Req.) │ FAILURE │ 4 │ 0 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ InRouteTest (Req.) │ SUCCESS │ 0 │ 0 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ OutsideRouteLanesTest (Req.) │ SUCCESS │ 0 │ 0 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ RunningRedLightTest (Req.) │ FAILURE │ 1 │ 0 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ RunningStopTest (Req.) │ SUCCESS │ 0 │ 0 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ lexus.utlexus (id=370) │ ActorSpeedAboveThresholdTest (Req.) │ SUCCESS │ 0 │ 0 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ │ Timeout (Req.) │ SUCCESS │ 255.95 │ 304 │�├────────────────────────┼─────────────────────────────────────┼─────────┼──────────────┼────────────────┤�│ │ GLOBAL RESULT │ FAILURE │ │ │�╘════════════════════════╧═════════════════════════════════════╧═════════╧══════════════╧════════════════╛

Pros and cons

Pros:

• Easy to run standard scenarios
• Can be automated as nightly test
• No need to manually set goal

Cons:

• Only limited number of pre-defined scenarios
• New scenarios need to be coded in Python
• Running full 3D simulation is actually expensive

Autoware Mini: CARLA ground truth

~/CARLA_0.9.13/CarlaUE4.sh -RenderOffScreen -prefernvidia

roslaunch autoware_mini start_carla.launch

1. Under Simulation enable Carla image view.
2. Wait till other vehicles are created and one passes you
3. Give the vehicle a goal with 2D Nav Goal, for example behind its back

Autoware Mini: CARLA lidar sensor

roslaunch autoware_mini start_carla.launch detector:=lidar_cluster

• Under Sensing enable Points raw center to see the raw pointcloud
• Give the vehicle a goal behind its back with 2D Nav Goal
• Try switching on Points ground, Points no ground, Points clustered
• Why the stability of closest object distance and closest object speed is worse than with ground truth?

Autoware Mini: CARLA camera traffic light detection

~/CARLA_0.9.13/CarlaUE4.sh -RenderOffScreen -prefernvidia -quality-level=Low

roslaunch autoware_mini start_carla.launch tfl_detector:=camera

• Under Simulation enable Carla image view.
• Under Detection enable Left ROI image and Right ROI image
• Give the vehicle a goal behind its back with 2D Nav Goal
• Notice the simulation being much slower.

Thank you!

Tambet Matiisen

UT Autonomous Driving Lab technology lead

tambet.matiisen@ut.ee

�Autonomous Driving Lab�University of Tartu Institute of Computer Science

Narva mnt 18, 51009 Tartu �

adl.cs.ut.ee