Traffic Danger Recognition�with Surveillance Cameras �Without Training Data

Lijun Yu

lijun@cmu.edu

11/13/2019

Content

Background

• Traffic danger

• Surveillance camera

• No training data

Goal

• Crash detection & proactive safety check
• Send first aid ASAP and potentially save lives

​

​

• Predictive danger recognition
• Other traffic analysis
• Pedestrians, bicycles, vehicles
• Provide insights about high-risk areas

Architecture

• Detect and track vehicles

​

​

​

​

​

• Collision detection based on �kinematics of rigid body
• No need for training data of crashes

• Estimate and predict location and speed
• Detect collision beforehand

• Get 3D bounding boxes
• Project vehicles from image to ground

Object Detection and Tracking

• Object detection
• Output: Vehicle type, score, �bounding box, mask
• Model: Mask-RCNN trained on COCO
• Object types: car, bus, truck
• Object filters
• Road area and vehicle size
• Complete visibility
• Tracking
• Output: Vehicle ID
• Algorithm: Deep-SORT

​

Detection result

Filter result

Camera Calibration

• Camera model*
• Vanishing points: directions of X Y Z on image
• Calibrated at installation
• Manual labeling on image
• 2 groups of parallel lines
• Intersection of each group: vanishing points
• Automatically on video
• Track the traffic flow for a while
• Vehicle moving direction as VP1
• Vehicle contours for VP2

*Sochor, Jakub, et al. "Comprehensive data set for automatic single camera visual speed measurement." IEEE Transactions on Intelligent Transportation Systems 20.5 (2018): 1633-1643.

3D Projection

• Contour from object mask
• Tangent lines from vanishing points

​

• Derive intersections and lines

• Handles different viewing angles

Trajectory Prediction

• Project bottoms of 3D bounding boxes to road plane
• Center location
• Speed (smoothed by moving average)
• Scale factor: to real world values of distance
• Assumptions
• Vehicle shapes do not change
• Vehicle location follows a normal distribution
• Linear prediction (currently):
• fixed acceleration and location deviation
• Calculate speed and coordinates according to kinematics

​

Danger Recognition

• Distance between vehicles
• Predicted location of a vehicle
• Calculated from shape and distribution of center
• Danger heatmap
• Aggregate predicted locations of all vehicles
• Probability of coexistence in the same location

Experiments

BrnoCompSpeed Dataset

*https://medusa.fit.vutbr.cz/traffic/research-topics/traffic-camera-calibration/brnocompspeed/

Evaluations on BrnoCompSpeed

• Detection & tracking recall: 94.0%
• Match with Lidar ground truth at temporal IoU > 0.5
• Distance error in projection
• Compare with distance ground truth in 2 directions
• 1.80% along the road, 2.06% vertical to the road
• Speed estimation error: 2.77km/h, 3.68%
• Compare with average speed from Lidar, ~75km/h
• Prediction error at 0.12s
• Location: 0.24m
• Speed: 2.53km/h

Proactive�Safety�Check�Demo

CADP Dataset

• A Novel Dataset for CCTV Traffic �Camera based Accident Analysis
• Collected by our group
• Accident video clips collected from Youtube
• Surveillance cameras / dash cameras / phone cameras
• Various resolution and quality
• Total length: 5.2 hours
• Average length of video: 366 frames
• Spatial-temporal annotation of accidents
• No calibration metadata, �not long enough for automatic calibration

Demo at https://www.youtube.com/watch?v=t7lxi7xa53U

MEVA Dataset

Vehicle-person�Interaction�Demo

Spacing between vehicles �and pedestrians

​

False alarm of crash �at interactions

Summary

• Reliable detection and tracking of vehicles
• Precise measuring of location, distance, �and speed
• Robust on arbitrary calibrated surveillance�camera

​

• Crash detection without training data
• Proactive safety checks on normal traffic �flow

​

Real World Demands 🡪 Extensions

Activity Detection

Object Detection �& Tracking

3D Projection

Trajectory Prediction

Danger Recognition

Activity Proposal

Detect activities based on kinematics

Upgraded

Upgraded 3D Projection

• Accurate calibration at installation
• Dynamic coordinate system (Not limited to straight road)
• Object moving direction estimated by 2D trajectory
• Reliable ground trajectory�of every vehicle and person

Activity Proposal

• Trigger-driven proposal
• Output: Vehicle activity
• Currently supported event types: �left/right/U-turn, start/stop
• Observed state parameters
• Ground speed
• 2D speed from Kalman filter
• 3D projection
• Polar coordinates: value + angle
• Slope of speed value/angle (acceleration)
• Local linear regression
• Trigger and border of activities

​

Start of a

turning activity

End of a

stop activity

Realtime Pipeline

• Stream Execution
• Frame-level parallelism
• Out-of-order detection
• Bottleneck: Mask-RCNN detector
• Reorder results from multiple detectors
• Throughput: 18fps @ 1920 * 1080
• With 4 2080Ti GPUs
• Usually works at 15fps
• Latency: 0.2s

​

Loader

Reorder buffer

Tracker

3D Projector

Activity�Proposer

1 frame

Activities

1 video

Buffered frames

Performance

Probability of missing detection at 15% of false alarm

Probability of missing detection at 15% of time-based false alarm

Demo at https://drive.google.com/file/d/18kmbYtb6R871NgZ8Ih8vqtRFxv-kVRWh/view?usp=sharing

Automatic Calibration & Trajectory Prediction

Object Detection �& Tracking

Camera Calibration �& 3D Projection

Trajectory Prediction

Danger Recognition

One-shot Automatic Calibration

Learning based prediction

Predict horizon line and vanishing point�on single image, �trained with synthetic data

Arbitrary, moving camera

Thank you!