1

Àlex Vicente Sola

Àlex Palomo Domínguez

Joan Sintes Marcos

Roger Marí Molas

M6 - VIDEO ANALYSIS 2018

​

VIDEO SURVEILLANCE FOR ROAD TRAFFIC MONITORING

TEAM 7

2

Pipeline / Objective

Video

stabilization

Background

Subtraction

Post-processing

Object Tracking

- Using OF from BM

​

- Fix reference point

- Adaptive Gaussian

Modelling

​

​

- Hole filling

​

- Area filtering

​

- Morph. filters

​

​

- Kalman filtering

​

- Other method:

MedianFlow tracker

​

​

Speed Estimaton

• Extract relevant data for traffic monitoring.

​

• Make the world a safer place by...

Our own study:

- Identify vehicles surpassing the speed limit

- Estimate vehicle density (cars/frame)

- Traffic rate (cars/min)

- Homography

transformation

​

- Distance Y / time

​

​

Speaker: Àlex V.

3

Video stabilization

1. Get Optical Flow with BM w.r.t reference frame.

​

2. Define the Translation matrix M,

where (u_p,v_p) is the OF at the selected point.

​

3. Rectify the frame by applying M.

Take 1st frame as reference frame. Select an static point.

For each frame:

Original sequence

Stabilized seq.

Background subtraction

Foreground

Foreground

Background

Original sequence

Background subtraction

• Adaptive Gaussian modelling with grayscale frames

𝝰

​

small → + FG

big → + BG

Relevance of current frame in the update

𝞺

​

Speaker: Àlex V.

4

Post-processing

Hole filling

Area filtering

Input

Closing + Hole filling

PR curve highway

PR curve traffic

Post-processing improved AUC ~ 10%

- 𝝰 and 𝞺 re-estimated after post-processing

​

- Use of shadow removal was discarded.

Structuring element: rectangle 4x2

Speaker: Àlex V.

5

Vehicle Tracking with Kalman Filter

Given the foreground segmentation mask,

track the moving objects (vehicles) along the sequence

How?

​

• Find centroid of each connected component

​

• Centroid position prediction for frame K+1 by means of Kalman Filter using:

- previous Kalman predictions

- position at current frame K

​

• IF the nearest centroid at frame K+1 satisfies a distance threshold, we track object to that position

​

ELSE consider a new object

Kalman Filter

​

Consider a measured value Z_k, the prediction X_k and a gain K_k

​

​

​

​

Using the prior error covariance and the measurement variance R we get the gain K_k:

​

​

Then, the following error covariance is:

Kalman

63.30 FPS

Suitable for real time!

Speaker: Roger

6

Vehicle Tracking with Kalman Filter

​

• What happens if a car is split in two CC?

​

• We need a method to 'group' close CC to the same ID!

​

Compute min distance between the bounding boxes

and merge if lower than a threshold (e.g. 20 pixels).

Speaker: Roger

7

Vehicle Tracking

​

(2) How to know if 2 CC are part of the same car or not?

​

• Take perspective into account.
• TOP cars are far away: CC of different cars can be close.
• BOTTOM cars are closer: CC of different cars are far

decrease thresholds!

Still fails if the 2 cars are part of the same CC !!!

Speaker: Roger

8

Vehicle Tracking - Alternative Technique

Same idea: Given the vehicle detection (mask), track the vehicles along the sequence

Difference: The tracking method (OpenCV)

• 5 different single object trackers in OpenCV 3.1 → BOOSTING, MIL, KCF, TLD, MEDIANFLOW

Main idea: Merge and Overlap still used!

• Initialize a tracker for every new detection (located on the bbox)
• Predict next location of each bbox. HOW?
• Motion and appearance model
• Motion model: Estimation of the direction of motion to predict current location
• Appearance model: How object looks like to provide more accurate location
• Abrupt changes in appearance! → Solution: Online Classifier (learning at a run time)

MEDIAN-FLOW tracker: Minimizes the forward-backward error. Photo-consistency based.

​

​

​

​

​

​

​

​

Need to adapt to

Multiple object tracking!

Pros and Cons

​

• Needs method to initialize for new detection

​

• Good tracking failure reporting

​

• Fails under large motion

​

• Works very well when the motion is highly predictable

​

• Sometimes tracker loses object → New tracker

26.97 FPS

Z Kalal et al. Forward-Backward Error: Automatic Detection of Tracking Failures.

ICPR. (2010). Link here.

Reference:

Speaker: Àlex P.

9

Speed Estimation

SPEED = DISTANCE [meters] / TIME [seconds]

The next step was to compute the speed of the vehicles. Speed is defined as displacement in a period of time:

→ Perspective distortion (1)

→ Measurement distance in pixels (2)

→ Rectify parallel lines by an affine rectification:

→ Estimate the factor that relates distance in pixels and distance in meters:

PROBLEMS:

1. Pick manually 2 pairs of points that conform two parallel lines

​

2. Compute two lines that passes through the corresponding pair of points

​

3. Compute the vanishing point as intersection of two lines

​

4. Estimate the homography H that projects the vanishing point to at infinity

1. Pick manually 2 points on the non-rectified image that are distanced 1 meter

​

2. Rectify both points with H and compute the distance between them

​

3. Compute the Pixels to Meters factor as:

​

​

​

Without affine rectification

After affine rectification

Factor = 1 / distance

To solve (1)

To solve (2)

Distance in meters between consecutives centroids divided by the elapsed time

SPEED

=

Inspired by

Team2, 2016

Speaker: Joan

10

Highway

Traffic

Speed Estimation

Bad results in Traffic dataset!

Possible reasons:

​

→ Too much perspective distortion

​

→ Poor rectification (image above)

​

→ Jitter is not 100% corrected

by stabilization... False positives!

24 fps assumed in both datasets

Speaker: Joan

11

Our own study

Extract relevant statistics to monitor road traffic

Density

​

Total number of cars that are surpassed the speed limit

LOW DENSITY

MEDIUM DENSITY

HIGH DENSITY

Checks number of cars in the actual frame

1 or less cars

​

2 or 3 cars

​

More than 3 cars

Infractions

​

NO INFRACTION →

INFRACTION →

Infractions

Traffic Rate

​

The average number of cars on the road in 1 minute [cars/min]

Traffic

Speed limit = 80km/h

Speaker: Joan

12

Best results

Highway sequence

Own dataset (source)

Speaker: Joan

13

Conclusions

Simple idea. We can get important data for a real-life situation in a relatively simple manner.

​

​

• Results are highly dependent on the parameter values and the input masks.

​

• We tried to adjust them but the optimal values differ a lot between different sequences, which indicates that the tested systems are not very robust...

​

• Speed estimation is inconsistent at some points, in particular when the vehicles get closer to the camera. Further work is required to remove the influence of apparent motion.

Speaker: Roger

Thanks!

Any questions?

You can find us at:

​

roger.mari01@estudiant.upf.edu

alex.vicente01@estudiant.upf.edu

joan.sintes01@estudiant.upf.edu

alex.palomo01@estudiant.upf.edu

Any questions?

Thanks!