Queen Creek Video Count Analysis Project Summary�

Completed by the Town of Queen Creek with support from:

​

Dr. Jeff Jenq, MAG

Dr. Sarath Joshua, MAG (retired)

Tricia Brown, ADOT

Dr. Sarah Simpson, UCG

Arizona ITE-IMSA Annual Spring Conference

April 2024

Presentation Overview

Motivation

​

Process

​

Findings

​

Conclusions

2

Motivation

• Perform unbiased study using statistical analysis
• ATSPMs and signal optimization are fully dependent on the use of traffic counts
• Video detection systems are commonly used for traffic signal operation, and for a variety of good reasons
• Video analytics / ATSPMs are now available from a variety of suppliers
• Are ATSPMs from video analytics acceptable?
• What kinds of situations lead to poor count performance?

3

Motivation

• Need to analyze all types of conditions
• High Volume Periods
• Low Volume Periods
• Sunrise
• Sunset
• Day of the Equinox (Sunset/Sunrise aligns with East/West lanes)
• Different lane types (Lane by lane)
• Four manual count observers to mitigate human error
• Binning as small as practicable to mimic detection needs for operations

4

Process

• TOQC recorded videos and video lane-by-lane counts for
• 8 intersections
• 30 cameras
• Videos from September 2022 – around Equinox and worst sun scenarios
• Three time periods (~10 hours total recording each)
• AM
• Mid-day
• PM
• Four human observers counted lane-by-lane in 5-minute bins during green phase crossing stop bar
• ~300 hours of videos, 4 people. A lot, lot, lot of time to manually observe.
• A few video sections had to be thrown out due to glitches, but not many.

5

Process – Reconciliation of Human Counts

• With 14,000 5-minute bins, cannot afford the level of effort to do recounts
• Reasonable process:
• Four people match, OK
• Three people match, OK
• Two people match, two people match and off by one 🡪 use average
• Two people match and one of those is the most experienced counter 🡪 use this number
• Note: if average is the “ground truth”, video counted as correct if off by 0.5
• Example: Average is 20.5, video is “right” if 20 or 21. If 22 or 19, “wrong”.
• “Wrong” here means not a 100% match. There are many use cases for the data where 117 instead of 119 will not cause adverse affects.

​

6

Statistical Comparisons

• Pearson Goodness of Fit test
• Histogram comparisons
• Tabulation of over- and under-counting bins
• Qualitative judgment of unacceptable error severity

7

Pearson Test

• Compares every observation to each other (video/humans) to determine how closely a “straight line” can be drawn to connect the two
• The lower the value of the Pearson, the worse the match between video and ground truth
• Over- and under-counts drive down the Pearson value, even though the average might be just fine (squaring the differences in the denominator causes this)

8

Pearson Test

• Good example (0.98)

​

​

​

​

​

​

​

​

​

​

Ocotillo and Power NB Left

• Poor example (0.81)

​

​

​

​

​

​

​

​

Ocotillo and Ellsworth Loop NB Thru 2

Note that Pearson of 0.98 does not mean “98% correct”

9

Where’s our good friend the T-test?

• #1 – Traffic is not “normally distributed”
• Counts cannot be less than zero
• #2 – comparisons at low number of observations (<100) have high level of standard error, i.e. the significance of a finding can be weak.
• #3 – findings that the means are probably the same are not usually helpful for traffic operations uses of the information, need to dig deeper than this
• Skewness is key: inspect histograms and look at under/over counting errors instead of t-values and standard error

10

Tabulating over and under counts

• Identifies bias, or lack of bias
• Careful here: “off by one” and “off by 10” are counted the same in this analysis
• “30% undercount” is not a traffic volume, it is the number of bins undercounted in the data set (e.g. 3 out of 10 bins are undercounted by the video)
• 30% over, 30% under, 40% exact with Pearson 0.98 is generally fine (off by 1 errors)
• 30% over, 30% under 40% exact with Pearson 0.92 🡪 generally indicates some problem
• Unequal distribution of under/over count percentages 🡪 bias
• Combine with Pearson to determine degree of bias
• We also tabulated cumulative under/over count total volume, but decided not to consider a numerical/statistical test for this
• Example: total count is 1,500. Undercount is 150, Overcount is 50

11

Qualitative assignment of significant findings

• Any Pearson below 0.94 seems to indicate some kind of issue
• Determined by engineering judgment
• Acceptable: Almost all Pearson 0.95+
• Slight undercount: Inspect graph
• Undercount: Pearson < 0.95 and/or % counts under very high
• Slight overcount: Inspect graph
• Overcount: Pearson < 0.95 and/or % counts over very high
• Both over/undercount: inspect graph

Reminder: Pearson of 0.94 does not mean “94% correct”

​

Pearson indicates how closely a “straight line” can be drawn to connect the video and person counts

12

Acceptable

Sossaman Road and Riggs Road

Count

Frequency of Bin Count

Count range

Bin Start Time

13

Slight Undercount

Ocotillo Road and Power Road

Count

Frequency of Bin Count

Count range

Bin Start Time

14

Undercount

Ocotillo Road and Ellsworth Loop Road

Count

Frequency of Bin Count

Count range

Bin Start Time

15

Both Under/Over in the Same Period

Ocotillo Road and 226^th Road

Count

Frequency of Bin Count

Count range

Bin Start Time

16

Over/undercount at the same time

• Could it be caused by bin breakpoints?
• Humans concluded the vehicle crossing the stop bar at the exact time the bin ended were in “Bin 1”
• Video concluded the vehicle crossing the stop bar at the exact time the bin ended were in “Bin 2”
• Does not seem prevalent
• <2% of occurrences appear as off-by-one “ping-pong” behavior possibly due to bin breakpoints occurring during the green/yellow interval of the approach/phase being counted
• Many occurrences of multiple sequential “under” bins and then later in the time period many/several occurrences of “over” bins in this category

17

Findings

• Generally the video system “gets it exactly right” for 60% of the bins
• Many (10%-20%) additional bin errors are just “off by one”
• At the highest level of total aggregation for all eight intersections/30 cameras for all observations the video system undercounts by 2% of the total volume (~180,000 versus ~176,000 total vehicles)

18

Each Intersection, All TOD Periods

Note generally how high the Pearson values are at this level of total aggregation

Denotes finding of concern

19

Each intersection, Each TOD Period

Denotes finding of concern

20

Each Approach, Each TOD Period

Denotes finding of concern

N/A indicates approach does not exist

21

Each Lane, Each TOD Period, Each Approach

Denotes finding of concern

N/A indicates lane does not exist

22

Each Lane,�Each TOD Period, Each Approach…

Skipping 10+ tables like previous…see report

23

Conclusions

• 309 individual time-period-lane findings (103 lanes, 3 periods per lane)

​

​

​

​

​

​

​

​

• Undercount 16%, Overcount 17%
• Although slight bias to undercount total volume (~2%), Over/Under error rates are approximately the same
• Probably a 2%-3% margin of error introduced by counting process artifacts (bin start/end times occurring during a green phase). i.e. Acceptable percentage might be 66% instead of 63%.

�

Time periods deemed “inconclusive” due to lack of enough information (<1%) were counted as “acceptable” for the purpose of this table.

Acceptable =

Pearson >0.95, per lane, per time period

24

Conclusions: West-pointing cameras

• Three times more likely to Overcount than Undercount when wrong
• Through counts are more often incorrect than exactly correct
• Occlusion causes double-counting?

�

Time periods deemed “inconclusive” due to lack of enough information (<1%) were counted as “acceptable” for the purpose of this table.

Acceptable =

Pearson >0.95, per lane, per time period

25

Conclusions: East-pointing cameras

• Twice as likely to overcount than undercount
• Error rate (25%) is half of error rate of west-pointing cameras (52%)

​

�

Time periods deemed “inconclusive” due to lack of enough information (<1%) were counted as “acceptable” for the purpose of this table.

Acceptable =

Pearson >0.95, per lane, per time period

26

Why is pointing West apparently harder than pointing East (in this data set)?

• In the AM period, the volumes are significantly lower than the mid-day and PM periods
• The higher the volume being counted, the more difficult it becomes for the camera system to perform acceptably
• Since the traffic volume coming westbound in the AM (sunrise) is 3x lower than the volume going eastbound in the PM (sunset), the error rate for cameras pointing into the sun during sunrise is significantly lower than for cameras pointing into the sun during sunset

The unacceptable performance during sunset (west-pointing cameras) or sunrise (east-pointing cameras) tends to be to overcount.

​

27

Conclusions: Why is Ellsworth/Ocotillo least acceptable?

​

​

�

28

Summary of Conclusions

• The Video Detection counts are an exact match to manual counts ~65% in 5-minute bins
• Another 10%-20% is probably good enough (“off by one” kind of errors)
• Sunset is difficult to count correctly when pointing into the sun
• Higher volumes are more difficult to count correctly to get right than lower volumes (more vehicles to track/identify in the scene)
• Sunrise seemed to not be as difficult as sunset, but this is probably due to significantly lower volumes in Queen Creek in the AM period
• No systematic bias of this particular system to generally under/over count, although total volume was ~2% lower than actual volume
• Possibly a 2%-3% margin of error introduced by counting process artifacts

29

Questions?

Backups

Jump to Conclusions

• The 5-minute binned Video Detection counts are an exact match to manual counts ~65% of the time
• Another ~10% of the 5-minute binned video counts are only “off by one”
• Sunset is difficult to count correctly when pointing into the sun
• Higher volumes are more difficult to count correctly than lower volumes (more vehicles to track/identify in the scene)
• Sunrise seemed to not be as difficult as sunset, but this is probably due to significantly lower volumes in Queen Creek in the AM period
• No systematic bias of this particular system to generally under/over count, although total volume was ~2% lower than manual count or ground truth volume
• Probably a 2%-3% margin introduced by counting process artifacts

33

Slight Overcount

Sossaman Road and Riggs Road

34

Overcount

Rittenhouse Road and Ellsworth Loop Road

35

All intersections/cameras by TOD

36

Each Approach, All TOD Periods

Denotes finding of concern

37

Each Lane, All TOD Periods (EB)

Denotes finding of concern

N/A indicates lane does not exist

38

Each Lane, All TOD Periods (WB)

Denotes finding of concern

N/A indicates lane does not exist

39

Each Lane, All TOD Periods (NB)

Denotes finding of concern

N/A indicates lane does not exist

40

Each Lane, All TOD Periods (SB)

Denotes finding of concern

N/A indicates lane does not exist

41

Conclusions: South-pointing cameras

• Overcount left-turns twice as often as undercount
• Undercount thru and shared lanes

​

�

Time periods deemed “inconclusive” due to lack of enough information (<1%) were counted as “acceptable” for the purpose of this table.

Acceptable =

Pearson >0.95, per lane, per time period

42

Conclusions: North-pointing cameras

• Overcount left-turns twice as often as undercount
• Undercount thru and shared lanes
• Essentially the same type of errors for North and South pointing cameras
• No directional N/S bias observed (shadows and light/dark transition moving in one direction versus the other have no systematic effect)

​

�

Time periods deemed “inconclusive” due to lack of enough information (<1%) were counted as “acceptable” for the purpose of this table.

Acceptable = Pearson >0.95, per lane, per time period

43