Evalu luatin ing c commercia ial p l probe d data o on arteria - - PowerPoint PPT Presentation

Evalu luatin ing c commercia ial p l probe d data o

• n arteria

ial f l facili ilitie ies: Insi nsights s from the V he Veh ehicle P Probe be P Projec ect validation pr progr gram

ZACH VANDER LAAN UMD CENTER FOR ADVANCED TRANSPORTATION TECHNOLOGY

Adventures in Crowdsourcing: Verifying Crowdsourced Data

Agenda

• VPP Validation Program Background
• Data Validation Approach
• Arterial Case Studies
• Conclusions and Next Steps

Agenda

• VPP Validation Program Background
• Data Validation Approach
• Arterial Case Studies
• Conclusions and Next Steps

Vehicle Probe Project (VPP) History

VPPI (2008)

• I-95 Corridor Coalition (now Eastern Transportation Coalition)

established first and largest multi-jurisdictional Traffic Monitoring System sourced with Industry data

• Established accuracy, latency, availability standards for probe-

based traffic data

• Developed validation methodology and program

VPPII (2014)

• Established multi-vendor marketplace
• Speed/travel-time standards extended to signalized roadways

VPPI and VPPII Validation

• Have consistently validated VPP speed and

travel time data since 2009

• VPPI (2009-2014)
• 1 vendor (INRIX)
• 45 validation reports
• Focused primarily on freeways at first, but

started exploring arterials at the end

• VPPII (2014-current)
• 3 vendors (HERE, INIRIX, TomTom)
• 24 total reports
• Freeways & arterials

Agenda

• VPP Validation Program Background
• Data Validation Approach
• Arterial Case Studies
• Conclusions and Next Steps

Validation Process

Main idea:

• Collect ground truth travel time data
• Compare with speed/travel time data reported by VPP

vendors

• Compute error metrics, visuals, and summarize results in

reports

Evaluate data quality:

• On various road types (e.g., freeway, arterial) and

geographic areas

• From multiple perspectives:
• “Point in time” vs repeatable patterns
• Overall performance vs. during aberrations

Wireless Re-identification Technology (WRTM) used to collect ground truth travel time samples

Traditional Validation

• Compare Vendor & WRTM speeds in 5-minute bins
• Average Absolute Speed Error (AASE): Measures deviation

from ground truth (10 MPH spec)

• Speed Error Bias (SEB): Measures consistent
• ver/underestimation of reported speed (+/- 5 MPH spec)
• Error metrics are computed for four flow regimes
• Specs are applied against Standard Error of the

Mean (SEM) band (interval estimate of mean)  Works well on freeways, but doesn’t tell the whole story on arterials

Arterials are more complex…

Arterial characteristics (relative to freeway)

• Lower volume
• Lower average traffic speeds
• Interrupted flow (traffic signals, mid-block friction)

○

Bi-modal speed distributions

○

Higher variance in speeds

Implications for Traditional Validation

• Average (i.e., space-mean) speed is used for

evaluation – unexpected results when WRTM speeds have high variance / multiple modes

• Error measures need to be carefully interpreted

○

High variance can mask performance (wide band) Distinct speed modes (but no one travels at the average speed)

Slowdown Analysis

• Major slowdown events identified in reference data
• For each slowdown, vendor data is graded based on how

well it captures the magnitude and duration:

• Fully captured
• Partially captured
• Failed to capture
• Evaluates data quality specifically during anomalies

(traditional method weights all 5-min periods the same)

 This approach turns out to be useful on arterials

Agenda

• VPP Validation Program Background
• Data Validation Approach
• Arterial Case Studies
• Conclusions and Next Steps

Original Arterial Report (VPPI)

• Original arterial report produced in 2015

○ 13 separate data collections during 2013-2014 ○ VPPI data only (1 vendor)

• Key findings:

○ Data quality depended heavily on road characteristics (signal density, and to a lesser extent volume) ○ Slowdown analysis provided the most insight into data quality ○ Fundamental issues across all case studies (erring towards faster speeds, complex flow patterns can’t be

• bseved)

Recommendations from **ORIGINAL** VPPI report

Arterial Report Update – VPPII Follow-up

• ETC Coalition commissioned an update based on

VPPII data

○ Has data quality improved over time? ○ Are there major differences in data quality across vendors? ○ Is data quality still linked to road characteristics (signal density)?

• Updated report was produced in November 2019

○ 14 separate arterial data collections between 2014-2018 ○ 3 vendors (called VPPII Vendor 1,2,3)

Traditional Analysis Results

All vendors in VPPII:

• Are highly compliant with contract specs
• Have improved error measures (AASE and

SEB) across speed bins relative to VPPI levels  Encouraging results, but recall traditional validation does not tell the whole story on arterials

Slowdown Analysis Results

• Drastic improvement for all vendors
• Fully captured: 33%  59-66%
• Failed to capture: 25%  6-10%

Impact of Road characteristics

• VPPI data quality was closely linked to road

characteristics (especially signal density)

• This is NOT the case with VPPII data (all vendors)
• But.. AADT and signal density are still worth

considering

○ Lower AADT = fewer observations (harder to characterize ground truth conditions) ○ Higher signal density = more complex traffic flow

Agenda

• VPP Validation Program Background
• Data Validation Approach
• Arterial Case Studies
• Conclusions and Next Steps

Conclusions

• Performance has improved dramatically over time

○

All 3 vendors much more accurate now than VPPI

• Within the observed range of road conditions (0-3 signals / mile, >20k AADT):

○ All 3 vendors’ data is suitable for planning and many operational use cases ○ Data quality is no longer tied to road characteristics (encouraging, but harder to provide

“rules of thumb”)

• Existing challenges:

○

Data errs towards faster speeds during congested periods

○

Complex flow patterns can’t be captured in single value

○

Low volume roads difficult to validate

Next steps

• Evaluate arterial probe data under conditions that fall
• utside current case studies
• Two 2020 low-volume deployments (<20k AADT)
• Refine analysis techniques
• Focus on developing methods to quantifying repeatable

patterns, rather than just “point-in-time”

• Prepare for VPPIII launch in 2021

○ Speed and travel time data remain a core product ○ Traffic volume & other products added (e.g., trajectory, O-D) ○ Currently developing validation strategies to streamline process and accommodate new data

E.g., Comparing time-of-day travel time distributions to quantify repeatable patterns