Data for Official Statistics Marco Puts, Piet Daas, Martijn - PowerPoint PPT Presentation

High Frequency Road Sensor Data for Official Statistics Marco Puts, Piet Daas, Martijn Tennekes

Road sensors Road sensor data – Passing vehicle counts for each minute (24/7) at about 60.000 sensors in the Netherlands – Types of sensors: ‐ Induction loop ‐ Camera ‐ Bluetooth – Length categories (e.g. small, medium, long vehicles) – Large volume: approx. 230 mln records/day 2

Dutch highways 3

Dutch highways with road sensors 4

Final Process Transform & Select Frame Clean Aggregate

Big Data Processes Quality feedback loop Q i Compare Q o output input Process 6

Low quality of data 7

Quality of the Data 8

Cleaning the data Hidden Markov Model state X 1 X 2 X 3 X 4 Y 1 Y 2 Y 3 Y 4 observation 9

Cleaning the Data Recursive Bayesian Estimation state X 1 X 2 X 3 X 4 Y 1 Y 2 Y 3 Y 4 observation Update 10

Cleaning the Data Recursive Bayesian Estimation state X 1 X 2 X 3 X 4 Y 1 Y 2 Y 3 Y 4 observation Prediction 11

Cleaning the Data Recursive Bayesian Estimation state X 1 X 2 X 3 X 4 Y 1 Y 2 Y 3 Y 4 observation Missing Data 12

Cleaning the Data Recursive Bayesian Estimation state X 1 X 2 X 3 X 4 Y 1 Y 2 Y 3 Y 4 observation Smoothing 13

Results of the filter Vehicle count Time [minutes] 14

Precision/Accuracy The filter does not introduce extra errors: – Precision: 3.6% – Accuracy:+0.13% 15

Questions ? 16