Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic - PowerPoint PPT Presentation

Heterogeneous Stream Processing and Crowdsourcing for Urban Traffic Management Alexander Artikis 1 , Matthias Weidlich 2 , Francois Schnitzler 3 , Ioannis Boutsis 4 , Thomas Liebig 5 , Nico Piatkowski 5 , Christian Bockermann 5 , Katharina Morik 5 , Vana Kalogeraki 4 , Jakub Marecek 6 , Avigdor Gal 3 , Shie Mannor 3 , Dermot Kinane 7 and Dimitrios Gunopulos 8 1 NCSR Demokritos, Greece, 2 Imperial College London, UK, 3 Technion, Israel, 4 Athens University of Economics and Business, Greece, 5 Technical University Dortmund, Germany, 6 IBM Research, Ireland, 7 Dublin City Council, Ireland, 8 University of Athens, Greece http://www.insight-ict.eu/

Urban Traffic Management Challenges ◮ Volume ◮ Increasing number of data sources. ◮ Variety ◮ Heterogeneous data sources. ◮ Veracity ◮ Inaccurate measurements, network failures, interference of mediators. ◮ Sparsity ◮ Traffic in several locations is never/infrequently monitored.

Addressing the Challenges ◮ Volume ◮ Stream processing. ◮ Variety ◮ Complex event processing. ◮ Veracity ◮ Crowdsourcing. ◮ Sparsity ◮ Traffic modelling.

The Insight System

Complex Event Processing ◮ Data variety problem: heterogeneous event sources ◮ Buses: position, direction, route, congestion. ◮ SCATS sensors: traffic flow, traffic density. ◮ Solution: complex event processing ◮ Compute bus punctuality, bus driving quality, traffic congestion trend, traffic congestion. ◮ Engine: Event Calculus for Run-Time reasoning (RTEC) ◮ Formal, declarative semantics. ◮ Interval-based reasoning. ◮ Succinct & intuitive representation of complex event patterns. ◮ Highly efficient (for complex event hierarchies). ◮ Machine learning support for automated construction of complex event patterns

Complex Event Processing Buses reporting congestion at some location ( Lon , Lat ) of interest: busCongestion ( Lon , Lat ) initiated iff move ( Bus , Lon B , Lat B , 1 ) happens , close ( Lon B , Lat B , Lon , Lat ) busCongestion ( Lon , Lat ) terminated iff move ( Bus , Lon B , Lat B , 0 ) happens , close ( Lon B , Lat B , Lon , Lat )

Complex Event Processing Identifying mismatches among different streams: disagree ( Bus , Lon I , Lat I , 1 ) happens if move ( Bus , Lon B , Lat B , 1 ) happens , close ( Lon B , Lat B , Lon I , Lat I ) , not ( scatsCongestion ( Lon I , Lat I ) = true holds ) disagree ( Bus , Lon I , Lat I , 0 ) happens if move ( Bus , Lon B , Lat B , 0 ) happens , close ( Lon B , Lat B , Lon I , Lat I ) , scatsCongestion ( Lon I , Lat I ) = true holds

Complex Event Processing Dealing with event source disagreement: noisy ( Bus ) = true initiatedAt T iff disagree ( Bus , Lon I , Lat I , BusVal ) happensAt T , crowd ( Lon I , Lat I , CrowdVal ) happensAt T ′ , BusVal � = CrowdVal , 0 < T ′ − T < threshold noisy ( Bus ) = true terminated if agree ( Bus ) happens noisy ( Bus ) = true terminatedAt T if disagree ( Bus , Lon I , Lat I , BusVal ) happensAt T , crowd ( Lon I , Lat I , CrowdVal ) happensAt T ′ , BusVal = CrowdVal , 0 < T ′ − T < threshold

Self-Adaptive Complex Event Processing Discarding temporarily unreliable event sources: busReportedCongestion ( Lon , Lat ) initiated iff move ( Bus , Lon B , Lat B , 1 ) happens , ¬ ( noisy ( Bus ) holds ) , close ( Lon B , Lat B , Lon , Lat ) busReportedCongestion ( Lon , Lat ) terminated iff move ( Bus , Lon B , Lat B , 0 ) happens , ¬ ( noisy ( Bus ) holds ) , close ( Lon B , Lat B , Lon , Lat )

Complex Event Processing in Dublin Static Event Recognition Self-Adaptive Event Recognition 10 9 8 7 Time (sec) 6 5 4 3 2 1 0 10 min = 30 min = 50 min = 70 min = 90 min = 110 min = 12,5K SDE 40,5K SDE 67K SDE 94,5K SDE 124K SDE 152K SDE Working Memory

Crowdsourcing ◮ Data veracity problem: Inaccurate measurements, network failures, interference of mediators. ◮ Solution: Query human volunteers (imperfect experts) close to the location of event source disagreement. ◮ Model the reliability of each participant ◮ Online Expectation-Maximisation. ◮ Use participant reliability to improve the aggregation of answers.

Crowdsourcing 1 1 5 9 Relative estimation error of p i 2 6 10 3 7 0.5 4 8 0 -0.5 -1 0 200 400 600 800 1000 Number of queries to participant i

Crowdsourcing—Query Execution Engine ◮ Communicate queries to participants reliably & efficiently. ◮ MapReduce is used. ◮ Each participant registers using a mobile device. ◮ Select a list of participants based on reliability, location, etc. ◮ Disseminate the query. ◮ Aggregate answers.

Crowdsourcing—Query Execution Engine Trigger Task Send Push Notification Communication Time 1000 900 800 Latency (ms) 700 600 500 400 300 200 100 0 2G 3G WiFi

Traffic Modelling ◮ Data sparsity problem: Several parts of the city are never/infrequently monitored. ◮ Solution: Generalise observations of monitored locations to produce estimates for locations without sensors. ◮ Scalability to city-sized areas is achieved by modelling the usual case. ◮ Traffic network is represented with a Gaussian Process regression framework ◮ SCATS intersections: observed traffic flow values. ◮ Variables are highly correlated if they are adjacent in the traffic network.

Traffic Modelling: Map of Dublin

Traffic Modelling: Street Network & SCATS locations

Traffic Modelling: Traffic Flow Estimates

Summary Insight solution to Urban Traffic Management: ◮ Volume ◮ Stream processing. ◮ Variety ◮ Complex event processing. ◮ Veracity ◮ Crowdsourcing. ◮ Sparsity ◮ Traffic modelling.