[PPT] - Streets4MPI (Parallel Programming Project) Julian Fietkau Joachim PowerPoint Presentation

SLIDE 1

Streets4MPI (Parallel Programming Project)

Julian Fietkau Joachim Nitschke

University of Hamburg

April 4th, 2012

SLIDE 2

Agenda Julian Fietkau, Joachim Nitschke

Agenda

Introduction Simulation Concept Traffic load Traffic jam tolerance Implementation Parallelization Results Summary

2 / 34

SLIDE 3

Introduction Julian Fietkau, Joachim Nitschke

Project task, revisited

Decide on a problem that may be solved using parallel processing, and implement a solution. → Street traffic simulation

Main Caveat

Realistic traffic predictions can only be made using an exceedingly detailed model. This makes things prohibitively complicated.

3 / 34

SLIDE 4

Introduction Julian Fietkau, Joachim Nitschke

Streets4MPI is here

We can simulate thousands of cars on the streets of Hamburg: We use OpenStreetMap data for navigation. We incorporate shortest-path algorithms. We model road congestion and its effect on the actual driving speed. We optimize the road system based on which roads are heavily used

and which one are empty.

We can visualize all of this as dynamic heatmaps. Also, we can do most of this in parallel using MPI. 4 / 34

SLIDE 5

Simulation: Concept Julian Fietkau, Joachim Nitschke

Revision: Discrete macroscopic simulation

Simulation runs in steps: Traffic load in one step influences the

driver’s behavior in the next step

Abstract from single cars, traffic lights etc. to daily traffic Display traffic development over longer time periods and influences

n street network

5 / 34

SLIDE 6

Simulation: Concept Julian Fietkau, Joachim Nitschke

Street network

6 / 34

SLIDE 7

Simulation: Concept Julian Fietkau, Joachim Nitschke

Trips

Representation for a resident’s daily traffic Shortest path between two nodes 7 / 34

SLIDE 8

Simulation: Concept Julian Fietkau, Joachim Nitschke

Trips

8 / 34

SLIDE 9

Simulation: Traffic load Julian Fietkau, Joachim Nitschke

Traffic load: Effect on driving speed

Heavy traffic slows cars down How can we calculate the deceleration? Assumption: Cars keep safe braking distances

⇒ By looking at the braking distance, we calculate the actual speed

Braking distance and actual speed

lbraking = lstreet

ntrips − lcar

vactual =

lbraking · abraking · 2

9 / 34

SLIDE 10

Simulation: Traffic load Julian Fietkau, Joachim Nitschke

Oscillation

Problem: Drivers show oscillating behavior 10 / 34

SLIDE 11

Simulation: Traffic jam tolerance Julian Fietkau, Joachim Nitschke

Idea

Solution: Each driver gets an individual traffic jam tolerance

Traffic jam tolerance

vperceived = vactual + (videal − vactual) · ftolerance

Compromise: We assign a (random) traffic jam tolerance to each

process and all its residents

This is because one copy of the street network graph can accomodate

ne traffic jam tolerance factor, and each MPI process has its own

copy anyway

11 / 34

SLIDE 12

Simulation: Traffic jam tolerance Julian Fietkau, Joachim Nitschke

Result

12 / 34

SLIDE 13

Simulation: Traffic jam tolerance Julian Fietkau, Joachim Nitschke

Improvement: Reworking redundant nodes

Problem: Redundant nodes are used to model curved streets Solution: Merge edges 13 / 34

SLIDE 14

Implementation Julian Fietkau, Joachim Nitschke

Python

Streets4MPI is written in Python (2.6 compatible) External modules: imposm.parser, pygraph, mpi4py, PIL 14 / 34

SLIDE 15

Implementation Julian Fietkau, Joachim Nitschke

OSM parser

Import data from OpenStreetMap XML-based semi-structured data format: OSM May provide additional information: street types/sizes, speed

limits, residential/industrial/commercial zones

15 / 34

SLIDE 16

Implementation Julian Fietkau, Joachim Nitschke

<?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="CGImap 0.0.2"> <bounds minlat="54.0889580" minlon="12.2487570" maxlat="54.0913900" maxlon="12.2524800"/> <node id="298884269" lat="54.0901746" lon="12.2482632" user="SvenHRO" uid="46882" visible="true" version="1" changeset="676636" timestamp="2008-09-21T21:37:45Z"/> <node id="261728686" lat="54.0906309" lon="12.2441924" user="PikoWinter" uid="36744" visible="true" version="1" changeset="323878" timestamp="2008-05-03T13:39:23Z"/> ... <node id="298884272" lat="54.0901447" lon="12.2516513" user="SvenHRO" uid="46882" visible="true" version="1" changeset="676636" timestamp="2008-09-21T21:37:45Z"/> <way id="26659127" user="Masch" uid="55988" visible="true" version="5" changeset="4142606" timestamp="2010-03-16T11:47:08Z"> <nd ref="292403538"/> <nd ref="298884289"/> ... <nd ref="261728686"/> <tag k="highway" v="unclassified"/> <tag k="name" v="Pastower Straße"/> </way> ... </osm>

16 / 34

SLIDE 17

Implementation Julian Fietkau, Joachim Nitschke

Visualization

Simulation and visualization run independantly Visualization uses the Python Imaging Library to render the map

and traffic load data to image files

Supports many different data modes and two color modes

(heatmap and grayscale)

17 / 34

SLIDE 18

SLIDE 19

Parallelization Julian Fietkau, Joachim Nitschke

mpi4py

Object oriented interface on top of the MPI specifications Provides all usual MPI routines

communicator = MPI.COMM_WORLD

bject = None

if communicator.Get_rank() == 0:

bject = Object()
bject = communicator.bcast(object, root=0)

Single program multiple data (mostly) MPI code contained within our main class 19 / 34

SLIDE 20

Parallelization Julian Fietkau, Joachim Nitschke

Algorithm

Each MPI process. . . . . . generates its own copy of the street network . . . generates trips for its (equally divided) subset of all residents . . . gets its own traffic jam resistance . . . calculates the shortest paths for its residents and the resulting

traffic load

After every simulation step, each process gets sent the traffic loads

from all other processes (via mpi.allgather)

Complete results are saved to disk by process #0 20 / 34

SLIDE 21

Parallelization Julian Fietkau, Joachim Nitschke

Algorithm: visualization

...

MPI OSM

results results results results

python #0 python #1 python #n

time

iteration 0 iteration 1 iteration 2 iteration 3 iteration 0 iteration 1 iteration 2 iteration 3 iteration 0 iteration 1 iteration 2 iteration 3

exchange traffic data exchange traffic data exchange traffic data

21 / 34

SLIDE 22

Parallelization Julian Fietkau, Joachim Nitschke

Performance (I)

500 1000 1500 2000 2500 3000 1 4 8 12 16 25 50 75 100 total time [s] efficiency [%] # of processes Run times of Streets4MPI (Stellingen dataset, 1000 residents, 50 simulation steps) total time efficiency

22 / 34

SLIDE 23

Parallelization Julian Fietkau, Joachim Nitschke

Performance (II)

200 400 600 800 1000 1200 1 4 8 12 16 25 50 75 100 total time [s] efficiency [%] # of processes Run times of Streets4MPI (Stellingen dataset, 100 residents, 200 simulation steps) total time efficiency

23 / 34

SLIDE 24

Parallelization Julian Fietkau, Joachim Nitschke

Weaknesses

Some activities (e.g. initial I/O, road construction simulation) are

not easily parallelized using our current model

Disk activity by process #0 makes it drag behind and leave others

waiting for synchronization

Shortest path calculation is not optimal for the distributed case 24 / 34

SLIDE 25

Parallelization Julian Fietkau, Joachim Nitschke

Improvement: Shortest path revisited

Highest calculation costs are due to the shortest path calculations Current implementation: Dijkstra’s algorithm Complexity: O(n2

nodes)

Executed ∼ nresidents

nprocesses times

Static shortest path vs. dynamic shortest path 25 / 34

SLIDE 26

Parallelization Julian Fietkau, Joachim Nitschke

Dynamic shortest path

Idea: Calculate shortest paths once and update them only when

the edge weights change

Performance gain through local influence of changes 26 / 34

SLIDE 27

Parallelization Julian Fietkau, Joachim Nitschke

Dynamic shortest path: Increasing a weight

+𝚬w

27 / 34

SLIDE 28

Parallelization Julian Fietkau, Joachim Nitschke

Dynamic shortest path: Decreasing a weight

𝚬w

28 / 34

SLIDE 29

Results Julian Fietkau, Joachim Nitschke

Simulation speed-up results

Results are mostly satisfactory, but could very likely be improved There are constant time elements not yet parallelized It bears mentioning that using the current model (traffic jam

resistance per process) increases simulation quality with number of processes, so real efficiency is slightly better than measured

29 / 34

SLIDE 30

Results Julian Fietkau, Joachim Nitschke

Project Goals

Simulation working and producing nontrivial results Parallel processing in Python working Visualization working Further work needed: better parallelization(?), documentation 30 / 34

SLIDE 31

Summary Julian Fietkau, Joachim Nitschke

Most Important Points

Simple traffic simulation Macro level with congestion analysis, street development,

visualization

MPI on Python 31 / 34

SLIDE 32

Miscellaneous: Literature Julian Fietkau, Joachim Nitschke

Literature

Weber, B.; MÃŒller, P.; Wonka, P.; Gross, M.: Interactive Geometric Simulation of 4D Cities

In: EUROGRAPHICS 28 (2009), Nr. 2

Chandy, K.M.; Misra, J.: Distributed computation on graphs: Shortest path algorithms

In: Commun. ACM, vol. 25, no. 11, pp. 833 – 837, Nov. 1982

Antonio, J. K.; Huang, G. M.; Tsai, W. K.: A fast distributed shortest path algorithm for a class of hierar- chically clustered data networks

In: IEEE Trans. Comput., vol. 41, pp. 710 – 724, June 1992

32 / 34

SLIDE 33

Miscellaneous: Weblinks Julian Fietkau, Joachim Nitschke

Weblinks

Project website

http://jfietkau.github.com/Streets4MPI/ (some time soon)

GitHub repository

http://github.com/jfietkau/Streets4MPI (available right now!)

Project wiki

http://pwiki.julian-fietkau.de/ (might go offline soon-ish)

33 / 34

SLIDE 34

Miscellaneous: Download and Usage Julian Fietkau, Joachim Nitschke

Download and Usage

These slides are published under the CC-BY-SA 3.0 license.

All pictures and illustrations not created by Streets4MPI are based on content from the OpenClipArt Project.

Download these slides and give feedback:

http://www.julian-fietkau.de/streets4mpi_final

34 / 34