SLIDE 1
Rendering maps without Database Thomas Skowron Previously (if you - - PowerPoint PPT Presentation
Rendering maps without Database Thomas Skowron Previously (if you - - PowerPoint PPT Presentation
Rendering maps without Database Thomas Skowron Previously (if you happen to speak German) berblick ber Pipelinebasierte Rendering-Techniken Erzeugung von und So fu ware Karten FOSSGIS-Konferenz 2017 FOSSGIS-Konferenz 2018
SLIDE 2
SLIDE 3
Turning OSM Data into a graphical map
SLIDE 4
Bitmap/Vector Tiles, Maps with larger extent, …
SLIDE 5
State of the art
SLIDE 6
OSM Planet PostgreSQL Renderer
Bottleneck Bottleneck Bottleneck Awesome, powerful stuff
SLIDE 7
PostgreSQL
Do we really need all of that stuff?
+ SQL + ACID, MVCC & transactions + Indexes + Role permission management + scriptable + fail over + …
- Performance (PostGIS)
- operational cost
- memory consumption
SLIDE 8
Attempts to improve the situation…
SLIDE 9
OSM Planet PostgreSQL Vector File “Renderer“
Bottleneck Bottleneck Bottleneck
Client Based Renderer
SLIDE 10
Pre-rendered Vector Tiles Client Based Renderer
All features already baked in, flexibility mostly gone
SLIDE 11
Alternative Approaches
SLIDE 12
tippecanoe
SLIDE 13
OSM File mbtiles vector tile set
SLIDE 14
Clever features to keep vector tiles small
SLIDE 15
has a gazillion of options
SLIDE 16
still very limited to filtering
SLIDE 17
Tilemaker
SLIDE 18
flexibility through lua scripting
SLIDE 19
not scalable to larger extracts
SLIDE 20
But why does one tool needs to do everything?
SLIDE 21
Generally, we are all doing almost the same stuff.
SLIDE 22
Step 1
Convert OSM data into geo data
SLIDE 23
Step 2
Filter
SLIDE 24
Step 3
Transform/map data
SLIDE 25
Step 4
Convert into target format
SLIDE 26
Suggestion:
parse | map-reduce | render
SLIDE 27
But how?
SLIDE 28
With Tools, which each do
- ne thing well
SLIDE 29
and a portable data format
SLIDE 30
Let’s do Shapefiles!
SLIDE 31
Let’s do Shapefiles!
SLIDE 32
Let’s do OSMPBF!
SLIDE 33
Let’s do OSMPBF!
SLIDE 34
What does a suitable file format need?
SLIDE 35
Performance
linear writes, parallelizable reads
SLIDE 36
Scalable
small to huge data sets
SLIDE 37
Tag structures
No tables no more!
SLIDE 38
Future proof
adaptable to change
SLIDE 39
Shapefile
Performance moderate Scalable no, 2 GB size limit Tag Strucutre no Future Proof no
SLIDE 40
GeoJSON
Performance moderate Scalable moderately, single threaded Tag Structure yes Future Proof limited
SLIDE 41
GeoPackage
Performance bad (SQLite) Scalable moderately Tag Structure yes Future Proof yes
SLIDE 42
Performance Flexibility
SLIDE 43
We need something new
There is no progress without change
SLIDE 44
How would a new file format look like?
- binary
- blocks, streamable
- single stream, not multiple files
- not SQLite
- not overly obscure
- open and extendable
SLIDE 45
Suggestion
SLIDE 46
SLIDE 47
Based on Protocol Buffers and WKB
SLIDE 48
Open Spec on
https://thomas.skowron.eu/spaten/
SLIDE 49
Reference implementation in Go
github.com/thomersch/grandine/lib/spaten
SLIDE 50
Around 50% smaller than GeoJSON*
* YMMV
SLIDE 51
Version 0
SLIDE 52
Feedback and Ideas are welcome
SLIDE 53
What could we do with it?
SLIDE 54
grandine-spatialize -in planet.osm.pbf -mapping roads.yml | grandine-tiler -out tiles/roads/ -zoom 14
SLIDE 55
- smium export -f spaten planet.osm.pbf |
gradine-converter -mapping roads.yml | grandine-tiler -out tiles/roads/ -zoom 14
(not yet)
SLIDE 56
- smium export -f spaten planet.osm.pbf |
your-tool-here -fancify | magic-renderer
(not yet)
SLIDE 57
Interchangeable tools
SLIDE 58
Future
SLIDE 59
Greater flexibility with less programming work
SLIDE 60
Faster processing with less hardware
SLIDE 61
Less points of failure
SLIDE 62
There is still lots to do
Data format, tools, markup, …
SLIDE 63
Let’s build the future together!
SLIDE 64