OSM in Loation Siene Jaak Laineste @jaakl CARTO 8 What is CAbTO - - PowerPoint PPT Presentation

▶
osm in lo ation s ien e
SMART_READER_LITE
LIVE PREVIEW

OSM in Loation Siene Jaak Laineste @jaakl CARTO 8 What is CAbTO - - PowerPoint PPT Presentation

Mar 27, 2024 139 likes •415 views

8 OSM in Loation Siene Jaak Laineste @jaakl CARTO 8 What is CAbTO CAbTO PIONEEbS IN LOCATION INTELLIGENCE K + , End-users Customers Team members CAbTO IS THE PLATFObM FOb

slide-1
SLIDE 1

8

Jaak Laineste @jaakl

OSM in Loation Siene

slide-2
SLIDE 2

8 CARTO

What is CAbTO

slide-3
SLIDE 3

CAbTO

Customers

PIONEEbS IN LOCATION INTELLIGENCE

,

End-users

K

Team members

+

slide-4
SLIDE 4

CAbTO IS THE PLATFObM FOb TUbNING LOCATION DATA INTO BUSINESS OUTCOMES

slide-5
SLIDE 5

BUILDEb

A self-service business user application for spatial analysis and visualization.

  • Built in drag and drop analytics or

custom functions

  • Both in the cloud and on premise
  • Auto-styling and Publishing
  • Rapid application deployment
  • Publish interactive dashboards that

update analysis and filter live

BbINGING LOCATION INTELLIGENCE TO THE MASSES

slide-6
SLIDE 6

ENGINE

POWEb YOUb APPS WITH LOCATION INTELLIGENCE

The one-stop shop for developers to power location applications in their organization.

  • Easy-to-use, open source APIs & SDKs
  • Location Data Services
  • Built for developers and designers
  • Native and custom analysis libraries
slide-7
SLIDE 7

ENGINE APIs

slide-8
SLIDE 8

DATA OBSEbVATObY

DON’T LET YOUb DATA LIMIT YOUb ANALYSIS

Augment your own data and broaden your analysis with thousands of datasets and measurements.

  • Demographic segments
  • Income, employment, and

family datasets

  • Real estate and financial data
  • Many more…
  • But : no OSM ”yet) !
slide-9
SLIDE 9

8 CARTO

Loation Siene tools

slide-10
SLIDE 10

8 CARTO

  • Most popular language for data

scientists

  • Extremely flexible
  • Huge + innovative community

Wh Pthon?

Image: xkcd.com/353

slide-11
SLIDE 11

8 CARTO

  • pen-source web application that

allows you to create and share documents that contain live code, equations, visualizations and narrative text

Jupter

slide-12
SLIDE 12

8 CARTO Image: Bayesian Methods for Hackers

  • The de facto standard for

communicating work

  • Discovery environment of choice

for many data scientists

  • Clearly shows reproducible

workflows

Jupter noteooks

slide-13
SLIDE 13

8 CARTO

Geo pthon oodies

slide-14
SLIDE 14

8 CARTO

Given enough time and code, you can probably do anything

Matplotli

from mpl_toolkits.basemap import Basemap import matplotlib.pyplot as plt import numpy as np map = Basemap(projection='ortho', lat_0 = 50, lon_0 = -100, resolution = 'l', area_thresh = 1000.) map.drawcoastlines() map.drawcountries() map.fillcontinents(color = 'coral') map.drawmapboundary() map.drawmeridians(np.arange(0, 360, 30)) map.drawparallels(np.arange(-90, 90, 30)) plt.show()

slide-15
SLIDE 15

8 CARTO

GeoPandas

import geopandas as gpd boroughs = gpd.datasets.get_path('nybb') df = gpd.read_file(boroughs) df.plot(column='Shape_Area', figsize=(10, 10), alpha=0.5)

slide-16
SLIDE 16

8 CARTO

Leaflet and Python integrated

Folium

import folium import pandas as pd state_data = pd.read_csv('data.csv') state_geo = 'us_states.geojson' m = folium.Map(location=[48, -102]) m.choropleth( geo_data=state_geo, name='choropleth', data=state_data, columns=['State', 'Unemployment'], key_on='feature.id', fill_color='YlGn', fill_opacity=0.7, line_opacity=0.2, legend_name='Unemployment Rate (%)' )

slide-17
SLIDE 17

Countin Your Chiken Beore The Hath CAbTOrames

slide-18
SLIDE 18

8 CARTO

OSM in CartoFrames

slide-19
SLIDE 19

8 CARTO

1. Extract data from OSM global database 2. Reduce data size, preprocess, filter, geocode 3. Do analysis with Python 4. Make a map! 5. Rinse, repeat

Case stud

What are the most popular names in different countries?

slide-20
SLIDE 20

8 CARTO

Data etrat - the hard part

1. Planet → Imposm3 → PostGIS → SQL → result UPDATE osm_roads AS r SET admin2 = (SELECT a.name_iso FROM adm0 AS a WHERE r.geometry && a.geom LIMIT 1); SELECT admin2, name, count(*) FROM osm_roads GROUP BY name, admin2; 2. Overpass API. Can get names, even countries, but output format is tricky Hard to write queries, too big result 3. Big Data as a service providers. AWS has OSM Planet, weekly updated, queryable via Athena SQL. No polygon query or custom functions. Google BigQuery does not have OSM ”yet). But it has user functions, can do point-in-polygon

slide-21
SLIDE 21

8 CARTO

Winnin method

1. Download per-country packages from Geofabrik 2. Osmconvert to o5m 3. Osmfilter for key stats to CSV 4. Sort and head 5. Py: transpose data 6. Py: make a CartoFrames map Fast! Convert: few seconds for small country, Italy ”1.3G) ~1

  • minute. Filter: also ~1 minute
slide-22
SLIDE 22

8 CARTO

Noteook:

https://github.com/jaakla/osm-name-stats Final map in CARTO: https://cartomobile-team.carto.com/u/jaakl /builder/166ca721-7c3a-47d6-80d8-7d340e 22b0ab/embed

slide-23
SLIDE 23

8 CARTO Locations

Ke learnins

  • Reduce big data to small
  • Use preprocessed data
  • Preprocess with proper tools
  • There are no universal tools

for big datasets

  • Most tools are not ok for the

Planet queries. E.g. PostGIS

  • Use optimized formats
  • Test with samples
  • Some steps remain manual
slide-24
SLIDE 24

8 CARTO

bead aout CAbTOrames

Silas Toms, Eric van Rees, and Paul Crickard

Includes a full chapter on CARTOframes

slide-25
SLIDE 25

8 CARTO

Tr oursel

Visit https://github.com/CartoDB/cartoframe s and click launch binder

slide-26
SLIDE 26

Thanks!

8

Jaak Laineste jaak@carto.com / jaak@nutiteq.com @jaakl Credits: Michelle Ho @ CARTO