Working with geodata in Go Andrew Minkin Who am I TeamLead at - - PowerPoint PPT Presentation

▶
working with geodata in go
SMART_READER_LITE
LIVE PREVIEW

Working with geodata in Go Andrew Minkin Who am I TeamLead at - - PowerPoint PPT Presentation

Sep 23, 2023 167 likes •766 views

Working with geodata in Go Andrew Minkin Who am I TeamLead at maddevs.io 10 years of experience https:/ /github.com/meshbird/meshbird Bishkek 1kk citizens 100++ taxi services Namba Taxi 8k orders 600+ online drivers

slide-1
SLIDE 1

Working with geodata in Go

Andrew Minkin

slide-2
SLIDE 2

Who am I

  • TeamLead at maddevs.io
  • 10 years of experience
  • https:/

/github.com/meshbird/meshbird

slide-3
SLIDE 3

Bishkek

  • 1kk citizens
  • 100++ taxi services
slide-4
SLIDE 4

Namba Taxi

  • 8k orders
  • 600+ online drivers
  • 500k clients
slide-5
SLIDE 5

What’s a taxi

  • Clients
  • Voice
  • Text
  • Mobile App
  • ….
  • Drivers
  • Operators
  • Managers
slide-6
SLIDE 6

AVG response time

  • Drivers 20 ms
  • Operators 2.5 ms
slide-7
SLIDE 7

Prehistory

slide-8
SLIDE 8
slide-9
SLIDE 9

First challenge

  • Decrease update interval on driver’s device
  • Interval 15 secs
slide-10
SLIDE 10

First try

  • Make request -> save coordinates
  • Make request -> animate the car
slide-11
SLIDE 11

First problems

  • We can’t animate car properly
  • Car moves through fields, forests and quarters
slide-12
SLIDE 12

First problems

slide-13
SLIDE 13

Solution

  • OSRM
slide-14
SLIDE 14

Try again

  • Timeout – 15 secs
  • Make request -> save coordinates
  • Send coordinates
  • Building route via OSRM
  • Get route -> animate marker
slide-15
SLIDE 15
slide-16
SLIDE 16

Solution

  • Check for 20 meters
slide-17
SLIDE 17

Application Released!

slide-18
SLIDE 18

Needs improvement

  • Trip cost calculations on driver side
  • We need more tracks and 1 track at 15 seconds is few
  • GPS problems on driver
slide-19
SLIDE 19

GPS problems

  • Bad device/Bad module
  • GPS going to die during the time
  • Pits and “Jams”
slide-20
SLIDE 20

Tasks

  • Start to collect more tracks from drivers
  • Show animated cars on the main screen
  • Store intermediate trip cost on the server side
  • Save mobile data
  • Collect each track per one second
slide-21
SLIDE 21

What’s the track?

  • Latitude
  • Longitude
  • Session
  • OrderID
  • TripCost
slide-22
SLIDE 22

Trafic economy

  • 1 track = 100 bytes
slide-23
SLIDE 23

Tell me all the options, please

  • HTTP
  • WebSockets
  • TCP
  • UDP
slide-24
SLIDE 24

Tell me all the options, please

  • HTTP
  • WebSockets
  • TCP
  • UDP
slide-25
SLIDE 25

Why we choosed UDP?

  • We send only datagrams
  • We don’t need guarantees
  • Minimalism
  • Save lots of data
  • We have only 20 bytes overhead
  • Not blocked in our country
slide-26
SLIDE 26

What about data serialization?

  • JSON L
  • MsgPack L
  • Protocol Buffers J
slide-27
SLIDE 27

Data size

137 127 42 JSON MsgPack Protobuff

slide-28
SLIDE 28

Total

  • 42 bytes of payload
  • + 20 bytes of IP headers
  • = 62 bytes per track
  • = PROFIT!
slide-29
SLIDE 29

Data storage

slide-30
SLIDE 30

What data to store?

  • Driver’s session
  • Cab number
  • Order ID
  • Trip cost
  • Last location
  • N last locations
slide-31
SLIDE 31

Which storages do we use?

  • Percona
  • Redis
  • Elasticsearch
slide-32
SLIDE 32

We need geoindex

  • KD-tree
  • R-tree
slide-33
SLIDE 33

What requirements for geoindex

  • Search of N nearest points
  • Balanced tree
slide-34
SLIDE 34

KD-Tree

slide-35
SLIDE 35

KD-tree cons

  • Unbalanced tree
  • Can search only one nearest point
slide-36
SLIDE 36

R-tree

slide-37
SLIDE 37

Where can I get it?

  • https:/

/github.com/dhconnelly/rtreego

slide-38
SLIDE 38

What’s wrong with drivers?

  • Bad internet connectivity
  • Turned off the phone
  • Low battery
  • Removed application from RAM
  • And lots of other reasons
slide-39
SLIDE 39
  • We need expire mechanism
  • We need LRU data structure for storing coordinates
slide-40
SLIDE 40

Storage architecture

  • In-memory
  • R-tree
  • Map with drivers(key is the session)
  • Map with drivers(key is cab number)
slide-41
SLIDE 41

What algorithm on backend

  • Get data by UDP
  • Try to get driver from storage
  • If doesn’t exist – get driver from redis
  • Check and validate data
  • Set driver to storage
  • If doesn’t exist – initialize LRU
  • Update r-tree
slide-42
SLIDE 42

Go

  • Strong typed and compiled
  • Small size of docker containers
  • Few resource usage
  • Less own zoo J
slide-43
SLIDE 43

HTTP API

  • Return nearest drivers
  • Remove driver from storage(by cab number or

session)

  • Get information about trip
  • Get information about driver
slide-44
SLIDE 44

How to maintain

  • Logging into stdout/stderr
  • Metrics to Graphite
  • Checks to sensu
  • Usefull /status
  • Bots
slide-45
SLIDE 45

Usefull /status

  • Uptime since
  • HTTP Statuses counters
  • Total requests
slide-46
SLIDE 46

Bots

  • Emulate of driver’s workflow
  • Emulate of client’s workflow
  • Running near Marokko or Congo
slide-47
SLIDE 47

How it looks now?

  • We get client location from sensors
  • We get nearest drivers with routes
  • Animate each car
  • Update interval 15 seconds
slide-48
SLIDE 48

Main slide of first story

  • UDP+Protobuf for data savings
  • In-memory storage
  • R-tree for nearest drivers
  • LRU cache for storing last locations
  • OSRM for map matching and building routes
slide-49
SLIDE 49

Challenges in geocoding

slide-50
SLIDE 50

What challenges do we have?

  • Lack of data
  • Crossroads navigation
  • We can’t trust GPS
slide-51
SLIDE 51

Which map providers available?

  • Yandex
  • Google
  • 2GIS
slide-52
SLIDE 52

What we have to solve issue

  • OpenStreetMap data
  • Own database with addresses with coordinates
slide-53
SLIDE 53

Search format example

  • Chui ave 139
  • Chui / Manasa
  • Red centre
  • AUCA
  • 5-15
  • 7 marksa 181
slide-54
SLIDE 54

We need own geocoder

  • Available to search in different formats
  • Support for auto translated values
  • Support of synonyms
slide-55
SLIDE 55

Elasticsearch index

  • Addresses(alias)
  • OSM_timestamp
  • Drivers_data
slide-56
SLIDE 56

./ariadna update

  • Download file
  • Create index in Elasticserach
  • Populate with data from osm
  • Search intersections and populate data
  • Change aliases
  • Removes old index
slide-57
SLIDE 57

Features

  • Geocode
  • By the name
  • By crossroad
  • By the name of institution
  • Synonyms support
  • Reverse geocode
slide-58
SLIDE 58

Where to get it?

  • https:/

/github.com/maddevsio/ariadna

slide-59
SLIDE 59

Questions?

  • https:/

/github.com/maddevsio

  • https:/

/github.com/maddevsio/ariadna

  • @gen1us2k (twitter,github,facebook,skype,telegram)