From check-ins to recommendations Jon Hoffman @hoffrocket QCon NYC - - PowerPoint PPT Presentation

▶
from check ins to recommendations
SMART_READER_LITE
LIVE PREVIEW

From check-ins to recommendations Jon Hoffman @hoffrocket QCon NYC - - PowerPoint PPT Presentation

Dec 17, 2023 145 likes •564 views

From check-ins to recommendations Jon Hoffman @hoffrocket QCon NYC June 11, 2014 About Foursquare Scaling in two parts Part one: data storage Part two: application complexity Part 1: Data Storage 2009 Table splits DB.A Venues

slide-1
SLIDE 1

From check-ins to recommendations

Jon Hoffman @hoffrocket QCon NYC – June 11, 2014

slide-2
SLIDE 2

About Foursquare

slide-3
SLIDE 3

Scaling in two parts

  • Part one: data storage
  • Part two: application complexity
slide-4
SLIDE 4

Part 1: Data Storage

2009

slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7

Table splits

Venues Checkins Users Friends Users Friends Venues Checkins DB DB.A DB.B

slide-8
SLIDE 8

Replication

Master RW Slave RO Slave RO

slide-9
SLIDE 9

Outgrowing our hardware

  • Not enough RAM for indexes and

working data set

  • 100 writes/second/disk
slide-10
SLIDE 10

Sharding

  • Manage ourselves in application code on

top of postgres?

  • Use something called Cassandra?
  • Use something called HBase?
  • Use something called Mongo?
slide-11
SLIDE 11
slide-12
SLIDE 12

Besides Mongo

  • Memcache
  • Elastic search

– nearby venue search – user search

  • Custom data services

– Read only key value server – in memory cache with business logic

slide-13
SLIDE 13

HFile Service: Read only KV Store

MR HDFS hfile_0_b hfile_1_b hfile_0 hfile_1

Hadoop HFile Servers

Zookeeper:

  • data type to machine mapping
  • key hash to shard mapping

Application Servers hfile_1_a hfile_0_a

slide-14
SLIDE 14

Caching Services

Mongo Oplog Tailer Kafka Kafka Consumers Redis Application Servers Cache Servers getUserVenueCounts( 1: list<i64> userIds 2: list<ObjectId> venues)

slide-15
SLIDE 15

Part 2: application complexity

2009

slide-16
SLIDE 16

RPC Tracing

slide-17
SLIDE 17
slide-18
SLIDE 18

Throttles

slide-19
SLIDE 19

Remember the goats?

slide-20
SLIDE 20
slide-21
SLIDE 21

Monolithic problems

  • Compiling all the code, all the time
  • Deploying all the code all the time
  • Hard to isolate cause of performance

regressions and resource leaks

slide-22
SLIDE 22

SOA Infancy

  • Single codebase, Multiple builds

Web API Offline

slide-23
SLIDE 23

Finagle Era

  • Twitter’s scala based RPC library

service ¡Geocoder ¡{ ¡ ¡ ¡GeocodeResponse ¡geocode( ¡ ¡ ¡ ¡1: ¡GeocodeRequest ¡r ¡ ¡ ¡) ¡ } ¡

slide-24
SLIDE 24

Benefits

  • Independent compile targets
  • Fined grained control on releases and

bug fixes

  • Functional isolation
slide-25
SLIDE 25
slide-26
SLIDE 26

Problems

  • Duplication in packaging and

deployment efforts

  • Hard to trace execution problems
  • Hard to define/change where things live
  • Networks aren’t reliable
slide-27
SLIDE 27

Builds and deploys

  • single service definition file
  • consistent build packaging
  • simple deployment of canary & fleet

./service_releaser ¡–j ¡service_name ¡

¡

slide-28
SLIDE 28

Monitoring

  • healthcheck endpoint over http
  • consistent metric names
  • dashboard for every service
slide-29
SLIDE 29

Distributed Tracing

slide-30
SLIDE 30

Exception Aggregation

slide-31
SLIDE 31

Application Discovery

  • Finagle Server Sets + ZK
slide-32
SLIDE 32

Circuit Breaking

  • Fast failing RPC calls after some error

rate threshold

  • Loosely based on Netflix’s hystrix
slide-33
SLIDE 33

SOA Problem Recap

  • Duplication in packaging and deployment efforts

– Build and deploy automation

  • Hard to trace execution problems

– Monitoring consistency – Distributed Tracing – Error aggregation

  • Hard to define/change where things live

– Application discovery with zookeeper

  • Networks aren’t reliable

– Circuit breaking

slide-34
SLIDE 34

Organization

  • Smaller teams owning front to back

implementation of features

  • Desire to have quick deploy cycles on

new API endpoints

slide-35
SLIDE 35

Remote Endpoints

Wouldn’t it be cool if a developer could expose a new API endpoint without redeploying our still monolithic API server?

slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

Remote Endpoint Benefits

  • Very easy to experiment with new

endpoints

  • Tight contract for service interaction

– JSON responses – all http params passed along

  • Clear path to breaking off more chunks

from API monolith

slide-39
SLIDE 39

Future work: Part 3?

  • Further isolating services with

independent storage layers?

  • Completely automated continuous

deployment

  • Hybrid immutable/mutable data storage

– mongo & hfile & cache service

slide-40
SLIDE 40

Thanks!

  • Want to build these things?

https://foursquare.com/jobs

  • jon@foursquare.com