SLIDE 1
timelines at scale
@raffi qcon sf 2012
timelines at scale @ra ffi qcon sf 2012 Pull Push Targeted - - PowerPoint PPT Presentation
timelines at scale @ra ffi qcon sf 2012 Pull Push Targeted - - PowerPoint PPT Presentation
timelines at scale @ra ffi qcon sf 2012 Pull Push Targeted twitter.com User / Site Streams home_timeline API Mobile Push (SMS, etc.) Queried Search API Track / Follow Streams the challenge > 150M world wide active users > 300K
SLIDE 2
SLIDE 3
SLIDE 4
Pull Push Targeted twitter.com
home_timeline API User / Site Streams Mobile Push (SMS, etc.)
Queried Search API
Track / Follow Streams
SLIDE 5
the challenge
⇢> 150M world wide active users ⇢> 300K QPS for timelines ⇢naïve timeline “materialization” can be slow
SLIDE 6
Timeline Service Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Redis
SLIDE 7
Timeline Service Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Redis Social Graph Service
SLIDE 8
Timeline Service Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Redis Social Graph Service
insert
⇢keyed off “recipient” ⇢pipelined 4k “destinations” at a time ⇢replicated
SLIDE 9
Timeline Service Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Redis
using redis
⇢native list structure
Tweet ID Bits User ID 8 bytes 4 bytes 8 bytes
SLIDE 10
Timeline Service Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Redis
using redis
⇢native list structure ⇢RPUSHX to
- nly add to
cached timelines
Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Bits User ID Tweet ID Tweet ID Tweet ID
SLIDE 11
Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Timeline Service Redis
SLIDE 12
Timeline Service Write API Fanout Redis Redis Timeline Cache Redis TweetyPie Gizmoduck
SLIDE 13
Pull Push Targeted twitter.com
home_timeline API User / Site Streams Mobile Push (SMS, etc.)
Queried Search API
Track / Follow Streams
SLIDE 14
Ingester Search Cache Redis Redis Earlybird Blender Push Compute HTTP Push Mobile Push Batch Compute Hadoop Write API Fanout Redis Redis Timeline Cache Timeline Service Redis
SLIDE 15
Push Compute HTTP Push Mobile Push Batch Compute Hadoop Search Index Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis Timeline Cache Redis Earlybird
blender
⇢queries one replica of all indexes ⇢merges & ranks results
SLIDE 16
Push Compute HTTP Push Mobile Push Batch Compute Hadoop Search Index Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis Timeline Cache Redis Earlybird
SLIDE 17
Write API Redis Redis Redis Write API Earlybird Earlybird Earlybird API Cache Read API Redis Redis Redis Read API Earlybird Earlybird Earlybird API Cache
⇢O(n) write ⇢O(1) write ⇢O(1) read ⇢O(n) read
SLIDE 18
the challenge (part #2)
⇢fanout can be really slow! ⇢...especially for high follower counts
SLIDE 19
SLIDE 20
@barackobama 23 million followers 31 million followers @katyperry 28 million followers @justinbieber 28 million followers @raffi 0.019 million followers @ladygaga
SLIDE 21
there are over 400 million tweets a day
SLIDE 22
a second 4600 tweets 0.2 ms a twee ≈
SLIDE 23
SLIDE 24
Write API Ingester Fanout Search Index Redis Earlybird Earlybird Redis Redis Redis Timeline Cache
search index ⇢[‘hello’,‘world’] fanout index ⇢[@danadanger, ...]
SLIDE 25
User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi
SLIDE 26
User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline user_timeline:nelson OR user_timeline:danadanger
SLIDE 27
User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi
SLIDE 28
User Intent Query Expansion “Hello, world” “Hello” AND “world” @raffi’s home timeline home_timeline:raffi OR user_timeline:taylorswift13
SLIDE 29
Batch Compute Hadoop Push Compute HTTP Push Search Index Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis Timeline Cache Redis Earlybird Mobile Push
SLIDE 30
Asynchronous Path Query Path Batch Compute Hadoop Synchronous Path Push Compute HTTP Push Search Index Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis Timeline Cache Redis Earlybird Mobile Push
SLIDE 31
Synchronous Path Query Path Batch Compute Hadoop Asynchronous Path Push Compute HTTP Push Search Index Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis Timeline Cache Redis Earlybird Mobile Push
SLIDE 32
Asynchronous Path Synchronous Path Batch Compute Hadoop Query Path Push Compute HTTP Push Search Index Blender Redis Timeline Service Ingester Earlybird Write API Fanout Redis Redis Timeline Cache Redis Earlybird Mobile Push
SLIDE 33
timeline query statistics
⇢>150m active users worldwide ⇢>300k qps poll-based timelines @ 1ms p50 / 4ms p99 ⇢>30k qps search-based timelines
SLIDE 34
tweet input
⇢~400m tweets per day ⇢~5K/sec daily average ⇢~7K/sec daily peak ⇢>12K/sec during large events
SLIDE 35
timeline delivery statistics
⇢30b deliveries / day (~21m / min) ⇢3.5 seconds @ p50 to deliver to 1m ⇢~300k deliveries / sec
SLIDE 36
SLIDE 37
SLIDE 38
SLIDE 39
SLIDE 40
SLIDE 41
SLIDE 42