Timelines @ Twitter QCon London 2012 Arya Asemanfar Thursday, - - PowerPoint PPT Presentation

▶
timelines twitter
SMART_READER_LITE
LIVE PREVIEW

Timelines @ Twitter QCon London 2012 Arya Asemanfar Thursday, - - PowerPoint PPT Presentation

Jun 12, 2023 757 likes •2.46k views

Timelines @ Twitter QCon London 2012 Arya Asemanfar Thursday, March 8, 2012 Poll-based Timeline Thursday, March 8, 2012 Poll-based Timeline Thursday, March 8, 2012 Search Thursday, March 8, 2012 Streaming (aka Push) Thursday, March 8,

slide-1
SLIDE 1

Timelines @ Twitter

QCon London 2012 Arya Asemanfar

Thursday, March 8, 2012
slide-2
SLIDE 2

Poll-based Timeline

Thursday, March 8, 2012
slide-3
SLIDE 3

Poll-based Timeline

Thursday, March 8, 2012
slide-4
SLIDE 4

Search

Thursday, March 8, 2012
slide-5
SLIDE 5

Streaming (aka Push)

Thursday, March 8, 2012
slide-6
SLIDE 6

Timeline Delivery

100M+ active users worldwide Poll-based

⇢200K QPS ⇢latency* 1ms median, 4ms @ p99

Search

⇢30K QPS

* to determine which tweets you should see

Thursday, March 8, 2012
slide-7
SLIDE 7

Timeline Delivery

250M New Tweets Per Day

⇢3K/sec - daily average ⇢5k/sec - daily peak ⇢10K+/sec - several large events in February

Thursday, March 8, 2012
slide-8
SLIDE 8

new tweet users’ home timelines

Thursday, March 8, 2012
slide-9
SLIDE 9

Timeline Delivery

26 billion deliveries every day

⇢18 million per minute

Latency to deliver to 1 million followers

⇢3.5 seconds average ⇢or 285K per second

Thursday, March 8, 2012
slide-10
SLIDE 10

Architecture

Thursday, March 8, 2012
slide-11
SLIDE 11 Thursday, March 8, 2012
slide-12
SLIDE 12

HTTP Proxy

Thursday, March 8, 2012
slide-13
SLIDE 13

HTTP Proxy tweet API

Thursday, March 8, 2012
slide-14
SLIDE 14

HTTP Proxy tweet API queue

Thursday, March 8, 2012
slide-15
SLIDE 15

HTTP Proxy tweet API tweet daemon queue

Thursday, March 8, 2012
slide-16
SLIDE 16

HTTP Proxy tweet API tweet daemon queue fanout

Thursday, March 8, 2012
slide-17
SLIDE 17

HTTP Proxy tweet API tweet daemon queue fanout delivery

Thursday, March 8, 2012
slide-18
SLIDE 18

fanout

Pipelined Delivery

Thursday, March 8, 2012
slide-19
SLIDE 19

fanout delivery 1st to 4000th followers

Pipelined Delivery

Thursday, March 8, 2012
slide-20
SLIDE 20

fanout delivery 1st to 4000th followers delivery 4001st to 8000th followers

Pipelined Delivery

Thursday, March 8, 2012
slide-21
SLIDE 21

fanout delivery 1st to 4000th followers delivery 4001st to 8000th followers delivery 8001st to 12000th followers

Pipelined Delivery

Thursday, March 8, 2012
slide-22
SLIDE 22

HTTP Proxy tweet API tweet daemon queue fanout delivery

Thursday, March 8, 2012
slide-23
SLIDE 23

HTTP Proxy tweet API tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-24
SLIDE 24

HTTP Proxy tweet API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-25
SLIDE 25

redis redis redis redis tweet ID RPUSHX to timeline ID user ID bitfield 8 bytes 8 bytes 4 bytes

Thursday, March 8, 2012
slide-26
SLIDE 26

tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield Redis list structure:

Thursday, March 8, 2012
slide-27
SLIDE 27

retweet ID tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield Redis list structure: retweet ID

Thursday, March 8, 2012
slide-28
SLIDE 28

retweet ID tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield tweet ID user ID bitfield Redis list structure: retweet ID retweet bit set

Thursday, March 8, 2012
slide-29
SLIDE 29

HTTP Proxy tweet API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-30
SLIDE 30

Poll-based Timelines

Thursday, March 8, 2012
slide-31
SLIDE 31

HTTP Proxy tweet API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-32
SLIDE 32

HTTP Proxy tweet API timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-33
SLIDE 33

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-34
SLIDE 34

three pieces of data

⇢timeline ⇢tweets ⇢users

Loading your home timeline

Thursday, March 8, 2012
slide-35
SLIDE 35

three pieces of data

⇢timeline ⇢tweets ⇢users

Loading your home timeline

Thursday, March 8, 2012
slide-36
SLIDE 36

three pieces of data

⇢timeline ⇢tweets ⇢users

Loading your home timeline

Thursday, March 8, 2012
slide-37
SLIDE 37

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-38
SLIDE 38

tweet service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-39
SLIDE 39

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-40
SLIDE 40

Hot Spots

Thursday, March 8, 2012
slide-41
SLIDE 41

users timelines

Thursday, March 8, 2012
slide-42
SLIDE 42

users timelines @diarioas timeline

Thursday, March 8, 2012
slide-43
SLIDE 43

users timelines

Thursday, March 8, 2012
slide-44
SLIDE 44

World

users timelines

Thursday, March 8, 2012
slide-45
SLIDE 45

timeline service large sustained QPS for a single timeline

Thursday, March 8, 2012
slide-46
SLIDE 46

timeline service timeline service timeline service timeline service

Thursday, March 8, 2012
slide-47
SLIDE 47

timeline service timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-48
SLIDE 48

timeline service timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-49
SLIDE 49

timeline service timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-50
SLIDE 50

timeline service timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

partitioning layer

Thursday, March 8, 2012
slide-51
SLIDE 51

timeline service redis redis redis redis timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

partitioning layer

Thursday, March 8, 2012
slide-52
SLIDE 52

timeline service redis redis redis redis timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-53
SLIDE 53

timeline service redis redis redis redis timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-54
SLIDE 54

timeline service redis redis redis redis timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-55
SLIDE 55

timeline service redis redis redis redis timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

Thursday, March 8, 2012
slide-56
SLIDE 56

timeline service redis redis redis redis timeline cache timeline cache timeline cache timeline cache timeline service timeline service timeline service

in-process cache

Thursday, March 8, 2012
slide-57
SLIDE 57

In-Process Cache

Thursday, March 8, 2012
slide-58
SLIDE 58

timeline service

Thursday, March 8, 2012
slide-59
SLIDE 59

timeline cache timeline service

Thursday, March 8, 2012
slide-60
SLIDE 60

timeline cache timeline service

Thursday, March 8, 2012
slide-61
SLIDE 61

timeline cache timeline service

Thursday, March 8, 2012
slide-62
SLIDE 62

timeline cache timeline service

detects high-frequency requests

Thursday, March 8, 2012
slide-63
SLIDE 63

timeline cache timeline service in-proc cache

detects high-frequency requests

Thursday, March 8, 2012
slide-64
SLIDE 64

Search

Thursday, March 8, 2012
slide-65
SLIDE 65 Thursday, March 8, 2012
slide-66
SLIDE 66

HTTP Proxy

Thursday, March 8, 2012
slide-67
SLIDE 67

HTTP Proxy tweet API

Thursday, March 8, 2012
slide-68
SLIDE 68

HTTP Proxy tweet API queue

Thursday, March 8, 2012
slide-69
SLIDE 69

HTTP Proxy tweet API queue ingester

Thursday, March 8, 2012
slide-70
SLIDE 70

HTTP Proxy tweet API queue search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-71
SLIDE 71

Search

Thursday, March 8, 2012
slide-72
SLIDE 72

HTTP Proxy tweet API queue search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-73
SLIDE 73

search API HTTP Proxy tweet API queue search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-74
SLIDE 74

search API search blender HTTP Proxy tweet API queue search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-75
SLIDE 75

search API search blender HTTP Proxy tweet API queue search index search index search index search index search index ingester user service

Thursday, March 8, 2012
slide-76
SLIDE 76

search index search index search index search index search index

Thursday, March 8, 2012
slide-77
SLIDE 77

search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index

Thursday, March 8, 2012
slide-78
SLIDE 78

search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index

New Tweet

Thursday, March 8, 2012
slide-79
SLIDE 79

search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search blender

Thursday, March 8, 2012
slide-80
SLIDE 80

search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search index search blender search blender search blender search blender

Thursday, March 8, 2012
slide-81
SLIDE 81

search index search index search index search index search blender search blender search blender search blender

Thursday, March 8, 2012
slide-82
SLIDE 82

search index search index search index search index search blender search blender search blender search blender

Thursday, March 8, 2012
slide-83
SLIDE 83

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-84
SLIDE 84

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Durable Stores

Thursday, March 8, 2012
slide-85
SLIDE 85

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Geo

Durable Stores

Thursday, March 8, 2012
slide-86
SLIDE 86

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Geo

Durable Stores

URL expansion

Thursday, March 8, 2012
slide-87
SLIDE 87

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Geo

Durable Stores

URL expansion

Thursday, March 8, 2012
slide-88
SLIDE 88

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Geo

Durable Stores

URL expansion

Thursday, March 8, 2012
slide-89
SLIDE 89

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Geo

Durable Stores

URL expansion

Thursday, March 8, 2012
slide-90
SLIDE 90

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Geo

Durable Stores

URL expansion

Thursday, March 8, 2012
slide-91
SLIDE 91

HTTP Proxy timeline API

Thursday, March 8, 2012
slide-92
SLIDE 92

Finagle - JVM RPC Library

  • pen source:

http://github.com/twitter/finagle

Thursday, March 8, 2012
slide-93
SLIDE 93

Finagle – RPC Common Library

Components

⇢connection management ⇢protocol codecs ⇢transient error handling ⇢distributed tracing ⇢service discovery ⇢observability

HTTP Proxy timeline API

Thursday, March 8, 2012
slide-94
SLIDE 94

Finagle – RPC Common Library

Components

⇢connection management ⇢protocol codecs ⇢transient error handling ⇢distributed tracing ⇢service discovery ⇢observability

HTTP Proxy timeline API

Thursday, March 8, 2012
slide-95
SLIDE 95

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-96
SLIDE 96

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-97
SLIDE 97

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-98
SLIDE 98

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-99
SLIDE 99

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-100
SLIDE 100

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-101
SLIDE 101

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-102
SLIDE 102

Building a client

ClientBuilder() .cluster(TimelineServiceCluster) .hostConnectionCoresize(5) .hostConnectionLimit(10) .hostConnectionIdleTime(5.seconds) .failureAccrualParams(5, 10.seconds) .retries(3) .timeout(500.milliseconds)

Thursday, March 8, 2012
slide-103
SLIDE 103

Finagle – RPC Common Library

Components

⇢connection management ⇢protocol codecs ⇢transient error handling ⇢distributed tracing ⇢service discovery ⇢observability

HTTP Proxy timeline API

Thursday, March 8, 2012
slide-104
SLIDE 104

Finagle – RPC Common Library

Components

⇢connection management ⇢protocol codecs ⇢transient error handling ⇢distributed tracing ⇢service discovery ⇢observability

HTTP Proxy timeline API

Thursday, March 8, 2012
slide-105
SLIDE 105

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-106
SLIDE 106

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache HTTP

Thursday, March 8, 2012
slide-107
SLIDE 107

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache HTTP Thrift

Thursday, March 8, 2012
slide-108
SLIDE 108

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache HTTP Thrift Thrift

Thursday, March 8, 2012
slide-109
SLIDE 109

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache HTTP Thrift Thrift Redis

Thursday, March 8, 2012
slide-110
SLIDE 110

Finagle – RPC Common Library

Components

⇢connection management ⇢protocol codecs ⇢transient error handling ⇢distributed tracing ⇢service discovery ⇢observability

HTTP Proxy timeline API

Thursday, March 8, 2012
slide-111
SLIDE 111

Looking Forward

Thursday, March 8, 2012
slide-112
SLIDE 112

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-113
SLIDE 113

tweet service user service HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-114
SLIDE 114

Case Study

Thursday, March 8, 2012
slide-115
SLIDE 115

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache

Thursday, March 8, 2012
slide-116
SLIDE 116

Create Tweet

Thursday, March 8, 2012
slide-117
SLIDE 117

Create Tweet Load Followers

Thursday, March 8, 2012
slide-118
SLIDE 118

Create Tweet Load Followers Write to Redis

Thursday, March 8, 2012
slide-119
SLIDE 119

Create Tweet Load Followers Write to Redis Sort

Thursday, March 8, 2012
slide-120
SLIDE 120

Create Tweet Load Followers Write to Redis API Sort

Thursday, March 8, 2012
slide-121
SLIDE 121

search API search blender HTTP Proxy tweet API queue search index search index search index search index search index ingester user service

Thursday, March 8, 2012
slide-122
SLIDE 122

Create Tweet

Thursday, March 8, 2012
slide-123
SLIDE 123

Create Tweet Tokenize

Thursday, March 8, 2012
slide-124
SLIDE 124

Create Tweet Tokenize Write to Lucene

Thursday, March 8, 2012
slide-125
SLIDE 125

Create Tweet Tokenize Write to Lucene Merge + Rank

Thursday, March 8, 2012
slide-126
SLIDE 126

Create Tweet Tokenize Write to Lucene Search Query Merge + Rank

Thursday, March 8, 2012
slide-127
SLIDE 127

Create Tweet Index Store Read

Aggregate

Thursday, March 8, 2012
slide-128
SLIDE 128

@joe tweets “hello world” @joe followed by @sam and @frank search index -> [“hello”, “world”] fanout index -> [@sam, @frank]

Thursday, March 8, 2012
slide-129
SLIDE 129

hello world

Redis Lucene

@sam @frank

@joe: hello world

Thursday, March 8, 2012
slide-130
SLIDE 130

Redis Lucene

@sam hello

1726371 1615234 1371241 1012357 1726371 1415044 1056831 98124

Thursday, March 8, 2012
slide-131
SLIDE 131

"QCon London" @sam's home timeline User Intent

Thursday, March 8, 2012
slide-132
SLIDE 132

"QCon London"

"QCon" AND "London"

@sam's home timeline User Intent Query Expansion

Thursday, March 8, 2012
slide-133
SLIDE 133

"QCon London"

"QCon" AND "London"

@sam's home timeline

"home_timeline:@sam"

User Intent Query Expansion

Thursday, March 8, 2012
slide-134
SLIDE 134

"QCon London"

"QCon" AND "London"

Search Index

@sam's home timeline

"home_timeline:@sam"

Redis

User Intent Query Expansion

Thursday, March 8, 2012
slide-135
SLIDE 135

"QCon London"

"QCon" AND "London"

Search Index

@sam's home timeline

"user_timeline:@joe" OR "user_timeline:@frank"

Redis

User Intent Query Expansion

union of timelines

Thursday, March 8, 2012
slide-136
SLIDE 136

Fan-out vs Fan-in

Thursday, March 8, 2012
slide-137
SLIDE 137

Delivery Type Read-time Cost Write-time Cost

Fan-out Fan-in ???

Thursday, March 8, 2012
slide-138
SLIDE 138

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Fan-in ???

Thursday, March 8, 2012
slide-139
SLIDE 139

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Low to High O(N) Fan-in ???

Thursday, March 8, 2012
slide-140
SLIDE 140

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Low to High O(N) Fan-in Low to High O(N) ???

Thursday, March 8, 2012
slide-141
SLIDE 141

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Low to High O(N) Fan-in Low to High O(N) Low O(1) ???

Thursday, March 8, 2012
slide-142
SLIDE 142

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Low to High O(N) Fan-in Low to High O(N) Low O(1) Hybrid

Thursday, March 8, 2012
slide-143
SLIDE 143

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Low to High O(N) Fan-in Low to High O(N) Low O(1) Hybrid Low

Thursday, March 8, 2012
slide-144
SLIDE 144

Delivery Type Read-time Cost Write-time Cost

Fan-out Low O(1) Low to High O(N) Fan-in Low to High O(N) Low O(1) Hybrid Low Low to Medium

Thursday, March 8, 2012
slide-145
SLIDE 145

@sam's home timeline

"home_timeline:@sam"

Redis

Thursday, March 8, 2012
slide-146
SLIDE 146

@sam's home timeline

"home_timeline:@sam" OR "user_timeline:@barackobama"

Redis

Thursday, March 8, 2012
slide-147
SLIDE 147

search API search blender tweet API queue search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-148
SLIDE 148

search API search blender tweet API queue search index search index search index search index search index ingester high-freq searches redis

Thursday, March 8, 2012
slide-149
SLIDE 149

"Super Bowl"

"Super" AND "Bowl"

Search Index

Thursday, March 8, 2012
slide-150
SLIDE 150

"Super Bowl"

"Super Bowl"

Redis

Thursday, March 8, 2012
slide-151
SLIDE 151

Redis vs Search Index

Thursday, March 8, 2012
slide-152
SLIDE 152

home_timeline:12512351 home_timeline:14653230 home_tiimeline:6014086 home_timeline:15123512

Redis Eviction Model

Thursday, March 8, 2012
slide-153
SLIDE 153

home_timeline:12512351 home_timeline:14653230 home_tiimeline:6014086 home_timeline:15123512

Redis Eviction Model

Thursday, March 8, 2012
slide-154
SLIDE 154

home_timeline:12512351 home_timeline:14653230 home_tiimeline:6014086 home_timeline:15123512

Redis Eviction Model

Thursday, March 8, 2012
slide-155
SLIDE 155

home_timeline:12512351 home_timeline:14653230 home_tiimeline:6014086 home_timeline:15123512

Redis Eviction Model

Thursday, March 8, 2012
slide-156
SLIDE 156

home_timeline:12512351 home_timeline:14653230 home_tiimeline:6014086 home_timeline:15123512

Redis Eviction Model

Thursday, March 8, 2012
slide-157
SLIDE 157

home_timeline:14653230 home_tiimeline:6014086 home_timeline:15123512

Redis Eviction Model

Thursday, March 8, 2012
slide-158
SLIDE 158

hello QCon London world

Search Index Eviction Model

Thursday, March 8, 2012
slide-159
SLIDE 159

hello QCon London world

Search Index Eviction Model

Thursday, March 8, 2012
slide-160
SLIDE 160

hello QCon London world

Search Index Eviction Model

Thursday, March 8, 2012
slide-161
SLIDE 161

hello QCon London world

Search Index Eviction Model

Thursday, March 8, 2012
slide-162
SLIDE 162

hello QCon London world

Search Index Eviction Model

Thursday, March 8, 2012
slide-163
SLIDE 163

Conclusion

Thursday, March 8, 2012
slide-164
SLIDE 164

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-165
SLIDE 165

HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search API search blender search index search index search index search index search index ingester

Thursday, March 8, 2012
slide-166
SLIDE 166

ingester search blender search API HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search index search index search index search index search index

Thursday, March 8, 2012
slide-167
SLIDE 167

ingester search blender search API HTTP Proxy tweet API timeline service timeline API redis redis redis redis tweet daemon queue fanout delivery timeline cache search index search index search index search index search index

Thursday, March 8, 2012
slide-168
SLIDE 168

Questions?

Arya Asemanfar @a_a

Thursday, March 8, 2012