SLIDE 1
Extreme Programming meets Real Time Data
Gel Goldsby & Tom Johnson, Unruly
Title position here
Your title sit over it
Extreme Programming meets Real Time Data Gel Goldsby & Tom - - PowerPoint PPT Presentation
Extreme Programming meets Real Time Data Gel Goldsby & Tom - - PowerPoint PPT Presentation
Extreme Programming meets Real Time Data Gel Goldsby & Tom Johnson, Unruly When Santa Got Stuck Up The Chimney Title position here Your title sit over it When Data Got Stuck Up The Chimney Title position here Your title sit over it
SLIDE 2
SLIDE 3
Title position here
Your title sit over it
When Data Got Stuck Up The Chimney
SLIDE 4
Title position here
Your title sit over it
- Hello. My name is...
Gel Goldsby Reporting and Data Team Lead Tom Johnson Senior Developer
SLIDE 5
Title position here
Your title sit over it
We Believe In XP
SLIDE 6
Title position here
Your title sit over it
Extreme Programming Values
- Communication
- Simplicity
- Feedback
- Courage
SLIDE 7
Title position here
Your title sit over it
Simplicity
SLIDE 8
Title position here
Your title sit over it
Simplicity
SLIDE 9
Title position here
Your title sit over it
Simplicity
SLIDE 10
Title position here
Your title sit over it
Simplicity
SLIDE 11
Title position here
Your title sit over it
Simplicity
SLIDE 12
Title position here
Your title sit over it
Simplicity
SLIDE 13
Title position here
Your title sit over it
Our Reporting Pipeline
pipeline events
SLIDE 14
Title position here
Your title sit over it
Our Reporting Pipelines
super duper wizzy pipeline events
- ld pipeline
SLIDE 15
Title position here
Your title sit over it
Shut It Off!
super duper wizzy pipeline events
- ld pipeline
SLIDE 16
Title position here
Your title sit over it
A Closer Look At Our Pipeline
pipeline events events consumer
SLIDE 17
Title position here
Your title sit over it
It’s Not A Truck, It’s A Series of Tubes
events sequencer parser consumer nginx
SLIDE 18
Title position here
Your title sit over it
Queueing with S3
events consumer S3 S3 S3 parser sequencer nginx
SLIDE 19
Title position here
Your title sit over it
Queueing with S3
events consumer S3 S3 S3 S3 S3 S3 parser sequencer nginx
SLIDE 20
Title position here
Your title sit over it
We Need More Power, Cap’n
events sequencer parser consumer nginx
SLIDE 21
Title position here
Your title sit over it
nginx parser
We Need More Power, Cap’n
events sequencer parser consumer nginx
SLIDE 22
Title position here
Your title sit over it
nginx nginx parser parser
We Need More Power, Cap’n
events sequencer parser consumer nginx
SLIDE 23
Title position here
Your title sit over it
nginx nginx nginx parser parser parser
We Need More Power, Cap’n
events sequencer parser consumer nginx
SLIDE 24
Title position here
Your title sit over it
Two Writes Can Make A Wrong
events sequencer parser consumer nginx
SLIDE 25
Title position here
Your title sit over it
Two Writes Can Make A Wrong
events sequencer parser consumer nginx
SLIDE 26
Title position here
Your title sit over it
Christmas was saved!
SLIDE 27
Title position here
Your title sit over it
Simplicity
- Each component does one thing
and does it well
SLIDE 28
Title position here
Your title sit over it
Just Another Report, Right?
- Improving targeting
- Correlate events for same ad call
- Need to join on session id
- Needs disaggregated data
SLIDE 29
Title position here
Your title sit over it
Aggregation
Campaign Site Acme Zombo.com Acme Zombo.com Acme Zombo.com Acme Nyan.cat Brawndo Zombo.com Brawndo Nyan.cat Brawndo Nyan.cat
SLIDE 30
Title position here
Your title sit over it
Aggregation
Campaign Site Acme Zombo.com Acme Zombo.com Acme Zombo.com Acme Nyan.cat Brawndo Zombo.com Brawndo Nyan.cat Brawndo Nyan.cat
SLIDE 31
Title position here
Your title sit over it
Aggregation
Campaign Site Acme Zombo.com Acme Zombo.com Acme Zombo.com Acme Nyan.cat Brawndo Zombo.com Brawndo Nyan.cat Brawndo Nyan.cat
SLIDE 32
Title position here
Your title sit over it
Aggregation
Count Campaign Site 1 Acme Zombo.com 1 Acme Zombo.com 1 Acme Zombo.com 1 Acme Nyan.cat 1 Brawndo Zombo.com 1 Brawndo Nyan.cat 1 Brawndo Nyan.cat
SLIDE 33
Title position here
Your title sit over it
Aggregation
Count Campaign Site 3 Acme Zombo.com 1 Acme Nyan.cat 1 Brawndo Zombo.com 2 Brawndo Nyan.cat
SLIDE 34
Title position here
Your title sit over it
Aggregation
Count Campaign Site Lots More 3 Acme Zombo.com ... ... 1 Acme Nyan.cat ... ... 1 Brawndo Zombo.com ... … 2 Brawndo Nyan.cat ... ...
SLIDE 35
Title position here
Your title sit over it
Lots of buckets
SLIDE 36
Title position here
Your title sit over it
Micro-Aggregations
- Roughly 20k events per second
- Batched: window size 20s
- x7 reduction factor
- Reduces writes to db
SLIDE 37
Title position here
Your title sit over it
Make America Aggregate Again
- Daily
- From ~800 million events
- Compacts to ~2 million rows
- 400x reduction
- Reduces disk usage
- Speeds up queries
SLIDE 38
Title position here
Your title sit over it
Querying data
view historic data today’s data user query
SLIDE 39
Title position here
Your title sit over it
Aggregatable facts
Campaign Site Acme Zombo.com Acme Zombo.com Acme Zombo.com Acme Nyan.cat Brawndo Zombo.com Brawndo Nyan.cat Brawndo Nyan.cat
SLIDE 40
Title position here
Your title sit over it
Add in session ids
Campaign Site Session Id Acme Zombo.com Wo5Meiri Acme Zombo.com Xotaipu6 Acme Zombo.com Xu1goor7 Acme Nyan.cat eVai6OhS Brawndo Zombo.com
- iMoo7Du
Brawndo Nyan.cat aiSh1eej Brawndo Nyan.cat rae8ieY5
SLIDE 41
Title position here
Your title sit over it
Does not aggregate well
Campaign Site Session Id Acme Zombo.com Wo5Meiri Acme Zombo.com Xotaipu6 Acme Zombo.com Xu1goor7 Acme Nyan.cat eVai6OhS Brawndo Zombo.com
- iMoo7Du
Brawndo Nyan.cat aiSh1eej Brawndo Nyan.cat rae8ieY5
SLIDE 42
Title position here
Your title sit over it
What next?
SLIDE 43
Title position here
Your title sit over it
What next? Spikes!
SLIDE 44
Title position here
Your title sit over it
Big Data!
SLIDE 45
Title position here
Your title sit over it
Big data: big choices
- Many options
- Available documentation was:
○ Academic ○ Evangelical ○ Naive/Trivial
SLIDE 46
Title position here
Your title sit over it
Spark!
SLIDE 47
Title position here
Your title sit over it
Big data: big costs
- Infrastructure
- Language (Scala)
- Incompatible with current approach
- Performance tradeoffs
SLIDE 48
Title position here
Your title sit over it
Why we could step away
- Understood our data better
- Underestimated costs
- We know our code
- We can change our code
SLIDE 49
Title position here
Your title sit over it
Feedback
- Regular retrospectives
- Shared understanding of “research”
- Shared understanding of value
SLIDE 50
Title position here
Your title sit over it
Courage
- Not afraid to try new things
- Not afraid to change direction
- Not lured by what we “ought” to do
SLIDE 51
Title position here
Your title sit over it
The Shape of our Data
SLIDE 52
Title position here
Your title sit over it
The Shape of our Data Disaggregated
SLIDE 53
Title position here
Your title sit over it
The Shape of our Data Disaggregated Unsampled
SLIDE 54
Title position here
Your title sit over it
The Shape of our Data Disaggregated Unsampled Real Time
SLIDE 55
Title position here
Your title sit over it
Programmatic Pacing Disaggregated Unsampled Real Time
SLIDE 56
Title position here
Your title sit over it
Operational Debugging Disaggregated Unsampled Real Time
SLIDE 57
Title position here
Your title sit over it
Auction Data Disaggregated Unsampled Real Time
SLIDE 58
Title position here
Your title sit over it
Advertising 101
user loads page payments auction ad call user interaction
SLIDE 59
Title position here
Your title sit over it
Funnel of data
user loads page payments auction ad call user interaction
SLIDE 60
Title position here
Your title sit over it
Pipelines to match data shape
user loads page payments auction ad call user interaction
SLIDE 61
Title position here
Your title sit over it
Our Actual Reporting Pipelines
payments pipeline events ad call pipeline user interaction pipeline auction pipeline
SLIDE 62
Title position here
Your title sit over it
When We Get Overloaded...
payments pipeline events ad call pipeline user interaction pipeline auction pipeline
SLIDE 63
Title position here
Your title sit over it
When We Get Overloaded...
payments pipeline events ad call pipeline user interaction pipeline auction pipeline
SLIDE 64
Title position here
Your title sit over it
When We Get Overloaded...
payments pipeline events ad call pipeline user interaction pipeline auction pipeline
SLIDE 65
Title position here
Your title sit over it
Ensuring real time performance
SLIDE 66
Title position here
Your title sit over it
Ensuring real time performance
SLIDE 67
Title position here
Your title sit over it
Communication
- How data was used
- Performance requirements
○ What was needed ○ What wasn’t needed ○ Hard vs soft requirements
SLIDE 68
Title position here
Your title sit over it
Simplicity
- Green cards
- 10 pair-days total
- Incremental
- Separable
SLIDE 69
Title position here
Your title sit over it
Let's talk about our databases
SLIDE 70
Title position here
Your title sit over it
Row-based database
Column A Column B Column C Column D Column E
SLIDE 71
Title position here
Your title sit over it
Row-based database
Column A Column B Column C Column D Column E
SLIDE 72
Title position here
Your title sit over it
Columnar database
Column A Column B Column C Column D Column E
SLIDE 73
Title position here
Your title sit over it
Row-based database
Column A Column B Column C Column D Column E
SLIDE 74
Title position here
Your title sit over it
Columnar database
Column A Column B Column C Column D Column E
SLIDE 75
Title position here
Your title sit over it
Vectorwise or Postgres?
SLIDE 76
Title position here
Your title sit over it
Query-based routing
api user query Vectorwise Postgres
SLIDE 77
Title position here
Your title sit over it
Query-based routing
api user query Vectorwise Postgres
SLIDE 78
Title position here
Your title sit over it
Query-based routing
api user query Vectorwise Postgres
SLIDE 79
Title position here
Your title sit over it
Conclusion
SLIDE 80
Title position here
Your title sit over it
Conclusion
- Simplicity
- Communication
- Feedback
- Courage
SLIDE 81
Title position here
Your title sit over it
Thank you!
SLIDE 82
Title position here
Your title sit over it
Questions?
(this space intentionally left blank)