SLIDE 1
LESSONS & PITFALLS
DATA AT SWEDEN'S TELEVISION
Ismail Elouafiq
DATA AT SWEDEN'S TELEVISION Ismail Elouafiq A wide spectrum of Apps - - PowerPoint PPT Presentation
DATA AT SWEDEN'S TELEVISION Ismail Elouafiq A wide spectrum of Apps - - PowerPoint PPT Presentation
LESSONS & PITFALLS DATA AT SWEDEN'S TELEVISION Ismail Elouafiq A wide spectrum of Apps A wide spectrum of Apps Running on different platforms A wide spectrum of Users STRATEGY ANALYSTS PRODUCT OWNERS A wide spectrum of Users STRATEGY
SLIDE 2
SLIDE 3
SLIDE 4
SLIDE 5
SLIDE 6
SLIDE 7
SLIDE 8
SLIDE 9
A wide spectrum of Apps
SLIDE 10
A wide spectrum of Apps
Running on different platforms
SLIDE 11
PRODUCT OWNERS STRATEGY
ANALYSTS
A wide spectrum of Users
SLIDE 12
PRODUCT OWNERS STRATEGY
ANALYSTS
AUTHORS/ EDITORS DEVELOPERS
A wide spectrum of Users
SLIDE 13
tl;dr:
Defining what to prioritise
SLIDE 14
tl;dr:
Defining what to prioritise
“Data: I could be chasing an untamed
- rnithoid without cause.”
― Star Trek The Next Generation
SLIDE 15
tl;dr:
Defining what to prioritise
“Data: I could be chasing an untamed
- rnithoid without cause.”
― Star Trek The Next Generation
SLIDE 16
tl;dr:
Spoilers: how and why we now use protobuf, functional data engineering and ETL practices Experimenting and iterating in small increments Defining what to prioritise
SLIDE 17
AI Deep reinforcement learning
tl;dr:
Spoilers: how and why we now use protobuf, functional data engineering and ETL practices Experimenting and iterating in small increments Defining what to prioritise
BLOCKCHAIN
SLIDE 18
tl;dr:
Experimenting and iterating in small increments Defining what to prioritise
SLIDE 19
ismail.land/velocity
tl;dr:
Experimenting and iterating in small increments Defining what to prioritise
SLIDE 20
tl;dr:
Experimenting and iterating in small increments Defining what to prioritise
ismail.land/velocity
SLIDE 21
What events should you collect?
SLIDE 22
What events should you collect?
SLIDE 23
what we want to know
How many people read the article per day
SLIDE 24
what we want to know
click scroll share
How many people read the article per day
what we can
- bserve
SLIDE 25
what we want to know
click scroll share
How many people read the article per day
what we can
- bserve
events
SLIDE 26
what we want to know
click scroll share
How many people read the article per day
what we can
- bserve
explicit model events
SLIDE 27
what we want to know
How many people read the article per day
click scroll share
what we can
- bserve
explicit model events
SLIDE 28
SLIDE 29
let's start with views
SLIDE 30
If you could do anything with data... What would you actually use for decision making
SLIDE 31
If you could do anything with data... What would you actually use for decision making
A/B tests... Hell yeah!
SLIDE 32
tl;dr:
Experimenting and iterating in small increments Defining what to prioritise
ismail.land/velocity
SLIDE 33
1
COLLECT
2
INGEST
SDK
First we need to collect data
SLIDE 34
1
COLLECT
2
INGEST
SDK
events
Event API
SLIDE 35
1
COLLECT
2
INGEST
SDK
events
Event API
publish
SLIDE 36
1
COLLECT
2
INGEST
SDK
events
Event API
publish
pub/sub
SLIDE 37
2
INGEST pub/sub
SLIDE 38
2
INGEST pub/sub
2
STORE
SLIDE 39
2
INGEST pub/sub
2
STORE Events table
SLIDE 40
2
INGEST pub/sub
2
STORE Events table
subscribe
judge-judi
write
SLIDE 41
2
INGEST pub/sub
2
STORE Events table
subscribe
judge-judi
write
SLIDE 42
3
STORE
2
INGEST
1
COLLECT
SLIDE 43
3
STORE
2
INGEST
1
COLLECT
{event_type: click} { eventType: click} {eventType: klick}
SLIDE 44
3
STORE
2
INGEST
1
COLLECT
{event_type: click} { eventType: click} {eventType: klick}
SLIDE 45
3
STORE
2
INGEST
1
COLLECT
More Issues
Multiple teams/platforms =>takes time to update the clients The schema is sent with every event Unclear types (arbitrary memory allocation)
SLIDE 46
3
STORE
2
INGEST
1
COLLECT
More Issues
Multiple teams/platforms =>takes time to update the clients The schema is sent with every event Unclear types (arbitrary memory allocation) We know the schema on all levels we have a common model for the data.. how can we make use of that...
SLIDE 47
ENTER PROTOBUF
Keepign a centralized Event Schema
SLIDE 48
ENTER PROTOBUF
Keepign a centralized Event Schema
person.proto
SLIDE 49
ENTER PROTOBUF
Keepign a centralized Event Schema
person.proto person.go person.js person.f compiler
SLIDE 50
ENTER PROTOBUF
Keepign a centralized Event Schema
person.js Person Client Person Server person.js binary serialize deserialize
SLIDE 51
1 - Define the Schema As a .proto file
ENTER PROTOBUF
Keepign a centralized Event Schema
event.proto
SLIDE 52
1 - Define the Schema As a .proto file
ENTER PROTOBUF
Keepign a centralized Event Schema
event.proto
2 - Publish libraries Publish using CI pipeline
go, js, java, swift
SLIDE 53
1 - Define the Schema As a .proto file
ENTER PROTOBUF
Keepign a centralized Event Schema
event.proto
2 - Publish libraries Publish using CI pipeline 2 - Fetch Fetch in SDKs (serialization) Fetch in Judy (deserialization) Use to generate table
go, js, java, swift
SLIDE 54
3
STORE
2
INGEST
1
COLLECT DEFINE
SLIDE 55
My work here is done!
SLIDE 56
Not really... Backward and forward compatibility Table changes Language agnostic but nor really Lack of support
SLIDE 57
The Data Pyramid
Collection and ingestion Storage, transformation, monitoring
SLIDE 58
The Data Pyramid
Collection and ingestion Storage, transformation, monitoring
SLIDE 59
The Data Pyramid
Metrics, aggregations, KPIs Collection and ingestion Learn, Optimise, Experiment Storage, transformation, monitoring
SLIDE 60
The Data Pyramid
Nirvana AI, machine learning Metrics, aggregations, KPIs Collection and ingestion Learn, Optimise, Experiment Storage, transformation, monitoring
SLIDE 61
The Data Pyramid
"The pyramids of Egypt could be explained as symbolic stairways to the stars, according to a British scientist" _ The Guardian
SLIDE 62
The Data Pyramid
"The pyramids of Egypt could be explained as symbolic stairways to the stars, according to a British scientist" _ The Guardian "The data pyramid could be explained as a symbolic stairway to the A.I., according to myself" _ Me
SLIDE 63
Endorse me on Linkedin
SLIDE 64
3
STORE
2
INGEST
1
COLLECT DEFINE
We have the data
Now what?
SLIDE 65
3
STORE
2
INGEST
1
COLLECT DEFINE
We have the data
Now what?
5 4
Batch jobs etl Streaming
Analyze
Service/API Dashboard Reports
Present
SLIDE 66
3
STORE
2
INGEST
1
COLLECT DEFINE
We have the data
Now what?
5 4
Batch jobs etl Streaming
Analyze
Service/API Dashboard Reports
Present
SLIDE 67
Everybody ETLs
SLIDE 68
Everybody ETLs
SLIDE 69
Some data to be aggregated
Inputs
Our mysterious job pipeline
Aggregated Table (article reads) Per DAY
Output
article reads per day
click events article titles
SLIDE 70
today- partition magic job Append
SLIDE 71
today- partition Failed magic job
SLIDE 72
today- partition magic job Append
Immutable data partitions Versioned logic
Principle: Ensuring reproducibility
SLIDE 73
On ETL design
Ensure reproducibility Practice failure in small increments Defining conventions in one place
SLIDE 74
ISMAIL.LAND/VELOCITY
keeping a tidy pipeline
SLIDE 75
Br3Ak 'em rULeS
¯\_(ツ)_/¯
SLIDE 76
summary...
SLIDE 77
summary...
SLIDE 78
summary...
SLIDE 79
summary... (what worked for us)
SLIDE 80
summary...
DATA DATA DATA
(what worked for us)
SLIDE 81
ismail.land/velocity