SLIDE 1
From batch to streaming to both Herman Schaaf, Senior Software - - PowerPoint PPT Presentation
From batch to streaming to both Herman Schaaf, Senior Software - - PowerPoint PPT Presentation
From batch to streaming to both Herman Schaaf, Senior Software Engineer A Story About me Herman Schaaf, Senior Software Engineer Data Platform Tribe The Cube From batch to streaming The Single Unified Log The Single Unified Log
SLIDE 2
SLIDE 3
About me
Herman Schaaf, Senior Software Engineer Data Platform Tribe
SLIDE 4
“The Cube”
SLIDE 5
From batch to streaming
SLIDE 6
The Single Unified Log
SLIDE 7
The Single Unified Log
SLIDE 8
“Organizations which design data platforms are constrained to produce designs which are copies of their communication structures” Lesson 1: Conway’s Law is true for data platforms
SLIDE 9
…but then metadata is critical Being self-serve is good
SLIDE 10
So let’s talk about metadata
SLIDE 11
A simple convention
prod.identity-service.AuditLog.identity.AuditMessage prod.flyingcircus.applog.applog.Message prod.raccoon_bandit.experiment.bandit.Metric
SLIDE 12
Descriptive Structural Administrative
we had some of this Some, from using protobuf schemas nope.
SLIDE 13
Lesson 2: Metadata is Critical
- Especially relationships
- Ideally automated
- Ideally from the start
- Tools like Schema Registry are a
start, but not the full solution
SLIDE 14
SLIDE 15
SLIDE 16
Lesson 3: Data Engineers Control the Plot Line
SLIDE 17
business events
SLIDE 18
From streaming to both
SLIDE 19
SLIDE 20
Lesson 4: Repeatability is important
- Streams have to choose between
replays and accepting errors as permanent
- Batch processing can be done again
any time
- Going straight to the archive in small
batches gets the benefits of both.
SLIDE 21
Key Takeaways
- Conway’s Law is true for data
platforms
- Metadata is Critical
- Data Engineers Control the Plot Line
- Repeatability is important
SLIDE 22
Contact If you have any questions regarding Skyscanner please contact: Herman Schaaf herman.schaaf@skyscanner.net