SLIDE 1
The Architecture of Wemlin Hub Ognen Ivanovski, Netcetera Jug.ch - - PowerPoint PPT Presentation
The Architecture of Wemlin Hub Ognen Ivanovski, Netcetera Jug.ch - - PowerPoint PPT Presentation
The Architecture of Wemlin Hub Ognen Ivanovski, Netcetera Jug.ch 15 Wemlin Data Data Planning Software Planning Software AVCS AVCS Wemlin Hub g Wemlin Hub Plan Data transport schedule (plan) over certain time package in nature
SLIDE 2
SLIDE 3
SLIDE 4
SLIDE 5
Data
SLIDE 6
Data
Planning Software
SLIDE 7
Planning Software
SLIDE 8
AVCS
SLIDE 9
AVCS
SLIDE 10
Wemlin Hub
SLIDE 11
g
Wemlin Hub
SLIDE 12
Plan Data
transport schedule (plan) over certain time package in nature Formats: HAFAS, GTFS, VDV 453 REF, VDV 454 REF, REST
SLIDE 13
RT Streams
updates on planned data stream in nature Formats: GTFS RT, VDV-453, VDV-454, REST
SLIDE 14
SLIDE 15
Getting Data
different protocols & implementations remote system availability issues
SLIDE 16
Figuring things out
different sources, no referential guarantees every data source is different data quality issues Example: station refeferences Example: trip references
SLIDE 17
Data Enrichment
line colors location station metadata (e.g. which lines stop there, which stations are near by)
SLIDE 18
Serve many users
SLIDE 19
Key Concerns
RT
low latency throughput available excllent failure management
Batch
throughput
SLIDE 20
Key Concerns
processing programs the same for both batch and RT data
SLIDE 21
Key Concerns
fast reaction to load changes
SLIDE 22
Architecture
Constraints decisions hard to change
SLIDE 23
SLIDE 24
Microkernel Architectural Style
SLIDE 25
tap filter transform aggregate sink
SLIDE 26
tap sink
SLIDE 27
filter
public interface Filter { boolean accept(Object obj); }
stateless
SLIDE 28
transform
public interface Transformer { Object transform(Object original); }
stateless
SLIDE 29
aggregate
public interface Aggregator { Optional<?> aggregate(Object obj); }
SLIDE 30
Composition
pipeElement = PipelineBuilder.from(gtfsInputJunction()) .transform(new StoppingPlaceResolver()) .filter(new InvalidStopsFilter()) .transform(new UnresolvedLineVehicleTypeAdder()) .transform(new UnresolvedLineNameAdjuster()) .transform(new LineResolver()) .transform( new LineColorsEnricher(...)) .aggregate(new CacheAggregator(cache())) .to(nullSink());
SLIDE 31
Model
immutable: model objects are values (changing means a new object) algebraic: each object identity is defined by it's contents. pure: in the sense of no external dependencies
SLIDE 32
Schedule Trip Stop Station Line
Projection
Stopping Place
SLIDE 33
Architecture
Model
Pure Java Immutable Algebraic Inverse References
Pipeline
Functional Microkernel Filter (stateless, pure function) Transformer (stateless, pure function) Aggregator (stateful, function) Sink (consumer) / Tap (producer)
SLIDE 34
pipeElement = PipelineBuilder.from(gtfsInputJunction()) .transform(new StoppingPlaceResolver()) .filter(new InvalidStopsFilter()) .transform(new UnresolvedLineVehicleTypeAdder()) .transform(new UnresolvedLineNameAdjuster()) .transform(new LineResolver()) .transform( new LineColorsEnricher(...)) .aggregate(new CacheAggregator(cache())) .to(nullSink());
SLIDE 35
SLIDE 36
pipeline “compile” assembly
SLIDE 37
SLIDE 38
Assemblies
Spring Integration based Thread-pool based Fork-join based
SLIDE 39
Segments
parallel / serial separated by queues segmentation based on the used interfaces
SLIDE 40
Fork-Join
to be effective, one must batch batching introduces latency (in RT) trick batch mode RT mode
SLIDE 41
Memoization at construction time
Problem: immutable objects —> lots of created objects @DesignatedFactoryMethod AspectJ runtime weaver
SLIDE 42
hornet backends frontend
SLIDE 43
Scaling
SLIDE 44
Storage
In-memory custom store replacable just an Aggregator makes blue-green deployments a beeze
SLIDE 45
Wins
Constraint
immutable model algebraic model Inverse references controlled state (functional) microkernel style
Gain
execution stragety freedom parallelism scalability memoization at constructor time cacheability composability
SLIDE 46
Wins
Constraint
immutable model algebraic model Inverse references controlled state (functional) microkernel style
Gain
execution stragety freedom parallelism scalability memoization at constructor time cacheability composability
SLIDE 47
Wins
Constraint
immutable model algebraic model Inverse references controlled state (functional) microkernel style
Gain
execution stragety freedom parallelism scalability memoization at constructor time cacheability composability
SLIDE 48
Wins
Constraint
immutable model algebraic model Inverse references controlled state (functional) microkernel style
Gain
execution stragety freedom parallelism scalability memoization at constructor time cacheability composability
SLIDE 49
Wins
Constraint
immutable model algebraic model Inverse references controlled state (functional) microkernel style
Gain
execution stragety freedom parallelism scalability memoization at constructor time cacheability composability
SLIDE 50
Wins
Constraint
immutable model algebraic model Inverse references controlled state (functional) microkernel style
Gain
execution stragety freedom parallelism scalability memoization at constructor time cacheability composability
SLIDE 51
Architecture is Important
It’s the set of constraints you choose for your system It how those constraints work in concert It happens on a smaller scale than you usually think
SLIDE 52
Acknowledgments
Clojure (especially Rich Hickey’s talk on values, state and identity) Lamina https://github.com/ztellman/lamina Apache Storm https://storm.incubator.apache.org Casading http://www.cascading.org Akka http://akka.io
SLIDE 53
Thank You
- gnen.ivanovski@netcetera.com