Evolving a data system Patterns & Antipatterns Simon Metson - - PowerPoint PPT Presentation

▶
evolving a data system
SMART_READER_LITE
LIVE PREVIEW

Evolving a data system Patterns & Antipatterns Simon Metson - - PowerPoint PPT Presentation

Nov 30, 2022 10 likes •395 views

Evolving a data system Patterns & Antipatterns Simon Metson Any views or opinions expressed in this presentation are those of the author, and do not necessarily represent official positions, strategies or opinions of International

slide-1
SLIDE 1

Evolving a data system

Patterns & Antipatterns Simon Metson

slide-2
SLIDE 2

Any views or opinions expressed in this presentation are those of the author, and do not necessarily represent official positions, strategies or opinions of International Business Machines (IBM) Corporation. No guarantees are offered as to the timeliness, accuracy or validity of information presented.

slide-3
SLIDE 3

Evolving a data system

Patterns & Antipatterns Simon Metson

slide-4
SLIDE 4

Outline

  • Terms
  • Patterns, anti-patterns & tricks
  • Process
slide-5
SLIDE 5

Evolution

slide-6
SLIDE 6
slide-7
SLIDE 7

Data systems

slide-8
SLIDE 8

???

slide-9
SLIDE 9

DBMS

slide-10
SLIDE 10

DBMS

Web Application

slide-11
SLIDE 11

DBMS

Web Application Web Client

slide-12
SLIDE 12

DBMS

Web Application Web Client Cache

slide-13
SLIDE 13

DBMS #2

ETL system

DBMS

Web Application Web Client Cache

slide-14
SLIDE 14

DBMS #2

ETL system Custom reports

DBMS

Web Application Web Client Cache

slide-15
SLIDE 15

DBMS #2

ETL system Custom reports

DBMS

Web Application Web Client Cache

slide-16
SLIDE 16

DBMS #2

ETL system Custom reports

DBMS

Web Application Web Client Cache

slide-17
SLIDE 17

DBMS #2

ETL system Custom reports

DBMS

Web Application Web Client Cache

slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20

Canonical data sources

slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24

Patterns

slide-25
SLIDE 25

http://lambda-architecture.net/

slide-26
SLIDE 26

Polyglot persistency

slide-27
SLIDE 27

Lava layers

Mike Hadlow

http://mikehadlow.blogspot.co.uk/2014/12/the-lava-layer-anti-pattern.html

slide-28
SLIDE 28

Data silos

slide-29
SLIDE 29

What’s in a name?

slide-30
SLIDE 30

/Cosmics/Commissioning2014-HcalCalHOCosmics- PromptReco-v3/ ALCARECO#c3f3b916-28a1-11e4-87ab-0023aefe9c6 8/a8098c1a-f86e-11da-bd1a-00112444be1e.root

slide-31
SLIDE 31
slide-32
SLIDE 32

Build solution Identify problem Evaluate solution

slide-33
SLIDE 33

?

slide-34
SLIDE 34
slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37

Build solution Identify problem Evaluate solution

slide-38
SLIDE 38

Take aways

  • Start from a real problem & get buy in
  • Pick off pieces you can solve - don’t leave lava
  • Follow an experimental process, with a good

feedback loop

  • Institutionalise knowledge, engage your team
  • Learn from mistakes, capitalise on successes