Big data in critical infrastructure: Production and failover - - PowerPoint PPT Presentation

▶
big data in critical infrastructure production and
SMART_READER_LITE
LIVE PREVIEW

Big data in critical infrastructure: Production and failover - - PowerPoint PPT Presentation

Aug 06, 2023 36 likes •558 views

Big data in critical infrastructure: Production and failover infrastructure in DWD's central data management Daniel Lee, German Weather Service (DWD) TI12b Sep. 2015 Agenda 1. DWD's goals 2. Operational systems 3. Technical

slide-1
SLIDE 1

TI12b – Sep. 2015

Big data in critical infrastructure: Production and failover infrastructure in DWD's central data management

Daniel Lee, German Weather Service (DWD)

slide-2
SLIDE 2

TI12b – Sep. 2015

  • 1. DWD's goals
  • 2. Operational systems
  • 3. Technical infrastructure
  • 4. Current challenges
  • 5. Future plans

Agenda

slide-3
SLIDE 3

TI12b – Sep. 2015

What does DWD do?

slide-4
SLIDE 4

TI12b – Sep. 2015

  • Monitoring of meteorological interactions

between the atmosphere and other environmental systems

  • Prediction of meteorological events
  • Monitoring and prediction of the movements
  • f radioactive trace particles
  • Operation of the necessary observation

systems

  • Storage, archival and documentation of

meteorological data and products

The DWD law

slide-5
SLIDE 5

TI12b – Sep. 2015

  • DWD aids in protecting lives and property, as

well as in planning and maintaining critical infrastructure in the areas of: – Aviation – Seafaring – Agriculture – Energy – Climate – Weather warnings – Protection and recovery from high impact weather – Etc.

Target audiences

slide-6
SLIDE 6

TI12b – Sep. 2015

Past Future

  • Yearly, seasonal and monthly forecasts

100 years 10 years 1 year 1 month today

  • Weather and climate observation
  • Longterm forecasts (72-360 hours)
  • Midterm forecasts (12-72 hours)
  • Shorterm forecasts (2-12 hours)
  • Nowcasting ( < 2 hours)
  • Climate projection

Multiscale topics

slide-7
SLIDE 7

TI12b – Sep. 2015

Example: Model output

slide-8
SLIDE 8

TI12b – Sep. 2015

Example: Radar map

slide-9
SLIDE 9

TI12b – Sep. 2015

Example: Short-term storm cell forecast

slide-10
SLIDE 10

TI12b – Sep. 2015

Example: Ensemble probabilities forecast

slide-11
SLIDE 11

TI12b – Sep. 2015

Example: Flight cross-section

slide-12
SLIDE 12

TI12b – Sep. 2015

Example storm event

slide-13
SLIDE 13

TI12b – Sep. 2015

Example storm event

slide-14
SLIDE 14

TI12b – Sep. 2015

Automated product generation

slide-15
SLIDE 15

TI12b – Sep. 2015

Operational systems

slide-16
SLIDE 16

TI12b – Sep. 2015

COSMO-DE ICON-EU ICON

Models for multiple spatiotemporal scales

slide-17
SLIDE 17

TI12b – Sep. 2015

Physical observation system

slide-18
SLIDE 18

TI12b – Sep. 2015

RMDCN RMDCN EUMETCast EUMETCast Dissemination Dissemination Radar- Radar- daten daten Messnetz Messnetz Bundeswehr Bundeswehr Radio- Radio- logie logie Satelliten- Satelliten- daten daten

Bringing the data together

slide-19
SLIDE 19

TI12b – Sep. 2015

ICON x = 13 km ICON-EU x = 6.5 km COSMO-DE x = 2.2 km

ICON: Grid spacing: 13 km Vertical layers: 90 Forecast range: 174 / 78 hours Runs per day: 4 ICON-EU: Grid spacing: 6.5 km Vertical layers: 54 Forecast range: 78 hours Runs per day: 8 COSMO-DE (-EPS): Grid spacing: 2.2 km Vertical layers: 80 Forecast range: 27 hours Runs per day: 8 EPS members: 20

Daily deterministic output: ~ 2.5 TByte Daily probabilistic output: ~ 3.5 TByte

DWD's weather models

slide-20
SLIDE 20

TI12b – Sep. 2015

Global observations International data exchange Database persistence + archival Numerical weather prediction Model output persistence Data distribution Visualization + interpretation

24 / 7

DIN EN ISO 9001:2008

Routine operation

slide-21
SLIDE 21

TI12b – Sep. 2015

Routine operation

slide-22
SLIDE 22

TI12b – Sep. 2015

Type of data Source Producer # reports

  • Approx. daily

data volume

Observation in-situ remote sensing Manned ground station Automatic ground station Ship observation Buoy observation Radiosonde Aircraft Radar Lidar Wind profiler Satellite 12,000 50,000 2,500 750 900 3,000 17 2 4 20 350 GB Model output model COSMO-DE COSMO-DE-EPS ICON-EU ICON Wave model / other models 8 8 8 4 12 6,000 GB 500 GB

Daily data ingress / egress

slide-23
SLIDE 23

TI12b – Sep. 2015

Physical infrastructure

slide-24
SLIDE 24

TI12b – Sep. 2015

1 TeraFLOP/s 1 MegaFLOP/s 1 GigaFLOP/s

CDC-3800 Cyber 76 Cray YMP IBM pSeries Cray T3E NEC SX-9 Moore's law

Dauerleistung

/ XC40

DWD's processing power

slide-25
SLIDE 25

TI12b – Sep. 2015

Multiple supercomputers

Production Research & Development

  • 24 / 7 routine production
  • Data assimilation for computing model

initial states

  • All numerical weather prediction

models

  • Parallel routine for evaluating model

changes

  • R&D
  • NUMEX: Numerical experiments
  • Backup if routine computer

unavailable

slide-26
SLIDE 26

TI12b – Sep. 2015

  • 2 Cray XC40 supercomputers
  • Each: 796 nodes, 17,648 compute modules,

79 TB RAM

  • Xeon prozessors, connected with Aries

network

  • Top performance: 550 TeraFLOP/s per

computer

  • 2 Linux Cluster (Megware Slashtwo)
  • Each: 523 nodes, 4.5 TB RAM
  • Each: 500 Xeon compute modules connected with

Infiniband network

  • Top performance: 16,7 TeraFLOP/s per computer

HPC specifications

slide-27
SLIDE 27

TI12b – Sep. 2015

Start of probabilistic forecasts

Volume in meteorological databases

slide-28
SLIDE 28

TI12b – Sep. 2015

  • Cray Sonexion & NEC/NetApp E5500 with ca. 4.2 PB storage capacity
  • 1.2 PB global storage for HPC
  • Write speed: 15 GB/s
  • Also:
  • 6 data servers (NEC/SUN Fire X2-8 with 80 Intel Xeon compute

modules & 1 TB RAM each)

  • 2 mirrored data servers
  • Availability: 99,9%
  • Server group manages up to 3 PB meteorological data
  • Archive of 2 StorageTek SL8500 tape libraries with

10,000 tape casettes each

  • > 40 casettes can be read and written to simultaneously
  • 16 robots physically access the tapes and insert them

into the archive server

  • Estimated data volume by 2016: 60 PB

Storage systems

slide-29
SLIDE 29

TI12b – Sep. 2015

Konrad-Zuse-Institut Berlin (ZiB)

Quelle: mygeo.info

DMRZ Offenbach Hochleistungsrechner Entwicklung Halle Ost

(Übernahme der Produktion bei Ausfall Halle West)

XC40 364 Ivy Bridge Knoten + 432 Haswell Knoten 35296 Cores 77 TiB Hauptspeicher Meteorologie des Geoinformationsdienstes der Bundeswehr in Euskirchen (MetBw)

Datenbankserver Sun X2-4 Datenzugriffserver Sun X2-4 Geteilte Gesamtkapazität 150 TiB

ECMWF Reading Modelle XC40 364 Knoten Ivy Bridge + 432 Knoten Haswell 35296 Cores 77 TiB Hauptspeicher DMRZ Offenbach Hochleistungsrechner Produktion Halle West

Concept: A. Pielicke, M. Jonas Current: November 2014

Sonexion Lustre- Filesystem mit 1012 TiB Kapazität Datenbankserver Sun X2-4 240 TiB Kapazität Datenzugriffsserver Sun X2-8 1300 TiB Kapazität

DMRZ Offenbach Hochleistungsrechner Produktion Halle West

Archivsystem SUN/IBM-HPSS SUN STK SL8500 2 Kassettensilos 20000 Stellplätze Datenvolumen 6 PiB 36 Laufwerke IBM X3650 9 Knoten 72 Prozessorkerne 216 GiB Hauptspeicher 600 TiB Plattensysteme Megware 18 Knoten mit 128GB RAM 4 Knoten mit 512GB RAM Panasas Filesystem mit 120 TiB

DMRZ Offenbach Hochleistungsrechner Entwicklung Halle Ost

(Übernahme der Produktion bei Ausfall Halle West)

Sonexion Lustre- Filesystem mit 2668 TiB Kapazität Megware 24 Knoten mit 128GB RAM 4 Knoten 512GB RAM Panasas Filesystem mit 171 TiB

Quelle: ZGeoBw EUS

Datenbankserver Sun X2-4 240 TiB Kapazität Datenzugriffsserver Sun X2-8 1600 TiB Kapazität Megware 6 Knoten mit 128 GB RAM + 2 Knoten mit 32 GB RAM fürs PBS 2x NFS Server mit 64GB RAM 2x8 Cores (inkl. 16 Hyperthreads)

DMRZ Offenbach XCT

XC40 8 Sandy Bridge Knoten + 8 Haswell Knoten 640 Cores 1 TiB Hauptspeicher

Georedundancy

slide-30
SLIDE 30

TI12b – Sep. 2015

Software configuration

slide-31
SLIDE 31

TI12b – Sep. 2015

Job dispatchment: SMS / ecFlow

  • Timed job execution
  • Interjob dependency
  • Status reports + output capture
  • Manual starts, restarts, aborts
  • Transferability between halls &

computing centers

Job management

slide-32
SLIDE 32

TI12b – Sep. 2015

Several monitoring systems, depending on target components

  • Nagios
  • Icinga
  • Big Brother
  • Custom software

Testing and building with Jenkins

Monitoring & integration

slide-33
SLIDE 33

TI12b – Feb. 2015

WISO (WMO data exchange) MIRAKEL Relational weather data Data conversion (BUFR-TO-ROUTKLI) DWD stations NinJo Server

(uses GloBUS)

NinJo meteorological workstations

Data management overview

Decoding (GloBUS) Internal model data Numerical weather prediction Obs. External model data Binary DBs External users

slide-34
SLIDE 34

TI12b – Feb. 2015

WISO (WMO data exchange) MIRAKEL Relational weather data Data conversion (BUFR-TO-ROUTKLI) DWD stations NinJo Server

(uses GloBUS)

NinJo meteorological workstations

AFD: Automated file distributor

Decoding (GloBUS) Internal model data Numerical weather prediction Obs. External model data Binary DBs External users

slide-35
SLIDE 35

TI12b – Feb. 2015

WISO (WMO data exchange) MIRAKEL Relational weather data Data conversion (BUFR-TO-ROUTKLI) DWD stations NinJo Server

(uses GloBUS)

NinJo meteorological workstations

GloBUS and BUFR-TO-ROUTKLI

Decoding (GloBUS) Internal model data Numerical weather prediction Obs. External model data Binary DBs External users

slide-36
SLIDE 36

TI12b – Feb. 2015

WISO (WMO data exchange) MIRAKEL Relational weather data Data conversion (BUFR-TO-ROUTKLI) DWD stations NinJo Server

(uses GloBUS)

NinJo meteorological workstations

SKY

Decoding (GloBUS) Internal model data Numerical weather prediction Obs. External model data Binary DBs External users

slide-37
SLIDE 37

TI12b – Sep. 2015

Meteorological databases

CIRRUS

Metadata

  • 1. Time critical (binary) databases

Applications: NWP, postprocessing, etc.

  • Near-real-time availability
  • proprietary (in-house software SKY)
  • Metadata in Oracle-RDBMS, payload

data in files MIRAKEL

General meteorological data

Others: LARS, LAURA, etc.

Application specific databases

Observations

(BUFR, STRING, etc.)

Modelldaten

(GRIB)

OMO

Foreign models

File systems

Reports

NUMEX

Numerical experiments

ROMA

DWD models

PARMA

Parallel routine

File systems

Results

  • 2. Relational databases

Applications: climate, web services, etc.

  • Lower availability
  • Easier access
  • Data and metadata stored together in-DB
slide-38
SLIDE 38

TI12b – Sep. 2015

  • 60,000 – 92,000 files saved per day
  • 4,200 – 12,000 files accessed per day
  • 35 – 75 TiB saved per day
  • 5 – 17 TiB accessed per day
  • 280,000 – 950,000 read

requests per day (often overlapping)

Access patterns

slide-39
SLIDE 39

TI12b – Sep. 2015

  • WCS, WFS, WMS, etc.

served by Geoserver

  • File-based access for

rasters

  • Vectors stored directly

in DB

Geo web services

slide-40
SLIDE 40

TI12b – Sep. 2015

Current challenges

slide-41
SLIDE 41

TI12b – Sep. 2015

Example: Growth of satellite data

Observation data volume

slide-42
SLIDE 42

TI12b – Sep. 2015

Example: Growth of radiosonde data

Observation data volume

slide-43
SLIDE 43

TI12b – Feb. 2015

slide-44
SLIDE 44

TI12b – Sep. 2015

SKY access patterns are changing:

  • 2015: Factor 2.5 – 5 increase in I/O
  • Some estimates for 2017 estimate increase

by factor 41 (worst case scenario)

Increase in post-processing DB access

slide-45
SLIDE 45

TI12b – Sep. 2015

  • Tiles (high resolution ground level modeling)
  • EDA (ensemble data assimilation)
  • Ensemble ICON (global & EU)
  • COSMO-D2(-EPS, 40 mem.): 3.2x more

data (higher resolution in all dims)

  • KENDA (km-scale Ensemble Data

Assimilation)

  • More ensemble members

Developments in NWP

slide-46
SLIDE 46

TI12b – Sep. 2015

More and more partner networks want to provide data in non-standard formats (XML, NetCDF, etc.) Custom applications for end users also require output data in new formats (GeoTiFF, etc.) Many users want higher data volumes or data on-demand

Increased contact with external users

slide-47
SLIDE 47

TI12b – Sep. 2015

Future plans

slide-48
SLIDE 48

TI12b – Sep. 2015

Post-acquisition is pre-acquisition

HPC expansion

slide-49
SLIDE 49

TI12b – Feb. 2015

Primary interface to file system, SKY, has several extensions in the pipeline:

  • Higher parallelization
  • More intelligent caching strategies

Encoding / decoding are also being reworked to move away from monolithic to more disparate software components

Data access optimization

slide-50
SLIDE 50

TI12b – Feb. 2015

New projects serving data to end users will be increasingly quarantined from production resources

Separation of concerns

slide-51
SLIDE 51

TI12b – Feb. 2015

Cooperation and coordination: IT and research

slide-52
SLIDE 52

TI12b – Feb. 2015

Thanks! Questions?