Data Management, In-Situ Workflows and Extreme Scales Manish - - PowerPoint PPT Presentation
Data Management, In-Situ Workflows and Extreme Scales Manish Parashar, Ph.D . Director, Rutgers Discovery Informatics Institute RDI 2 Distinguished Professor, Department of Computer Science Philip Davis, Shaohua Duan, Yubo Qin, Melissa Romanus,
Director, Rutgers Discovery Informatics Institute RDI2 Distinguished Professor, Department of Computer Science
ROSS 2018 @ HPDC’18, Tempe, AZ, USA June 12, 2018
– Data placement – Resilience
scientific process ss has evolved to include co computation & da data
data rich”
Oceanography OOI Biology: Sequencing Personalized Medicine Crisis Management Fusion: KSTAR Physics: LHC Sociology: The Web Economics: POS Terminals Neuroscience: EEG, fMRI Internet of Things Astronomy: LSST
Source: Hyperion (IDC) Paints a Bullish Picture of HPC Future By John Russell
generate huge amounts of data!
– If a single core produces 2MB/minute on average, one of these machines could generate simulation data between ~170TB per hour -> ~700PB per day -> ~1.4EB per year
How we enable the computation scientists to efficiently manage and explore extreme scale data: “find the needles in haystack” ??
– Run large-scale simulation workflows on large supercomputers – Dump data to parallel disk systems – Export data to archives – Move data to users’ sites – usually selected subsets – Perform data manipulations and analysis on mid-size clusters – Collect experimental / observational data – Move to analysis sites – Perform comparison of experimental/observational to validate simulation data
Analysis/Visualization Cluster Simulation Machines Storage Servers
…
S i m u l a t i
R a w D a t a
significant concern
Computer”
Energy_move_data = bitrate*length2 cross_section_area_of_wire performance gap
memory and persistent storage is slow!
Traditional data analysis pipeline
– Reduce data movement – Move computation/analytics closer to the data – Add value to simulation data along the IO path
– Workflow coordination, all applications share an MPI context, which is split by an execution wrapper
– Embed analysis/viz in simulation processes using time division.
– Flexible I/O abstractions for end to end data pipelines
– Independent task coordination across processes.
– Decoupled dataflow middleware for in-situ workflows
– Semantic data redistribution of complex data structures for in-situ applications
– Standardizing glue components for HPC workflows
– Leverage heterogeneous compute node resources like GPUs to run in-situ workflow
– Leverages dedicated cores in multicore nodes to offload data management tasks
– RPC and bulk message passing across applications.
– Communication between MPI applications using a reliable transport.
l
Virtual shared-space programming abstraction
l
Simple API for coordination, interaction and messaging
l
Distributed, associative, in-memory object store
l
Online data indexing, flexible querying
l
Adaptive cross-layer runtime management
l
Hybrid in-situ/in-transit execution
l
Efficient, high-throughput/low-latency asynchronous data transport
The DataSpaces Abstraction
15
Data staging Runtime data coupling Online data analysis and processing
16
– Same cores as the simulation (in situ) – Some (dedicated) cores on the same nodes – Some dedicated nodes on the same machine – Dedicated nodes on an external resource
– Direct access to simulation data structures – Shared memory access via hand-off / copy – Shared memory access via non-volatile near node storage (NVRAM) – Data transfer to dedicated nodes or external resources
– Execute synchronously with simulation every nth simulation time step – Execute asynchronously
Processing data on remote nodes Using distinct cores on same node Sharing cores with the simulation
DRAM DRAM
DRAM Simulation Node DRAM Staging Node
Network Communication
Analysis Tasks Simulation Visualization
DRAM NVRAM SSD Hard Disk CPUs DRAM NVRAM SSD Hard Disk CPUs Network Node 1 Node 2 Node N
...
Staging option 1 Staging option 2 Staging option 3
17
SRAM: Latency ~1X DRAM: Latency ~10X 3D-RAM: Latency ~100X NAND-SSDs: Latency ~100,000X Disks: Latency ~10 MillionX
18
19
Credit: Gary Grider LANL
20
– Data placement – Resilience
support code coupling and data management in data-intensive simulation workflows.
target data objects to DRAM prior to a read request from a coupled component
dynamically place data objects at different memory hierarchy levels, i.e., SSD or DRAM
Anticipate data read pattern
Quantify utility
near future have higher utility
Place data
remains in the DRAM level longer Utility quantifies the relative value of data objects in the staging area, for example, based
An illustration of spatial-temporal data read patterns for a 2D data domain with N time steps. Gray regions: data written into the staging area; yellow regions and checkered regions: data read by two different applications. An illustration of feature tracking case about subtle vortical structures identified in a large and complex flow field of turbulent combustion.
○ Reduce data access costs ○ Example
○ P4 accesses data d4 ○ P4 is mapped to S1 ○ d4 is placed on S1
○ Efficient usage of storage space ○ Example
○ d2 is replicated and placed ○
– Which data elements to place; Where to place data elements
– Asynchronous data generation – Estimated Execution Time – EET
– Computation load
– Data affinity
– T. Jin, et al, "Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows", 29th IEEE International Parallel & Distributed Processing Symposium (IPDPS'15), May 2015.
– Q. Sun, et al, "Adaptive Data Placement For Staging-based Coupled Scientific Workflows", ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC'15), Nov, 2015
– Q. Sun, et al, “In-Staging Data Placement for Asynchronous Coupling of Task-Based Scientific Workflows,” 2nd International Workshop on ESPM2'16 in conjunction with SC’16, Nov, 2016. (Best Paper)
– Supervised/ Unsupervised Learning – Training of neural network or determining weights takes a long time (& needs sufficient input data)
– Use n-grams models to predict not
variable, but also predict which variable – Build a hash-table for each variable and predict upcoming sequence of accesses
50 55 60 65 70 75 80 85 1 2 3 4 5 6 7 8 9 10 256+4096+4096 512+8192+8192 1024+16384+16384 % of Read Requests going to Disk Cumulative Read Time for 20 Time Steps
In-memory ML-Based Locality-Based No-prefetching ML-Based Disk Access Locality-based Disk Access 50 55 60 65 70 75 80 85 90 95 5 10 15 20 25 256+4096+4096 256+8192+8192 256+16384+16384 % of Read Requests going to Disk Cumulative Read Time for 20 Time Steps
In-memory ML-Based Locality-Based No-prefetching ML-Based Disk Access Locality-based Disk Access
Experiments were run on Titan Supercomputer with the S3D Workflow with DNS - LES coupling
a) Increase of servers and clients in same ratio b) Increase in clients only
Resilience Resilience
– Data placement – Resilience
Histograms of inter-failure arrival time
Data based on available public records in: D. Tiwari, S. Gupta, S. S. Vazhkudai. “Lazy checkpointing: Exploiting temporal locality in failures to mitigate checkpointing overheads on extreme-scale systems.” DSN 2014
Metric: average MTBF – e.g. 8h in Titan Longest period without any failures – e.g. 24h in Titan
Node Failure Frequency in Current Systems
Estimated MTBF for an exascale system would be in minutes
q Unfortunately, traditional HPC fault tolerance techniques can not be directly used to implement resilient data staging services.
Data Objects In Memory Data Staging Resiliency approach (ULFM) Simulation C Resiliency approach (Checkpoint/restart) Simulation B Resiliency approach (Fenix) Coupled Simulation F Resiliency approach (Checkpoint/restart) Simulation A Resiliency approach (ULFM) Analytic D Resiliency approach (Fenix) Analytic E Vulnerable for failure
q Ideally, data resilience in the staging service should be transparent to workflows and provide acceptable storage/compute overheads.
Impact of checkpointing on staging-based in-situ workflows
50 100 150 200 250 300 1G 2G 4G 8G Time (sec) Data Size Exec Exec-CoREC Exec-check Checkpoint Restart
ü Spatial/Temporal Locality ü Hot data: Replication ü Cold Data: Erasure Coding
Dynamic hybrid Erasure Codes & Replication
Data Set Fault Tolerance Method Set Hot Write Data Cold Data Hot Read Data Erasure code N-way Replication frequency performance Data Objects In Memory Data Staging Redundant Data Objects Hybrid Erasure Coding Cold Data Object: Hot Data Object: Parity Object: Replica: 34
q A time complexity of CoREC:
!" !#: Time Complexity of replication / erasure coding $
% $ &: Frequency of updates for hot / cold data
'
% ' &: Percentage of hot / cold data
Cerasure: fully erasure coding Creplica: fully replication Chybrid: simple hybrid erasure coding CCoREC: with varying miss ratios ((
))
= !"$
% − !#$ & + !# − !" $ %( ) -'% + !#$ &-
!./01. = !"$
%-'% + !#$ &-' &
= !"$
% − !#$ & -'% + !#$ &-
'
& = 1 − ' %
!# − !" $
%( )-' %
Relative time complexity Ph (%) 100 1 2 Ph = Pr Percentage of hot data
q Factors affecting relative time complexity of CoREC:
ü The difference of hot and cold data frequencies ($
% − $ &)
ü The difference of replication and erasure coding complexity (!#−!") ü The scale of workload - ü The accuracy of data classification (
)
35
q Hot/Cold data:
If a data object has been recently accessed more than a threshold number of times within a certain time interval it is considered as hot data, otherwise it is considered as cold data.
q Hot/Cold data classification:
Based on Spatial and temporal data locality.
An illustration of spatial and temporal data write/update patterns for a 2D data domain with N + 1 time steps. The red regions and slash regions (hot data) indicate data written into the staging area, while the blue regions (cold data) are not updated since time step i.
TS 1 TS i TS n TS n+1 Y X Hot data Cold data Query data TS 1 TS 2 TS i TS n TS n+1 Y X
(a) Single time step data locality case (b) Multi time steps data locality case
Result: CoREC reduces the write response time by 7.3%, 14.8%, and 5.4% as compared to full erasure coding
CoREC reduces the read response time by up to 40.8% and 37.4% for one and two failures respectively.
0.5 1 1.5 2 2.5 3 3.5 4 4480 8960 17920
Read Time (sec)
S3D disk DataSpaces Replicate Erasure CoREC Erasure+1f CoREC+1f Erasure+2f CoREC+2f 21.5436 sec 22.7551 sec 26.8691 sec 2 4 6 8 10 12 14 16 18 20 22 4480 8960 17920
Write Time (sec)
S3D disk DataSpaces Replicate Erasure CoREC Erasure+1f CoREC+1f Erasure+2f CoREC+2f 166.46 sec 234.42 sec 346.68 sec
Comparison of the cumulative data read/write response time using S3D and analysis workflow on Titan
S3D combustion simulation and analysis workflow on Titan Cray XK7
4480 8960 17920
16x16x16= 4096 32x16x16= 8192 32x32x16=16384
256 512 1024
128 256 512 Volume size 1024x1024x1024 2048x1024x1024 2048x2048x1024 Data size (GB) 160 320 640
1 1 1
3 3 3
1 1 1 Storage efficiency 67% 67% 67%
XGC GENE
Interpolator
Reduction Reduction XGC Viz. XGC
GENE Viz. GENE
Comparative Viz.
NVRAM
PFS TAPE
Performance Viz.
– One-way data exchange of two types of data: Particle data (large size, single iteration) and Turbulence data (small size, multiple iterations) – Data generated in each cycle needs to be cached for further usage
XGC-1 Execution Particle Turbulence Particle Particle Turbulence XGC-1 Execution XGC-a Execution XGC-a Execution I/O I/O I/O I/O
Performance of Reading Particle Data*
* XGC1 and XGCa averages shown on same plot
Approach Setup 1 Setup 2 Setup 3 Disk-based
(sec)
35.792 90.062 425.059 Server- based (sec) 1.865 2.283 3.781 On-node
(sec)
0.097 0.131 0.316 Total Time of Writing Particle Data*
ß Decreases the total time for writing particle data by avg. 99% compared to disk-based and by
server-based Decreases the total time for reading particle data by avg. 98% compared to disk-based and by
server-based à
17 * XGC1 and XGCa averages shown in same table
– Data costs (performance, latency, energy) are quickly dominating – Traditional data management/analytics pipelines are breaking down
– Users to efficiently intertwine applications, libraries, middleware for complex analytics
– The DataSpaces project explores solutions at various levels