Visualization Frameworks for Data Staging and In- Situ Environments David Pugmire Scientific Computing Group, Oak Ridge National Laboratory Thanks to: H. Abbasi, S. Ahern, C. Chang, J. Choi, S. Ku, S. Klasky, J. Kress, J. Logan, Q. Liu, J. Meredith, K. Mu, G. Ostrouchov, N. Podhorszki, R. Sisneros, Y. Tian + many more 16 November 2014 XVis Sunday, November 16, 14
XVis Sunday, November 16, 14
XVis Sunday, November 16, 14
or XVis Sunday, November 16, 14
Data Driven Science and Scientific Visualization Increasing mesh resolutions Volume Increasing temporal resolution Increasing temporal resolution Velocity Frequency of data Multi-variate Variety Ensembles Increasing complexity Uncertainty Veracity Errors Approximations Visualization and Analysis Value Feature detection Scientific insight XVis Sunday, November 16, 14
Today’s Tools in Data Driven Science World Increasing mesh resolutions Volume Increasing temporal resolution Increasing temporal resolution Velocity Frequency of data Multi-variate Variety Ensembles Increasing complexity Uncertainty Veracity Errors Approximations Visualization and Analysis Value Feature detection Scientific insight XVis Sunday, November 16, 14
Today’s Tools in Data Driven Science World Increasing mesh resolutions Volume Increasing temporal resolution Increasing temporal resolution Velocity Frequency of data Multi-variate Variety Ensembles Increasing complexity Uncertainty Veracity Errors Approximations Visualization and Analysis Value Feature detection Scientific insight Focused on Volume Others V’s are harder, and often a function of Volume XVis Sunday, November 16, 14
Scalability of Visualization Tools Research Questions: • Can current visualization tools survive at the exascale? • What are the bottlenecks at the largest scales? • What differences do architectures make? Methodology: • “Create” exascale data (trillions of zones) • Execute a simple workflow : • Read data • Volume render / contour data • Render and composite see: Extreme Scaling of Production Visualization Core-collapse supernova simulation. Data Software on Diverse Architectures , IEEE CG&A, 2010 courtesy of T. Mezzacappa (GenASiS) XVis Sunday, November 16, 14
Scalability of Visualization Tools XVis Sunday, November 16, 14
Scalability of Visualization Tools XVis Sunday, November 16, 14
Challenges at Exascale: 100-200 From: Exascale Workshop on Data Analysis, Management and Visualization. DOE ASCR 2011 XVis Sunday, November 16, 14
Challenges at Exascale: 100-200 From: Exascale Workshop on Data Analysis, Management and Visualization. DOE ASCR 2011 XVis Sunday, November 16, 14
Challenges at Exascale: 100-200 From: Exascale Workshop on Data Analysis, Management and Visualization. DOE ASCR 2011 XVis Sunday, November 16, 14
Challenges at Exascale: 100-200 From: Exascale Workshop on Data Analysis, Management and Visualization. DOE ASCR 2011 XVis Sunday, November 16, 14
Challenges at Exascale: 100-200 From: Exascale Workshop on Data Analysis, Management and Visualization. DOE ASCR 2011 I/O Caveats: System System Peak I/O Peak I/O Reality I/O Hero JaguarPF 2PF 200 GB/s 1 GB/s 60 GB/s Titan 20PF 1.2 TB/s 1 GB/s 120 GB/s Future 1000PF 10 TB/s (?) ?? ?? XVis Sunday, November 16, 14
Challenges at Exascale: 100-200 From: Exascale Workshop on Data Analysis, Management and Visualization. DOE ASCR 2011 I/O Caveats: System System Peak I/O Peak I/O Reality I/O Hero JaguarPF 2PF 200 GB/s 1 GB/s 60 GB/s Titan 20PF 1.2 TB/s 1 GB/s 120 GB/s Future 1000PF 10 TB/s (?) ?? ?? We will get less of what we want We will get more of what we don’t know how to use XVis Sunday, November 16, 14
Impacts on Visualization Massive Concurrency Complex Memory • Production tools of today cannot • Visualization APIs not available fully utilize • New algorithms and programming • Challenges of new programming models models Decreased I/O Memory Constraints Performance • Cannot rely on storage system in • Expressive and flexible data models workflows • Efficient data models become • In situ methods become critical imperative, especially for zero-copy in situ applications XVis Sunday, November 16, 14
Path Forward Massive Concurrency Complex Memory • Library that supports/abstracts: • Advanced data model • Heterogeneous computing • API that manages/abstracts the complexities • Fine grained parallelism Decreased I/O Memory Constraints Performance • Data management and movement • Advanced, expressive data model library • Efficient data model • Flexible in situ interface • Representation and execution XVis Sunday, November 16, 14
Path Forward Extreme-Scale Analysis and Visualization Library (EAVL) TO • Advanced visualization and analysis for next generation computer architectures • Part of DOE funded VTK-m efforts Adaptable I/O System (ADIOS) • Middleware abstraction of I/O for HPC systems • Provides increased performance for disk based I/O, and in situ processing XVis Sunday, November 16, 14
Data Management Framework: ADIOS • An I/O abstraction framework • Provides portable, fast, easy-to-use metadata rich output • Change I/O method on-the-fly • Abstract the API from the method • Looks to provide support for “90% of applications” ¡h#p://www.nccs.gov/user-‑support/center-‑projects/adios/ Interface)to)apps)for)descrip/on)of)data)(ADIOS,)etc.)) Data)Management)Services) • Astrophysics • Neutron Science • Climate Feedback) Buffering) )Schedule) • Nuclear Science • Combustion Mul/Bresolu/on) Data)Compression) Data)Indexing) • Quantum Turbulence • CFD methods) methods) (FastBit))methods) • Relativity • Environmental Science Plugins)to)the)hybrid)staging)area) • Seismology • Fusion Provenance) Workflow))Engine) Run/me)engine) Data)movement) • Sub-surface Modeling • Earthquake Analysis)Plugins) Visualiza/on)Plugins) • Weather • Material Science • Satellite Processing AdiosBbp) IDX) HDF5) pnetcdf ) “raw”)data) Image)data) • Medical: Pathology Parallel)and)Distributed)File)System) Viz.)Client) XVis Sunday, November 16, 14
Sunday, November 16, 14
I/O in ADIOS • Carefully manage movement of ADIOS API data in network and I/O system Data Management Services • Data format agnostic • Allows simulations to spend more Scheduling Buffering time in compute, or allows more I/O Transports frequent output of data • Visualization is especially sensitive raw image BP pnetcdf HDF5 to I/O performance Parallel Filesystem Sunday, November 16, 14
Data Staging in ADIOS ADIOS API Data Management Services Scheduling Buffering I/O Transports BP raw image pnetcdf Staging HDF5 Parallel Filesystem Staging Area Sunday, November 16, 14
Data Staging in ADIOS • Same application API can be used to do more advanced data movement • Plugins will operate on data streams in user-defined ways Reader HPC Application ADIOS ADIOS Staging Server Sunday, November 16, 14
Extreme-scale Analysis and Visualization Library (EAVL) EAVL enables advanced visualization and analysis for the next generation scientific simulations, supercomputing systems, and end-user analysis tools. New$Mesh$Layouts$ Greater&Memory&Efficiency& • More%accurately%represent%simula1on% • Support'future'low,memory'systems' data%in%analysis%results% • Minimize'data'movement'and' • Support%novel%simula1on%applica1ons% transforma7on'costs' Parallel&Algorithm&Framework& In#Situ#Support# • Direct'zero*copy'mapping'of'data'' • Accelerator)based-system-support- from'simula6on'to'analysis'codes' • Pervasive-parallelism-for-mul6)core- • Heterogeneous'processing'models' and-many)core-processors- allow'broad'pla:orm'support' J.S. Meredith, S. Ahern, D. Pugmire, R. Sisneros, "EAVL: The Extreme-scale Analysis and Visualization Library", Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), 2012. http://ft.ornl.gov/eavl XVis Sunday, November 16, 14
Gaps in Current Data Models • Traditional data set models target only common combinations of cell and point arrangements • This limits their expressiveness and flexibility Poin Point Arrangement ment Cells Coordinates Explicit Logical Implicit Strided Structured Grid Structured Structured Separated Rectilinear Grid Image Data Unstructured Strided Grid Unstructured Unstructured Separated XVis Sunday, November 16, 14
Arbitrary Compositions for Flexibility • EAVL allows clients to construct data sets from cell and point arrangements that exactly match their original data • In effect, this allows for hybrid and novel mesh types • Native data results in great accuracy and efficiency Poin Point Arrangement ment Cells Coordinates Explicit Logical Implicit Strided Structured Structured Separated Strided Unstructured Unstructured Separated XVis Sunday, November 16, 14
Other Data Model Gaps Addressed in EAVL A B# Low/high dimensional Multiple cell groups in Multiple coordinate data (7D GenASiS) one mesh systems (lat/lon + XY) H H C C H H Novel and hybrid Non-physical data Mixed topology mesh types (quadtree (graphs, sensor, etc) (atoms+bonds) grid from MADNESS) XVis Sunday, November 16, 14
Recommend
More recommend
