In-Situ Data Analysis and Visualization: ParaView, Calalyst and - - PowerPoint PPT Presentation

▶
in situ data analysis and visualization
SMART_READER_LITE
LIVE PREVIEW

In-Situ Data Analysis and Visualization: ParaView, Calalyst and - - PowerPoint PPT Presentation

Jan 04, 2024 125 likes •785 views

In-Situ Data Analysis and Visualization: ParaView, Calalyst and VTK-m GTC, San Jose, CA March, 2015 Robert Maynard Marcus D. Hanwell 1 Agenda Introduction to ParaView Catalyst Run example Catalyst Script Creating your own Catalyst

slide-1
SLIDE 1

Robert Maynard Marcus D. Hanwell

GTC, San Jose, CA March, 2015

1

In-Situ Data Analysis and Visualization: ParaView, Calalyst and VTK-m

slide-2
SLIDE 2

Agenda

  • Introduction to ParaView Catalyst
  • Run example Catalyst Script
  • Creating your own Catalyst Script
  • Rendering grid and field information from

live simulation

2

slide-3
SLIDE 3

Going From Data to Visualization

3

slide-4
SLIDE 4

The Visualization Pipeline

  • A sequence of algorithms that operate on

data objects to generate geometry

4

Source Data Data Filter Filter Data Data Mapper Mapper Actor Actor

Render on screen

slide-5
SLIDE 5

Data Types

Uniform Rectilinear (vtkImageData) Non-Uniform Rectilinear (vtkRectilinearData) Curvilinear (vtkStructuredData) Polygonal (vtkPolyData) Unstructured Grid (vtkUnstructuredGrid)

Multi-block

Hierarchical Adaptive Mesh Refinement (AMR) Hierarchical Uniform AMR

slide-6
SLIDE 6

Why In Situ?

6

Need a supercomputer to analyze results from a hero run 2 orders of magnitude difference between each level

slide-7
SLIDE 7

Access to More Data

7

Post-processing In situ processing

CTH (Sandia) simulation with roughly equal data stored at simulation time Reflections and shadows added in post-processing for both examples

Dump Times

slide-8
SLIDE 8

Quick and Easy Run-Time Checks

MPAS-O (LANL) simulation

slide-9
SLIDE 9

Faster Time to Solution

CTH (Sandia) simulations comparing different workflows

9

slide-10
SLIDE 10

5 10 15 20 25 16 32 64 128 256 512 1024 2048

Seconds per Time Step Cores data generation annotations write render cell to point contour

Small Run-Time Overhead

XRAGE (LANL) simulation

10

slide-11
SLIDE 11

Reduced File IO Costs

Time of Processing Type of File Size per File Size per 1000 time steps Time per File to Write at Simulation Post Restart 1,300 MB 1,300,000 MB 1-20 seconds Post Ensight Dump 200 MB 200,000 MB > 10 seconds In Situ PNG .25 MB 250 MB < 1 second

XRAGE (LANL) simulation

11

slide-12
SLIDE 12

What is ParaView Catalyst?

12

slide-13
SLIDE 13

ParaView Catalyst Architecture

ParaView Catalyst Python Wrappings

ParaView Server

Parallel Abstractions and Controls

VTK

Core Visualization Algorithms

13

slide-14
SLIDE 14

ParaView User Interface

Menu Bar Toolbars Pipeline Browser Properties Panel 3D View Advanced Toggle

slide-15
SLIDE 15

Geometry Representations

Points Wireframe Surface Surface with Edges Volume

Toggle Color Legend Mapped Variable Representation Vector Component Edit Colors Reset Scalar Range Custom Scalar Range

slide-16
SLIDE 16

Simulation Disk Storage Visualization Simulation Disk Storage Visualization

Separate MPI

Running Simulation with Catalyst

In Situ In Transit

16

slide-17
SLIDE 17

Running Simulation with Catalyst

Today we are going to be running In Situ and interactively with Catalyst

17

Simulation Visualization

slide-18
SLIDE 18

Simulation

Polygonal Output with Field Data Script Export Augmented script in input deck. Rendered Images 18

# Create the reader and set the filename. reader = sm.sources.Reader(FileNames=path) view = sm.CreateRenderView() repr = sm.CreateRepresentation(reader, view) reader.UpdatePipeline() dataInfo = reader.GetDataInformation() pinfo = dataInfo.GetPointDataInformation() arrayInfo = pInfo.GetArrayInformation( "displacement9")

Output Processed Data

User Perspective

Series Data

slide-19
SLIDE 19

GETTING READY TO RUN INTERACTIVELY

19

slide-20
SLIDE 20

Two Ways for Live In Situ Analysis and Visualization

  • Without blocking

– Simulation proceeds while the user interacts with the data from a specific time step – “At scale” mode

  • With blocking (new in ParaView 4.2)

– Simulation is blocked while the user interacts with the data – “Debugging” mode

  • Can switch between interactive and batch during run as well as

disconnecting from the run

20

slide-21
SLIDE 21

Getting Ready to Run Interactively

  • From the command shell run

– “~/insitu_demo/demo1.sh”

  • This runs a fake simulation with a prebuilt

catalyst pipeline

21

slide-22
SLIDE 22

Run Interactively

Load ParaView and connect to the simulation

22

slide-23
SLIDE 23

Run Interactively

23

Enable Extraction of Contour Visualize the Contour

slide-24
SLIDE 24

Run Interactively

24

slide-25
SLIDE 25

CREATING CATALYST OUTPUT

25

slide-26
SLIDE 26

Creating Catalyst Output

  • ParaView GUI plugin to create Python

scripts

  • Developer generated “canned” scripts

– See ParaView Catalyst User’s Guide

26

slide-27
SLIDE 27

Create Python Scripts from ParaView

Load the Catalyst script generator GUI plugin

27

slide-28
SLIDE 28

Create Python Scripts from ParaView

Load File

~/Demo2/filename_20_0.vti

Create Contour Filter

28

slide-29
SLIDE 29

29

Create Python Scripts from ParaView

Using the Writers menu add a writer

slide-30
SLIDE 30

30

Create Python Scripts from ParaView

  • 1. Load the Catalyst script generator GUI plugin
  • 2. Load File
  • 3. Create Contour Filter
  • 4. Using the Writers menu add a writer
slide-31
SLIDE 31

31

Export Catalyst Live Script

Using the CoProcessing menu start the export process

slide-32
SLIDE 32

32

Export Catalyst Live Script

  • 1. Using the CoProcessing menu start the export

process

  • 2. Select ‘filename_20_0.pvti’ as the simulation

inputs

  • 3. Select Live Visualization
slide-33
SLIDE 33

Other Catalyst Export Options

  • Options:

– Live visualization – Output to Cinema – Output Views

33

slide-34
SLIDE 34

Other Catalyst Export Options

  • Output Views

– Image Type – File Name – Write Frequency – Magnification – Fit to Screen – %t for timestep

34

slide-35
SLIDE 35

Other Catalyst Export Options

  • Output to Cinema

– Static vs Spherical Camera

35

Θ φ

slide-36
SLIDE 36

GETTING READY TO RUN OUR SCRIPT

36

slide-37
SLIDE 37

Getting Ready to Run Interactively

  • run ~/insitu_demo/demo2.sh

We are running our premade python script, if you want to run the version you made change driver_path variable in demo2.sh

37

slide-38
SLIDE 38

Run Interactively

Load ParaView and connect to the simulation

38

slide-39
SLIDE 39

Run Interactively ( Update with new Images)

39

Enable Extraction of Contour Visualize the Contour

slide-40
SLIDE 40

Developers Section Overview

  • Most work is done in creating the adaptor between the

simulation code and Catalyst

  • Small footprint in main simulation code base
  • Needs to be efficient both in memory and computationally
  • Fortran, C++ and Python examples:

– https://github.com/Kitware/ParaViewCatalystExampleCode

40

slide-41
SLIDE 41

Interaction Overview

  • Simulation has separate data structures from

VTK data structures

  • Use an adaptor to bridge the gap

– Try to reuse existing memory – Also responsible for other interactions between simulation code and Catalyst

Solver Adaptor

41

slide-42
SLIDE 42

Developer Perspective

Simulation

42

Adaptor

INITIALIZE(in pipelines) COPROCESS(in time, in grid, in fields) FINALIZE()

INITIALIZE() ADDPIPELINE(in pipeline) REQUESTDATADESCRIPTION(in time,

  • ut fields)

COPROCESS(in vtkDataSet) FINALIZE()

slide-43
SLIDE 43

Online Help

  • ParaView Catalyst User’s Guide:

– http://paraview.org/Wiki/images/4/48/CatalystUsersGuide.pdf

  • Email list:

– paraview@paraview.org

  • Doxygen:

– http://www.vtk.org/doc/nightly/html/classes.html – http://www.paraview.org/ParaView3/Doc/Nightly/html/classes.html

  • Sphinx:

– http://www.paraview.org/ParaView3/Doc/Nightly/www/py-doc/index.html

  • Websites:

– http://www.paraview.org – http://www.paraview.org/in-situ/

  • Examples:

– https://github.com/Kitware/ParaViewCatalystExampleCode

43

slide-44
SLIDE 44

VTK-m Project Goals

  • A single place for the visualization community to

collaborate, contribute, and leverage massively threaded algorithms.

  • Reduce the challenges of writing highly

concurrent algorithms by using data parallel algorithms

slide-45
SLIDE 45

VTK-m Architecture

In-Situ

Execution

Data Parallel Algorithms

Arrays

Post Processing

Worklets

DataModel

Filters

  • Combines strengths of multiple projects:

– EAVL, Oak Ridge National Laboratory – DAX, Sandia National Laboratory – PISTON, Los Alamos National Laboratory

slide-46
SLIDE 46

Execution Environment

Cell Operations Field Operations Basic Math Make Cells

Control Environment

Grid Topology Array Handle Invoke

Device Adapter

Allocate Transfer Schedule Sort …

Worklet

VTK-m Framework

cont exec

slide-47
SLIDE 47

VTK-m Arbitrary Composition

  • VTK-m allows clients to access different memory layouts through the

Array Handle and Dynamic Array Handle. –Allows for efficient in-situ integration –Allows for reduced data transfer

Control Environment Execution Environment

Transfer

Control Environment Execution Environment

slide-48
SLIDE 48

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-49
SLIDE 49

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-50
SLIDE 50

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-51
SLIDE 51

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-52
SLIDE 52

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-53
SLIDE 53

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-54
SLIDE 54

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-55
SLIDE 55

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

Execution Environment Control Environment

vtkm::cont::ArrayHandle<vtkm::Float32> inputHandle = vtkm::cont::make_ArrayHandle(input); vtkm::cont::ArrayHandle<vtkm::Float32> sineResult; vtkm::worklet::DispatcherMapField<Sine> dispatcher; dispatcher.Invoke(inputHandle, sineResult);

slide-56
SLIDE 56

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } }; vtkm::cont::ArrayHandle<vtkm::Float32> inputHandle = vtkm::cont::make_ArrayHandle(input); vtkm::cont::ArrayHandle<vtkm::Float32> sineResult; vtkm::worklet::DispatcherMapField<Sine> dispatcher; dispatcher.Invoke(inputHandle, sineResult);

Execution Environment Control Environment

slide-57
SLIDE 57

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } }; vtkm::cont::ArrayHandle<vtkm::Float32> inputHandle = vtkm::cont::make_ArrayHandle(input); vtkm::cont::ArrayHandle<vtkm::Float32> sineResult; vtkm::worklet::DispatcherMapField<Sine> dispatcher; dispatcher.Invoke(inputHandle, sineResult);

Execution Environment Control Environment

slide-58
SLIDE 58

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } }; vtkm::cont::ArrayHandle<vtkm::Float32> inputHandle = vtkm::cont::make_ArrayHandle(input); vtkm::cont::ArrayHandle<vtkm::Float32> sineResult; vtkm::worklet::DispatcherMapField<Sine> dispatcher; dispatcher.Invoke(inputHandle, sineResult);

Execution Environment Control Environment

slide-59
SLIDE 59

struct Sine: public vtkm::worklet::WorkletMapField { typedef void ControlSignature(FieldIn<>, FieldOut<>); typedef _2 ExecutionSignature(_1); template<typename T> VTKM_EXEC_EXPORT T operator()(T x) const { return vtkm::math::Sin(x); } };

slide-60
SLIDE 60

Dispatcher Invoke Operations

  • Convert polymorphic types to static types
  • Check types
  • Dispatcher-specific operations

– Find domain length – Build index arrays

  • Transport data from control to execution
  • Run worklet invoke kernel
  • Fetch thread-specific data
  • Invoke worklet
  • Push thread-specific data

DispatcherMapField<Sine> dispatcher; dispatcher.Invoke(inputHandle, sineResult);

slide-61
SLIDE 61

struct ImagToPolar: public vtkm::worklet::WorkletMapField { typedef void ControlSignature( FieldIn<vtkm::TypeListTagScalar>, FieldIn<vtkm::TypeListTagScalar>, FieldOut<vtkm::TypeListTagScalar>, FieldOut<vtkm::TypeListTagScalar>); typedef void ExecutionSignature(_1, _2, _3, _4); template<typename RealType, typename ImaginaryType, typename MagnitudeType, typename PhaseType> VTKM_EXEC_EXPORT void operator()(RealType real, ImaginaryType imag, MagnitudeType &magnitude, PhaseType &phase) const { magnitude = vtkm::math::Sqrt(real*real + imag*imag); phase = vtkm::math::ATan2(imaginary, real); } };

slide-62
SLIDE 62

What We Have So Far

  • Features

– Statically and Dynamically Typed Arrays – Device Interface (Serial, Cuda, TBB under development) – Basic Worklet and Dispatcher

slide-63
SLIDE 63

What We Have So Far

  • Compiles with

– gcc (4.8+), clang, msvc (2010+), icc, and pgi

  • User Guide
  • Ready for larger collaboration
slide-64
SLIDE 64

Thank You!

Marcus D. Hanwell

  • mhanwell@kitware.com
  • @mhanwell
  • +MarcusHanwell

Robert Maynard

  • robert.maynard@kitware.com
  • @robertjmaynard

64

Please complete the Presenter Evaluation sent to you by email or through the GTC Mobile App. Your feedback is important!

Checkout out Kitware @ www.kitware.com and ParaView @ www.paraview.org