Streaming Realtime Workflows at the Light Sources Harinarayan - PowerPoint PPT Presentation

Streaming Realtime Workflows at the Light Sources Harinarayan Krishnan, Computer Systems Engineer Computational Research Division, Data Analysis & Visualization Group

What is a workflow?

A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic organization of resources into processes that transform materials, provide services, or process information.

Post processing Workflows Simulation http://www.gridprovenance.org/applications/DLR.html Page

Insitu Workflows In-situ workflows Realtime Steering https://www.researchgate.net/publication/320237199_Development_of_Advanced_Analysis_Toolkit_for_Turbulent_Bubbly_Flow_Sim ulations/figures?lo=1 Page

Realtime Workflows https://www.semanticscholar.org/paper/Improved-%24%24T_%7B2%7D%5E%7B*%7D%24%24-T-2-% E2%88%97-determination-in-23Na%2C-Niesporek-Umathum/c16507726ba126426a58db5172aa222a57b50e13/figure/2 Page

Bright and coherent X-rays ALS, Berkeley, US Sprint8/SACLA, Japan Max IV, Lund, Sweden …and many more SSRL/LCLS, Stanford, US PSI, Switzerland Elettra/FERMI, Trieste, Italy Page 7

Data Acquisition Page

Data Rates Current Data Rates: 400 megabytes per second and can now generate a few terabytes of data per day – enough to store about 500 to 1,000 feature-length movies. Next-generation detectors, will produce data 100 times faster Page

Higher Contrast & Time Dependence Future DAQs are also higher contrast than current ALS beamlines. This brightness can translate to nanoscale resolution, and can also enable far more precision in time-dependent experiments Page

“ Under the ideal situation, where we have very high- contrast samples, we’ll be able to image at the x-ray wavelength, which nobody else can do. COSMIC is going to bring x-ray microscopy much closer to the capabilities of electron microscopy, but with the added benefit of x-rays, which is that you can penetrate lots of material. ” ~ David Shapiro (COSMIC, ALS) Page

Why Streaming? Complexity • Workflows • Data analysis • Resource management • Scalability and Portability Resources • Experiment Timeframe is limited • Beam time is not free We start with a use case: Ptychography is one of the most data intensive beamlines. Page 12

What is Ptychography? Page

Ptychography is similar to Scanning Microscope but trades greater complexity for higher resolution 40 horizontal 1600 diffraction images positions 260 40 vertical positions 260 Making use of redundant data! Page 14

Ptychography Pipeline Scanned Sample Diffraction Pattern Beamline Ptychography Zone Plate Lens Frame Stack X-ray Beam Final CCD Output Scan Detector Direction For each pixel replace magnitude Multiply Object Iteration i with experimental value with Probes FFT Overlap and Algorithm Inverse FFT average frames. Page 15

SHARP: Scalable Heterogeneous Adaptive Real-time Ptychography Ptychography image using the same data. Traditional STXM SEM image. image . Resolution of about 10 nm. CAMERA developed SHARP: A collection of algorithms packaged as useable software for Ptychographic reconstruction • Developed New Accelerated Solvers (RAAR and ADMM) – MPI & Multi-GPU • Combined Phase Retrieval and Denoising • Proving Convergence and Stability with First Order ADMM RAAR: Relaxed Averaged Alternating Reflections ADMM: Alternating Direction of Method of Multipliers Page 16

Why is this important? US researchers claim X-ray microscope record US researchers have used soft X-ray Ptychography to image structures at 5 nm scale . The resolution, obtained at Berkeley Lab's Advanced Light Source, is the highest resolution ever achieved with X-ray microscopy https://microscopy-analysis.com/editorials/editorial-listings/us-researchers-claim-x-ray-microscopy-record Page

Ptychography Workflow 5 UDP Trigger 4 1 2 3 Experiment Framegrabber / Reconstruction Pre-processing Control Camera control TIFF CXI file metadata Page 18

Ptychography Time Budget 5 UDP Trigger 4 1 2 3 Experiment Framegrabber / Reconstruction Pre-processing Control Camera control ~Minutes 1 frame / second time / scan (detection AND exposure): 300 (s) • Beamlines are exploratory and is run until sample or time / sample setup (s) 600 feature is found. samples or time points / user 25 • In a post processing workflow: Preprocessing & time/ user setup (s) 1200 Reconstruction can only run after all frames have been Total: 1200 + (600*25) 4.5 Hours! acquired! Page 19

Ideal Streaming Workflow 5 UDP Trigger 4 1 2 3 SHARP Experiment Framegrabber / Pre-processing reconstruction Control Camera control TIFF TIFF TIFF Page 20

Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Output i CCD Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration i with Probes FFT Overlap and lFFT average frames. Page 21

Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Output j CCD Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration j with Probes FFT Overlap and lFFT average frames. Page 22

Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Output k CCD Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration k with Probes FFT Overlap and lFFT average frames. Page 23

Iterative Reconstruction Scanned Sample Diffraction Pattern Ptychography Zone Plate Lens Frame Stack X-ray Beam Final CCD Output Scan Detector Direction For each pixel replace magnitude with experimental value Multiply Object Iteration n with Probes FFT Overlap and lFFT average frames. Page 24

How can we build a general real-time streaming pipeline?

Goals What • Data movement • Data Tagging/Cataloging/Querying • Analysis & Visualization • Data Access When • Pre-planning – Simulation/Modeling • During Experiment – Quick Data Movement, Simple analysis, or iterative analysis • Post-planning – Archive & Compute: Typical HPC analysis Page 26

Requirements • An infrastructure to create and run complex analytical pipelines • Ability to analyze and evaluate algorithms • Get results (or partial results) for real-time decision making Page 27

Requirements User Compute Facilities Resources Performance Algorithms • Scalable • Tomography • Portable • Ptychography • Production-Ready • GiSAXS • Image Processing Page 28

CAM-Link What it is: A distributed task execution library • Interface-based: Enabling different workflows to run underneath • Coordinates communication over a distributed set of resources • Provides tasks with information and consistent environment • Handles security to communicate with services behind firewalls and batch systems. Enables developers to create and run custom workflow environments Page 29

Data Processing Pipeline Page 30

Real-time Analysis Task N Task 2 Execution steps Task 1 1.Identify and setup resources 2.Launch services Handler / event loop 3.Connect network 4.Execute graph User / local computer Compute cluster Experiment / data acquisition Page 31

Real-time Analysis Experiment / data acquisition remote event loop Compute cluster remote event loop Execution steps 1.Identify and setup resources master event loop 2.Launch services 3.Connect network 4.Execute graph User / local computer Page 32

Real-time Analysis Experiment / data acquisition remote event loop process images ptych Compute cluster o SHAR P remote frame event loop grabber Control Network exp. contr ol elog Execution steps remot e 1.Identify and setup resources contr master ol event loop 2.Launch services Graphical User Interface (GUI) 3.Connect network 4.Execute graph User / local computer Page 33

Real-time Analysis Experiment / data acquisition remote event loop process images ptych Compute cluster o SHAR P remote frame event loop grabber Data Network exp. contr ol elog Execution steps remot e 1.Identify and setup resources contr master ol event loop 2.Launch services Graphical User Interface (GUI) 3.Connect network 4.Execute graph User / local computer Page 34

Real-time Analysis Experiment / data acquisition remote event loop process images ptych Compute cluster o SHAR P 10G Network remote frame event loop grabber Visualize Results exp. contr ol Write Output elog Execution steps remot e 1.Identify and setup resources contr master ol event loop 2.Launch services Graphical User Interface (GUI) 3.Connect network 4.Execute Graph User / local computer Page 35

Real-time Analysis Page 36

Streaming workflows are necessary for many other applications