SLIDE 1
Streaming Realtime Workflows at the Light Sources
Harinarayan Krishnan, Computer Systems Engineer Computational Research Division, Data Analysis & Visualization Group
Streaming Realtime Workflows at the Light Sources Harinarayan - - PowerPoint PPT Presentation
Streaming Realtime Workflows at the Light Sources Harinarayan - - PowerPoint PPT Presentation
Streaming Realtime Workflows at the Light Sources Harinarayan Krishnan, Computer Systems Engineer Computational Research Division, Data Analysis & Visualization Group What is a workflow? A workflow consists of an orchestrated and
SLIDE 2
SLIDE 3
A workflow consists of an orchestrated and repeatable pattern
- f business activity enabled by the systematic organization of
resources into processes that transform materials, provide services, or process information.
SLIDE 4
Page
Post processing Workflows
http://www.gridprovenance.org/applications/DLR.html
Simulation
SLIDE 5
Page
Insitu Workflows
https://www.researchgate.net/publication/320237199_Development_of_Advanced_Analysis_Toolkit_for_Turbulent_Bubbly_Flow_Sim ulations/figures?lo=1
In-situ workflows
Realtime Steering
SLIDE 6
Page
Realtime Workflows
https://www.semanticscholar.org/paper/Improved-%24%24T_%7B2%7D%5E%7B*%7D%24%24-T-2-% E2%88%97-determination-in-23Na%2C-Niesporek-Umathum/c16507726ba126426a58db5172aa222a57b50e13/figure/2
SLIDE 7
Page 7
Bright and coherent X-rays
Max IV, Lund, Sweden ALS, Berkeley, US SSRL/LCLS, Stanford, US Sprint8/SACLA, Japan Elettra/FERMI, Trieste, Italy PSI, Switzerland …and many more
SLIDE 8
Page
Data Acquisition
SLIDE 9
Page
Data Rates
Current Data Rates: 400 megabytes per second and can now generate a few terabytes of data per day – enough to store about 500 to 1,000 feature-length
- movies. Next-generation detectors, will produce data
100 times faster
SLIDE 10
Page
Higher Contrast & Time Dependence
Future DAQs are also higher contrast than current ALS beamlines. This brightness can translate to nanoscale resolution, and can also enable far more precision in time-dependent experiments
SLIDE 11
Page
“Under the ideal situation, where we have very high- contrast samples, we’ll be able to image at the x-ray wavelength, which nobody else can do. COSMIC is going to bring x-ray microscopy much closer to the capabilities of electron microscopy, but with the added benefit of x-rays, which is that you can penetrate lots of material.” ~ David Shapiro (COSMIC, ALS)
SLIDE 12
Page
Why Streaming?
Complexity
- Workflows
- Data analysis
- Resource management
- Scalability and Portability
Resources
- Experiment Timeframe is limited
- Beam time is not free
We start with a use case: Ptychography is one of the most data intensive beamlines.
12
SLIDE 13
Page
What is Ptychography?
SLIDE 14
Page
40 horizontal positions 40 vertical positions 260 1600 diffraction images 260
Ptychography is similar to Scanning Microscope but trades greater complexity for higher resolution
Making use of redundant data!
14
SLIDE 15
Page
Ptychography Pipeline
Overlap and average frames. FFT For each pixel replace magnitude with experimental value Inverse FFT Multiply Object with Probes
Iterationi
Zone Plate Lens
Ptychography Frame Stack
X-ray Beam Scan Direction
Diffraction Pattern
Scanned Sample CCD Detector
Final Output
Algorithm Beamline
15
SLIDE 16
Page
SHARP: Scalable Heterogeneous Adaptive Real-time Ptychography
CAMERA developed SHARP: A collection of algorithms packaged as useable software for Ptychographic reconstruction
- Developed New Accelerated Solvers (RAAR and ADMM) – MPI & Multi-GPU
- Combined Phase Retrieval and Denoising
- Proving Convergence and Stability with First Order ADMM
RAAR: Relaxed Averaged Alternating Reflections ADMM: Alternating Direction of Method of Multipliers
16
Ptychography image using the same data. Traditional STXM image. SEM image .
Resolution of about 10 nm.
SLIDE 17
Page
Why is this important?
US researchers have used soft X-ray Ptychography to image structures at 5 nm scale. The resolution, obtained at Berkeley Lab's Advanced Light Source, is the highest resolution ever achieved with X-ray microscopy
https://microscopy-analysis.com/editorials/editorial-listings/us-researchers-claim-x-ray-microscopy-record
US researchers claim X-ray microscope record
SLIDE 18
Page 18
Ptychography Workflow
Experiment Control 1 Trigger Reconstruction 4 CXI file Framegrabber / Camera control 2 UDP TIFF Pre-processing 3 metadata 5
SLIDE 19
Page 19
Ptychography Time Budget
Experiment Control 1 Trigger Reconstruction 4 Framegrabber / Camera control 2 UDP Pre-processing 3 5 1 frame / second
time / sample setup (s) 600
samples or time points / user
25 time/ user setup (s) 1200 Total: 1200 + (600*25) 4.5 Hours!
time / scan (detection AND exposure): 300 (s)
- Beamlines are exploratory and is run until sample or
feature is found.
- In a post processing workflow: Preprocessing &
Reconstruction can only run after all frames have been acquired! ~Minutes
SLIDE 20
Page 20
Experiment Control 1 Trigger SHARP reconstruction 4 Framegrabber / Camera control 2 UDP Pre-processing 3 5 TIFF TIFF TIFF
Ideal Streaming Workflow
SLIDE 21
Page
Iterative Reconstruction
Overlap and average frames. FFT For each pixel replace magnitude with experimental value lFFT Multiply Object with Probes
Zone Plate Lens
Ptychography Frame Stack
X-ray Beam Scan Direction
Diffraction Pattern
Scanned Sample CCD Detector
Outputi Iterationi
21
SLIDE 22
Page
Iterative Reconstruction
Overlap and average frames. FFT For each pixel replace magnitude with experimental value lFFT Multiply Object with Probes
Zone Plate Lens
Ptychography Frame Stack
X-ray Beam Scan Direction
Diffraction Pattern
Scanned Sample CCD Detector
Outputj Iterationj
22
SLIDE 23
Page
Iterative Reconstruction
Overlap and average frames. FFT For each pixel replace magnitude with experimental value lFFT Multiply Object with Probes
Zone Plate Lens
Ptychography Frame Stack
X-ray Beam Scan Direction
Diffraction Pattern
Scanned Sample CCD Detector
Outputk Iterationk
23
SLIDE 24
Page
Iterative Reconstruction
Overlap and average frames. FFT For each pixel replace magnitude with experimental value lFFT Multiply Object with Probes
Zone Plate Lens
Ptychography Frame Stack
X-ray Beam Scan Direction
Diffraction Pattern
Scanned Sample CCD Detector
Final Output Iterationn
24
SLIDE 25
How can we build a general real-time streaming pipeline?
SLIDE 26
Page
Goals
What
- Data movement
- Data Tagging/Cataloging/Querying
- Analysis & Visualization
- Data Access
When
- Pre-planning – Simulation/Modeling
- During Experiment – Quick Data Movement, Simple
analysis, or iterative analysis
- Post-planning – Archive & Compute: Typical HPC analysis
26
SLIDE 27
Page
Requirements
- An infrastructure to create and run complex analytical
pipelines
- Ability to analyze and evaluate algorithms
- Get results (or partial results) for real-time decision making
27
SLIDE 28
Page
- Scalable
- Portable
- Production-Ready
- Tomography
- Ptychography
- GiSAXS
- Image Processing
User Facilities Compute Resources Algorithms Performance
Requirements
28
SLIDE 29
Page 29
CAM-Link
What it is: A distributed task execution library
- Interface-based: Enabling different workflows to run underneath
- Coordinates communication over a distributed set of resources
- Provides tasks with information and consistent environment
- Handles security to communicate with services behind firewalls and
batch systems. Enables developers to create and run custom workflow environments
SLIDE 30
Page 30
Data Processing Pipeline
SLIDE 31
Page 31
Real-time Analysis
1.Identify and setup resources 2.Launch services 3.Connect network 4.Execute graph Execution steps
Handler / event loop Task 1 Task 2 Task N
User / local computer Compute cluster Experiment / data acquisition
SLIDE 32
Page
remote event loop remote event loop
32
User / local computer Compute cluster Experiment / data acquisition
1.Identify and setup resources 2.Launch services 3.Connect network 4.Execute graph Execution steps
master event loop
Real-time Analysis
SLIDE 33
Page
Graphical User Interface (GUI)
remote event loop
frame grabber exp. contr
- l
remote event loop
process images ptych
- SHAR
P
33
User / local computer Compute cluster Experiment / data acquisition
1.Identify and setup resources 2.Launch services 3.Connect network 4.Execute graph Execution steps
master event loop
remot e contr
- l
elog
Control Network
Real-time Analysis
SLIDE 34
Page
Graphical User Interface (GUI)
remote event loop
frame grabber exp. contr
- l
remote event loop
process images ptych
- SHAR
P
34
User / local computer Compute cluster Experiment / data acquisition
Execution steps
master event loop
remot e contr
- l
elog
1.Identify and setup resources 2.Launch services 3.Connect network 4.Execute graph
Data Network
Real-time Analysis
SLIDE 35
Page
Graphical User Interface (GUI)
remote event loop
frame grabber exp. contr
- l
remote event loop
process images ptych
- SHAR
P
35
User / local computer Compute cluster Experiment / data acquisition
Execution steps
master event loop
remot e contr
- l
elog
1.Identify and setup resources 2.Launch services 3.Connect network 4.Execute Graph
10G Network Visualize Results Write Output
Real-time Analysis
SLIDE 36
Page 36
Real-time Analysis
SLIDE 37
Streaming workflows are necessary for many
- ther applications
SLIDE 38
Page
(GUI)
master event loop
Dask Client
remote event loop
Dask Server
38
Decision-based: Streaming Tomography
User / local computer Compute cluster
Execution steps 1.Send Workflow 2.Execute Tasks 3.Dynamically decide which tasks to execute next
Write Output
Dask Workers Dask Workers Dask Workers Tomogram Stream Task N Task 3 Task 2 Task 2 Task 2
Conditional
Task N
Experiment / data acquisition
SLIDE 39
Page
Graphical User Interface
master event loop
Dask Client
remote event loop
Dask Server
39
Remote Execution: Tomography
User / local computer Compute cluster Experiment / data acquisition
Execution steps 1.Send Workflow 2.Execute Graph 3.Return result 4.Visualization updates
Write Output
Dask Workers Dask Workers Dask Workers
1.Read 2.Normalize 3.Remote Outlier 4.Remove Stripe 5.Padding 6.Reconstruction 7.Crop 8.Circular Mask 9.Output
Tomography
SLIDE 40
Page 40
Scalable Execution: Grazing-incidence small-angle scattering (GISAXS)
User / local computer
Compute cluster
Execution steps
1.Locally Construct Model 2.Run algorithm at Compute Facility 3.Return result
Write Output
(GUI)
master event loop
Dask Client
remote event loop
Dask Server
Model1
Dask Workers Dask Workers
Piz-Daint
- Automated Port Discovery
- Resource Management on Super Computer
- Results directed to visualization system
(GUI)
master event loop
Dask Client
remote event loop
Dask Server
Modelj
Dask Workers Dask Workers
(GUI)
master event loop
Dask Client
remote event loop
Dask Server
Modeln
Dask Workers Dask Workers Dask Workers
Over 200+ Miles!
SLIDE 41
Page
SLIDE 42
Page
3D: Ptycho-Tomography
https://www.nature.com/articles/s41467-018-03401-x Ptycho-Tomography scan resolving chemical states in three dimensions at 11 nm spatial resolution
SLIDE 43
Page 43
Conclusions
- Easy startup and use
- Distributed processing environments
- Complex communication patterns
- Support for consistent software ecosystem
- Stand alone or extended into other frameworks
A framework for real-time data processing
- B. J. Daurer, H. Krishnan, T. Perciano, F. R. N. C. Maia, D. A. Shapiro, J. A. Sethian and S. Marchesini
Nanosurveyor: a framework for real-time data processing Advanced Structural and Chemical Imaging 3:7 (2017).
SLIDE 44
Page