Learning Similarity Metrics for Numerical Simulations Georg Kohl - PowerPoint PPT Presentation

Learning Similarity Metrics for Numerical Simulations Georg Kohl Kiwon Um Nils Thuerey Technical University of Munich

Overview – Motivation Similarity assessment of scalar 2D simulation data from PDEs

Overview – Motivation Similarity assessment of scalar 2D simulation data from PDEs → → →

Overview – Motivation Typical metrics like L¹ or L² operate locally → structures and patterns are ignored Recognition of spatial contexts with CNNs Mathematical metric properties should be considered → 0.1 0.1 0.1 0.2

Overview – Method Numerical Simulation [ p i p i + 1 Δ i p i + 2 Δ i p i + 10 Δ i ] ⋯ Initial condition varied in isolation Variation defines Simulation with PDE Solver ⋯ Data sequence [ 0.0 0.1 0.2 1.0 ] ⋯ Ground truth distances

Overview – Method Numerical Simulation Learning CNN [ p i p i + 1 Δ i p i + 2 Δ i p i + 10 Δ i ] ⋯ Initial condition varied in isolation Feature Shared Pairing normalization weights Feature CNN comparison Variation defines Simulation with PDE Solver [ 0.01 0.12 0.24 0.99 ] ⋯ Learned distances ⋯ Data sequence [ 0.0 0.1 0.2 1.0 ] ⋯ Ground truth distances

Overview – Method Numerical Simulation Learning CNN [ p i p i + 1 Δ i p i + 2 Δ i p i + 10 Δ i ] ⋯ Initial condition varied in isolation Feature Shared Pairing normalization weights Feature CNN comparison Variation defines Simulation with PDE Solver Optimization [ 0.01 0.12 0.24 0.99 ] ⋯ Loss Learned distances ⋯ Data sequence [ 0.0 0.1 0.2 1.0 ] ⋯ Ground truth distances

Overview – Method Numerical Simulation Learning CNN [ p i p i + 1 Δ i p i + 2 Δ i p i + 10 Δ i ] ⋯ Initial condition varied in isolation Feature Shared Pairing normalization weights Feature CNN comparison Variation defines Simulation with PDE Solver Optimization [ 0.01 0.12 0.24 0.99 ] ⋯ Loss Learned distances ⋯ Data sequence Evaluation [ 0.0 0.1 0.2 1.0 ] ⋯ [ 0.95 ] Ground truth distances Spearman’s ranking correlation

Overview – Results Single example: distance comparison Plume (a) Reference Plume (b) Distance to Reference 1 Plume (a) Plume (b) 0 LSiM (ours) L² Ground Truth

Overview – Results Single example: distance comparison Combined test data: correlation evaluation Plume (a) Reference Plume (b) 0.80 Spearman's rank correlation 0.70 Distance to Reference 0.60 1 Plume (a) 0.50 Plume (b) 0 0.40 LSiM (ours) L² Ground Truth L² SSIM LPIPS LSiM (ours)

Related Work “Shallow” vector space metrics Metrics induced by L p -norms, peak signal-to-noise ratio (PSNR) – Structural similarity index (SSIM) [Wang04] – Evaluation with user studies for PDE data Liquid simulations [Um17] – Non-oscillatory discretization schemes [Um19] – Image-based deep metrics with CNNs E.g. learned perceptual image patch similarity (LPIPS) [Zhang18] – [Wang04] Wang, Bovik, Sheikh, and Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing , 2004 [Um17] Um, Hu, and Thuerey. Perceptual Evaluation of Liquid Simulation Methods. ACM Transactions on Graphics , 2017 [Um19] Um, Hu, Wang, and Thuerey. Spot the Difference: Accuracy of Numerical Simulations via the Human Visual System. CoRR, abs/1907.04179 , 2019 [Zhang18] Zhang, Isola, Efros, Shechtman, and Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR , 2018

Data Generation Time depended, motion-based PDE with one varied initial condition t 1 t 2 t t o 0 [ p 0 p 1 p i ] ⋯ Initial conditions Finite difference solver with time discretization Output

Data Generation Time depended, motion-based PDE with one varied initial condition t 1 t 2 t t o 0 [ p 0 p 1 p i ] ⋯ Increasing parameter change Decreasing output similarity t 1 t 2 t t o 1 [ p 0 p 1 p i +Δ i ] ⋯ t 1 t 2 t t [ p 0 p 1 p i + n ⋅Δ i ] ⋯ o n Initial conditions Finite difference solver with time discretization Output

Data Generation Time depended, motion-based PDE with one varied initial condition Chaotic behavior in controlled environment → added noise to adjust data difficulty t 1 t 2 t t o 0 [ p 0 p 1 p i ] ⋯ Increasing parameter change Decreasing output similarity noise 1,1 ( s ) noise 1,2 ( s ) noise 1 ,t ( s ) t 1 t 2 t t o 1 [ p 0 p 1 p i +Δ i ] ⋯ noise 2,1 ( s ) noise 2,2 ( s ) noise 2 ,t ( s ) t 1 t 2 t t [ p 0 p 1 p i + n ⋅Δ i ] ⋯ o n noise n ,1 ( s ) noise n ,2 ( s ) noise n,t ( s ) Initial conditions Finite difference solver with time discretization Output

Training Data Eulerian smoke plume Liquid via FLIP [Zhu05] Advection-diffusion transport Burger’s equation [Zhu05] Zhu and Bridson. Animating sand as a fluid. ACM SIGGRAPH , 2005

Test Data Liquid (background noise) Advection-diffusion transport (density) Shape data Video data TID2013 [Ponomarenko15] [Ponomarenko15] Ponomarenko, Jin, Ieremeiev, et al. Image database TID2013: Peculiarities, results and perspectives. Signal Processing-Image Communication , 2015

Method – Base Network Siamese architecture (shared weights) → Convolution + ReLU layers Feature extraction from both inputs Existing network possible → specialized model works better Base Network Base Network Feature maps: sets of 3 rd order tensors

Method – Feature Normalization Adjust value range of feature vectors along channel dimension Unit length normalization → cosine distance (only angle comparison) – Element-wise std. normal distribution → angle and length in global length distribution – Base Feature map Network normalization Base Feature map Network normalization Feature maps: sets of 3 rd order tensors

Method – Latent Space Difference Actual comparison of feature maps → element-wise distance Must be a metric w.r.t. the latent space → ensure metric properties | ~ x −~ 2 (~ x −~ y | y ) or are useful options Base Feature map Network normalization Element-wise latent space difference Base Feature map Network normalization Feature maps: Difference maps: sets of 3 rd order tensors set of 3 rd order tensors

Method – Aggregations Compression of difference maps to scalar distance prediction Learned channel aggregation via weighted average Simple aggregations with sum or average 1 Learned weight per feature map Base Feature map Network normalization Distance Element-wise Channel Spatial Layer output latent space aggregation: aggregation: aggregation: difference weighted avg. average summation Base Feature map Network normalization d 1 d 2 d 3 d Feature maps: Difference maps: Average maps: Layer distances: Result: sets of 3 rd order tensors set of 3 rd order tensors set of 2 nd order tensors set of scalars scalar

Loss Function Ground truth distances c and predicted distances d 2 + λ 2 ( 1 − ( c −¯ c )⋅ ( d −¯ d ) 2 ) L ( c ,d ) = λ 1 ( c − d ) ‖ c −¯ c ‖ 2 ‖ d −¯ d ‖ Mean squared error term → minimize distance deviation directly Inverted correlation term → maximize linear distance relationship

Results Evaluation with Spearman’s rank correlation Ground truth against predicted distances Validation data sets Test data sets Metric Smo Liq Adv Bur TID LiqN AdvD Sha Vid All L 2 0.66 0.80 0.74 0.62 0.82 0.73 0.57 0.58 0.79 0.61 SSIM 0.69 0.74 0.77 0.71 0.77 0.26 0.69 0.46 0.75 0.53 LPIPS 0.63 0.68 0.68 0.72 0.86 0.50 0.62 0.84 0.83 0.66 LSiM 0.78 0.82 0.79 0.75 0.86 0.79 0.58 0.88 0.81 0.73

Real-world Evaluation ScalarFlow [Eckert19] WeatherBench [Rasp20] Johns Hopkins Turbulence Database (JHTDB) [Perlman07] [Eckert19] Eckert, Um, and Thuerey. Scalarflow: A large-scale volumetric data set of real-world scalar transport flows [...]. ACM Transactions on Graphics , 2019 [Rasp20] Rasp, Dueben, Scher, Weyn, Mouatadid, and Thuerey. Weatherbench: A benchmark dataset for data-driven weather forecasting. CoRR, abs/2002.00469, 2020 [Perlman07] Perlman, Burns, Li, and Meneveau. Data exploration of turbulence simulations using a database cluster. ACM/IEEE Conference on Supercomputing , 2007

Real-world Evaluation ScalarFlow JHTDB WeatherBench Retrieve order of spatial and temporal frame translations Average Spearman correlation 1.0 Six interval spacings per data repository 0.9 180-240 sequences each Mean and standard deviation over 0.8 correlation of each spacing 0.7 0.6 L² SSIM LPIPS LSiM (ours)

Future Work Accuracy assessment of new simulation methods Parameter reconstructions of observed behavior Guiding generative models of physical systems Extensions to other data 3D flows and further PDEs – Multi-channel turbulence data –

Thank you for your attention! Join the live-sessions for questions and discussion Source code available at https://github.com/tum-pbs/LSIM