CNN-based Feature Descriptors Mengyu Chu, Nils Thuerey Technical - PowerPoint PPT Presentation

Data-Driven Synthesis of Smoke Flows with CNN-based Feature Descriptors Mengyu Chu, Nils Thuerey Technical University of Munich

Introduction • High resolution smoke generation • Numerical viscosity • Expensive calculations

Introduction • Related work

Proposed approach

Overview Descriptor learning CNN CNN

Overview Descriptor learning CNN CNN Deformation- limiting advection

Overview Descriptor learning Fluid repository CNN CNN Deformation- limiting advection ...

Overview Descriptor learning Volumetric Synthesis Fluid repository CNN CNN Deformation- limiting advection ...

Overview Descriptor learning Volumetric Synthesis Fluid repository CNN CNN Deformation- limiting advection ...

Learning flow similarity • Descriptor learning

Learning flow similarity • Descriptor learning CNN CNN

Learning flow similarity • Descriptor learning – Input: pair of fluid data CNN CNN

Learning flow similarity • Descriptor learning – Input: pair of fluid data – Output: similarity (scalar) CNN CNN

Learning flow similarity • Descriptor learning – Input: pair of fluid data – Output: similarity (scalar) – Flow similarity, 1 as similar, -1 as dissimilar CNN CNN

Learning flow similarity • Descriptor learning – Input: pair of fluid data – Output: similarity (scalar) – Flow similarity, 1 as similar, -1 as dissimilar – Labelled input pairs CNN CNN CNN CNN 𝑧 = 1 𝑧 = −1

Learning flow similarity

Learning flow similarity

Learning flow similarity 𝑧 = 1

Learning flow similarity 𝑧 = −1

Learning flow similarity • Structure

Learning flow similarity • Structure Siamese structure

Learning flow similarity • Structure Siamese structure Shared L 2 weights

Learning flow similarity • Structure Siamese structure Descriptor learning – Invariants • resolution • Shared numerical L 2 weights viscosity

Learning flow similarity • CNN structure —— Siamese structure • Loss function 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1

Learning flow similarity • CNN structure —— Siamese structure • Loss function 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 P N Descriptor space

Learning flow similarity • CNN structure —— Siamese structure • Loss function 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 P N N Descriptor space

Learning flow similarity • CNN structure —— Siamese structure • Loss function 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 P N N Descriptor space

Learning flow similarity • CNN structure —— Siamese structure • Loss function 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 P N N Descriptor space

Learning flow similarity • CNN structure —— Siamese structure • Loss function 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 P N N Descriptor space loss t training

Learning flow similarity • CNN structure —— Siamese structure • Loss function —— Hinge loss 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 P N N

Learning flow similarity • CNN structure —— Siamese structure • Loss function —— Hinge loss 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 0 𝑏 𝑞 P N N Descriptor space 𝑏 𝑜 2

Learning flow similarity • CNN structure —— Siamese structure • Loss function —— Hinge loss 𝑚 𝑓 𝑦 1 , 𝑦 2 = ൝ max 0, −𝑏 𝑞 + 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = 1 max 0, 𝑏 𝑜 − 𝑒 𝑥 𝑦 1 − 𝑒 𝑥 𝑦 2 , 𝑧 = −1 0 𝑏 𝑞 P N N Descriptor space loss 𝑏 𝑜 t 2 training

Patch advection • Error minimization problem 𝐹 = 𝜇𝐹 𝑒𝑓𝑔𝑝 + 𝐹 𝑏𝑒𝑤

Patch advection • Error minimization problem 𝐹 = 𝜇𝐹 𝑒𝑓𝑔𝑝 + 𝐹 𝑏𝑒𝑤 – 𝐹 𝑏𝑒𝑤 = σ 𝑤 𝑗 − 𝑤 𝑗 ′ 2 , 𝑤 ′ = 𝑏𝑒𝑤 𝑤 𝑢−1 2 – 𝐹 𝑒𝑓𝑔𝑝 = σ 𝑤 𝑗 − 𝑤 𝑗 ∗ 2 = σ 𝑤 𝑗 − σ 𝐵 𝑘 𝑤 𝑘 • 𝑤 𝑗∗ , based on Laplacian coordinates [Sorkine et al. 2004] ' V 3 V V 1 3 3 ' V V 0 V 3 V 2

Patch advection

Patch anticipation • Fading in → Anticipation

Patch anticipation • Fading in → Anticipation • Fading out ill-suited ones

Patch anticipation • Fading in → Anticipation • Fading out ill-suited ones Normal fading in Patch anticipation

Patch advection • Fluid repository Fluid Volumetric Synthesis – Space-time data repository • Synthesis – Reusing the repository • Lagrangian – Stable & reusable – Resolution independent ... ...

Overview Descriptor learning Volumetric Synthesis Fluid repository CNN CNN Deformation- limiting advection ...

Synthesis Simulation: • Forward pass – Sampling, matching • Backward pass

Synthesis Simulation: • Forward pass – Sampling, matching • Backward pass

Synthesis Simulation: • Forward pass – Sampling, matching • Backward pass

Synthesis Simulation: • Forward pass – Sampling, matching • Backward pass

Synthesis Simulation: • Forward pass – Sampling, matching – Forward advection • Backward pass

Synthesis Simulation: • Forward pass – Sampling, matching – Forward advection – Fading out ill-suited • Backward pass

Synthesis Simulation: • Forward pass – Sampling, matching – Forward advection – Fading out ill-suited • Backward pass – Backward anticipation & advection

Synthesis Simulation: • Forward pass – Sampling, matching – Forward advection – Fading out ill-suited • Backward pass – Backward anticipation & advection Advantages: – Calculation: Coarse resolution – Storage: • Descriptors only • Output: patch ID, cage vertices' pos, fading weights

Synthesis Rendering: • Loading patches, – fading weights – spatial weights

Synthesis Rendering: • Loading patches, – fading weights – spatial weights • Normalization >1

Synthesis Rendering: • Loading patches, – fading weights – spatial weights • Normalization >1

Synthesis Rendering: • Loading patches, – fading weights – spatial weights • Normalization • Independent frames >1

Evaluation

Descriptor evaluation • Recall over rank —— the percentage of correctly matched pairs within a given rank 2D 3D More discriminative! % % 90 90 80 80 70 70 60 60 50 50 40 40 HOG descriptors CNN density descriptors 30 30 CNN density descriptors 20 20 CNN density and curl combined CNN density and curl combined 10 10 descriptor descriptors 0 0 Rank Rank 1 11 21 31 41 51 61 71 1 11 21 31 41 51 61 71

Descriptor evaluation Density and curl descriptors Input Density descriptor only

Descriptor evaluation Density and curl descriptors Input Density descriptor only

Descriptor evaluation Density descriptor only Density and curl descriptors

More results

Results

Results

Results

Results

Conclusion

Discussions • Contributions • Limitations – Fully divergence-free – CNN fluid descriptors • Velocity synthesis – Patch advection – Spatial blending – Fluid repository – Storage – Synthesis

Future directions CNN Descriptors Repository Neural networks Synthesis Patch Advection • More data-driven approaches • Neural networks

Thank you! More information: http://ge.in.tum.de/publications/2017-sig-chu/ Code online: https://github.com/RachelCmy/mantaPatch/