CNN-based Feature Descriptors Mengyu Chu, Nils Thuerey Technical - - PowerPoint PPT Presentation

Mengyu Chu, Nils Thuerey

Technical University of Munich

Data-Driven Synthesis of Smoke Flows with CNN-based Feature Descriptors

Introduction

• High resolution smoke generation
• Numerical viscosity
• Expensive calculations

Introduction

• Related work

Proposed approach

Overview

Descriptor learning

CNN CNN

Overview

Descriptor learning Deformation- limiting advection

CNN CNN

Overview

Fluid repository

...

Descriptor learning Deformation- limiting advection

CNN CNN

Overview

Fluid repository

...

Descriptor learning Volumetric Synthesis Deformation- limiting advection

CNN CNN

Overview

Fluid repository

...

Volumetric Synthesis Deformation- limiting advection Descriptor learning

CNN CNN

• Descriptor learning

Learning flow similarity

• Descriptor learning

CNN CNN

Learning flow similarity

• Descriptor learning

– Input: pair of fluid data

CNN CNN

Learning flow similarity

• Descriptor learning

– Input: pair of fluid data – Output: similarity (scalar)

CNN CNN

Learning flow similarity

• Descriptor learning

– Input: pair of fluid data – Output: similarity (scalar) – Flow similarity, 1 as similar, -1 as dissimilar

CNN CNN

Learning flow similarity

• Descriptor learning

– Input: pair of fluid data – Output: similarity (scalar) – Flow similarity, 1 as similar, -1 as dissimilar – Labelled input pairs

CNN CNN

𝑧 = 1

CNN CNN

𝑧 = −1

Learning flow similarity

Learning flow similarity

Learning flow similarity

Learning flow similarity

𝑧 = 1

Learning flow similarity

𝑧 = −1

Learning flow similarity

• Structure

Learning flow similarity

• Structure

Siamese structure

L2

Learning flow similarity

• Structure

Siamese structure

Shared weights

L2

Learning flow similarity

• Structure

Siamese structure Descriptor learning

– Invariants

• resolution
• numerical

viscosity

Shared weights

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

Descriptor space

Descriptor space

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

N

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

Descriptor space N

Descriptor space

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

N

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

Descriptor space N t loss training

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function —— Hinge loss

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

N

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function —— Hinge loss

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

N Descriptor space 𝑏𝑞 𝑏𝑜 2

Learning flow similarity

• CNN structure —— Siamese structure
• Loss function —— Hinge loss

P N

𝑚𝑓 𝑦1, 𝑦2 = ൝max 0, −𝑏𝑞 + 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = 1 max 0, 𝑏𝑜 − 𝑒𝑥 𝑦1 − 𝑒𝑥 𝑦2 , 𝑧 = −1

N Descriptor space 𝑏𝑞 𝑏𝑜 2 t loss training

Patch advection

• Error minimization problem

𝐹 = 𝜇𝐹𝑒𝑓𝑔𝑝 + 𝐹𝑏𝑒𝑤

Patch advection

• Error minimization problem

𝐹 = 𝜇𝐹𝑒𝑓𝑔𝑝 + 𝐹𝑏𝑒𝑤 – 𝐹𝑏𝑒𝑤 = σ 𝑤𝑗 − 𝑤𝑗′ 2 , 𝑤′ = 𝑏𝑒𝑤 𝑤𝑢−1 – 𝐹𝑒𝑓𝑔𝑝 = σ 𝑤𝑗 − 𝑤𝑗∗ 2

• 𝑤𝑗∗, based on Laplacian coordinates

[Sorkine et al. 2004]

V

3

V

1

V2 V

V

3'

V

3

V

3

'

= σ 𝑤𝑗 − σ 𝐵𝑘𝑤𝑘

2

Patch advection

Patch anticipation

• Fading in → Anticipation

Patch anticipation

• Fading in → Anticipation
• Fading out ill-suited ones

Patch anticipation

• Fading in → Anticipation
• Fading out ill-suited ones

Normal fading in Patch anticipation

Fluid repository

... ...

Volumetric Synthesis

Patch advection

• Fluid repository

– Space-time data

• Synthesis

– Reusing the repository

• Lagrangian

– Stable & reusable – Resolution independent

Overview

Fluid repository

...

Descriptor learning Deformation- limiting advection

CNN CNN

Volumetric Synthesis

Synthesis

Simulation:

• Forward pass

– Sampling, matching

• Backward pass

Synthesis

Simulation:

• Forward pass

– Sampling, matching

• Backward pass

Synthesis

Simulation:

• Forward pass

– Sampling, matching

• Backward pass

Synthesis

Simulation:

• Forward pass

– Sampling, matching

• Backward pass

Synthesis

Simulation:

• Forward pass

– Sampling, matching – Forward advection

• Backward pass

Synthesis

Simulation:

• Forward pass

– Sampling, matching – Forward advection – Fading out ill-suited

• Backward pass

Synthesis

Simulation:

• Forward pass

– Sampling, matching – Forward advection – Fading out ill-suited

• Backward pass

– Backward anticipation & advection

Synthesis

Simulation:

• Forward pass

– Sampling, matching – Forward advection – Fading out ill-suited

• Backward pass

– Backward anticipation & advection

Advantages: – Calculation: Coarse resolution – Storage:

• Descriptors only
• Output: patch ID, cage vertices' pos, fading weights

Synthesis

Rendering:

• Loading patches,

– fading weights – spatial weights

Synthesis

Rendering:

• Loading patches,

– fading weights – spatial weights

• Normalization

>1

Synthesis

Rendering:

• Loading patches,

– fading weights – spatial weights

• Normalization

>1

Synthesis

Rendering:

• Loading patches,

– fading weights – spatial weights

• Normalization
• Independent frames

>1

Evaluation

Descriptor evaluation

• Recall over rank

10 20 30 40 50 60 70 80 90 1 11 21 31 41 51 61 71

% Rank

2D

HOG descriptors CNN density descriptors CNN density and curl combined descriptors

10 20 30 40 50 60 70 80 90 1 11 21 31 41 51 61 71

% Rank

3D

CNN density descriptors CNN density and curl combined descriptor

—— the percentage of correctly matched pairs within a given rank More discriminative!

Descriptor evaluation

Input Density descriptor only Density and curl descriptors

Descriptor evaluation

Input Density descriptor only Density and curl descriptors

Descriptor evaluation

Density descriptor only Density and curl descriptors

More results

Results

Results

Results

Results

Conclusion

Discussions

• Contributions

– CNN fluid descriptors – Patch advection – Fluid repository – Synthesis

• Limitations

– Fully divergence-free

• Velocity synthesis

– Spatial blending – Storage

Future directions

• More data-driven approaches
• Neural networks

Patch Advection Synthesis CNN Descriptors Repository

Neural networks

Thank you!

More information: http://ge.in.tum.de/publications/2017-sig-chu/ Code online: https://github.com/RachelCmy/mantaPatch/