Lecture 12 - Filtering Welcome! , = (, ) , - - PowerPoint PPT Presentation

β–Ά
lecture 12 filtering
SMART_READER_LITE
LIVE PREVIEW

Lecture 12 - Filtering Welcome! , = (, ) , - - PowerPoint PPT Presentation

Nov 21, 2022 331 likes β€’807 views

INFOMAGR Advanced Graphics Jacco Bikker - November 2019 - February 2020 Lecture 12 - Filtering Welcome! , = (, ) , + , , ,

slide-1
SLIDE 1

𝑱 π’š, π’šβ€² = 𝒉(π’š, π’šβ€²) 𝝑 π’š, π’šβ€² + ΰΆ±

𝑻

𝝇 π’š, π’šβ€², π’šβ€²β€² 𝑱 π’šβ€², π’šβ€²β€² π’†π’šβ€²β€²

INFOMAGR – Advanced Graphics

Jacco Bikker - November 2019 - February 2020

Lecture 12 - β€œFiltering”

Welcome!

slide-2
SLIDE 2

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-9
SLIDE 9

Path Tracing

The rendering equation allows us to accurately simulate light transport: β–ͺ Given a sensible scene description β–ͺ Given a sensible camera model β–ͺ Ignoring some complex effects. We evaluate it using Monte Carlo integration:

βž” The result is always an estimate,

with variance.

Noise

Advanced Graphics – Filtering 9

en.wikipedia.org/wiki/Path_tracing

slide-10
SLIDE 10

Path Tracing

The rendering equation allows us to accurately simulate light transport: β–ͺ Given a sensible scene description β–ͺ Given a sensible camera model β–ͺ Ignoring some complex effects. We evaluate it using Monte Carlo integration:

βž” The result is always an estimate,

with variance.

Noise

Advanced Graphics – Filtering 10

www.mitsuba-renderer.org

slide-11
SLIDE 11

Noise in the Real World

Noise in a digital photo can be the result

  • f short exposure.

Noise

Advanced Graphics – Filtering 11

www.photoreview.com.au/tips/shooting/how-to-control-image-noise

slide-12
SLIDE 12

Noise in the Real World

Noise in a digital photo can be the result

  • f short exposure.

We solve this by lowering our ISO settings, so we gather more photons per pixel. (that obviously has consequences)

Noise

Advanced Graphics – Filtering 12

www.photoreview.com.au/tips/shooting/how-to-control-image-noise

slide-13
SLIDE 13

Reducing Noise in Path Tracing

We can increase β€˜exposure’ in a pathtracer by taking more samples.

Noise

Advanced Graphics – Filtering 13

aras-p.info/blog/2018/03/28/Daily-Pathtracer-Part-1-Initial-C-/

slide-14
SLIDE 14

Reducing Noise in Path Tracing

We can increase β€˜exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling.

Noise

Advanced Graphics – Filtering 14

slide-15
SLIDE 15

Reducing Noise in Path Tracing

We can increase β€˜exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling. Or Next Event Estimation. However, at interactive rates it will be quite hard to reach a converged image.

Noise

Advanced Graphics – Filtering 15

slide-16
SLIDE 16

Noise

Advanced Graphics – Filtering 16 well-lit, diffuse: low noise depth of field: moderate noise Indirect light: moderate noise

? ?

slide-17
SLIDE 17

Not All Noise is Equal

In general, if we can importance sample it, noise will be low: β–ͺ sample BRDF proportional to cos πœ„; β–ͺ sample a light proportional to potential contribution. The β€˜fireflies’ are the result of a random bounce, followed by a specular bounce, which then hit the light: a direction is sampled with a low probability, but yields a high value.

Noise

Advanced Graphics – Filtering 17

slide-18
SLIDE 18

Noise

Advanced Graphics – Filtering 18 ΰΆ± 𝑔 𝑦 𝑒𝑦 = 𝐹 𝑔 𝑦 β‰ˆ 1 𝑂 ෍

𝑗=1 𝑂

𝑔 π‘Œπ‘— 𝐹 𝑔 𝑦 β‰ˆ 1 𝑂 ෍

𝑗=1 𝑂 𝑔 π‘Œπ‘—

π‘ž π‘Œπ‘—

π‘šπ‘π‘ π‘•π‘“ π‘€π‘π‘šπ‘£π‘“ π‘‘π‘›π‘π‘šπ‘š π‘€π‘π‘šπ‘£π‘“ = 𝑀𝑓𝑠𝑧 π‘šπ‘π‘ π‘•π‘“ π‘šπ‘π‘ π‘•π‘“ π‘€π‘π‘šπ‘£π‘“ π‘šπ‘π‘ π‘•π‘“ π‘€π‘π‘šπ‘£π‘“ β‰ˆ 1

slide-19
SLIDE 19

Not All Noise is Equal

A good RNG is important when producing samples.

Noise

Advanced Graphics – Filtering 19

slide-20
SLIDE 20

Not All Noise is Equal

A good RNG is important when producing samples. We can do better than a high quality uniform RNG however. Pseudo-random numbers can greatly improve the uniformity, while keeping the relevant properties of random numbers*. Relevant property: avoid correlation.

*: C. Waechter. Quasi-Monte Carlo Light Transport Simulation by Efficient Ray Tracing. PhD thesis, Ulm, 2009.

Noise

Advanced Graphics – Filtering 20

slide-21
SLIDE 21

Not All Noise is Equal

Our eyes expect noise. β€˜Good noise’ is uniform, high-frequent, and… blue*.

*: www.arnoldrenderer.com/arnold/research

Noise

Advanced Graphics – Filtering 21

slide-22
SLIDE 22

Noise

Advanced Graphics – Filtering 22

slide-23
SLIDE 23

Noise

Advanced Graphics – Filtering 23

slide-24
SLIDE 24

Noise

Advanced Graphics – Filtering 24

slide-25
SLIDE 25

Noise

Advanced Graphics – Filtering 25

slide-26
SLIDE 26

Suppressing Fireflies

β€œA firefly is easily recognized in the final image: it is a pixel with a value that differs significantly from its neighbors.” β–ͺ Is this always true? β–ͺ How to fix it? β–ͺ Is that still correct? Firefly suppression introduces bias in our estimator. β–ͺ Spread out the removed energy over the image / neighborhood β–ͺ Just wait it out (additional samples will improve the average) β–ͺ Do some adaptive sampling (detect high variance) β–ͺ Just Just acc accept it it.

Noise

Advanced Graphics – Filtering 26

slide-27
SLIDE 27

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-28
SLIDE 28

Reducing the Problem - Filtering

Core idea: Exploit the fact that illumination is typically low-frequent: Nearby pixels tend to converge to similar values, so we should be able to use information gathered for one pixel to improve the estimate of the next. Note: Unless neighboring pixels actually converge to the same value, filtering will introduce bias.

Ingredients

Advanced Graphics – Filtering 28

slide-29
SLIDE 29

Filter kernels

For the actual filtering, we apply a kernel.

Pixel FilteredValue( 𝑗𝑦, 𝑗𝑧, β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž ) 𝑑𝑣𝑛 = 0 π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’ = 0 for π‘˜π‘¦ = 𝑗𝑦 - β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž to 𝑗𝑦 + β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž for π‘˜π‘§ = 𝑗𝑧 - β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž to 𝑗𝑧 + β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž 𝑑𝑣𝑛 += ReadPixel( π‘˜π‘¦, π‘˜π‘§ ) * weight( π‘˜π‘¦, π‘˜π‘§ ) π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’ += weight( π‘˜π‘¦, π‘˜π‘§ ) return 𝑑𝑣𝑛 / π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’

Ingredients

Advanced Graphics – Filtering 29

ΖΈ 𝑑𝑗 = Οƒπ‘˜βˆˆπ’ͺ𝑗 𝑑

π‘˜π‘₯(𝑗, π‘˜)

Οƒπ‘˜βˆˆπ’ͺ𝑗 π‘₯(𝑗, π‘˜)

slide-30
SLIDE 30

Filter kernels

For the actual filtering, we apply a kernel.

Pixel FilteredValue( 𝑗𝑦, 𝑗𝑧, β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž ) 𝑑𝑣𝑛 = 0 π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’ = 0 for π‘˜π‘¦ = 𝑗𝑦 - β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž to 𝑗𝑦 + β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž for π‘˜π‘§ = 𝑗𝑧 - β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž to 𝑗𝑧 + β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž 𝑑𝑣𝑛 += ReadPixel( π‘˜π‘¦, π‘˜π‘§ ) * weight( π‘˜π‘¦, π‘˜π‘§ ) π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’ += weight( π‘˜π‘¦, π‘˜π‘§ ) return 𝑑𝑣𝑛 / π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’

Here, weight or π‘₯ is the weight function. We could simply use the Gaussian kernel: π‘₯ 𝑗, π‘˜ = exp

βˆ’ π‘žπ‘—βˆ’π‘žπ‘˜

2

2πœπ‘’

2

,

Ingredients

Advanced Graphics – Filtering 30 where π‘žπ‘— and π‘žπ‘˜ are screen space positions and πœπ‘’ is the spatial standard deviation of the Gaussian kernel.

slide-31
SLIDE 31

Filter kernels

A Gaussian filter (as well as other low-pass filters) blurs out high frequency details. We can improve on this using a non-linear bilateral filter*. π‘₯ 𝑗, π‘˜ = exp βˆ’ π‘žπ‘— βˆ’ π‘žπ‘˜

2

2πœπ‘’

2

Γ— exp βˆ’ 𝑑𝑗 βˆ’ 𝑑

π‘˜ 2

2πœπ‘ 

2

*: Tomasi & Manduchi, Bilateral filtering for gray and color images. ICCV ’98.

Ingredients

Advanced Graphics – Filtering 31

slide-32
SLIDE 32

Filter kernels

The bilateral filter takes the color of nearby pixels into account. We can take this further, by taking an arbitrary set of features into account. The cross bilateral filter*: π‘₯ 𝑗, π‘˜ = exp βˆ’ π‘žπ‘— βˆ’ π‘žπ‘˜

2

2πœπ‘’

2

Γ— ΰ·‘

𝑙=1 𝐿

exp βˆ’ 𝑔

𝑙,𝑗 βˆ’ 𝑔 𝑙,π‘˜ 2

2πœπ‘™

2

Here, 𝑔

𝑙,𝑗 is the 𝑙’th feature vector at pixel 𝑗 and πœπ‘™ is the bandwidth parameter for feature 𝑙.

Note that we can use noise-free features to smooth noisy features. Example of a low-noise feature: normals at the primary intersection point. Example of a noisy feature: indirect illumination at the primary intersection point.

*: Eisemann & Durand. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23, 3 (Aug. 2004).

Ingredients

Advanced Graphics – Filtering 32

slide-33
SLIDE 33

Ingredients

Advanced Graphics – Filtering 33 Indirect illumination as a feature: A path tracer allows us to conveniently split direct from indirect, and bounce 1 from bounce 2. Separating illumination into layers allows us to filter each layer separately. This prevents bleeding, and allows for layer-specific kernel sizes. We can also separate albedo from illumination.

slide-34
SLIDE 34

Ingredients

Advanced Graphics – Filtering 34

= X

Adding this separation to an existing renderer: β–ͺ store albedo at the primary intersection (simple material property); β–ͺ at the end of the pipeline: illumination = sample / max( epsilon, albedo ).

slide-35
SLIDE 35

Ingredients

Advanced Graphics – Filtering 35

sinmantyx.wordpress.com/2015/12/02/denoising-cycles-renders-in-blender-theory

Example of denoising using a cross-bilateral filter in Blender.

slide-36
SLIDE 36

Ingredients

Advanced Graphics – Filtering 36

Reference 4096 spp Input 4 spp Bilateral 265ms Γ€-Trous 64ms Separable 28ms

Not shown here: β–ͺ Worst-case scenarios β–ͺ Speculars, glossy’s β–ͺ Dielectrics β–ͺ Indirect contact shadows β–ͺ Temporal stability β–ͺ 1spp, 2spp, 3spp β–ͺ Depth of field β–ͺ …

slide-37
SLIDE 37

Manifold Filtering*

Core idea: If we split up samples in depth layers, we can filter each layer separately.

*: Sample-based Manifold Filtering for Interactive Global Illumination and Depth of Field. Bauszat et al., 2014.

Ingredients

Advanced Graphics – Filtering 37

slide-38
SLIDE 38

Reprojection

Core idea: In an animation, samples taken for the previous frame are meaningful for the current frame. We can supply the filter with more data by looking back in time.

Ingredients

Advanced Graphics – Filtering 38

slide-39
SLIDE 39

Reprojection

Core idea: In an animation, samples taken for the previous frame are meaningful for the current frame. We can supply the filter with more data by looking back in time. Problem: in an animation, the camera and/or the geometry moves. We need to find the location of a pixel in the previous frame(s). Solution: use the camera matrices. 𝑁4𝑦4 𝑦π‘₯π‘π‘ π‘šπ‘’ 𝑧π‘₯π‘π‘ π‘šπ‘’ 𝑨π‘₯π‘π‘ π‘šπ‘’ 1 = π‘¦π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘§π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘¨π‘‘π‘‘π‘ π‘“π‘“π‘œ 1 βž” 𝑁4𝑦4

βˆ’1

π‘¦π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘§π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘¨π‘‘π‘‘π‘ π‘“π‘“π‘œ 1 = 𝑦π‘₯π‘π‘ π‘šπ‘’ 𝑧π‘₯π‘π‘ π‘šπ‘’ 𝑨π‘₯π‘π‘ π‘šπ‘’ 1 (finally, apply the matrix of the previous frame to obtain the screen location in the previous frame.)

Ingredients

Advanced Graphics – Filtering 39

https://www.shadertoy.com/view/ldtGWl

slide-40
SLIDE 40

Reprojection

Reprojection using camera matrices: β–ͺ fails if we have animation β–ͺ will not work with depth of field β–ͺ will not work with speculars. For a (partial) solution, we look to the field of video compression: We derive per-pixel motion vectors*.

*: Shan Zhu and Kai-Kuang. A New Diamond Search Algorithm for Fast Block-matching Motion Estimation. Trans. Img. Proc. 9, 2 (Feb. 2000). And: Victor Voorhuis, Beyond Spatiotemporal Variance-Guided Filtering: Temporally Stable Filtering of Path-Traced Reflections in Real-Time. Master thesis, UU, 2018.

Ingredients

Advanced Graphics – Filtering 40

slide-41
SLIDE 41

Ingredients

Advanced Graphics – Filtering 41 https://dspace.library.uu.nl/bitstream/handle/1874/366198/Beyond%20SVGF.pdf

slide-42
SLIDE 42

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-43
SLIDE 43

State of the Art

Advanced Graphics – Filtering 43

slide-44
SLIDE 44

State of the Art

Advanced Graphics – Filtering 44

slide-45
SLIDE 45

Digest

Advanced Graphics – Filtering 45

Filtering, Ingredients:

β–ͺ Separate albedo and illumination β–ͺ Separate direct and indirect light β–ͺ Filter kernel (box filter < Gaussian < bilateral) β–ͺ Feature buffers (albedo, illumination, normals, world pos, …) β–ͺ Filtered features β–ͺ Reprojection (camera matrix (per mesh?), diamond search) β–ͺ Adaptive sampling β–ͺ High quality samples Open problems: β–ͺ Depth of field, motion blur, AA β–ͺ Transparency β–ͺ Glossy surfaces (~)

slide-46
SLIDE 46

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-47
SLIDE 47

INFOMAGR – Advanced Graphics

Jacco Bikker - November 2019 – February 2020

END of β€œFiltering”

next lecture: β€œBidirectional”