Lecture 15 - Filtering Welcome! , = (, ) , - - PowerPoint PPT Presentation

β–Ά
lecture 15 filtering
SMART_READER_LITE
LIVE PREVIEW

Lecture 15 - Filtering Welcome! , = (, ) , - - PowerPoint PPT Presentation

May 11, 2023 446 likes β€’857 views

INFOMAGR Advanced Graphics Jacco Bikker - November 2018 - February 2019 Lecture 15 - Filtering Welcome! , = (, ) , + , , ,

slide-1
SLIDE 1

𝑱 π’š, π’šβ€² = 𝒉(π’š, π’šβ€²) 𝝑 π’š, π’šβ€² + ΰΆ±

𝑻

𝝇 π’š, π’šβ€², π’šβ€²β€² 𝑱 π’šβ€², π’šβ€²β€² π’†π’šβ€²β€²

INFOMAGR – Advanced Graphics

Jacco Bikker - November 2018 - February 2019

Lecture 15 - β€œFiltering”

Welcome!

slide-2
SLIDE 2

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-3
SLIDE 3

Path Tracing

The rendering equation allows us to accurately simulate light transport: β–ͺ Given a sensible scene description β–ͺ Given a sensible camera model β–ͺ Ignoring some complex effects. We evaluate it using Monte Carlo integration:

βž” The result is always an estimate,

with variance.

Noise

Advanced Graphics – Filtering 3

en.wikipedia.org/wiki/Path_tracing

slide-4
SLIDE 4

Path Tracing

The rendering equation allows us to accurately simulate light transport: β–ͺ Given a sensible scene description β–ͺ Given a sensible camera model β–ͺ Ignoring some complex effects. We evaluate it using Monte Carlo integration:

βž” The result is always an estimate,

with variance.

Noise

Advanced Graphics – Filtering 4

www.mitsuba-renderer.org

slide-5
SLIDE 5

Noise in the Real World

Noise in a digital photo can be the result

  • f short exposure.

Noise

Advanced Graphics – Filtering 5

www.photoreview.com.au/tips/shooting/how-to-control-image-noise

slide-6
SLIDE 6

Noise in the Real World

Noise in a digital photo can be the result

  • f short exposure.

We solve this by lowering our ISO settings, so we gather more photons per pixel. (that obviously has consequences)

Noise

Advanced Graphics – Filtering 6

www.photoreview.com.au/tips/shooting/how-to-control-image-noise

slide-7
SLIDE 7

Reducing Noise in Path Tracing

We can increase β€˜exposure’ in a pathtracer by taking more samples.

Noise

Advanced Graphics – Filtering 7

aras-p.info/blog/2018/03/28/Daily-Pathtracer-Part-1-Initial-C-/

slide-8
SLIDE 8

Reducing Noise in Path Tracing

We can increase β€˜exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling.

Noise

Advanced Graphics – Filtering 8

slide-9
SLIDE 9

Reducing Noise in Path Tracing

We can increase β€˜exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling. Or Next Event Estimation. However, at interactive rates it will be quite hard to reach a converged image.

Noise

Advanced Graphics – Filtering 9

slide-10
SLIDE 10

Noise

Advanced Graphics – Filtering 10 well-lit, diffuse: low noise depth of field: moderate noise Indirect light: moderate noise

? ?

slide-11
SLIDE 11

Not All Noise is Equal

In general, if we can importance sample it, noise will be low: β–ͺ sample BRDF proportional to cos πœ„; β–ͺ sample a light proportional to potential contribution. The β€˜fireflies’ are the result of a random bounce, followed by a specular bounce, which then hit the light: a direction is sampled with a low probability, but yields a high value.

Noise

Advanced Graphics – Filtering 11

slide-12
SLIDE 12

Not All Noise is Equal

A good RNG is important when producing samples.

Noise

Advanced Graphics – Filtering 12

slide-13
SLIDE 13

Not All Noise is Equal

A good RNG is important when producing samples. We can do better than a high quality uniform RNG however. Pseudo-random numbers can greatly improve the uniformity, while keeping the relevant properties of random numbers*. Relevant property: avoid correlation.

*: C. Waechter. Quasi-Monte Carlo Light Transport Simulation by Efficient Ray Tracing. PhD thesis, Ulm, 2009.

Noise

Advanced Graphics – Filtering 13

slide-14
SLIDE 14

Not All Noise is Equal

Our eyes expect noise. β€˜Good noise’ is uniform, high-frequent, and… blue*.

*: www.arnoldrenderer.com/arnold/research

Noise

Advanced Graphics – Filtering 14

slide-15
SLIDE 15

Noise

Advanced Graphics – Filtering 15

slide-16
SLIDE 16

Noise

Advanced Graphics – Filtering 16

slide-17
SLIDE 17

Noise

Advanced Graphics – Filtering 17

slide-18
SLIDE 18

Noise

Advanced Graphics – Filtering 18

slide-19
SLIDE 19

Suppressing Fireflies

β€œA firefly is easily recognized in the final image: it is a pixel with a value that differs significantly from its neighbors.” β–ͺ Is this always true? β–ͺ How to fix it? β–ͺ Is that still correct? Firefly suppression introduces bias in our estimator. β–ͺ Spread out the removed energy over the image / neighborhood β–ͺ Just wait it out (additional samples will improve the average) β–ͺ Do some adaptive sampling (detect high variance) β–ͺ Just Just acc accept it it.

Noise

Advanced Graphics – Filtering 19

slide-20
SLIDE 20

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-21
SLIDE 21

Reducing the Problem - Filtering

Core idea: Exploit the fact that illumination is typically low-frequent: Nearby pixels tend to converge to similar values, so we should be able to use information gathered for one pixel to improve the estimate of the next. Note: Unless neighboring pixels actually converge to the same value, filtering will introduce bias.

Ingredients

Advanced Graphics – Filtering 21

slide-22
SLIDE 22

Filter kernels

For the actual filtering, we apply a kernel.

Pixel FilteredValue( 𝑗𝑦, 𝑗𝑧, β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž ) 𝑑𝑣𝑛 = 0 π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’ = 0 for π‘˜π‘¦ = 𝑗𝑦 - β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž to 𝑗𝑦 + β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž for π‘˜π‘§ = 𝑗𝑧 - β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž to 𝑗𝑧 + β„Žπ‘π‘šπ‘”π‘‹π‘—π‘’π‘’β„Ž 𝑑𝑣𝑛 += ReadPixel( π‘˜π‘¦, π‘˜π‘§ ) * weight( π‘˜π‘¦, π‘˜π‘§ ) π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’ += weight( π‘˜π‘¦, π‘˜π‘§ ) return 𝑑𝑣𝑛 / π‘‘π‘£π‘›π‘›π‘“π‘’π‘‹π‘“π‘—π‘•β„Žπ‘’

Here, weight or π‘₯ is the weight function. We could simply use the Gaussian kernel: π‘₯ 𝑗, π‘˜ = exp

βˆ’ π‘žπ‘—βˆ’π‘žπ‘˜

2

2πœπ‘’

2

,

Ingredients

Advanced Graphics – Filtering 22

ΖΈ 𝑑𝑗 = Οƒπ‘˜βˆˆπ’ͺ𝑗 𝑑

π‘˜π‘₯(𝑗, π‘˜)

Οƒπ‘˜βˆˆπ’ͺ𝑗 π‘₯(𝑗, π‘˜)

where π‘žπ‘— and π‘žπ‘˜ are screen space positions and πœπ‘’ is the spatial standard deviation of the Gaussian kernel.

slide-23
SLIDE 23

Filter kernels

A Gaussian filters (as well as other low-pass filters) blurs out high frequency details. We can improve on this using a non-linear bilateral filter*. π‘₯ 𝑗, π‘˜ = exp βˆ’ π‘žπ‘— βˆ’ π‘žπ‘˜

2

2πœπ‘’

2

Γ— exp βˆ’ 𝑑𝑗 βˆ’ 𝑑

π‘˜ 2

2πœπ‘ 

2

*: Tomasi & Manduchi, Bilateral filtering for gray and color images. ICCV ’98.

Ingredients

Advanced Graphics – Filtering 23

slide-24
SLIDE 24

Filter kernels

The bilateral filter takes the color of nearby pixels into account. We can take this further, by taking an arbitrary set of features into account. The cross bilateral filter*: π‘₯ 𝑗, π‘˜ = exp βˆ’ π‘žπ‘— βˆ’ π‘žπ‘˜

2

2πœπ‘’

2

Γ— ΰ·‘

𝑙=1 𝐿

exp βˆ’ 𝑔

𝑙,𝑗 βˆ’ 𝑔 𝑙,π‘˜ 2

2πœπ‘™

2

Here, 𝑔

𝑙,𝑗 is the 𝑙’th feature vector at pixel 𝑗 and πœπ‘™ is the bandwidth parameter for feature 𝑙.

Note that we can use noise-free features to smooth noisy features. Example of a low-noise feature: normals at the primary intersection point. Example of a noisy feature: indirect illumination at the primary intersection point.

*: Eisemann & Durand. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23, 3 (Aug. 2004).

Ingredients

Advanced Graphics – Filtering 24

slide-25
SLIDE 25

Ingredients

Advanced Graphics – Filtering 25 Indirect illumination as a feature: A path tracer allows us to conveniently split direct from indirect, and bounce 1 from bounce 2. Separating illumination into layers allows us to filter each layer separately. This prevents bleeding, and allows for layer-specific kernel sizes. We can also separate albedo from illumination.

slide-26
SLIDE 26

Ingredients

Advanced Graphics – Filtering 26

= X

Adding this separation to an existing renderer: β–ͺ store albedo at the primary intersection (simple material property); β–ͺ at the end of the pipeline: illumination = sample / max( epsilon, albedo ).

slide-27
SLIDE 27

Ingredients

Advanced Graphics – Filtering 27

sinmantyx.wordpress.com/2015/12/02/denoising-cycles-renders-in-blender-theory

Example of denoising using a cross-bilateral filter in Blender.

slide-28
SLIDE 28

Ingredients

Advanced Graphics – Filtering 28

Reference 4096 spp Input 4 spp Bilateral 265ms Γ€-Trous 64ms Separable 28ms

Not shown here: β–ͺ Worst-case scenarios β–ͺ Speculars, glossy’s β–ͺ Dielectrics β–ͺ Indirect contact shadows β–ͺ Temporal stability β–ͺ 1spp, 2spp, 3spp β–ͺ Depth of field β–ͺ …

slide-29
SLIDE 29

Manifold Filtering*

Core idea: If we split up samples in depth layers, we can filter each layer separately.

*: Sample-based Manifold Filtering for Interactive Global Illumination and Depth of Field. Bauszat et al., 2014.

Ingredients

Advanced Graphics – Filtering 29

slide-30
SLIDE 30

Ingredients

Advanced Graphics – Filtering 30

slide-31
SLIDE 31

Ingredients

Advanced Graphics – Filtering 31

Reference 4096 spp Input 4 spp Bilateral 265ms Γ€-Trous 64ms Separable 28ms

Not shown here: β–ͺ Worst-case scenarios β–ͺ Speculars, glossy’s β–ͺ Dielectrics β–ͺ Indirect contact shadows β–ͺ Temporal stability β–ͺ 1spp, 2spp, 3spp β–ͺ Depth of field β–ͺ …

slide-32
SLIDE 32

Reprojection

Core idea: In an animation, samples taken for the previous frame are meaningful for the current frame. We can supply the filter with more data by looking back in time. Problem: in an animation, the camera moves. We need to find the location of a pixel in the previous frame(s). Solution: use the camera matrices. 𝑁4𝑦4 𝑦π‘₯π‘π‘ π‘šπ‘’ 𝑧π‘₯π‘π‘ π‘šπ‘’ 𝑨π‘₯π‘π‘ π‘šπ‘’ 1 = π‘¦π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘§π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘¨π‘‘π‘‘π‘ π‘“π‘“π‘œ 1 βž” 𝑁4𝑦4

βˆ’1

π‘¦π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘§π‘‘π‘‘π‘ π‘“π‘“π‘œ π‘¨π‘‘π‘‘π‘ π‘“π‘“π‘œ 1 = 𝑦π‘₯π‘π‘ π‘šπ‘’ 𝑧π‘₯π‘π‘ π‘šπ‘’ 𝑨π‘₯π‘π‘ π‘šπ‘’ 1 (finally, apply the matrix of the previous frame to obtain the screen location in the previous frame.)

Ingredients

Advanced Graphics – Filtering 32

slide-33
SLIDE 33

Reprojection

Reprojection using camera matrices: β–ͺ fails if we have animation β–ͺ will not work with depth of field β–ͺ will not work with speculars. For a (partial) solution, we look to the field of video compression: We derive per-pixel motion vectors*.

*: Shan Zhu and Kai-Kuang. A New Diamond Search Algorithm for Fast Block-matching Motion Estimation. Trans. Img. Proc. 9, 2 (Feb. 2000). And: Victor Voorhuis, Beyond Spatiotemporal Variance-Guided Filtering: Temporally Stable Filtering of Path-Traced Reflections in Real-Time. Master thesis, UU, 2018.

Ingredients

Advanced Graphics – Filtering 33

slide-34
SLIDE 34

Ingredients

Advanced Graphics – Filtering 34

slide-35
SLIDE 35

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-36
SLIDE 36

State of the Art

Advanced Graphics – Filtering 36

slide-37
SLIDE 37

State of the Art

Advanced Graphics – Filtering 37

slide-38
SLIDE 38

Today’s Agenda:

β–ͺ Noise β–ͺ Ingredients β–ͺ State of the Art

slide-39
SLIDE 39

INFOMAGR – Advanced Graphics

Jacco Bikker - November 2018 – February 2019

END of β€œFiltering”

next lecture: β€œBits and Pieces”