Lecture 15 - Filtering Welcome! , = (, ) , - PowerPoint PPT Presentation

INFOMAGR – Advanced Graphics Jacco Bikker - November 2018 - February 2019 Lecture 15 - “Filtering” Welcome! 𝑱 𝒚, 𝒚 ′ = 𝒉(𝒚, 𝒚 ′ ) 𝝑 𝒚, 𝒚 ′ + න 𝝇 𝒚, 𝒚 ′ , 𝒚 ′′ 𝑱 𝒚 ′ , 𝒚 ′′ 𝒆𝒚′′ 𝑻

Today’s Agenda: ▪ Noise ▪ Ingredients ▪ State of the Art

Advanced Graphics – Filtering 3 Noise Path Tracing The rendering equation allows us to accurately simulate light transport: ▪ Given a sensible scene description ▪ Given a sensible camera model ▪ Ignoring some complex effects. We evaluate it using Monte Carlo integration: en.wikipedia.org/wiki/Path_tracing ➔ The result is always an estimate, with variance.

Advanced Graphics – Filtering 4 Noise Path Tracing The rendering equation allows us to accurately simulate light transport: ▪ Given a sensible scene description ▪ Given a sensible camera model ▪ Ignoring some complex effects. We evaluate it using Monte Carlo integration: www.mitsuba-renderer.org ➔ The result is always an estimate, with variance.

Advanced Graphics – Filtering 5 Noise Noise in the Real World Noise in a digital photo can be the result of short exposure. www.photoreview.com.au/tips/shooting/how-to-control-image-noise

Advanced Graphics – Filtering 6 Noise Noise in the Real World Noise in a digital photo can be the result of short exposure. We solve this by lowering our ISO settings, so we gather more photons per pixel. (that obviously has consequences) www.photoreview.com.au/tips/shooting/how-to-control-image-noise

Advanced Graphics – Filtering 7 Noise Reducing Noise in Path Tracing We can increase ‘exposure’ in a pathtracer by taking more samples. aras-p.info/blog/2018/03/28/Daily-Pathtracer-Part-1-Initial-C-/

Advanced Graphics – Filtering 8 Noise Reducing Noise in Path Tracing We can increase ‘exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling.

Advanced Graphics – Filtering 9 Noise Reducing Noise in Path Tracing We can increase ‘exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling. Or Next Event Estimation. However, at interactive rates it will be quite hard to reach a converged image.

Advanced Graphics – Filtering 10 Noise well-lit, diffuse: low noise Indirect light: depth of field: moderate noise moderate noise ? ?

Advanced Graphics – Filtering 11 Noise Not All Noise is Equal In general, if we can importance sample it, noise will be low: ▪ sample BRDF proportional to cos 𝜄 ; ▪ sample a light proportional to potential contribution. The ‘fireflies’ are the result of a random bounce, followed by a specular bounce, which then hit the light: a direction is sampled with a low probability, but yields a high value.

Advanced Graphics – Filtering 12 Noise Not All Noise is Equal A good RNG is important when producing samples.

Advanced Graphics – Filtering 13 Noise Not All Noise is Equal A good RNG is important when producing samples. We can do better than a high quality uniform RNG however. Pseudo-random numbers can greatly improve the uniformity, while keeping the relevant properties of random numbers*. Relevant property: avoid correlation . *: C. Waechter. Quasi-Monte Carlo Light Transport Simulation by Efficient Ray Tracing. PhD thesis, Ulm, 2009.

Advanced Graphics – Filtering 14 Noise Not All Noise is Equal Our eyes expect noise. ‘Good noise’ is uniform, high- frequent, and… blue*. *: www.arnoldrenderer.com/arnold/research

Advanced Graphics – Filtering 15 Noise

Advanced Graphics – Filtering 16 Noise

Advanced Graphics – Filtering 17 Noise

Advanced Graphics – Filtering 18 Noise

Advanced Graphics – Filtering 19 Noise Suppressing Fireflies “A firefly is easily recognized in the final image: it is a pixel with a value that differs significantly from its neighbors.” ▪ Is this always true? ▪ How to fix it? ▪ Is that still correct? Firefly suppression introduces bias in our estimator. ▪ Spread out the removed energy over the image / neighborhood ▪ Just wait it out (additional samples will improve the average) ▪ Do some adaptive sampling (detect high variance) ▪ Just Just acc accept it it.

Today’s Agenda: ▪ Noise ▪ Ingredients ▪ State of the Art

Advanced Graphics – Filtering 21 Ingredients Reducing the Problem - Filtering Core idea: Exploit the fact that illumination is typically low-frequent: Nearby pixels tend to converge to similar values, so we should be able to use information gathered for one pixel to improve the estimate of the next. Note: Unless neighboring pixels actually converge to the same value, filtering will introduce bias.

Ƹ Advanced Graphics – Filtering 22 Ingredients Filter kernels For the actual filtering, we apply a kernel. σ 𝑘∈𝒪 𝑗 𝑑 𝑘 𝑥(𝑗, 𝑘) Pixel FilteredValue( 𝑗 𝑦 , 𝑗 𝑧 , ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ ) 𝑑 𝑗 = 𝑡𝑣𝑛 = 0 σ 𝑘∈𝒪 𝑗 𝑥(𝑗, 𝑘) 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 = 0 for 𝑘 𝑦 = 𝑗 𝑦 - ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ to 𝑗 𝑦 + ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ for 𝑘 𝑧 = 𝑗 𝑧 - ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ to 𝑗 𝑧 + ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ 𝑡𝑣𝑛 += ReadPixel( 𝑘 𝑦 , 𝑘 𝑧 ) * weight( 𝑘 𝑦 , 𝑘 𝑧 ) 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 += weight( 𝑘 𝑦 , 𝑘 𝑧 ) return 𝑡𝑣𝑛 / 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 Here, weight or 𝑥 is the weight function. We could simply use the Gaussian kernel: 2 − 𝑞 𝑗 −𝑞 𝑘 𝑥 𝑗, 𝑘 = exp , where 𝑞 𝑗 and 𝑞 𝑘 are screen space positions and 𝜏 𝑒 is 2 2𝜏 𝑒 the spatial standard deviation of the Gaussian kernel.

Advanced Graphics – Filtering 23 Ingredients Filter kernels A Gaussian filters (as well as other low-pass filters) blurs out high frequency details. We can improve on this using a non-linear bilateral filter*. 2 2 𝑥 𝑗, 𝑘 = exp − 𝑞 𝑗 − 𝑞 𝑘 × exp − 𝑑 𝑗 − 𝑑 𝑘 2 2 2𝜏 𝑒 2𝜏 𝑠 *: Tomasi & Manduchi , Bilateral filtering for gray and color images. ICCV ’98.

Advanced Graphics – Filtering 24 Ingredients Filter kernels The bilateral filter takes the color of nearby pixels into account. We can take this further, by taking an arbitrary set of features into account. The cross bilateral filter*: 𝐿 2 2 𝑥 𝑗, 𝑘 = exp − 𝑞 𝑗 − 𝑞 𝑘 exp − 𝑔 𝑙,𝑗 − 𝑔 𝑙,𝑘 × ෑ 2 2 2𝜏 𝑒 2𝜏 𝑙 𝑙=1 Here, 𝑔 𝑙,𝑗 is the 𝑙’ th feature vector at pixel 𝑗 and 𝜏 𝑙 is the bandwidth parameter for feature 𝑙 . Note that we can use noise-free features to smooth noisy features. Example of a low-noise feature: normals at the primary intersection point. Example of a noisy feature: indirect illumination at the primary intersection point. *: Eisemann & Durand. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23, 3 (Aug. 2004).

Advanced Graphics – Filtering 25 Ingredients Indirect illumination as a feature: A path tracer allows us to conveniently split direct from indirect, and bounce 1 from bounce 2. Separating illumination into layers allows us to filter each layer separately. This prevents bleeding, and allows for layer-specific kernel sizes. We can also separate albedo from illumination.

Advanced Graphics – Filtering 26 Ingredients X = Adding this separation to an existing renderer: ▪ store albedo at the primary intersection (simple material property); ▪ at the end of the pipeline: illumination = sample / max( epsilon, albedo ).

Advanced Graphics – Filtering 27 Ingredients Example of denoising using a cross-bilateral filter in Blender. sinmantyx.wordpress.com/2015/12/02/denoising-cycles-renders-in-blender-theory

Advanced Graphics – Filtering 28 Ingredients Not shown here: ▪ Worst-case scenarios ▪ Speculars , glossy’s ▪ Dielectrics ▪ Indirect contact shadows ▪ Temporal stability ▪ 1spp, 2spp, 3spp ▪ Depth of field ▪ … Reference Input Bilateral À-Trous Separable 4096 spp 4 spp 265ms 64ms 28ms

Advanced Graphics – Filtering 29 Ingredients Manifold Filtering* Core idea: If we split up samples in depth layers, we can filter each layer separately. *: Sample-based Manifold Filtering for Interactive Global Illumination and Depth of Field. Bauszat et al., 2014.

Advanced Graphics – Filtering 30 Ingredients

Advanced Graphics – Filtering 31 Ingredients Not shown here: ▪ Worst-case scenarios ▪ Speculars , glossy’s ▪ Dielectrics ▪ Indirect contact shadows ▪ Temporal stability ▪ 1spp, 2spp, 3spp ▪ Depth of field ▪ … Reference Input Bilateral À-Trous Separable 4096 spp 4 spp 265ms 64ms 28ms