of Infinite Input Extensions Diego Nehab Andr Maximo IMPA GE - PowerPoint PPT Presentation

Parallel Recursive Filtering of Infinite Input Extensions Diego Nehab André Maximo IMPA GE Global Research GTC 2017

Linear time-invariant filters filter

Linear • Invariant to scale filter scale scale filter

Linear • Invariant to addition filter filter add add filter

Time invariant • Invariant to shift filter t t t t shift shift filter t t

Convolution • The convolution of sequences 𝐜 and 𝐢 is a sequence 𝐛 ∞ 𝐜, 𝐢, 𝐛: ℤ → ℝ 𝐛 = 𝐜 ∗ 𝐢 𝑏 𝑗 = ෍ 𝑐 𝑘 ℎ 𝑗−𝑘 𝑘=−∞ 𝐜 𝐢 𝐢 𝐢 𝐢 𝐛 = 𝐜 ∗ 𝐢

Linear shift-invariant filters are convolutions unit impulse filter filter’s impulse response convolve filter

Examples ∗ ∗ ∗

Outline of talk • Introduction • Recursive filters are very useful • Initialization at the boundaries is an important problem • Exact recursive filtering of infinite input extensions • Closed-form formulas available for the first time • Enable simple and effective algorithms • Parallelization • Fastest recursive filtering algorithms to date • First to filter infinite extensions exactly

General model for convolutions • Linear difference equations 𝐱 𝐴 𝐴 𝐴 filter 𝐁𝐴 = 𝐂𝐱 𝑏 𝑠 𝑨 𝑗−𝑠 + ⋯ + 𝑏 1 𝑨 𝑗−1 + 𝑏 0 𝑨 𝑗 + 𝑏 −1 𝑨 𝑗+1 + ⋯ + 𝑏 −𝑠 𝑨 𝑗+𝑠 = 𝑐 𝑒 𝑥 𝑗−𝑒 + ⋯ + 𝑐 1 𝑥 𝑗−1 + 𝑐 0 𝑥 𝑗 + 𝑐 −1 𝑥 𝑗+1 + ⋯ + 𝑐 −𝑒 𝑥 𝑗+𝑒 ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋮ ⋮ ⋱ 𝑏 0 𝑏 −1 ⋮ 𝑏 −𝑠 ⋱ 𝑐 0 𝑐 −1 ⋮ 𝑐 −𝑒 𝑨 𝑗−2 𝑥 𝑗−2 ⋱ 𝑏 1 𝑏 0 𝑏 −1 ⋮ 𝑏 −𝑠 ⋱ 𝑐 1 𝑐 0 𝑐 −1 ⋮ 𝑐 −𝑒 𝑨 𝑗−1 𝑥 𝑗−1 ⋱ ⋮ 𝑏 1 𝑏 0 𝑏 −1 ⋮ ⋱ = ⋱ ⋮ 𝑐 1 𝑐 0 𝑐 −1 ⋮ ⋱ 𝑨 𝑗 𝑥 𝑗 𝑨 𝑗+1 𝑥 𝑗+1 𝑏 𝑠 ⋮ 𝑏 1 𝑏 0 𝑏 −1 ⋱ 𝑐 𝑒 ⋮ 𝑐 1 𝑐 0 𝑐 −1 ⋱ 𝑨 𝑗+2 𝑥 𝑗+2 𝑏 𝑠 ⋮ 𝑏 1 𝑏 0 ⋱ 𝑐 𝑒 ⋮ 𝑐 1 𝑐 0 ⋱ ⋮ ⋮ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ constant diagonals

General model for convolutions • Decompose into direct and recursive parts 𝐱 𝐱 𝐱 𝐲 recursive recursive 𝐴 𝐴 𝐴 𝐴 𝐴 direct direct filter 𝐲 = 𝐂𝐱 𝐁𝐴 = 𝐂𝐱 𝐁𝐴 = 𝐲 𝑦 𝑗 = 𝑐 𝑡 𝑥 𝑗−𝑡 + ⋯ + 𝑐 1 𝑥 𝑗−1 + 𝑐 0 𝑥 𝑗 + 𝑐 −1 𝑥 𝑗+1 + ⋯ + 𝑐 −𝑡 𝑥 𝑗+𝑡 recursive part is the inverse of a convolution direct part is what we think of as convolution it is like a linear system 𝐁𝐴 = 𝐲 it is like a matrix multiplication 𝐲 = 𝐂𝐱 𝑏 𝑠 𝑨 𝑗−𝑠 + ⋯ + 𝑏 1 𝑨 𝑗−1 + 𝑏 0 𝑨 𝑗 + 𝑏 −1 𝑨 𝑗+1 + ⋯ + 𝑏 −𝑠 𝑨 𝑗+𝑠 = 𝑦 𝑗 𝑃(𝑡 𝑜) ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋮ ⋮ ⋮ ⋮ ⋱ 𝑐 0 𝑐 −1 ⋮ 𝑐 −𝑡 ⋱ 𝑏 0 𝑏 −1 ⋮ 𝑏 −𝑠 𝑦 𝑗−2 𝑥 𝑗−2 𝑨 𝑗−2 𝑦 𝑗−2 ⋱ 𝑐 1 𝑐 0 𝑐 −1 ⋮ 𝑐 −𝑡 ⋱ 𝑏 1 𝑏 0 𝑏 −1 ⋮ 𝑏 −𝑠 𝑨 𝑗−1 𝑦 𝑗−1 𝑦 𝑗−1 𝑥 𝑗−1 ⋱ ⋮ 𝑏 1 𝑏 0 𝑏 −1 ⋮ ⋱ = 𝑦 𝑗 = ⋱ ⋮ 𝑐 1 𝑐 0 𝑐 −1 ⋮ ⋱ 𝑥 𝑗 𝑨 𝑗 𝑦 𝑗 𝑨 𝑗+1 𝑦 𝑗+1 𝑦 𝑗+1 𝑥 𝑗+1 𝑏 𝑠 ⋮ 𝑏 1 𝑏 0 𝑏 −1 ⋱ 𝑐 𝑡 ⋮ 𝑐 1 𝑐 0 𝑐 −1 ⋱ 𝑦 𝑗+2 𝑥 𝑗+2 𝑨 𝑗+2 𝑦 𝑗+2 𝑐 𝑡 ⋮ 𝑐 1 𝑐 0 ⋱ 𝑏 𝑠 ⋮ 𝑏 1 𝑏 0 ⋱ ⋮ ⋮ ⋮ ⋮ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ infinite impulse response support (IIR) finite impulse response support (FIR)

General model for convolutions • Decompose recursive part into causal and anticausal passes 𝐲 𝐱 𝐲 𝐲 recursive 𝐳 𝐴 𝐴 𝐴 anticausal anticausal direct causal causal 𝐲 = 𝐂𝐱 𝐄𝐳 = 𝐲 𝐁𝐴 = 𝐲 𝐅𝐴 = 𝐳 1 𝑧 𝑗 = 𝑦 𝑗 − 𝑒 1 𝑧 𝑗−1 − ⋯ − 𝑒 𝑠 𝑧 𝑗−𝑠 √𝑏 0 causal part is forward-substitution 𝐄𝐳 = 𝐲 anticausal is back-substitution 𝐅𝐴 = 𝐳 𝑃(𝑠 𝑜) 1 𝑃(𝑠 𝑜) 𝑨 𝑗 = 𝑧 𝑗 − 𝑓 1 𝑨 𝑗+1 − ⋯ − 𝑓 𝑠 𝑨 𝑗+𝑠 √𝑏 0 ⋱ ⋱ ⋱ ⋱ ⋱ ⋱ 1 1 𝑓 1 ⋮ 𝑓 𝑠 ⋱ 𝑒 1 1 1 𝑓 1 ⋮ 𝑓 𝑠 𝐄𝐳 = √𝑏 0 ⋱ ⋮ 𝑒 1 1 𝐳 = 𝐲 𝐅𝐴 = √𝑏 0 𝐴 = 𝐳 1 𝑓 1 ⋮ ⋱ 𝑒 𝑠 ⋮ 𝑒 1 1 1 𝑓 1 ⋱ 𝑒 𝑠 ⋮ 𝑒 1 1 1 ⋱ ⋱ ⋱ ⋱ ⋱ ⋱

Downsampling with cardinal cubic B-splines input prefiltered Catmull-Rom t cubic B-spline post-processed output t cardinal cubic B-spline recursive filter t infinite support prefiltered [Nehab & Hoppe 2014]

Fast image blur Blur with FIR filter (given 𝜏 ) Blur with IIR filter (any 𝜏 ) 2 𝜏 𝑜 operations 6𝑜 operations input direct convolution 𝑡 = 2 𝜏 causal pass 𝑠 = 3 anticausal pass 𝑠 = 3 Blur with FFT 𝑜 log 𝑜 operations [van Vliet et al. 1998]

What do near input boundaries? ? ?

Infinite input extensions repeat periodically reflect periodically constant padding filter filter filter

Tileable textures • Textures designed to be tiled in a certain way • Filtering must respect the periodicity original texture filtered, tiled, and shifted periodic repetition clamp to border

Dealing with boundaries in practice • In the frequency domain • DFT/DCT imply infinite extensions • Computations are exact even for IIR filters • In the time domain • Direct convolution can decide out of bounds input arbitrarily • Recursive filters must define out of bounds outputs !

Approximation by input padding padded filtered padded filtered padded filtered input output approximation wasted computation wasted memory more wasted computation more wasted memory even more wasted memory even more wasted computation amount of padding depends on impulse response ”support”

Outline of talk • Introduction • Recursive filters are very useful • Initialization at the boundaries is an important problem • Exact recursive filtering of infinite input extensions • Closed-form formulas available for the first time • Enable simple and effective algorithms • Parallelization • Fastest recursive filtering algorithms to date • First to filter infinite extensions exactly

Exact recursive filtering of finite input input extension finite input input extension … 𝐲 12 … 𝐲 –5 𝐲 –4 𝐲 –3 𝐲 –2 𝐲 –1 𝐲 –0 𝐲 1 𝐲 2 𝐲 3 𝐲 4 𝐲 5 𝐲 6 𝐲 7 𝐲 8 𝐲 9 𝐲 10 𝐲 11 … 𝐳 12 … 𝐳 –5 𝐳 –4 𝐳 –3 𝐳 –2 𝐳 –1 𝐳 0 𝐳 1 𝐳 2 𝐳 3 𝐳 4 𝐳 5 𝐳 6 𝐳 7 𝐳 8 𝐳 9 𝐳 10 𝐳 11 𝐴 12 … 𝐴 8 𝐴 9 𝐴 10 𝐴 11 𝐴 1 𝐴 2 𝐴 3 𝐴 4 𝐴 5 𝐴 6 𝐴 7 finite output

Exact recursive filtering of finite input • Instead, first obtain the initial feedbacks • To initialize causal pass • To initialize subsequent anticausal pass • How to obtain these feedbacks in closed form? • Depends on the choice of infinite input extension input extension finite input input extension … 𝐲 12 … 𝐲 –5 𝐲 –4 𝐲 –3 𝐲 –2 𝐲 –1 𝐲 –0 𝐲 1 𝐲 2 𝐲 3 𝐲 4 𝐲 5 𝐲 6 𝐲 7 𝐲 8 𝐲 9 𝐲 10 𝐲 11 𝐳 0 𝐳 1 𝐳 2 𝐳 3 𝐳 4 𝐳 5 𝐳 6 initial causal feedback 𝐴 1 𝐴 2 𝐴 3 𝐴 4 𝐴 5 𝐴 6 𝐴 7 finite output initial anticausal feedback

Constant padding precomputed 𝑠 × 𝑠 matrix 𝑵 1 = 𝑻 𝐺 ഥ 𝑠 −1 𝐳 0 = 𝑵 1 𝐲 0 𝑩 𝐺 𝑻 𝐺 = 𝑱 − 𝑩 𝐺 𝑠 𝑵 1 𝑠 𝑵 3 = 𝑻 𝑆 ഥ 𝑵 2 = 𝑻 𝑆𝐺 𝑩 𝐺 𝐴 7 = 𝑵 2 𝐳 6 + 𝑵 3 𝐲 7 𝑩 𝑆 − 𝑻 𝑆𝐺 𝑩 𝐺 𝑠 𝑻 𝑆𝐺 𝑩 𝐺 𝑠 = ഥ 𝑠 −1 𝑻 𝑆 = 𝑱 − 𝑩 𝑆 𝑻 𝑆𝐺 − 𝑩 𝑆 𝑩 𝑆 precomputed 𝑠 × 𝑠 matrices input extension finite input input extension … 𝐲 7 … 𝐲 0 𝐲 0 𝐲 0 𝐲 0 𝐲 0 𝐲 0 𝐲 1 𝐲 2 𝐲 3 𝐲 4 𝐲 5 𝐲 6 𝐲 7 𝐲 7 𝐲 7 𝐲 7 𝐲 7 causal infinite series series … 𝐳 12 … 𝐳 –5 𝐳 –4 𝐳 –3 𝐳 –2 𝐳 –1 𝐳 0 𝐳 1 𝐳 2 𝐳 3 𝐳 4 𝐳 5 𝐳 6 𝐳 7 𝐳 8 𝐳 9 𝐳 10 𝐳 11 initial causal feedback anticausal infinite series 𝐴 12 … 𝐴 8 𝐴 9 𝐴 10 𝐴 11 𝐴 1 𝐴 2 𝐴 3 𝐴 4 𝐴 5 𝐴 6 𝐴 7 finite output initial anticausal feedback

ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ ሶ Periodic repetition 𝑜 −1 𝐳 6 = 𝑵 4 𝐳 6 𝑵 4 = 𝑱 − 𝑩 𝐺 𝑜 −1 𝐴 1 = 𝑵 5 𝐴 1 𝑵 5 = 𝑱 − 𝑩 𝑆 input extension finite input input extension … 𝐲 6 … 𝐲 1 𝐲 2 𝐲 3 𝐲 4 𝐲 5 𝐲 6 𝐲 1 𝐲 2 𝐲 3 𝐲 4 𝐲 5 𝐲 6 𝐲 1 𝐲 2 𝐲 3 𝐲 4 𝐲 5 2nd causal 1st causal 𝐳 6 𝐳 1 𝐳 2 𝐳 3 𝐳 4 𝐳 5 𝐳 6 𝐳 1 𝐳 2 𝐳 3 𝐳 4 𝐳 5 𝐳 6 0 initial causal feedback 2nd anticausal 1st anticausal 𝐴 1 𝐴 1 𝐴 2 𝐴 2 𝐴 3 𝐴 3 𝐴 4 𝐴 4 𝐴 5 𝐴 5 𝐴 6 𝐴 6 𝐴 1 0 finite output initial anticausal feedback