Sampling Resampling Warping Morphing Dr. Shai Avidan Faculty of - - PowerPoint PPT Presentation

Sampling Resampling Warping Morphing

• Dr. Shai Avidan

Faculty of Engineering Tel-Aviv University

Slide Credits (Partial List)

• Alyosha Efros
• Steve Seitz
• Rick Szeliski
• Bill Freeman
• Fredo Durand

Sampling/Resampling/Warping/Morphing/Retargeting

• Sampling

– Given a continuous signal g(x) and its sampled counterpart g_s(x), are the samples of g_s(x) sufficient to exactly describe g(x)? – If so, how can g(x) be reconstructed from g_s(x)?

• Resampling

– Image resampling is the process of transforming a sampled image from one coordinate system to another

• Warping

– Is a geometric transformation that redefines the spatial relationship between points in the image

• Morphing

– Is the process of blending two images

• Retargeting

– Is a content-aware resizing operation

Sampled representations

• How to store and compute with continuous functions?
• Common scheme for representation: samples

– write down the function’s values at many points

[FvDFH fig.14.14b / Wolberg]

Reconstruction

• Making samples back into a continuous function

– for output (need realizable method) – for analysis or processing (need mathematical method) – amounts to “guessing” what the function did in between

[FvDFH fig.14.14b / Wolberg]

1D Example: Audio

low high frequencies

Sampling in digital audio

• Recording: sound to analog to samples to disc
• Playback: disc to samples to analog to sound again

– how can we be sure we are filling in the gaps correctly?

Sampling and Reconstruction

• Simple example: a sine wave

Undersampling

• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave

– unsurprising result: information is lost

Undersampling

• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave

– unsurprising result: information is lost – surprising result: indistinguishable from lower frequency

Undersampling

• What if we “missed” things between the samples?
• Simple example: undersampling a sine wave

– unsurprising result: information is lost – surprising result: indistinguishable from lower frequency – also was always indistinguishable from higher frequencies – aliasing: signals “traveling in disguise” as other frequencies

Aliasing in video

Slide by Steve Seitz

Aliasing in images

What’s happening?

Input signal:

x = 0:.05:5; imagesc(sin((2.^x).*x))

Plot as image:

Alias! Not enough samples

Antialiasing

• What can we do about aliasing?
• Sample more often

– Join the Mega-Pixel craze of the photo industry – But this can’t go on forever

• Make the signal less “wiggly”

– Get rid of some high frequencies – Will loose information – But it’s better than aliasing

Preventing aliasing

• Introduce lowpass filters:

– remove high frequencies leaving only safe, low frequencies – choose lowest frequency in reconstruction (disambiguate)

Sampling and Reconstruction

Image Acquisition

Imaging subsystem *h(x,y) Sampling Subsytem *s(x,y) Quantizer

f(x,y) g(x,y) gs(x,y) gd(x,y)

Digitized Image Scene Image Sampled Image

g(x,y)=f(x,y)*h(x,y) The scene is mapped to the image via the point spread function of the camera h(x,y) gs(x,y)=g(x,y)s(x,y) The continuous image g(x,y) then enters A sampling subsystem 2D comb function s(x,y) given by:

• m

n

n y m x y x s , ,

• Therefore, gs(x,y) is now a discrete-continuous

image with intensity values defined only over integral indices of x and y After sampling, the quantizer takes the continuous Intensity values and quantizes them (either uniform or non-uniform)

Sampling (I)

We start with continuous signal f(x). Due to the psf We obtain a degraded, band-limited output g(x). The frequency content of g(x) is given by its spectrum, G(f), as determined by the Fourier transform:

• fx

ic fx e dx e x g f G

fx i fx i

• 2

sin 2 cos

2 2

• Where x represents the spatial position and f denotes

Spatial frequency. Since g(x) does not have any frequencies beyond fmax, we say that it is band-limited. |G(f)|

• fmax

fmax

Sampling (II)

The continuous output g(x) is then digitized by an ideal Impulse sampler, the comb function, to get the sampled signal gs(x). The ideal 1-D sampler is given by:

• n

s

nT x x s

• where \delta is the impulse function and T_s is the sampling
• period. The running index n is used with \delta to define the

impulse train of the comb function. We now have:

x s x g x gs

Sampling (III)

Taking the Fourier transform of gs(x) yields

• n

s s n s s s

nf f G f nf f f f G f S f G f G * *

• Where fs is the sampling frequency and * denotes convolution.

We make use of the following properties of the Fourier Transform 1. Multiplication in the spatial domain corresponds to convolution in the frequency domain. 2. The Fourier transform on an impulse train is itself an impulse train. 3. The spectrum of a signal sampled with frequency fs (Ts=1/fs) yields the

• riginal spectrum replicated in the frequency domain with period fs

|Gs(f)|

• fmax

fmax

• fs

fs

Sampling (IV)

The above result reveals that the sampling operation has left the

• riginal input spectrum intact, merely replicating it periodically in

the frequency domain with a spacing of f_s. This allows us to rewrite G_s(f) as a sum of two terms, the low frequency (baseband) and high frequency components. The baseband specturm is exactly G(f), and the high frequency components, G_high(f), consist of the remaining replicated versions of G(f). G_s(f) = G(f) + G_high(f) Exact signal reconstruction from sampled data requires us to discard the replicated spectra G_high(f), leaving only G(f), the spectrum of the signal we seek to recover.

Reconstruction Conditions

The only provision for exact reconstruction is that G(f) be undistorted due to overlap with G_high(f). Two conditions must hold for this to be true: 1. The signal must be bandlimited. This avoids spectra with infinite extent that are impossible to replicate without

• verlap. (In most imaging systems, the psf ensures this

condition is satisfied in practice). 2. The sampling frequency f_s must be greater than twice the maximum frequency f_max, present in the signal. This minimum sampling frequency, known as the Nyquist rate, is the minimum distance between spectra copies, each with bandwidth f_max. f_s>f_nyquist=2*f_max

The Ideal Low-Pass Filter

Given that it is theoretically possible to perform reconstruction, how may it be done? The act of reconstruction requires us to completely suppress G_high(f). This is done by multiplying G_s(g) with H(f) given as:

• max

max

1 f f f f f H

|Gs(f)|

• fmax

fmax

• fs

fs

H(f)

Sinc Function

In the spatial domain, the ideal low-pass filter is derived by computing the inverse Fourier transform of H(f). This yields the sinc function defined as: Since multiplication in the frequency domain is identical to convolution in the spatial domain, then we can convolve the sample signal g_s(x) with sinc(x) to obtain the continuous g(x). The problem is that sinc(x) requires infinite support which is impossible in practice.

• x

x x

• sin

sinc

• d

x g x g x x g

s s

• sinc

* sinc

Ringing effect

When using a truncated sinc function for reconstruction we get

• ringing. These artifacts, known as the Gibbs phenomenon, are

the overshoots and undershoots caused by reconstructing a signal with truncated frequency terms. Approximations to the ideal low pass filter balance between tampering frequencies below f_max and not suppressing fully frequencies above it.

Nonideal Reconstruction

Since H(f), the ideal filter cannot be computed in practice, we resort to nonideal filters.This bring up the problem of how to assess the quality of a filter

|G(f)|

• fmax

fmax

• fs

fs

Hr(f)

H_r(f) attenuate some frequencies that should remain intact and pass and pass some of the high frquency that should be suppressed. This leads to the problem of aliasing: aliasing: signals “traveling in disguise” as other frequencies high frequencies disguised as low frequencies

Aliasing

aliasing: signals “traveling in disguise” as other frequencies It can be solved in one of two ways: 1. Increasing sampling rate (to reach Nyquist rate) 2. Applying low pass filter (to bandlimit the signal)

Aliasing in video

Slide by Steve Seitz

Image Resampling

So, we have a discrete image and we want to warp it to another discrete

• image. The two coordinate systems are related by a spatial warping
• function. This permits the output image to be generated by the following

straightforward procedure. First, the inverse mapping function is applied to the output sampling grid, projecting it onto the input. The result is a resampling grid, specifying the locations at which the input is to be

• resampled. Then, the input image is sampled at these points and the

values are assigned to their respective output pixels. The problem is that resampling grid does not coincide with the input sampling grid. The solution is to reconstruct the input image to a continuous image and then do the sampling.

Ideal Image Resampling

There are four basic elements to ideal image resampling: reconstruction, warping, prefiltering and sampling.

The Bible

• http://www-2.cs.cmu.edu/~ph/texfund/texfund.pdf

Resampling

2D texture space 2D screen space warp due to perspective

Notations

• Input signal f(u)
• Forward mapping (texture-to-screen)

x=m(u)

• Output signal g(x)

f(u) g(x) m

Resampling

• What do we need to do?

f(u) g(x) m

Resampling

1. Reconstruct the continuous signal from the discrete input signal 2. Warp the domain of the continuous signal 3. Prefilter the warped continuous signal 4. Sample this signal

f(u) g(x) m

Resampling

Resampling: progression

• Discrete input texture f(u) for integer u
• Reconstructed input texture

fc(u) = f(u) * r(u) = _{k \in Z} f(k) r(u-k)

• Warped texture gc(x)=fc(m-1(x))
• Band-limited output g’c(x) = gc(x) * h(x) = \Integral gc(t) h(x-t) dt
• Discrete output g(x) = g’c(x) s(x)

f(u) g(x) m

Resampling

discrete input warp warped input reconstructed input reconstruct continuous output prefilter source space resampling with prefiltering [Heckbert 89] discrete output sample destination space

Put it together

• Discrete input texture f(u) for integer u
• Reconstructed input texture fc(u) = f(u) - r(u) = _k f(k) r(u-k)
• Warped texture gc(x)=fc(m-1(x))
• Band-limited output g’c(x)=gc(x)*h(x) = S gc(t) h(x-t) dt
• Discrete output g(x) = g’c(x)- s(x)
• g(x) = gc’(x)

= s gc(t) h(x-t) dt = s fc(m-1(t)) h(x-t) dt = s h(x-t) å f(k) r(m-1(t)-k) dt = _k f(k) (x, k)

• Where (x,k) = s

h(x – t) r (m-1(t) –k) dt

• Ignoring normalization
• The image space resampling filter combines a warped reconstruction

filter and a low-pass filter

• The signal x, in the output image is warped to the input image through

m^{-1}. Since it falls on non-integer pixel value we use the reconstruction filter r to interpolate the discrete input signal f, and to make sure the result conforms to the Nyquist rate we bandlimit it with the function h

Resampling – convolution view

• i

i i

x h x m r x f x g ) ( )) ( ( ) ( ) (

1

resampling filter

Resampling

reconstruction kernels reconstructed input position color irregular spacing

Resampling

Source Space

• 3. Filter

Destination Space

• 2. Warp

Destination Space

• 4. Sample

Destination Space

Resampling

low-pass filter convolution resampling filters sum of resampling filters warped reconstruction kernel

Source Space

• 3. Filter

Destination Space

• 2. Warp

Destination Space

• 4. Sample

Destination Space

• Ignoring normalization
• The image space resampling filter combines a warped

reconstruction filter and a low-pass filter

• This is great, but how do we

warp reconstruction filters?

Resampling – convolution view

• i

i i

x h x m r x f x g ) ( )) ( ( ) ( ) (

1

resampling filter

Resampling

• Use local affine approximation of warp
• Elliptical Gaussian kernels [Heckbert 89]

– Closed under affine mappings and convolution

• i

i i i

x h x m r c x g ) ( )) ( ~ ( ) (

1

• i

i i

x G c ) (

Gaussian resampling kernel (EWA resampling kernel)

Resampling filter

• Depends on local warp
• For perspective,

approximated by local affine at center of kernel

• Not bad approximation

because filter small at periphery

Resampling Filter

warped recon- struction kernel low-pass filter resampling filter

minification magnification

EWA resampling

Image Quality Comparison

• Trilinear mipmapping

EWA trilinear mipmapping

Minification/Magnification

In the special case of magnification, we may ignore the prefilter altogether, treating it instead as an impulse function. This is due to the fact that no high frequencies are introduced into the output upon magnification. So we only have to do reconstruction (i.e., interpolation). Conversely, minifcation introduces high frequencies and does not require any reconstruction (i.e., interpolation) filter and treat it simply as an impulse function. Therefore, 1) For magnification we only need to interpolate 2) For minification we need to low-pass filter and then sub-sample.

For magnification, the shape of the reconstruction filter does not change in response to the mapping function. In minifcation the reconstruction filter is replaced by a prefilter. Unlike reconstruction filters, though, the actual shape must be scaled by an amount linearly related to the minifcation

• factor. As the input in increasingly decimated, the prefilter must become

broader and shorter. It becomes broader in order to average more neighboring pixel together, thereby further bandlimiting the input. Since larger neighborhood are used to compute each output pixel, the normalized weights applied to the input decrease to reflect the diminishing impact of each input sample. As a result, the prefilter grows shorter.

Interpolation

Interpolation is the process of determining the values of a function at positions lying between its samples. I achieves this process by fitting a continuous function through the discrete input samples. For equally spaced data, interpolation can be expressed as: Where h is the interpolation kernel weighted by coefficients c_k and applied to K data sample x_k

• 1

K k k k

x x h c x f

Interpolation Kernels

Nearest neighbor method is given by the following interpolating polynomial: It can be achieved by convolving the image with a one-pixel width rectangle in the spatial domain. The interpolation kernel for the nearest neighbor algorithm is defined as Due to the prominent side lobes and infinite extent, a sinc function makes a poor low-pass filter.

• 2

2

1 1

• k

k k k k

x x x x x x f x f

• x

x x h 5 . 5 . 1

Interpolation Kernels

Linear interpolation method is given by the following interpolating polynomial: In the spatial domain, linear interpolation is equivalent to convolving the sampled input with the following interpolation kernel (tent filter)

• 1

1 1 1 1 1 1

polynomial ing interpolat following the gives This 1 1 : solving by determined are , where , f f x x x x f x f x x a a f f a a a x a x f

• !

! " #

• x

x x x h 1 1 1

Image Morphing History

Morphing is turning one image into another through a seamless transition

Michael Jackson’s “Black or White” Cross-fading