Super-Resolution via Image Recapture and Bayesian Effect Modeling - - PowerPoint PPT Presentation

Super-Resolution via Image Recapture and Bayesian Effect Modeling

Neil Toronto Oral Thesis Defense Department of Computer Science Brigham Young University December 2008

Super-Resolution via Image Recapture and Bayesian Effect Modeling 2

Single-Frame Super-Resolution (i.e. Good Image Magnification)

 Printing photos from a camera or the Internet  Compositing images  Signal conversion (e.g. DVD to HDTV)  Crime drama television, blackmail  Generally an underconstrained, ill-posed problem  General solution: make it well-posed and make

assumptions, then infer the extra information

Super-Resolution via Image Recapture and Bayesian Effect Modeling 3

Size vs. Apparent Resolution

Nearest neighbor (NN) Sinc ??? Bilinear Input

Super-Resolution via Image Recapture and Bayesian Effect Modeling 4

Nonadaptive Methods

 Assume the pixels are a function or signal sample  Families: function-fitting, frequency-domain  Implementation of both: sum up scaled copies of

the same fuzzy shape (kernel)

 Artifacts: Blocky: Blurry:  Only two possible solutions: get more data, make

stronger assumptions

Bilinear Bicubic Sinc

Super-Resolution via Image Recapture and Bayesian Effect Modeling 5

Adaptive Methods

 Make strong assumptions  Families: edge-preserving, training-

based, optimization

 Examples:

 Resolution synthesis (RS), local

correlation (LCSR): learn optimal kernels from examples, apply locally according to class

 Image analogies, Freeman’s MRFs:

Construct Frankenimages from Flickr

 Level-set reconstruction: optimize

upscaled result with respect to rewarding accuracy and penalizing jaggies

Local correlation (LCSR) Resolution synthesis (RS)

Super-Resolution via Image Recapture and Bayesian Effect Modeling 6

Quantitative Comparison



Popularly mean-squared error

• n downscaled and

reconstructed images, results compared against nonadaptive whipping-boys



Ouwerkerk 2006: First ever qualitative and quantitative survey of super-resolution

? ? ?

Decimation Super- resolution Original images Reconstructed images



Tested nine methods on seven test images using three measures

• f correctness



The winners: resolution synthesis (RS) and local correlation (LCSR)

Correctness measures

Super-Resolution via Image Recapture and Bayesian Effect Modeling 7

Motivation

RS Bayesian edge inference (BEI) 2x 4x LCSR

Objective: avoid these artifacts, be competitive on Ouwerkerk’s measures

Super-Resolution via Image Recapture and Bayesian Effect Modeling 8

Optimization Reconstruction Framework

 Reconstruction the mostly Bayesian way  Assumptions: an image I’ existed that was

degraded to produce I

 Task: given I, reconstruct I’  Reconstruct using Bayes’ Law:  Result is almost always argmax P(I’|I) (a “MAP

estimate”)

PI'∣I= PI∣I'PI' PI

Degradation Image prior

Super-Resolution via Image Recapture and Bayesian Effect Modeling 9

Modeling Mismatch 1: Printing a Full-Resolution Photo

There’s no higher-resolution original image to reconstruct

Super-Resolution via Image Recapture and Bayesian Effect Modeling 10

Modeling Mismatch 2: CCD Demosaicing

There’s no pristine, unfiltered original image to reconstruct

Super-Resolution via Image Recapture and Bayesian Effect Modeling 11

Recapture Reconstruction Framework



Assumptions: a scene S was captured with C to create image I



Task: given I (and possibly C), reconstruct S, and recapture it as I’ using a fictional C’

PI'∣I=⋯

Super-Resolution via Image Recapture and Bayesian Effect Modeling 12

Using the Recapture Framework

1 1 Define scene model S 2 2 Define capture process I|S,C and recapture process I’|S,C’ 3 3 Sample I’|I (the “posterior predictive” distribution) 3 3’s inference is familiar, but not always easy 1 1 2 2 3 3

Super-Resolution via Image Recapture and Bayesian Effect Modeling 13

Modeling Step Edges

 Goal: preserve edges and gradients  Assumption: scenes are mostly comprised

• f solid objects with coherent boundaries

 Capture ≈ global blur + sampling at discrete points  Scene model: a grid of linear discontinuities convolved

with blurring kernels (appx. spatially varying PSF)

*

Discrete sampling

= S

Discrete sampling Point-spread

I00 I01 I02 I10 I11 I12 I20 I21 I22

Super-Resolution via Image Recapture and Bayesian Effect Modeling 14

Spatially Varying Point-Spread

2x magnification Spatially varying point spread Dark = narrow BEI’s reconstruction

Super-Resolution via Image Recapture and Bayesian Effect Modeling 15

Scene Model: Causal or Noncausal?

Hierarchical (causal) Compatibility (noncausal)

Super-Resolution via Image Recapture and Bayesian Effect Modeling 16

Utility of Compatibility

No data Data No compatibility Compatibility Samples from prior predictive distribution I’ Samples from posterior predictive distribution I’|I 4x magnification

Super-Resolution via Image Recapture and Bayesian Effect Modeling 17

Bayesian Effect Modeling

 Compatibility is an

effect (noncausal)

 Capture is obviously

causal

 Need to mix the two

in the same model

 Solution: use the

transformation from MRFs to BNs, but locally

 Confines noncausal dependence to local subgraphs  Can’t create cycles

Super-Resolution via Image Recapture and Bayesian Effect Modeling 18

Minimum Blur and Decimation Blur

 Doing super-resolution without accounting for point-

spread gives blurry results

 I’ is naturally sharpened by adding minimum blur

variance in capture model and adding less in recapture

 Decimation: minimum blur converging on σ = 1/3  If decimation has occurred, model it by setting minimum

blur to 1/3 in capture and 1/(3s) in recapture

¼ ¼ ¼ ¼

*

¼ ¼ ¼ ¼

*

¼ ¼ ¼ ¼

* •••

Total σ2 = 1/9

*

Super-Resolution via Image Recapture and Bayesian Effect Modeling 19

Results: Boundary Coherence

4x magnification Decimated (NN) RS LCSR BEI

Super-Resolution via Image Recapture and Bayesian Effect Modeling 20

Results: Edges and Gradients

Original Decimated (NN) Bilinear NEDI LCSR RS BEI BEI 8x 4x magnification after two decimations

Super-Resolution via Image Recapture and Bayesian Effect Modeling 21

Results: Correctness Measures

Super-Resolution via Image Recapture and Bayesian Effect Modeling 22

Complementary Breakfast: CCD Demosaicing

 Missing data problem? Rev. Bayes

says it’s easy: just leave it out

 In CCD demosaicing tasks, 2/3 of the

data is missing: it was never collected

Original Bicubic interpolation BEI’s reconstruction

Super-Resolution via Image Recapture and Bayesian Effect Modeling 23

Complementary Breakfast: CCD Demosaicing

Original Bicubic interpolation BEI’s reconstruction

Super-Resolution via Image Recapture and Bayesian Effect Modeling 24

Complementary Breakfast: Inpainting and Restoration

 Inpainting can be seen as a missing-data problem  Could model defacement in the capture process

Defaced 33% BEI’s reconstruction

Super-Resolution via Image Recapture and Bayesian Effect Modeling 25

Complementary Breakfast: Inpainting and Restoration

Defaced 33% BEI’s reconstruction

Super-Resolution via Image Recapture and Bayesian Effect Modeling 26

Limitations and Future Work

 Slow: could just be Python’s fault, but there’s a

lot to compute

 Super-resolution’s canny ridge  Line and T-junction models  Throwing the recapture framework at every

reconstruction problem that still twitches

 Making Bayesian effect modeling into a

graphical model in its own right

Super-Resolution via Image Recapture and Bayesian Effect Modeling 27

Take-Home Messages

 In super-resolution (or reconstruction), make

strong assumptions cuz you ain’t gettin mo data

 Model capture, reconstruct scenes, recapture

results

 Potentially fits the actual process better  More flexible than modeling just degradation

 Explicitly model what you want to reason about

precisely (e.g. edges, sharpness, scenes)

 Compatibility is more tractible than hierarchy

and gives great results

 Missing data is no problem for Bayesians

Super-Resolution via Image Recapture and Bayesian Effect Modeling 28

Questions

? ?

Super-Resolution via Image Recapture and Bayesian Effect Modeling 29

An Intriguing Equivalence

≡

Reconstruction via recapture Supervised machine learning

Super-Resolution via Image Recapture and Bayesian Effect Modeling 30

Details: Effect Modeling

Compatibility to conditional density conversion: Joint density: Unnormalized complete conditional (Markov blanket) density:

Super-Resolution via Image Recapture and Bayesian Effect Modeling 31

Details: Definitions

Given an m x n image I. Image coordinates are properly a parameter of the capture process C. C’ also contains an array of coordinates. Compatibility and capture are defined in terms of nearest neighbors:

Super-Resolution via Image Recapture and Bayesian Effect Modeling 32

Details: Step Edges

Step edge geometry is expressed as an implicit line: Facet profiles are defined by Gaussian convolution and have an analytic solution: Because of symmetry, 2D step edges can be expressed in terms of their profiles.

Super-Resolution via Image Recapture and Bayesian Effect Modeling 33

Details: Scene Model Facets

The scene model random variables are a tuple of m x n arrays sufficient to determine step edges: It is helpful to think of the scene as a grid of functions, so these are defined for every index as A useful concept is the weighted expected scene output: This is used both in capture and compatibility.

Super-Resolution via Image Recapture and Bayesian Effect Modeling 34

Details: Scene Model Priors

Reasonable beliefs: edge geometries and intensities are uniformly probable and there are relatively few strong

• edges. These are summed up in the priors

Compatibility makes the model prefer regions of similar color and coherent object boundaries. It does both by making low output variance more probable.

Super-Resolution via Image Recapture and Bayesian Effect Modeling 35

Details: Capture and Blur

Discrete sampling is assumed to be Normally distributed (appx. white noise) about the expected scene output. Minimum blur Cσ is accounted for in both capture and recapture by summing variances. Slide 19 hints at this infinite series that computes variance due to decimation blur.

Super-Resolution via Image Recapture and Bayesian Effect Modeling 36

Details: Inference

Starting values seem to affect only time to convergence. BEI uses these, partially computed from the image data: BEI gets a MAP estimate for S|I using a hill-climbing algorithm based on Gibbs sampling. It samples at deterministic distances from current values and adapts using exponentially weighted moving variances: