Guided Signal Reconstruction with Application to Image Magnification - - PowerPoint PPT Presentation

MITSUBISHI ELECTRIC RESEARCH LABORATORIES Cambridge, Massachusetts

Guided Signal Reconstruction with Application to Image Magnification

Akshay Gadde (USC, MERL) Speaker: Andrew Knyazev (MERL) Hassan Mansour (MERL) Dong Tian (MERL)

c MERL December 16, 2015 1 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Outline

• 1. Introduction

Problem Definition and Motivation Related Work

• 2. Reconstruction Set

Geometric Interpretation Algorithm for Finding the Reconstruction Set Relation to Regularized Reconstruction

• 3. Experiments
• 4. Conclusion and Future Work

c MERL December 16, 2015 2 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Outline

• 1. Introduction

Problem Definition and Motivation Related Work

• 2. Reconstruction Set

Geometric Interpretation Algorithm for Finding the Reconstruction Set Relation to Regularized Reconstruction

• 3. Experiments
• 4. Conclusion and Future Work

c MERL December 16, 2015 3 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Problem Definition

sample reconstruct sampling subspace

• Lossy measurements
• Prior information about the signal ⇒ Guiding subspace T ⊂ H

f ∈ T

• r

T⊥f small

c MERL December 16, 2015 4 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Problem Definition

sample reconstruct sampling subspace

• Lossy measurements
• Prior information about the signal ⇒ Guiding subspace T ⊂ H

f ∈ T

• r

T⊥f small

Questions

Conditions on S and T for:

• Uniqueness of reconstruction
• Stability of reconstruction
• Efficient algorithm for reconstruction
• Effect of noise and model mismatch

c MERL December 16, 2015 4 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Motivation

• Image magnification
• S : 2 × 2 averaging
• T : low pass DCT

τ τ

c MERL December 16, 2015 5 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Motivation

• Image magnification
• S : 2 × 2 averaging
• T : low pass DCT
• Semi-supervised

learning

? ? ? ?

• S = {x|xU = 0}
• T : low pass GFT

T = K

• i=1

ciui

• ,

{ui} e.v.’s of graph L

τ τ

c MERL December 16, 2015 5 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Motivation

• Image magnification
• S : 2 × 2 averaging
• T : low pass DCT
• Semi-supervised

learning

? ? ? ?

• S = {x|xU = 0}
• T : low pass GFT

T = K

• i=1

ciui

• ,

{ui} e.v.’s of graph L

• Bandwidth expansion of

speech

τ τ

50 100 150 200 250 300 350 0Hz Khz 8Khz 0Hz Khz Khz 0Hz Khz Khz

(i) (ii) (iii)

• S : low pass DFT
• T : learned from data

c MERL December 16, 2015 5 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Related Work: Consistent Reconstruction

• Consistent reconstruction ˆ

f ⇔ Sˆ f = Sf (Unser and Aldroubi’94, Eldar’03)

Existence and Uniqueness

• Consistent reconstruction exists in T for any f ∈ H

iff T + S⊥ = H

• Consistent reconstruction is unique

iff T ∩ S⊥ = {0}

c MERL December 16, 2015 6 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Related Work: Consistent Reconstruction

• Consistent reconstruction ˆ

f ⇔ Sˆ f = Sf (Unser and Aldroubi’94, Eldar’03)

Existence and Uniqueness

• Consistent reconstruction exists in T for any f ∈ H

iff T + S⊥ = H

• Consistent reconstruction is unique

iff T ∩ S⊥ = {0}

• Under the above assumptions

ˆ f = PT ⊥Sf (oblique projection)

• If f ∈ T then ˆ

f = f

• If T ∩ S⊥ = {0} (non-unique consistent solutions), pick
• ne by imposing additional constraints

c MERL December 16, 2015 6 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Related Work: Generalized Reconstruction

• Existence of consistent reconstruction needs T + S⊥ = H
• Can lead to unstable reconstructions (if min. gap between T and S is large)
• Oversampling for stability can cause T + S⊥ ⊂ H

Generalized reconstruction

• Sample consistent plane Sf + S⊥
• ˆ

f ∈ T closest to Sf + S⊥ (relax Sf = Sˆ f) ˆ f = PT ⊥S(T )f

c MERL December 16, 2015 7 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Related Work: Generalized Reconstruction

• Existence of consistent reconstruction needs T + S⊥ = H
• Can lead to unstable reconstructions (if min. gap between T and S is large)
• Oversampling for stability can cause T + S⊥ ⊂ H

Generalized reconstruction

• Sample consistent plane Sf + S⊥
• ˆ

f ∈ T closest to Sf + S⊥ (relax Sf = Sˆ f) ˆ f = PT ⊥S(T )f Question: ˆ f ∈ Sf + S⊥ (consistent) or ˆ f ∈ T (generalized) or something else?

c MERL December 16, 2015 7 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Outline

• 1. Introduction

Problem Definition and Motivation Related Work

• 2. Reconstruction Set

Geometric Interpretation Algorithm for Finding the Reconstruction Set Relation to Regularized Reconstruction

• 3. Experiments
• 4. Conclusion and Future Work

c MERL December 16, 2015 8 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Reconstruction Set

• sample consistent place Sf + S⊥
• guiding subspace T

Reconstruction set

Shortest pathway between the consistent place and the guiding subspace min

ˆ f∈Sf+S⊥ min t∈T ˆ

f − t = min

ˆ f∈Sf+S⊥ t∈T

ˆ f − t = min

t∈T

min

ˆ f∈Sf+S⊥ ˆ

f − t

• ˆ

f: consistent reconstruction

• t: generalized reconstruction

c MERL December 16, 2015 9 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Iterative Consistent Reconstruction Using Cojugate Gradient

• Consistent reconstruction

inf

ˆ f

T⊥f subject to Sˆ f = Sf

c MERL December 16, 2015 10 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Iterative Consistent Reconstruction Using Cojugate Gradient

• Consistent reconstruction

inf

ˆ f

T⊥f subject to Sˆ f = Sf

Consistent reconstruction using CG

Define ˆ x = (ˆ f − Sf) ∈ S⊥. Then the above problem is equivalent to solving (S⊥T⊥)

• S⊥x = −S⊥T⊥Sf

(Sf : measurement)

• Restriction of S⊥T⊥ to S⊥ is self-adjoint
• Use CG with initialization x0 ∈ S⊥
• CG: most efficient iterative method for solving linear systems
• Frame-less algorithm: Needs only the (approximate) projector T

c MERL December 16, 2015 10 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Finding the Reconstruction Set

min

ˆ f∈Sf+S⊥ min t∈T ˆ

f − t = min

ˆ f∈Sf+S⊥ t∈T

ˆ f − t = min

t∈T

min

ˆ f∈Sf+S⊥ ˆ

f − t

• ˆ

f: consistent reconstruction

• t: generalized reconstruction
• Relation between ˆ

f and t t = Tˆ f

c MERL December 16, 2015 11 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Finding the Reconstruction Set

min

ˆ f∈Sf+S⊥ min t∈T ˆ

f − t = min

ˆ f∈Sf+S⊥ t∈T

ˆ f − t = min

t∈T

min

ˆ f∈Sf+S⊥ ˆ

f − t

• ˆ

f: consistent reconstruction

• t: generalized reconstruction
• Relation between ˆ

f and t t = Tˆ f Reconstruction Set = {αˆ f + (1 − α)Tˆ f, where α ∈ [0, 1]}

c MERL December 16, 2015 11 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Connection with Regularization Reconstruction by regularization

inf

ˆ fρ

• Sˆ

fρ − Sf

• 2

+ ρ

• ˆ

fρ − Tˆ fρ

• 2

, ρ > 0

c MERL December 16, 2015 12 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Connection with Regularization Reconstruction by regularization

inf

ˆ fρ

• Sˆ

fρ − Sf

• 2

+ ρ

• ˆ

fρ − Tˆ fρ

• 2

, ρ > 0

Theorem (Reconstruction set and Regularization)

Let ˆ f be the consistent reconstruction given by inf

ˆ f

T⊥f subject to Sˆ f = Sf. The reconstruction set is given by {ˆ fα = αˆ f + (1 − α)Tˆ f, where 0 ≤ α ≤ 1}. Then ˆ fα is a solution of the regularized reconstruction problem with ρ = (1 − α)/α.

• If a unique ˆ

f ∈ T ∩ (Sf + S⊥) exists, then ˆ fρ = ˆ f = Tˆ f ∀ ρ > 0

• No need to re-solve the regularization problem if ρ changes

c MERL December 16, 2015 12 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Reconstruction in the Presence of Noise

• Noisy measurements: Sf ′ = Sf + e ⇒ Original signal f /

∈ (Sf ′ + S⊥)

• Trust the guiding more than the samples
• Let ˆ

f ∈ Sf ′ + S⊥ be the consistent solution ⇒ Good solution is ˆ fα = αˆ f + (1 − α)Tˆ f with α > 0

Good choice of α

Noise energy e. Then pick α such that 1 − α = e ˆ f − Tˆ f ⇒ ˆ fα = ˆ f − e ˆ f − Tˆ f ˆ f − Tˆ f

• Assumes that noise is orthogonal to T

c MERL December 16, 2015 13 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Outline

• 1. Introduction

Problem Definition and Motivation Related Work

• 2. Reconstruction Set

Geometric Interpretation Algorithm for Finding the Reconstruction Set Relation to Regularized Reconstruction

• 3. Experiments
• 4. Conclusion and Future Work

c MERL December 16, 2015 14 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Experiment I: Problem Setting

• Original signal f: w × w image
• Sampling subspace S: Sf = BSB∗

Sf, where

B∗

S : r × r averaging then downsampling

BS : upsampling by copying each pixel in r × r block

• Guiding subspace T : k × k low pass bandlimited DCT
• kscale = (w/r)/k: relative dimensionality of S and T

kscale < 1 : undersampling kscale > 1 : oversampling

c MERL December 16, 2015 15 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Experiment II: Noiseless Reconstruction

ˆ fc: Consistent, ˆ fg = Tˆ fc: Generalized, ˆ fm = TSf: minimax regret (Eldar et al’06)

1 2 3 4 15 20 25 30 kscale PSNR (dB) fc fg fm fα=0.7

(a) α = 0.7

0.5 1 18 20 22 24 26 28 α PSNR (dB) fα

(b) kscale = 4

• kscale < 1 (undersampling): ˆ

fc = ˆ fg better than ˆ fm

• kscale > 1 (oversampling): ˆ

fc better than ˆ fg and ˆ fm

• Since samples are noiseless, reconstruction improves as α increases

c MERL December 16, 2015 16 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Experiment III: Reconstruction in the Presence of Noise

• Sf ′ = Sf + e, where e is iid N(0, 0.001) ⇒ αopt = 0.7

0.5 1 19 20 21 22 23 α PSNR (dB) fα

(a) kscale = 4

1 2 3 4 18 20 22 24 kscale PSNR (dB) fc fg fm fα=0.7

(b) α = 0.7 (a) Sf ′, 21.69dB (b) ˆ fg, 19.73dB (c) ˆ fc, 22.00dB (d) ˆ fα=0.7, 22.88dB

c MERL December 16, 2015 17 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Outline

• 1. Introduction

Problem Definition and Motivation Related Work

• 2. Reconstruction Set

Geometric Interpretation Algorithm for Finding the Reconstruction Set Relation to Regularized Reconstruction

• 3. Experiments
• 4. Conclusion and Future Work

c MERL December 16, 2015 18 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

Conclusion

• Unified view of different reconstruction methods
• Novel formulation of the reconstruction set
• Efficient reconstruction algorithm for finding the reconstruction set
• Connection with regularization and reconstruction with noisy samples

Future work

• Error bounds based on
• noise
• model mismatch
• relative positions of S and T
• Applications in other areas: speech, video, machine learning

c MERL December 16, 2015 19 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

References I

Akshay Gadde, Aamir Anis, and Antonio Ortega. Active semi-supervised learning using sampling theory for graph signals. Accepted in ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2014. Sunil K Narang, Akshay Gadde, Eduard Sanou, and Antonio Ortega. Localized iterative methods for interpolation in graph strutured data. In IEEE Global Conference on Signal and Information Processing (GlobalSIP), pages 491–494, 2013. Yonina C Eldar and Tsvi G Dvorkind. A minimum squared-error framework for generalized sampling. Signal Processing, IEEE Transactions on, 54(6):2155–2167, 2006. Yonina C Eldar. Sampling with arbitrary sampling and reconstruction spaces and oblique dual frame vectors. Journal of Fourier Analysis and Applications, 9(1):77–96, 2003.

c MERL December 16, 2015 20 / 21

MITSUBISHI ELECTRIC RESEARCH LABORATORIES

References II

Michael Unser and Akram Aldroubi. A general sampling theory for nonideal acquisition devices. Signal Processing, IEEE Transactions on, 42(11):2915–2925, 1994. Ben Adcock and Anders C Hansen. A generalized sampling theorem for stable reconstructions in arbitrary bases. Journal of Fourier Analysis and Applications, 18(4):685–716, 2012. Peter Berger and Karlheinz Grchenig. Sampling and reconstruction in different subspaces by using oblique projections. Technical Report arXiv:1312.1717 [math.NA], December 2013. Dhananjay Bansal, Bhiksha Raj, and Paris Smaragdis. Bandwidth expansion of narrowband speech using non-negative matrix factorization. In Ninth European Conference on Speech Communication and Technology, 2005.

• A. Hirabayashi and M. Unser.

Consistent sampling and signal recovery. Signal Processing, IEEE Transactions on, 55(8):4104–4115, Aug 2007.

c MERL December 16, 2015 21 / 21