Personal Photo Enhancement using Example Images Neel Joshi - - PowerPoint PPT Presentation

▶
personal photo enhancement using example images
SMART_READER_LITE
LIVE PREVIEW

Personal Photo Enhancement using Example Images Neel Joshi - - PowerPoint PPT Presentation

Aug 03, 2023 481 likes •851 views

Personal Photo Enhancement using Example Images Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research, Adobe Research, MERL, MIT CSAIL, and UCSD Motivation and Approach 2 It is difficult

slide-1
SLIDE 1

Personal Photo Enhancement using Example Images

Neel Joshi Wojciech Matusik, Edward H. Adelson, and David J. Kriegman Microsoft Research, Disney Research, Adobe Research, MERL, MIT CSAIL, and UCSD

slide-2
SLIDE 2

2

Motivation and Approach

  • It is difficult for most users to fix their

images

  • It’s easier for users to rate their good

photos

  • Use examples of a persons good photos

to fix the bad ones automatically

 X

slide-3
SLIDE 3

3

Our Approach

  • Focus on images with faces
  • Use a known face as a calibration object
  • Users provide good examples, instead

performing manual edits

X X

slide-4
SLIDE 4

4

Previous Work

  • Deblurring and Upsampling/Super-Resolution
  • Poisson image/noise models [Richardson 1972; Lucy 1974]; Sparse gradient

priors [Fergus et al. 2006; Levin 2006; Levin 2007]; Sparse wavelet coefficients [de Rivaz 2001]; Spatially Varying [Whyte et al. 2010; Gupta et al. 2010]; Baker and Kanade 2000; Freeman et al. 2000; Freeman et al. 2002; Liu et al. 2007; Dai et al. 2007; Fattal 2007

  • Denoising
  • Sparse wavelet coefficients [Simoncelli and Adelson 1996; Portilla et al. 2003],

Anisotropic diffusion [Perona and Malik 1990], Field of Experts [Roth and Black 2005];, Baker and Kanade 2000; Freeman et al. 2000; Freeman et al. 2002; Liu et

  • al. 2007; Dai et al. 2007; Fattal 2007
  • White-Balancing/Color Correction
  • Finlayson et al. 2004, 2005; Weijer et al. 2007
  • Using photo collections
  • Baker and Kanade 2000, Liu et al. 2007 , Dale et al. 2009
  • Hardware Methods
  • Joshi et al. 2010, Raskar et al. 2008, Levin et al. 2008, Veeraraghavan et al. 2007,

Levin et al. 2007, Raskar et al. 2006, Ben-Ezra et al. 2005, Ben-Ezra and Nayar 2004

slide-5
SLIDE 5

5

Specific vs. General Priors

  • We use an identity specific prior

Generic Image Prior Multi-Image

Field of Experts [Roth and Black] Sparse Prior [Levin et al.] Example Based [Freeman et al.] Our Approach

X

Photo Collections [Dale et al.]

slide-6
SLIDE 6

6

Facespace

  • Faces are a subspace of all images
  • Eigenfaces -- Turk and Petland 1987
  • Person-specific space is relatively small
  • The range of images can be captured

with a few good examples

slide-7
SLIDE 7

7

Personal Image Enhancement Pipeline

FACE DETECTION ALIGNMENT

GLOBAL AND LOCAL

ENHANCEMENT

FINAL ENHANCED IMAGE GOOD IMAGES BAD IMAGE INTRINSIC IMAGE DECOMPOSITION INTRINSIC IMAGE DECOMPOSITION

slide-8
SLIDE 8

8

Intrinsic Images [Land and McCann 1971,Barrow and Tenenbaum 1978]

  • Separation into Lighting, Texture, Color Layers
  • Use base/detail decomposition of Eisemann and Durand

2004

Input Image Chroma R Detail/Texture Chroma G Lighting

slide-9
SLIDE 9

9

Image Enhancements

  • Blur (Global)
  • Color/Exposure Balance

(Global)

  • Super-Resolution/Up-

sampling

slide-10
SLIDE 10

10

Image Enhancements

  • Blur
  • Color/Exposure Balance
  • Super-Resolution/Up-

sampling

slide-11
SLIDE 11

11

Blur Formation

= ⊗

Blurry image Blur kernel (Point-Spread Function)

+

Zero Mean Gaussian Noise Sharp image

Convolution

slide-12
SLIDE 12

12

Blur Estimation Goal

= ⊗

Blurry image Blur kernel

+

Zero Mean Gaussian Noise Sharp image Known Unknown Known σ

slide-13
SLIDE 13

13

Deblurring: Multiple Possible Solutions

= ⊗

Blurry image Sharp image Blur kernel

= ⊗ = ⊗

slide-14
SLIDE 14

14

Eigenspace

  • Identity Specific Images are used to build an aligned eigenspace

Mean Face Eigenvectors * 3 *σ + Mean Face Eigenvectors * -3 *σ + Mean Face

slide-15
SLIDE 15

15

( )

( ) ( )

2 4 3 2 8 . 1 2 ,

) ( min arg , K K I I I K I B K I

p T K I

∇ + + − + − Λ Λ + ∇ + ⊗ − = λ λ µ µ ρ λ λ σ ρ Eigenspace used for Blind Deconvolution

  • Eigenspace used as a linear constraint
  • Robust norm
  • Sparsity and smoothness priors on the Kernel
  • Solved using an Multi-Scale EM style algorithm

B = Blurry Image I = Sharp Prediction Λ = Eigenbasis vectors µ = Mean Vector ρ(.) = Robust Norm σ = Noise standard deviation λ = Regularization parameter p < 1

Data Term Sparse Prior

slide-16
SLIDE 16

16

Image Enhancements

  • Blur
  • Color/Exposure Balance
  • Super-Resolution/Up-

sampling

slide-17
SLIDE 17

17

Image Enhancements

  • Blur
  • Color/Exposure Balance
  • Super-Resolution/Up-

sampling

slide-18
SLIDE 18

18

Color Correction: Multiple Possible Solutions

= X

Observed image White-balanced Image Lighting Color

= X

slide-19
SLIDE 19

19

White Balance and Exposure Correction

  • Diagonal white balancing matrix (scales r and g

independently)

  • Exposure adjustment scales lighting layer

( )

r C C

r r C r

r

− = µ ρ min arg

Cr = r scale Cg = g scale CL = L scale µr = Mean r Vector µg = Mean g Vector µL = Mean L Vector ρ(.) = Robust Norm

( )

g C C

g g C g

g

− = µ ρ min arg

( )

L C C

L L C L

L

− = µ ρ min arg

slide-20
SLIDE 20

20

Image Enhancements

  • Blur
  • Color/Exposure Balance
  • Super-Resolution/Up-

sampling

slide-21
SLIDE 21

21

Image Enhancements

  • Blur
  • Color/Exposure Balance
  • Super-Resolution/Up-

sampling

slide-22
SLIDE 22

22

Face Correction: Patch Based [Freeman et al. 2000, Liu et al.

2007]

  • High-frequencies hallucinated by minimizing the energy of patch-based Markov

network

  • Two types of energies:
  • external potential — to model the connective statistics between two linked

patches in and .

  • internal potential — to make adjacent patches in

smooth.

  • Energy minimization by raster scan [Freeman et al. 2000]

g H

I ) (v I g

H

 ) (v N l

H

l H

I

) (v S 

) (v I l

H

I

L H

I

G H

I

L H

I

slide-23
SLIDE 23

Results

slide-24
SLIDE 24

24

Camera Motion Blur (Global Correction)

Good Example Images

slide-25
SLIDE 25

25

Exposure Correction and White-Balancing

Good Example Images

slide-26
SLIDE 26

26

Defocus Blur (Local Correction)

Good Example Images

slide-27
SLIDE 27

27

Upsampling (Local Correction)

Good Example Images

slide-28
SLIDE 28

Comparisons

slide-29
SLIDE 29

29

Comparisons to Previous Work

Our Result Fergus et al. 2006

slide-30
SLIDE 30

30

Comparisons to Color Constancy [Weijer et al. 2007 ]

Grayworld Shades of Gray Our Results Grayedge MaxRGB

slide-31
SLIDE 31

31

Using Generic Faces

Our Result Liu et al. 2007 Our Result Liu et al. Generic (10) Generic (50) Generic Faces (10) Generic Faces (50)

slide-32
SLIDE 32

32

Our Result Liu et al. 2007 Generic Faces (10) Generic Faces (50)

Using Generic Faces

Input

slide-33
SLIDE 33

33

Discussion/Future Work

  • Latent photo may not be well modeled by

the Eigenspace

  • All parts of the Eigenspace may not be

equally likely

  • A prior on the distribution within the

Eigenspace

  • Better non rigid alignment/morphable

model

  • Personalized Enhancement on

camera/phone

slide-34
SLIDE 34

34

Contributions

  • We use good examples of known face images

for corrections

  • Faces are used as calibration objects for

global corrections

  • We can further improve the faces in images
  • Identity-specific priors out-perform generic

priors

slide-35
SLIDE 35

35

Thank You! http://research.microsoft.com/en- us/um/people/neel/personal_photos/