Image Analysis Stuart Geman (with E. Borenstein, L.-B. Chang, W. - PowerPoint PPT Presentation

Generative Hierarchical Models for Image Analysis Stuart Geman (with E. Borenstein, L.-B. Chang, W. Zhang)

I. Image modeling II. Data likelihood III. Priors: content/context sensitivity

I. Image modeling • Red herrings? • Bayesian (generative) image models II. Data likelihood III. Priors: content/context sensitivity

Practical vision problems: What is the end-product of processing? machine vision: machine analysis human vision: “The more you look, the more you see”

Learning Theory: Pure learning ( label ) x Tree  ( image ) y ( ) black box No Tree      N , , ,..., Given y x produce so that  1 2 k k 1 N k       N black box OPTIMAL classifier

Performance of stressed biological systems: Super-rapid response… In this circumstance: machine vision achieves biological performance

I. Image modeling • Red herrings? • Bayesian (generative) image models II. Data likelihood III. Priors: content/context sensitivity

I. Bayesian (generative) image models Prior I set of possible "interpretations" or "parses"  x I a particular interpretation ( ) P x probability model on I * very structured and constrained * organizing principles: hierarchy and reusability (Amit, Buhmann, Felzenszwalb, Mumford, Pogio, Yuille, Zhu, etc.) * non-Markovian (context/content sensitive) Data likelihood y image ( | ) P y x conditional probability model Posterior  ( | ) ( | ) ( ) P x y P y x P x

I. Image modeling II. Data likelihood • Feature distributions and data distributions • Conditional modeling • Examples: learning templates III. Priors: content/context sensitivity

Feature distributions and data distributions  y pixel intensity at s S S s  { } y y  s s S image patch Given a category (e.g. edge, corner, eye, face, (eye,pose ),…), model patch through a feature model: ( ) f y "feature" e.g.  ( ) for short f y variance of patch histogram of gradients, sift features, etc. template correlation ( ) for short P f   ( ; ) P f proba bility model F F    1 ,..., , Problem: given y y samples of eye patches, learn and N

Use maximum likelihood…but what is the likelihood? ( ),..., ( ) f y f y Tempting to PRETEND that the data is : 1 N N     ( ( ),..., ( ); , ) ( ( )) L f y f y P f y 1 N F k  1 k caution: this is different from 1  ( ) ( ( )) Y P y P f y F Z   , ,..., BUT the data is y y and N 1   ( ) ( ( )) ( | ( )) P y P f y P y F f y Y F Y N       ( ,..., ; , ) ( ( )) ( | ( )) L y y P f y P y F f y 1 N F k Y k k  1 k  The first is fine for estimating P (i.e. ), F  but not fine for estimating (i.e. ) f

I. Image modeling II. Data likelihood • Feature distributions and data distributions • Conditional modeling • Examples: learning templates III. Priors: content/context sensitivity

Conditional modeling  ( ) For any category (e.g. "eye") and feature g F f Y   g g g ( ) ( ( )) ( | ( )) P y P f y P y F f y Y F Y  g g ( ( )) ( | ). Easy to model P f y ; hard to model P y F f F Y Proposal: start with a "null" or "background" o g ( ) ( ) distribution P y and choose P y Y Y g ( ) 1. consistent with P f , and F o ( ) 2. otherwise "as close as possible" to P y Y

Conditional modeling: a perturbation of the null distribution g o ( ) ( ), Specifically, given P f , and a null distribution P y F Y choose  g o ( ) arg min ( || ) P y D P P Y Y Y : ( ) P F Y has Y g P ( f ) distribution F    g g o ( ) ( ( )) ( | ( )) P y P f y P y F f y Y F Y   ( ) P y % ( || ) ( )log (where D P P P y dy is K-L divergence) % ( ) P y

Estimation    g g 1 ,..., ( ) ( ; ) y y P f P f Given , ), and N F F    g g o ( ) ( ( )) ( | ( )) P y P f y P y F f y Y F Y   : estimate and   argmax ( ,..., ; , ) L y y 1 N   , g N ( ( )) P f y    argmax F k .... o ( ( )) P f y    1 k F k ,

In fact, for arbitrary mixture (e.g. over poses, templates, vector quanta, …):    g g ( ) ( ; ) 1,2,..., P f P f ), m M  F m F m  m m m M     g g o ( ) ( ( )) ( | ( )) P y P f y P y F f y Y m F m Y m m m  1 m       argmax ( ,..., ; ,..., ,..., ,..., ) L y y 1 1 1 1 N m m m       ,..., ,..., ,..., 1 m 1 m 1 m g ( ( )) P f y N M      argmax F m k .... m m o ( ( )) P f y         1 1 k m F m k ,..., ,..., ,..., m 1 m 1 m 1 m

I. Image modeling II. Data likelihood • Feature distributions and data distributions • Conditional modeling • Examples: learning templates III. Priors: content/context sensitivity

Example: learning eye templates  y pixel intensity at s S S s  { } y y  s s S image patch   ( ) ( ) ( , ), Take f y c y corr T y and model eyes as a  T mixture: M     e e o ( ) ( ( )) ( | ( )) P y P c y P y C c y Y m C T Y T T T m m m m  m 1 M        (1 c ( )) y o ( | ( )) = e m Tm P y C c y  m Y T T m m m m=1

o : Null distribution, P for estimation Y o ( ) only P c C T matters...   2 o o ( ) (0, P sample P c N ) Y C T 1   iid | | S 10   random | | S image patch 15   random | | S smooth image patch

Example: learning eye templates Examples of faces from Feret database  With N 500 compute     argmax ( ,..., | ,..., , ,..., , ,..., ) L y y T T 1 1 1 1 N m m m     ,..., ,..., T ,..., T 1 1 1 m m m     (1 ( )) c y e m T k N M m      argmax m m o ( ( )) P c y       1 k 1 m ,..., ,..., ,..., C T k T T T m 1 m 1 m 1 m m

Example: learning eye templates, mixing over position, scale, and template samples from training set learned templates Top to bottom: EM iterations

Example: learning (right) eye templates What if we forget all this nonsense and just maximize     (1 ( )) c y e m T k N M N M m           (1 ( ))  c y ) e m T k (instead of ? m m  m m o ( ( )) P c y m     1 1 k 1 m k 1 m C T k T m m

How good are the templates? A classification experiment… Classify East Asian and South Asian * mixing over 4 scales, and 8 templates East Asian: (L) examples of training images (M) progression of EM (R) trained templates South Asian: (L) examples of training images (M) progression of EM (R) trained templates Classification Rate: 97%

Other examples: noses 16 templates multiple scales, shifts, and rotations samples from training set learned templates

Other examples: mixture of noses and mouths samples from training set 32 learned templates (1/2 noses, 1/2 mouths)

Other examples: train on 58 faces …half with glasses…half without samples from training set 32 learned templates 6 learned templates

Other examples: train on 58 faces …half with glasses…half without 6 learned templates random eight of the 58 faces row 2 to 5, top to bottom: templates ordered by posterior likelihood

Other examples: train on 58 faces …half with glasses…half without top row: the six learned templates row 2 to 5, top to bottom: Training images ordered by correlation

Other examples: train random patches (“sparse representation”) 500 random 15x15 training patches from 24 10x10 templates random internet images

Other examples: coarse representation   ( ) ( , ( )), use f y Corr T D y where D downconvert  ( ) ( ( ), )?) (go other way for super res.: f y Corr D T y training of 8 low-res (10x10) templates

Grenander : “pattern synthesis=pattern analysis” (approximate) sampling… 0 32 samples from mixture model with P white noise Y

(approximate) sampling…  0 32 samples from mixture model with P Caltech 101 Y

(approximate) sampling… 0 32 samples from mixture model with P population of Y smooth image patches

I. Image modeling II. Data likelihood III. Priors: content/context sensitivity • Hierarchical models and the Markov dilemma • Conditional modeling • Examples: detecting faces and reading license plates