9.54 Class 16 Features for recognition supervised, unsupervised - - PowerPoint PPT Presentation

9.54 Class 16

Features for recognition

supervised, unsupervised and innate

Shimon Ullman + Tomaso Poggio Danny Harari + Daniel Zysman + Darren Seibert

Visual recognition

The initial input is just image intensities

• - We perceive the world in term of objects and classes
• - Large variability within a each class

Object Categories

Individual Recognition

Object parts

Headlight Window Door knob Back wheel Mirror Front wheel Headlight Window Bumper

Categorization: dealing with class variability

Class Non-class

Natural for the brain, difficult computationally

Unsupervised Classification

Features and Classifiers

Features and Classifiers

Generic Features

Simple (wavelets) Complex (Geons)

Image features Classifier

Visual Class: Similar Configurations of Shared Image Components

What will be optimal image building- blocks for the class?

Optimal Class Components?

• Large features are too rare
• Small features are found

everywhere Find features that carry the highest amount of information

Mutual Information I(C,F) I(F,C) = H(C) – H(C|F)

  

  

   

C c F f F f

f c P Log f c P f p f F C H f p F C H )) ( ( ) ( ) ( ) ( ) ( ) (

• Definition of MI as the difference between

the class entropy and conditional entropy of the class given a feature:

• Definition of entropy:





 

C c

c P Log c P C H )) ( ( ) ( ) (

• Definition of conditional entropy:

Mutual Information I(C,F)

Class: 1 1 1 1 Feature 1 1 1 1

I(F,C) = H(C) – H(C|F)

Computing MI from Examples

• Mutual information can be measured from

examples:

• 100 Faces

100 Non-faces Feature: 44 times 6 times Mutual information: 0.1525

H(C) = 1, H(C|F) = 0.8475

Simple neural-network approximations

Optimal classification features

• Theoretically: maximizing delivered information minimizes classification

error

Error = H – I(C;F)

• In practice: informative object components can be identified in training images

Mutual Info vs. Threshold

0.00 20.00 40.00 Detection threshold Mutual Info

forehead hairline mouth eye nose nosebridge long_hairline chin twoeyes

Selecting Fragments

‘Imprinting’ many receptive fields and selecting a subset

Adding a New Fragment

(Avoiding redundancy by max-min selection) ?

ΔMI Maxi Mink ΔMI (Fi, Fk) Compare new fragments Fi to all the previous ones. Select F which maximizes the additional information

Competition between units with similar responses

Highly Informative Face Fragments

Optimal receptive fields for Faces

Ullman et al Nature Neuroscience 2002

Horse-class features Car-class features

Informative class features

Informative fragments with positions

∑wk Fk > θ

On all detected fragments within their regions

Star model

Detected fragments ‘vote’ for the center location Find location with maximal vote In variations, a popular state-of-the art scheme

Image parts informative for classification

Fergus, Perona, Zisserman 2003 Agarwal, Roth 2002

Ullman, Sali 1999

Variability of Airplanes Detected

Image representation for recognition HoG Descriptor

Dallal, N & Triggs, B. Histograms of Oriented Gradients for Human Detection

Object model using HoG

fMRI

Functional Magnetic Resonance Imaging

Looking for Class Features in the Brain: fMRI

Lerner, Epshtein Ullman Malach JCON 2008

Class-fragments and Activation

Malach et al 2008

EEG

Informative Fragments: ERP Study

Harel, Ullman, Epshtein, Bentin

ERP

FACE FEATURES milliseconds 200 400 600 200 400 600 milliseconds Left Hemisphere Right Hemisphere Posterior-Temporal sites FACE FEATURES milliseconds 200 400 600 200 400 600 milliseconds Left Hemisphere Right Hemisphere Posterior-Temporal sites

MI 1 — MI 2 — MI 3 — MI 4 — MI 5 —

Harel, Ullman,Epshtein, Bentin Vis Res 2007

Features for object segregation:

Innate mechanisms for unsupervised learning

Object Segregation

Object 1 Object 2 Background

Object segregation is learned

[Kellman & Spelke 1983; Spelke 1990; Kestenbaum et al., 1987]

5 months

Even basic Gestalt cues are initially missing

[Schmidt et al. 1986]

Object segregation is learned

Adults

It all begins with motion

It all begins with motion

Grouping by common motion precedes figural goodness

[Spelke 1990 - review]

Motion discontinuities provide an early cue for occlusion boundaries

[Granrud et al. 1984]

Our model

Static segregation Local occlusion boundaries Object form Motion discontinuities Common motion

Boundary General Accurate Noisy Incomplete Global Object-specific Complete Inaccurate

Motion-based segregation

Dorfman, Harari & Ullman, CogSci 2013

Intensity edges?

Boundary

Occlusion cues

Extremal edges Convexity T-junctions

[Ghose & Palmer 2010]

Boundary

Familiar object

Global

How does it actually work?

Moving object

Motion

Motion Boundary Global

Figure Ground Unknown

Need many examples for good results (1000+) Boundary

Informative boundary features

Prediction

Figure

• r

Ground? Figure

• r

Ground? Novel object, novel background

78% success Using 100,000 training examples

Boundary

Entire image

Boundary

Figure Background

Learning an object

Standard object recognition algorithm

Learns local features and their relative locations

Global

Detection

Global

Combining information sources

Combined Boundary

Accurate Noisy & Incomplete

Global

Complete Inaccurate

Figure Background

More complex algorithms

Default GrabCut With segregation cue

[Rother et al. 2004]

More complex algorithms

Default GrabCut With segregation cue

[Rother et al. 2004]

Object segregation - summary

• Static segregation is learned from motion
• Two simple mechanisms:

Boundary

Motion discontinuities  Occlusion boundaries

(Need a rich library, including extremal edges)

Global

Common motion  Object form

• These mechanisms work in synergy
• This is enough to get started,

adult segregation is much more complex

Summary

• Features are important for many visual tasks

such as object recognition and segregation.

• Features can be learned in a supervised

manner given labeled examples.

• Features can be also learned in an

unsupervised manner using statistical regularities or domain-specific cues.