Cue combinations, Bayesian models Thurs. March 1, 2018 1 Visual - - PowerPoint PPT Presentation

1

COMP 546

Lecture 15

Cue combinations, Bayesian models

• Thurs. March 1, 2018

Visual Cues:

image properties that can tell us about scene properties

Image

texture

• size, shape, density

shading binocular disparities motion (from moving observer) defocus blur

Scene

depth gradient

• slant, tilt

surface curvature depth

2

𝑞 𝐽 = 𝑗 𝑇 = 𝑡 )

• Probability of measuring image 𝐽 = 𝑗, when the scene is 𝑇 = 𝑡.

(called “likelihood” of scene 𝑇 = 𝑡, given the image 𝐽 = 𝑗).

• Maximum likelihood method:

Choose 𝑇 = 𝑡 that maximizes 𝑞 𝐽 = 𝑗 𝑇 = 𝑡 )

Last lecture: Likelihood

3

4

This lecture: How to combine cues ?

𝑞 𝐽1, 𝐽2 𝑇 )

5

Example:

stereo only texture and stereo

[Hillis 2004]

texture only (monocular)

6

𝑞 𝐽1, 𝐽2 𝑇 ) = 𝑞 𝐽1 𝑇 ) 𝑞 𝐽2 𝑇 )

Assume likelihood function is “conditionally independent”: e.g. 𝐽1 is texture. 𝐽2 is binocular disparity.

7

𝑇 = s

𝑞 𝐽2 𝑇 ) 𝑞 𝐽1 𝑇 )

Assume 𝑞 𝐽1 = 𝑗1 𝑇 = 𝑡 ) and 𝑞 𝐽2 = 𝑗2 𝑇 = 𝑡 ) are Gaussian shaped.

8

𝑇 = s

𝑞 𝐽2 𝑇 ) 𝑞 𝐽1 𝑇 )

Assume 𝑞 𝐽1 = 𝑗1 𝑇 = 𝑡 ) and 𝑞 𝐽2 = 𝑗2 𝑇 = 𝑡 ) are Gaussian shaped. Their maxima might occur at different values of 𝑡. Why ? 𝑡1 𝑡2

We want to find the 𝑡 that maximizes:

𝑞 𝐽1 | 𝑇 = 𝑡 𝑞 𝐽2 | 𝑇 = 𝑡 = 𝑓

− 𝑡 − 𝑡1 2 2 𝜏12

𝑓

− 𝑡 − 𝑡2 2 2 𝜏22

We want to find the 𝑡 that maximizes: So, we want to find the 𝑡 that minimizes:

𝑞 𝐽1 | 𝑇 = 𝑡 𝑞 𝐽2 | 𝑇 = 𝑡 = 𝑓

− 𝑡 − 𝑡1 2 2 𝜏12

𝑓

− 𝑡 − 𝑡2 2 2 𝜏22

The lecture notes show that the solution 𝑇 = 𝑡 is where 𝑡 = 𝑥1𝑡1 + 𝑥2𝑡2 𝑥1 + 𝑥2 = 1 0 < 𝑥𝑗 < 1

“Linear Cue Combination”

The lecture notes show that the solution 𝑇 = 𝑡 is where Thus, less reliable cue (larger 𝜏) get less weight. 𝑥1 = 𝜏22 𝜏12 + 𝜏22 𝑥2 = 𝜏12 𝜏12 + 𝜏22 𝑡 = 𝑥1𝑡1 + 𝑥2𝑡2 𝑥1 + 𝑥2 = 1 0 < 𝑥𝑗 < 1

13

Example:

stereo only

[Hillis 2004]

texture only (monocular)

Measure slant discrimination thresholds for cues in isolation. Estimate likelihood function parameters (𝑡1, 𝜏1 , 𝑡2, 𝜏2).

14

texture and stereo

… then

• present cues together
• measure thresholds for 𝑇
• convert thresholds to likelihood parameters (𝑡 , σ)

15

texture and stereo

… then

• present cues together
• measure thresholds for 𝑇
• convert thresholds to likelihood parameters (𝑡 , σ)
• examine if these values are consistent with the model*

*Model also makes prediction about σ in combined case.

𝑡 = 𝑥1𝑡1 + 𝑥2𝑡2

16

𝑇 = s

𝑞 𝐽2 𝑇 ) 𝑞 𝐽1 𝑇 )

𝑡1 𝑡2

Experimenter can manipulate 𝑡1 , 𝑡2 , 𝜏1 , 𝜏2 and predict effect on perception of slant.

texture and stereo

17

COMP 546

Lecture 15

Cue combinations, Bayesian models

• Thurs. March 1, 2018

𝑞 𝐽 = 𝑗 𝑇 = 𝑡) ≠ 𝑞 𝑇 = 𝑡 𝐽 = 𝑗)

18

Likelihood of scene 𝑡, given image 𝑗 Probability of scene 𝑡, given image 𝑗 What is the crucial difference ?

wire frame with independently chosen depths regular solid cube flat drawing

All scenes above have the same likelihood 𝑞( 𝐽 = 𝑗 | 𝑇 = 𝑡 ). Why do we prefer the regular solid cube?

[Kersten & Yuille 2003]

Some scenes may have a larger probability 𝑞(𝑇 = 𝑡 ). The marginal probably 𝑞(𝑇 = 𝑡) is called the "prior".

𝑞 𝐽 𝑇 ) ≡

𝑞(𝐽, 𝑇 ) 𝑞(𝑇)

𝑞 𝑇 𝐽 ) ≡

𝑞 (𝐽, 𝑇 ) 𝑞(𝐽)

𝑞 𝐽 𝑇 ) 𝑞 𝑇 = 𝑞 𝑇 𝐽 ) 𝑞 𝐽

Thus,

𝑞 𝐽 𝑇 ) ≡

𝑞(𝐽, 𝑇 ) 𝑞(𝑇)

𝑞 𝑇 𝐽 ) ≡

𝑞 (𝐽, 𝑇 ) 𝑞(𝐽)

𝑞 𝑇 𝐽 ) = 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝐽

Thus,

Bayes Theorem

posterior likelihood scene prior image prior

Maximum ‘a Posteriori’ (MAP)

Given an image, 𝐽 = 𝑗, find the scene 𝑇 = 𝑡 that maximizes 𝑞( 𝑇 = 𝑡 | 𝐽 = 𝑗 ).

𝑞 𝑇 𝐽 ) = 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝐽

posterior likelihood scene prior image prior

Maximum ‘a Posteriori’ (MAP)

Given an image, 𝐽 = 𝑗, find the scene 𝑇 = 𝑡 that maximizes 𝑞( 𝑇 = 𝑡 | 𝐽 = 𝑗 ).

𝑞 𝑇 𝐽 ) = 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝐽

posterior likelihood scene prior image prior

We don't care about 𝑞( 𝐽 = 𝑗 ). Why not ?

If the prior p(S) is uniform then maximum likelihood gives the same solution as maximum posterior (MAP). Interesting cases arise when the prior is non-uniform. 𝑞 𝑇 𝐽 ) = 𝑞 𝐽 𝑇 ) 𝑞 𝑇 𝑞 𝐽

posterior likelihood scene prior image prior constant

likelihood prior

http://www.youtube.com/watch?v=Ttd0YjXF0no

Ames Room

https://www.youtube.com/watch?v=gJhyu6nlGt8

Priors (“Natural Scenes Statistics”)

• intensity
• rientation of image lines, edges
• disparity
• motion
• surface slant, tilt

• rientation 𝜄 of lines, edges

[Girshick 2011]

𝑞(𝑇 = 𝜄)

People are indeed better at discriminating vertical and horizontal orientations than oblique orientations. Why? Because they use a prior ?

surface slant 𝜏 and tilt 𝜐

Here we represent (slant, tilt) using a concave hemisphere. See next slide.

floor ceiling

𝑞(𝑇 = (𝜏, 𝜐))

[Adams & Elder 2016]

represent slants and tilts using a concave

Each disk shows 𝑞(𝜏, 𝜐) for surfaces visible over a range of viewing direction elevations, relative to line of sight.

𝑞(𝑇 = (𝜏, 𝜐))

𝑞(𝑇 = (𝜏, 𝜐))

Maximum a Posteriori (MAP)

= ∗ Choose the S = (slant,tilt) that maximizes the posterior.

𝑞( 𝑇 ) 𝑞(𝐽 = 𝑗 | 𝑇) 𝑞 𝑇 𝐽 = 𝑗 )

posterior likelihood prior

i.e. convex or concave ?

𝑞(𝐽 = 𝑗 | 𝑇)

• verall

(slant, tilt)

Likelihood functions can have more than one maximum.

Depth Reversal Ambiguity and Shading

(see Exercise) A valley illuminated from the right produces the same shading as a hill illuminated from the left.

𝑞(𝐽 = 𝑗 | 𝑇) Likelihood (slant, tilt)

What “priors” does the visual system use to resolve such twofold ambiguities ? Let’s look at a few related examples.

You can perceive the center point as a hill or a valley. When you see it as a hill, you perceive the tilt as 180 deg (leftward). But when you see it as a valley, the slant is 0 (rightward).

We tend to see the center as a hill. Why ?

We tend to see the center as a valley. Why ?

The visual system uses three priors to resolve the depth reversal ambiguity:

• surface orientation: p(floor) > p(ceiling)
• light source direction:

p( above) > p( below)

• ‘global’ surface curvature: p(convex) > p(concave)

Example in which all three priors assumptions are met

light from above viewpoint from above (floor) shape is convex

Example in which all three prior assumptions fail

shape is concave viewpoint from below (ceiling) light from below

floor ceiling

Convex shape, illuminated from above the line of sight

floor ceiling

Concave shape, illuminated from below the line of sight

We showed how people combined the three different "priors": Percent correct in judging local "hill" or "valley": = 50 +/- 10 floor vs. ceiling +/- 10 light from above vs. below +/- 10 globally convex/concave

[Langer and Buelthoff, 2001]

Best (80%) Worst (20%)

These look weird, but in different ways. How ?

Reminder

• A2 is due tonight
• Midterm (optional) is first class after Study Break