Dense Random Fields Philipp Krhenbhl Stanford University Zoo of - - PowerPoint PPT Presentation

▶
dense random fields
SMART_READER_LITE
LIVE PREVIEW

Dense Random Fields Philipp Krhenbhl Stanford University Zoo of - - PowerPoint PPT Presentation

Jun 03, 2023 178 likes •1.62k views

Dense Random Fields Philipp Krhenbhl Stanford University Zoo of computer vision problems bottle tiger bottle bottle kitten wood car skin paper pumpkin cloth playing tennis Emma in her hat looking super cute 2 Zoo of computer

slide-1
SLIDE 1

Dense Random Fields

Philipp Krähenbühl Stanford University

slide-2
SLIDE 2

Zoo of computer vision problems

skin

paper cloth wood

kitten

Emma in her hat looking super cute

playing tennis

pumpkin tiger

car

bottle bottle bottle

2

slide-3
SLIDE 3

Zoo of computer vision problems

Labeling problems

skin

paper cloth wood

kitten

Emma in her hat looking super cute

playing tennis

pumpkin tiger

car

bottle bottle bottle

2

slide-4
SLIDE 4

Zoo of computer vision problems

Labeling problems

skin

paper cloth wood

kitten

Emma in her hat looking super cute

playing tennis

pumpkin tiger

car

bottle bottle bottle

>66%

2

slide-5
SLIDE 5

Zoo of computer vision problems

Labeling problems

skin

paper cloth wood

kitten

Emma in her hat looking super cute

playing tennis

pumpkin tiger

car

bottle bottle bottle

2

slide-6
SLIDE 6

skinpaper cloth wood

Labeling problems

kitten

sparse dense per image

Emma in her hat looking super cute playing tennis pumpkin tiger car bottle bottle bottle

3

slide-7
SLIDE 7

skinpaper cloth wood

Labeling problems

kitten

sparse dense per image

Emma in her hat looking super cute playing tennis pumpkin tiger car bottle bottle bottle

3

slide-8
SLIDE 8

skinpaper cloth wood

Labeling problems

kitten

sparse dense per image

Emma in her hat looking super cute playing tennis pumpkin tiger car bottle bottle bottle

3

slide-9
SLIDE 9

skinpaper cloth wood

Labeling problems

kitten

sparse dense per image

Emma in her hat looking super cute playing tennis pumpkin tiger car bottle bottle bottle

3

slide-10
SLIDE 10

skinpaper cloth wood

Labeling problems

kitten

sparse dense per image

Emma in her hat looking super cute playing tennis pumpkin tiger car bottle bottle bottle

33% 33%

3

slide-11
SLIDE 11

Dense labeling problems

skin paper cloth wood car bottle bottle bottle

4

slide-12
SLIDE 12

Dense labeling problems

  • pixel-wise labeling

skin paper cloth wood car bottle bottle bottle

4

slide-13
SLIDE 13

Dense labeling problems

  • pixel-wise labeling
  • spatial coherence

skin paper cloth wood car bottle bottle bottle

4

slide-14
SLIDE 14

Pixelswise vs Instance

5

skin paper cloth wood car bottle bottle bottle

slide-15
SLIDE 15

Pixelswise vs Instance

5

skin paper cloth wood car bottle bottle bottle

slide-16
SLIDE 16

Multi-class image segmentation

table chair background

6

slide-17
SLIDE 17

Multi-class image segmentation

sheep grass

7

slide-18
SLIDE 18

Classification

[1] TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context, Shotton et.al. 2009

𝜔(grass) 𝜔(sheep)

8

slide-19
SLIDE 19

Classification

  • Train classifier 𝜔(l)
  • for each class l
  • TextonBoost [1]

[1] TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context, Shotton et.al. 2009

𝜔(grass) 𝜔(sheep)

8

slide-20
SLIDE 20

Classification

  • Train classifier 𝜔(l)
  • for each class l
  • TextonBoost [1]
  • Pixels independent
  • noisy classification

[1] TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context, Shotton et.al. 2009

8

slide-21
SLIDE 21

Classification

  • Train classifier 𝜔(l)
  • for each class l
  • TextonBoost [1]
  • Pixels independent
  • noisy classification
  • Large regional context
  • inaccurate around

boundaries

[1] TextonBoost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context, Shotton et.al. 2009

8

slide-22
SLIDE 22

Random Field Models

9

slide-23
SLIDE 23

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

unary term pairwise term

10

slide-24
SLIDE 24

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj) P(X) = 1 Z exp(−E(X))

unary term pairwise term

  • Probabilistic interpretation

10

slide-25
SLIDE 25

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj) P(X) = 1 Z exp(−E(X))

unary term pairwise term

  • Probabilistic interpretation
  • MAP inference
  • most likely labeling
  • lowest energy

10

slide-26
SLIDE 26

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

𝜔ij(Xi,Xj) = 0 unary term

11

slide-27
SLIDE 27

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

𝜔ij(Xi,Xj) = [Xi≠Xj] conditional random field

12

slide-28
SLIDE 28

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

𝜔ij(Xi,Xj) = 100[Xi≠Xj] conditional random field

13

slide-29
SLIDE 29

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

𝜔ij(Xi,Xj) = 100[Xi≠Xj] conditional random field

14

slide-30
SLIDE 30

conditional random field

Random Field Models

𝜔ij(Xi,Xj) = wij[Xi≠Xj]

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

wij=exp(-α(ci-cj)2)

15

slide-31
SLIDE 31

Random Field Models

weight horizontal 𝜔ij(Xi,Xj) = wij[Xi≠Xj]

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

wij=exp(-α(ci-cj)2)

15

slide-32
SLIDE 32

Random Field Models

𝜔ij(Xi,Xj) = wij[Xi≠Xj]

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

wij=exp(-α(ci-cj)2) weight vertical

15

slide-33
SLIDE 33

Random Field Models

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

𝜔ij(Xi,Xj) = 100wij[Xi≠Xj] conditional random field color sensitive

16

slide-34
SLIDE 34

Random Field Models

17

slide-35
SLIDE 35

Random Field Models

Pros:

17

slide-36
SLIDE 36

Random Field Models

Pros:

  • Probabilistic interpretation

17

slide-37
SLIDE 37

Random Field Models

Pros:

  • Probabilistic interpretation
  • Parameter learning

17

slide-38
SLIDE 38

Random Field Models

Pros:

  • Probabilistic interpretation
  • Parameter learning
  • Combine with other models

17

slide-39
SLIDE 39

Random Field Models

Pros:

  • Probabilistic interpretation
  • Parameter learning
  • Combine with other models

Cons:

17

slide-40
SLIDE 40

Random Field Models

Pros:

  • Probabilistic interpretation
  • Parameter learning
  • Combine with other models

Cons:

  • Shrinking bias

17

slide-41
SLIDE 41

Random Field Models

Pros:

  • Probabilistic interpretation
  • Parameter learning
  • Combine with other models

Cons:

  • Shrinking bias
  • Models only local interactions

17

slide-42
SLIDE 42

Random Field Models

Pros:

  • Probabilistic interpretation
  • Parameter learning
  • Combine with other models

Cons:

  • Shrinking bias
  • Models only local interactions
  • hard for information to

propagate

17

slide-43
SLIDE 43

Filtering

classifier labeling

18

slide-44
SLIDE 44

Filtering

classifier log likelihood

19

slide-45
SLIDE 45

Filtering

blurred log likelihood Gaussian 𝜏s=2px

20

slide-46
SLIDE 46

Filtering

blurred labeling Gaussian 𝜏s=2px

21

slide-47
SLIDE 47

Filtering

blurred labeling Gaussian 𝜏s=6px

22

slide-48
SLIDE 48

Filtering

Conditional Random Field (CRF)

23

slide-49
SLIDE 49

Filtering

blurred labeling Gaussian 𝜏s=6px

24

slide-50
SLIDE 50

Filtering

blurred labeling Bilateral 𝜏s=60px 𝜏c=15

25

slide-51
SLIDE 51

Filtering

Conditional Random Field (CRF) color sensitive

26

slide-52
SLIDE 52

Filtering

˜ vi = X

j

wijvj

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-53
SLIDE 53

Filtering

˜ vi = X

j

wijvj

vj

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-54
SLIDE 54

Filtering

˜ vi = X

j

wijvj

vj

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-55
SLIDE 55

Filtering

˜ vi = X

j

wijvj

vj

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-56
SLIDE 56

Filtering

˜ vi = X

j

wijvj

vj

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-57
SLIDE 57

Filtering

˜ vi = X

j

wijvj

si sj vj (ci-cj)2=( - )2

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-58
SLIDE 58

Filtering

˜ vi = X

j

wijvj

si sj vj (c

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

w exp(-(si-sj)2/𝜏s) exp

slide-59
SLIDE 59

Filtering

˜ vi = X

j

wijvj

s s vj (ci-cj)2=( - )2

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

w exp exp(-(ci-cj)2/𝜏c)

slide-60
SLIDE 60

Filtering

  • Efficient convolution
  • Permutohedral lattice [2]
  • compute all ṽi in linear time
  • 50-100ms / image

˜ vi = X

j

wijvj

si sj vj (ci-cj)2=( - )2

[2] Fast High-Dimensional Filtering Using the Permutohedral Lattice, Adams et.al. 2010

ṽi

27

wij = exp(-(si-sj)2/𝜏s) exp(-(ci-cj)2/𝜏c)

slide-61
SLIDE 61

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

28

slide-62
SLIDE 62

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

Pros:

28

slide-63
SLIDE 63

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

Pros:

  • Propagates information over

large distances

  • up to 1/3 of image

28

slide-64
SLIDE 64

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

Pros:

  • Propagates information over

large distances

  • up to 1/3 of image

Cons:

28

slide-65
SLIDE 65

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

Pros:

  • Propagates information over

large distances

  • up to 1/3 of image

Cons:

  • No probabilistic interpretation

28

slide-66
SLIDE 66

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

Pros:

  • Propagates information over

large distances

  • up to 1/3 of image

Cons:

  • No probabilistic interpretation
  • No joint inference

28

slide-67
SLIDE 67

Filtering

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 ṽi

Pros:

  • Propagates information over

large distances

  • up to 1/3 of image

Cons:

  • No probabilistic interpretation
  • No joint inference
  • No learning

28

slide-68
SLIDE 68

Dense Random Fields

29

slide-69
SLIDE 69

Dense Random Fields

29

slide-70
SLIDE 70

Dense Random Fields

29

slide-71
SLIDE 71

Dense Random Fields

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

unary term pairwise term

30

slide-72
SLIDE 72

Dense Random Fields

E(X) = X

i

ψi(Xi) + X

i,j∈N

ψij(Xi, Xj)

unary term pairwise term

  • Every node is connected to every other node
  • Connections weighted differently

30

slide-73
SLIDE 73

Dense Random Fields

31

slide-74
SLIDE 74

Dense Random Fields

31

slide-75
SLIDE 75

Dense Random Fields

32

slide-76
SLIDE 76

Dense Random Fields

Pros:

32

slide-77
SLIDE 77

Dense Random Fields

Pros:

  • Long range interactions

32

slide-78
SLIDE 78

Dense Random Fields

Pros:

  • Long range interactions
  • No shrinking bias

32

slide-79
SLIDE 79

Dense Random Fields

Pros:

  • Long range interactions
  • No shrinking bias
  • Probabilistic interpretation

32

slide-80
SLIDE 80

Dense Random Fields

Pros:

  • Long range interactions
  • No shrinking bias
  • Probabilistic interpretation
  • Parameter learning

32

slide-81
SLIDE 81

Dense Random Fields

Pros:

  • Long range interactions
  • No shrinking bias
  • Probabilistic interpretation
  • Parameter learning
  • Combine with other models

32

slide-82
SLIDE 82

Dense Random Fields

33

slide-83
SLIDE 83

Dense Random Fields

Cons:

33

slide-84
SLIDE 84

Dense Random Fields

Cons:

  • Very large model
  • 50’000 - 100’000 variables
  • billions pairwise terms

33

slide-85
SLIDE 85

Dense Random Fields

Cons:

  • Very large model
  • 50’000 - 100’000 variables
  • billions pairwise terms
  • Traditional inference very slow
  • MCMC “converges” in 36h
  • GraphCuts and alpha-exp.: no

convergence in 3 days

33

slide-86
SLIDE 86

Dense Random Fields

  • Efficient inference
  • 0.2s / image
  • Pairwise term
  • linear combination of

Gaussians

34

slide-87
SLIDE 87

Dense Random Fields

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj)

35

slide-88
SLIDE 88

Dense Random Fields

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj) ψij(Xi, Xj) = X

m

µ(m)(Xi, Xj) k(m)(fi, fj)

35

slide-89
SLIDE 89

Dense Random Fields

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj) ψij(Xi, Xj) = X

m

µ(m)(Xi, Xj)

Gaussian kernel k(m)

k(m)(fi, fj)

35

slide-90
SLIDE 90

Dense Random Fields

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj) ψij(Xi, Xj) = X

m

µ(m)(Xi, Xj)

Gaussian kernel k(m) Label compatibility 𝜈(m)

⨂

𝜈 GRASS SHEEP WATER … GRASS

1 1 …

SHEEP

1 10 …

WATER

1 10 …

…

… … …

k(m)(fi, fj)

35

slide-91
SLIDE 91

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

36

slide-92
SLIDE 92

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility

36

slide-93
SLIDE 93

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility
  • Potts model: 𝜈(Xi,Xj) = [Xi≠Xj]

𝜈 GRASS SHEEP WATER … GRASS

1 1 1

SHEEP

1 1 1

WATER

1 1 1

…

1 1 1

36

slide-94
SLIDE 94

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility
  • Potts model: 𝜈(Xi,Xj) = [Xi≠Xj]
  • Learned from data

𝜈 GRASS SHEEP WATER … GRASS

? ? ?

SHEEP

? ? ?

WATER

? ? ?

…

? ? ?

36

slide-95
SLIDE 95

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility
  • Potts model: 𝜈(Xi,Xj) = [Xi≠Xj]
  • Learned from data
  • Appearance kernel

si sj (ci-cj)2=( - )2

36

slide-96
SLIDE 96

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility
  • Potts model: 𝜈(Xi,Xj) = [Xi≠Xj]
  • Learned from data
  • Appearance kernel
  • Color—sensitive

si sj (ci-cj)2=( - )2

36

slide-97
SLIDE 97

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility
  • Potts model: 𝜈(Xi,Xj) = [Xi≠Xj]
  • Learned from data
  • Appearance kernel
  • Color—sensitive
  • Local smoothness

si sj

36

slide-98
SLIDE 98

Dense Random Fields

ψij(Xi, Xj) =µ1(Xi, Xj) exp −|si − sj|2 2σ2

α

− |ci − cj|2 2σ2

β

! µ2(Xi, Xj) exp ✓ −|si − sj|2 2σ2

γ

◆ +

  • Label compatibility
  • Potts model: 𝜈(Xi,Xj) = [Xi≠Xj]
  • Learned from data
  • Appearance kernel
  • Color—sensitive
  • Local smoothness
  • Discourages single pixel noise

si sj

36

slide-99
SLIDE 99

Efficient inference

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj)

37

slide-100
SLIDE 100

Efficient inference

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj)

Find most likely assignment (MAP)

P(X) = 1 Z exp(−E(X)) ˆ x = arg max

X P(X) where

37

slide-101
SLIDE 101

Efficient inference

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj)

Find most likely assignment (MAP)

P(X) = 1 Z exp(−E(X)) ˆ x = arg max

X P(X) where

NP-Hard

37

slide-102
SLIDE 102

Efficient inference

E(X) = X

i

ψi(Xi)+ X

i>j

ψij(Xi, Xj)

Find most likely assignment (MAP)

P(X) = 1 Z exp(−E(X)) ˆ x = arg max

X P(X) where

NP-Hard

Mean Field approximation

Find Q(X)=∏iQi(Xi) close to P(X) in terms of KL-divergence D(Q||P)

ˆ x ≈ arg max

X Q(X)

37

slide-103
SLIDE 103

Mean-Field approximation

38

slide-104
SLIDE 104

Mean-Field approximation

38

slide-105
SLIDE 105

Mean-Field approximation

38

slide-106
SLIDE 106

Efficient inference

Mean Field algorithm

39

slide-107
SLIDE 107

Efficient inference

Mean Field algorithm

Initialize Qi(l) = 1

Zi exp(−ψi(l))

39

slide-108
SLIDE 108

Efficient inference

Mean Field algorithm

Initialize Until convergence:

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-109
SLIDE 109

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing: ˜

Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-110
SLIDE 110

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = 1 Zi exp(−ψi(l))

𝜈 GRASS SHEEP WATER … GRASS 1 1 … SHEEP 1 10 … WATER 1 10 … … … … …

39

slide-111
SLIDE 111

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l)) Qi(l) = 1 Zi exp(−ψi(l))

39

slide-112
SLIDE 112

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l)) Qi(l) = 1 Zi exp(−ψi(l))

39

slide-113
SLIDE 113

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N)

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-114
SLIDE 114

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N)

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-115
SLIDE 115

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N) O(N)

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-116
SLIDE 116

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N) O(N) O(N)

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-117
SLIDE 117

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N) O(N) O(N) O(N2)

Qi(l) = 1 Zi exp(−ψi(l))

39

slide-118
SLIDE 118

Efficient message passing

  • Update all variables simultaneously
  • Gaussian Convolution
  • Efficient approximation

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

40

slide-119
SLIDE 119

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N) O(N) O(N) O(N2)

Qi(l) = 1 Zi exp(−ψi(l))

41

slide-120
SLIDE 120

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N) O(N) O(N) O(N) High-dimensional filter

Qi(l) = 1 Zi exp(−ψi(l))

41

slide-121
SLIDE 121

Efficient inference

Mean Field algorithm

Initialize Until convergence:

  • Message passing:
  • Compatibility transform:
  • Local update:
  • Normalize Qi

˜ Q(m)

i

(l) = X

j

k(m)(fi, fj)Qj(l)

ˆ Q(m)

i

(l0) = X

l

µ(m)(l0, l) ˜ Q(m)

i

(l)

Qi(l) = exp(−ψi(l) − X

m

ˆ Q(m)

i

(l))

O(N) O(N) O(N) O(N) O(N) High-dimensional filter

linear in number of variables independent of number of pairwise terms

Qi(l) = 1 Zi exp(−ψi(l))

41

slide-122
SLIDE 122

Parallel Mean-Field

42

slide-123
SLIDE 123

Parallel Mean-Field

  • Not guaranteed to converge for general models

42

slide-124
SLIDE 124

Parallel Mean-Field

  • Not guaranteed to converge for general models
  • Guaranteed to converge for fully-connected models with

negative definite label compatibility

  • Potts models
  • L1 norms
  • ...

42

slide-125
SLIDE 125

Parallel Mean-Field

  • Not guaranteed to converge for general models
  • Guaranteed to converge for fully-connected models with

negative definite label compatibility

  • Potts models
  • L1 norms
  • ...
  • Proof see Thesis or [3]
  • Reduction of Parallel Mean-Field to CCCP

[3] Parameter Learning and Convergent Inference for Dense Random Fields,
 Krähenbühl and Koltun, ICML 2013

42

slide-126
SLIDE 126

How fast will it converge

5 10 15 20 KL-divergence Number of iterations θα=θβ=10 θα=θβ=30 θα=θβ=50 θα=θβ=70 θα=θβ=90

43

slide-127
SLIDE 127

0 iterations

Q(bird) Q(sky)

44

slide-128
SLIDE 128

1 iterations

Q(bird) Q(sky)

45

slide-129
SLIDE 129

2 iterations

Q(bird) Q(sky)

46

slide-130
SLIDE 130

10 iterations

Q(bird) Q(sky)

47

slide-131
SLIDE 131

Results - MSRC

bird water grass bird water grass tree bird water grass car road tree building sky car road tree building sky car road tree building sky cow grass cow grass cow grass

grid fully con. unary

48

slide-132
SLIDE 132

Results - MSRC

bird water grass bird water grass tree bird water grass car road tree building sky car road tree building sky car road tree building sky cow grass cow grass cow grass

grid fully con. unary

MSRC dataset

  • 591 images
  • 21 classes

TIME GLOBAL AVERAGE UNARY

  • 84.0

76.6

49

slide-133
SLIDE 133

Results - MSRC

bird water grass bird water grass tree bird water grass car road tree building sky car road tree building sky car road tree building sky cow grass cow grass cow grass

grid fully con. unary

MSRC dataset

  • 591 images
  • 21 classes

TIME GLOBAL AVERAGE UNARY

  • 84.0

76.6

GRID CRF

1s 84.6 77.2

49

slide-134
SLIDE 134

Results - MSRC

bird water grass bird water grass tree bird water grass car road tree building sky car road tree building sky car road tree building sky cow grass cow grass cow grass

grid fully con. unary

MSRC dataset

  • 591 images
  • 21 classes

TIME GLOBAL AVERAGE UNARY

  • 84.0

76.6

GRID CRF

1s 84.6 77.2

FC CRF

0.2s 86.0 78.3

49

slide-135
SLIDE 135

Results - MSRC

bird water grass bird water grass tree bird water grass car road tree building sky car road tree building sky car road tree building sky cow grass cow grass cow grass

grid fully con. unary

MSRC dataset

  • 591 images
  • 21 classes

TIME GLOBAL AVERAGE UNARY

  • 84.0

76.6

GRID CRF

1s 84.6 77.2

FC CRF

0.2s 86.0 78.3

FILTER

0.05s 85.0 77.5

49

slide-136
SLIDE 136

Results - MSRC

standard gt

tree sky grass

50

slide-137
SLIDE 137

accurate gt

tree sky grass

Results - MSRC

51

slide-138
SLIDE 138

accurate gt

tree sky grass

Results - MSRC

MSRC Accurate annotations

  • 94 images
  • hand annotated (30 min each)
  • unary train on standard anno.
  • 5-fold cross validation

GLOBAL AVERAGE UNARY

83.2±1.5 80.6±2.3

GRID CRF

84.8±1.5 82.4±1.8

FC CRF

88.2±0.7 84.7±0.7

grid crf tree sky grass

52

slide-139
SLIDE 139

accurate gt

tree sky grass

Results - MSRC

MSRC Accurate annotations

  • 94 images
  • hand annotated (30 min each)
  • unary train on standard anno.
  • 5-fold cross validation

GLOBAL AVERAGE UNARY

83.2±1.5 80.6±2.3

GRID CRF

84.8±1.5 82.4±1.8

FC CRF

88.2±0.7 84.7±0.7

grid crf tree sky grass fully connected tree sky grass

52

slide-140
SLIDE 140

Results - VOC 2010

TIME IOU ACCURACY UNARY

  • 27.6

GRID CRF

2.5s 28.3

FC CRF

0.5s 29.1

PASCAL VOC 2010

  • 1928 images
  • 20 classes + background

ground truth

boat background

fully connected

boat background

ground truth

sheep background

fully connected

sheep background

53

slide-141
SLIDE 141

54

slide-142
SLIDE 142

Questions?

fully connected

cat background

55

slide-143
SLIDE 143

Questions?

fully connected

cat background fully connect all the things

55