Graph Cuts, MRFs and Graphcut Textures CS448V Computational Video - - PowerPoint PPT Presentation

▶
graph cuts mrfs and graphcut textures
SMART_READER_LITE
LIVE PREVIEW

Graph Cuts, MRFs and Graphcut Textures CS448V Computational Video - - PowerPoint PPT Presentation

Jan 12, 2024 365 likes •1.44k views

Graph Cuts, MRFs and Graphcut Textures CS448V Computational Video Manipulation April 2019 What makes this a texture? What makes this a texture? stationary What makes this a texture? Local What makes this a

slide-1
SLIDE 1

Graph Cuts, MRFs and Graphcut Textures

CS448V — Computational Video Manipulation April 2019

slide-2
SLIDE 2
slide-3
SLIDE 3

What makes this a “texture”?

slide-4
SLIDE 4

What makes this a “texture”?

slide-5
SLIDE 5

What makes this a “texture”?

stationary

slide-6
SLIDE 6

What makes this a “texture”?

Local

slide-7
SLIDE 7

Stochastic Regular

slide-8
SLIDE 8

Stochastic Regular Texture?

slide-9
SLIDE 9

Textures are everywhere!

slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12

[Kwatra et al. 2003]

slide-13
SLIDE 13

[Wexler et al. 2004]

slide-14
SLIDE 14

[Wexler et al. 2004]

slide-15
SLIDE 15

[Wexler et al. 2004]

slide-16
SLIDE 16

[Wexler et al. 2004]

slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19

Graphcut Textures: Image and Video Synthesis Using Graph Cuts

Kwatra et al. 2003

slide-20
SLIDE 20

Graphcut Textures: Image and Video Synthesis Using Graph Cuts

Kwatra et al. 2003

slide-21
SLIDE 21

Graph Cuts

s t

slide-22
SLIDE 22

Graph Cuts

s t

Partition (S, T) ∣ s ∈ S, t ∈ T

slide-23
SLIDE 23

Graph Cuts

s t

Partition (S, T) ∣ s ∈ S, t ∈ T 5 4 2 11 Capacity(S, T) = sum of edge weights (leaving) S

slide-24
SLIDE 24

Max-flow Min-cut theorem

s t

5 4 2 11 12 1 3 20 3 1 4 7 3

slide-25
SLIDE 25

Max-flow Min-cut theorem

s t

5 4 2 11 12 1 3 20 3 1 4 7 3 What is a flow?…

slide-26
SLIDE 26

Max-flow Min-cut theorem

s t

5 4 2 11 12 1 3 3 1 Capacity of min-cut: 6 4 7 3 20

slide-27
SLIDE 27

Max-flow Min-cut theorem

s t

5 4 2 11 12 1 3 3 3 1 Capacity of min-cut: 6 4 7 6 4 2 4 3 3 3 20 Max-flow from s to t: 6

slide-28
SLIDE 28

Cuts & Flows

slide-29
SLIDE 29

Cuts & Flows

Many variants:

  • directed/undirected
  • with/without terminals
  • multi-cut
  • non integer weights
  • negative weights
  • …
slide-30
SLIDE 30

Cuts & Flows

Many variants:

  • directed/undirected
  • with/without terminals
  • multi-cut
  • non integer weights
  • negative weights
  • …

Many many applications!

slide-31
SLIDE 31

Max Bipartite Match

1 2 3 4 A B C D

slide-32
SLIDE 32

Max Bipartite Match

1 2 3 4 A B C D s t

slide-33
SLIDE 33

Max Bipartite Match

1 2 3 4 A B C D s t

slide-34
SLIDE 34

Max Bipartite Match

1 2 3 4 A B C D s t

we will use a similar trick…

slide-35
SLIDE 35

Back to Graphcut Textures…

slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

“Chernobyl harvest”

slide-39
SLIDE 39

Where to place next patch? Which pixels to use?

slide-40
SLIDE 40

Which pixels to use?

slide-41
SLIDE 41

Which pixels to use?

Graph cuts to the rescue

slide-42
SLIDE 42

Which pixels to use?

Graph cuts to the rescue

slide-43
SLIDE 43

Which pixels to use?

Graph cuts to the rescue

s t M(s, t)

slide-44
SLIDE 44

Which pixels to use?

Graph cuts to the rescue

M(s, t) = ∥A(s) − B(s)∥ + ∥A(t) − B(t)∥

s t M(s, t)

slide-45
SLIDE 45

Which pixels to use?

slide-46
SLIDE 46

Which pixels to use?

1 4 S1 New Patch B

slide-47
SLIDE 47

Which pixels to use?

1 4 S1 New Patch B

Old seam cost

slide-48
SLIDE 48

Which pixels to use?

1 4 S1 New Patch B

Old seam cost 1 from old 4 from new

slide-49
SLIDE 49

Which pixels to use?

1 4 S1 New Patch B

Old seam cost 1 from old 4 from new 4 from old 1 from new

slide-50
SLIDE 50

Which pixels to use?

1 4 S1 New Patch B

Old seam cost 1 from old 4 from new 4 from old 1 from new Cut at most one edge!

slide-51
SLIDE 51

Which pixels to use?

1 4 S1 New Patch B

Old seam cost 1 from old 4 from new 4 from old 1 from new Cut at most one edge! M should be a metric

slide-52
SLIDE 52

Which pixels to use?

slide-53
SLIDE 53

Which pixels to use?

Kept old seam

slide-54
SLIDE 54

Which pixels to use?

Updated seam

slide-55
SLIDE 55

Which pixels to use?

Removed seam

slide-56
SLIDE 56

Which pixels to use?

slide-57
SLIDE 57

Which pixels to use?

slide-58
SLIDE 58

Which pixels to use?

What might happen if we only connect a few pixels to B?

slide-59
SLIDE 59

Minor detour: MRFs

slide-60
SLIDE 60

Markov Random Field

slide-61
SLIDE 61

Markov Random Field

Reminder: Markov property

slide-62
SLIDE 62

Markov Random Field

Reminder: Markov property “memoryless”

slide-63
SLIDE 63

Markov Random Field

Reminder: Markov property “memoryless” For a discrete process: P(Xn = xn|Xn−1 = xn−1, …, X0 = x0) = P(Xn = xn|Xn−1 = xn−1)

slide-64
SLIDE 64

Markov Random Field

Reminder: Markov property “memoryless” For a discrete process: P(Xn = xn|Xn−1 = xn−1, …, X0 = x0) = P(Xn = xn|Xn−1 = xn−1) What about fields?

slide-65
SLIDE 65

Markov Random Field

Reminder: Markov property “memoryless” For a discrete process: P(Xn = xn|Xn−1 = xn−1, …, X0 = x0) = P(Xn = xn|Xn−1 = xn−1) What about fields? two non-adjacent variables are conditionally independent given all other variables

slide-66
SLIDE 66

Markov Random Field

Reminder: Markov property “memoryless” For a discrete process: P(Xn = xn|Xn−1 = xn−1, …, X0 = x0) = P(Xn = xn|Xn−1 = xn−1) What about fields? A variable is conditionally independent

  • f all other variables

given its neighbors

slide-67
SLIDE 67

Markov Random Field

Reminder: Markov property “memoryless” For a discrete process: P(Xn = xn|Xn−1 = xn−1, …, X0 = x0) = P(Xn = xn|Xn−1 = xn−1) What about fields? two subsets are conditionally independent given a separating subset

slide-68
SLIDE 68

Markov Random Field

What does it mean in our setting?

slide-69
SLIDE 69

Where to place next patch? Which pixels to use?

slide-70
SLIDE 70

Placing the next patch

slide-71
SLIDE 71

Placing the next patch

  • Random placement
slide-72
SLIDE 72

Placing the next patch

  • Random placement
  • Entire patch matching
slide-73
SLIDE 73

Placing the next patch

  • Random placement
  • Entire patch matching
  • Sub-patch matching
slide-74
SLIDE 74

Placing the next patch

  • Random placement
  • Entire patch matching
  • Sub-patch matching

What would be the “right” thing to do, assuming no runtime constraints?

slide-75
SLIDE 75

Results

slide-76
SLIDE 76
slide-77
SLIDE 77
slide-78
SLIDE 78
slide-79
SLIDE 79
slide-80
SLIDE 80
slide-81
SLIDE 81
slide-82
SLIDE 82
slide-83
SLIDE 83

Video synthesis

slide-84
SLIDE 84

Temporally stationary Spatio-temporally stationary

slide-85
SLIDE 85

Temporally stationary Spatio-temporally stationary

How should this affect patch search strategy?

slide-86
SLIDE 86

Temporally stationary

Video Textures

+

Seam optimization

Per-pixel transition timing

slide-87
SLIDE 87
slide-88
SLIDE 88
slide-89
SLIDE 89

Temporally stationary Spatio-temporally stationary

slide-90
SLIDE 90

Spatio-temporally stationary

Can search patches in time and space!

slide-91
SLIDE 91

Spatio-temporally stationary

Can search patches in time and space!

Robust results even for short sequences

slide-92
SLIDE 92

Spatio-temporally stationary

Can search patches in time and space!

Robust results even for short sequences Can make videos larger

slide-93
SLIDE 93

Harder to create loops. Why?

slide-94
SLIDE 94

Harder to create loops. Why?

time

slide-95
SLIDE 95

Harder to create loops. Why?

time

Solution: explicitly force beginning and end to match

slide-96
SLIDE 96

Harder to create loops. Why?

time time

Solution: explicitly force beginning and end to match

slide-97
SLIDE 97

Recap

slide-98
SLIDE 98

Recap

  • Textures are everywhere!
slide-99
SLIDE 99

Recap

  • Textures are everywhere!
  • Add to your tool belt: Graph Cuts
slide-100
SLIDE 100

Recap

  • Textures are everywhere!
  • Add to your tool belt: Graph Cuts
  • Graphcut Textures
slide-101
SLIDE 101

What didn’t we cover?

slide-102
SLIDE 102

What didn’t we cover?

  • Many things!
slide-103
SLIDE 103

What didn’t we cover?

  • Many things!
  • E.g., image analogies
slide-104
SLIDE 104

What didn’t we cover?

  • Many things!
  • E.g., image analogies
  • Convert between different

representations of an image

slide-105
SLIDE 105

What didn’t we cover?

  • Many things!
  • E.g., image analogies
  • Convert between different

representations of an image

  • Stylization
slide-106
SLIDE 106

What didn’t we cover?

  • Many things!
  • E.g., image analogies
  • Convert between different

representations of an image

  • Stylization
  • We will discuss these

applications later in the course (using more recent methods)