Lecture: Image Resizing Juan Carlos Niebles and Ranjay Krishna - - PowerPoint PPT Presentation

▶
lecture image resizing
SMART_READER_LITE
LIVE PREVIEW

Lecture: Image Resizing Juan Carlos Niebles and Ranjay Krishna - - PowerPoint PPT Presentation

Aug 29, 2022 274 likes •1.13k views

Image Resizing Lecture: Image Resizing Juan Carlos Niebles and Ranjay Krishna 15-Oct-2019 Stanford Vision and Learning Lab 1 St Stanfor ord University CS 131 Roadmap Image Resizing Pixels Segments Images Videos Web Neural networks

slide-1
SLIDE 1

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 1

Lecture: Image Resizing

Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab

slide-2
SLIDE 2

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 2

CS 131 Roadmap

Pixels Images

Convolutions Edges Descriptors

Segments

Resizing Segmentation Clustering Recognition Detection Machine learning

Videos

Motion Tracking

Web

Neural networks Convolutional neural networks

slide-3
SLIDE 3

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 3

Today’s agenda

  • Image resizing

– Seam carving – Dynamic programming

  • Applications
slide-4
SLIDE 4

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 4

Display Devices

slide-5
SLIDE 5

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 5

Content Retargeting

PC iPhone

slide-6
SLIDE 6

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 6

Page Layout

slide-7
SLIDE 7

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 7

?

Simple Media Retargeting Operators

?

Letterboxing Scaling

slide-8
SLIDE 8

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 8

Content-aware Retargeting Operators

Content- aware Content-

  • blivious

“Important” content

slide-9
SLIDE 9

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 9

Content-aware Retargeting

Input Crop Scale Content-aware “less-Important” content

slide-10
SLIDE 10

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 10

Image Retargeting

  • Problem statement:

– Input Image 𝐽 nxm, and new size n’xm’ – Output Image 𝐽′ of size n’xm’ which will be “good representative” of the original image 𝐽

  • To date, no agreed definition, or measure, as to what a good

representative is in this context!

slide-11
SLIDE 11

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 11

Image/Video Retargeting

  • In large, we would expect retargeting to:
  • 1. Adhere to the geometric constraints (display/aspect ratio)
  • 2. Preserve the important content and structures
  • 3. Limit artifacts
  • Very Ill-posed!

– How do we define important? Is there a universal ground truth? – Would different viewers think the same about a retargeted image? – What about artistic impression in the original content?

slide-12
SLIDE 12

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 12

Importance (Saliency) Measures

  • A function S: p à [0,1]

Wang et al. 2008 Judd et al. ICCV09 Learning to predict where people look

  • More sophisticated: attention models, eye tracking

(gazing studies), face detectors, …

slide-13
SLIDE 13

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 13

General Retargeting Framework

  • 1. Define an energy function E(I)

(interest, importance, saliency)

  • 2. Use some operator(s) to

change the image I Recompose Setlur et al. [2005] Crop Santella et

  • al. [2005]

Warp Gal et al. [2006]

slide-14
SLIDE 14

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 14

Previous Retargeting Approaches

  • Optimal Cropping Window
  • For videos: “Pan and scan”

Still done manually in the movie industry

Liu and Gleicher, Video Retargeting: Automating Pan and Scan (2006)

slide-15
SLIDE 15

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 15

Cropping

slide-16
SLIDE 16

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 16

Seam Carving

  • Assume m x n à m x n’, n’<n (summarization)
  • Basic Idea: remove unimportant pixels from the image

– Unimportant = pixels with less “energy”

  • Intuition for gradient-based energy:

– Preserve strong contours – Human vision more sensitive to edges – so try remove content from smoother areas – Simple enough for producing some nice results

slide-17
SLIDE 17

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 17

Pixel Removal

Optimal Least-energy pixels (per row) Least-energy columns

slide-18
SLIDE 18

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 18

A Seam

  • A connected path of pixels from top to bottom (or left to right). Exactly one in

each row

slide-19
SLIDE 19

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 19

A Seam

  • A connected path of pixels from top to bottom (or left to right). Exactly one in

each row

slide-20
SLIDE 20

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 20

Finding the Seam?

slide-21
SLIDE 21

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 21

The Optimal Seam

) ( min arg

*

s E s

S

=

| | | | ) ( I I I y x E ¶ ¶ + ¶ ¶ =

slide-22
SLIDE 22

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 22

The Optimal Seam

  • The recursion relation
  • Can be solved efficiently using dynamic programming in

(s=3 in the original algorithm)

( )

) 1 , 1 ( ), , 1 ( ), 1 , 1 ( min ) , ( ) , ( +

  • +

= j i j i j i j i E j i M M M M

) ( m n s O × ×

slide-23
SLIDE 23

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 23

Dynamic Programming

  • Invariant property:

– M(i,j) = minimal cost of a seam going through (i,j) (satisfying the seam properties)

5 8 12 3 9 2 5 8 12 3 4 2 3 9 7 3 4 2 5 5 7 8

Energy - E(I,j)

slide-24
SLIDE 24

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 24

Dynamic Programming

5 8 12 3 9

2+5 ( )

) 1 , 1 ( ), , 1 ( ), 1 , 1 ( min ) , ( ) , ( +

  • +

= j i j i j i j i E j i M M M M

5 8 12 3 4 2 3 9 7 3 4 2 5 5 7 8

Energy - E(I,j)

slide-25
SLIDE 25

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 25

Dynamic Programming

5 8 12 3 9 7

3+3 ( )

) 1 , 1 ( ), , 1 ( ), 1 , 1 ( min ) , ( ) , ( +

  • +

= j i j i j i j i E j i M M M M

5 8 12 3 4 2 3 9 7 3 4 2 5 5 7 8

Energy - E(I,j)

slide-26
SLIDE 26

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 26

Dynamic Programming

5 8 12 3 9 7 6 12 14 9 10 8 14 14 15

8+8 ( )

) 1 , 1 ( ), , 1 ( ), 1 , 1 ( min ) , ( ) , ( +

  • +

= j i j i j i j i E j i M M M M

5 8 12 3 4 2 3 9 7 3 4 2 5 5 7 8

Energy - E(I,j)

slide-27
SLIDE 27

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 27

Searching for Minimum

  • Backtrack (can store choices along the path, but do not have to)

5 8 12 3 9 7 6 12 14 9 10 8 14 14 15 16

slide-28
SLIDE 28

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 28

Backtracking the Seam

5 8 12 3 9 7 6 12 14 9 10 8 14 14 15 16

slide-29
SLIDE 29

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 29

Backtracking the Seam

5 8 12 3 9 7 6 12 14 9 10 8 14 14 15 16

slide-30
SLIDE 30

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 30

Backtracking the Seam

5 8 12 3 9 7 6 12 14 9 10 8 14 14 15 16

slide-31
SLIDE 31

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 31

H & V Cost Maps

Low cost High cost

Vertical Cost Horizontal Cost

slide-32
SLIDE 32

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 32

Seam Carving

slide-33
SLIDE 33

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 33

The Seam-Carving Algorithm

SEAM-CARVING(im, n’) // size(im) = m x n 1. Do (n-n’) times

2.1. E ß Compute energy map on im 2.2. s ß Find optimal seam in E 2.3. im ß Remove s from im

2. Return im

  • For vertical resize: transpose the image
  • Running time:

2.1 O(mn) 2.2 O(mn) 2.3 O(mn) èO(dmn) d=n-n’

slide-34
SLIDE 34

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 34

Changing Aspect Ratio

slide-35
SLIDE 35

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 35

Changing Aspect Ratio

Seam Carving Scaling Original

slide-36
SLIDE 36

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 36

Changing Aspect ratio

Seams Scaling Cropping

slide-37
SLIDE 37

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 37

Changing Aspect Ratio

Scaling

slide-38
SLIDE 38

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 38

Changing Aspect Ratio

Scaling

slide-39
SLIDE 39

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 39

Example seam carving

slide-40
SLIDE 40

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 40

Another example

slide-41
SLIDE 41

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 43

Questions?

  • Q: Will the result be the same if the image is flipped upside down?
  • A: Yes (up to numerical stability)
  • Q: Can we improve the running time?
  • A: Yes, by accounting for locality of operations
slide-42
SLIDE 42

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 44

A Local Operator

slide-43
SLIDE 43

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 45

Questions?

  • Q: Will the result be the same if the image is flipped upside down?
  • A: Yes (up to numerical stability)
  • Q: Can we improve the running time?
  • A: Yes, by accounting for locality of operations
  • Q: What happens to the overall energy in the image during seam

carving?

slide-44
SLIDE 44

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 46

Preserved Energy

10% 30% 40% 75%

Energy While resizing: remove as many low energy pixels and as few high energy pixels!

slide-45
SLIDE 45

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 47

Preserved Energy

While resizing: remove as many low energy pixels and as few high energy pixels! If we measure the average energy of pixels in the image after applying a resizing operator… …the average should increase! Average Pixel Energy Image Reduction

slide-46
SLIDE 46

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 48

  • ptimal

Preserved Energy

crop column seam Image Reduction Average Pixel Energy pixel

slide-47
SLIDE 47

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 49

Extension to Both Dimensions?

  • m x n à m’ x n’
  • Remove horizontal seam first?
  • Remove vertical seams first?
  • Alternate between the two?
  • The optimal order can be found! à Dynamic Prog (again)
slide-48
SLIDE 48

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 55

Image Expansion (Synthesis)

slide-49
SLIDE 49

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 56

Image Expansion – take 2

Scaling

slide-50
SLIDE 50

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 58

Combined Insert and Remove

Scaling Insert & remove seams

slide-51
SLIDE 51

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 59

Questions?

slide-52
SLIDE 52

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 60

Multi-Size Images

  • We can create a new representation of an image that will allow

adapting it to different sizes!

1. Precompute all seams once 2. Realtime resizing / transmit with content

slide-53
SLIDE 53

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 61

Multi-Size Images

First to be removed Last to be removed

…

slide-54
SLIDE 54

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 62

Multi-Size Image Representation

+

slide-55
SLIDE 55

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 63

Multi-Size Image Representation

slide-56
SLIDE 56

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 65

Object Removal

slide-57
SLIDE 57

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 66

Auxiliary Energy

  • Recall our seam equation

( )

) 1 , 1 ( ), , 1 ( ), 1 , 1 ( min ) , ( ) , ( +

  • +

= j i j i j i j i E j i M M M M

slide-58
SLIDE 58

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 67

Object Removal

slide-59
SLIDE 59

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 68

Find the Missing Shoe!

slide-60
SLIDE 60

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 69

Solution

slide-61
SLIDE 61

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 70

Limitations

slide-62
SLIDE 62

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 71

With face detector

slide-63
SLIDE 63

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 72

With User Constraints

slide-64
SLIDE 64

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 73

Preserved Energy - Revisited

crop column seam pixel

  • ptimal

Image Reduction Average Pixel Energy

slide-65
SLIDE 65

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 74

Inserted Energy

slide-66
SLIDE 66

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 75

Minimize Inserted Energy

  • Instead of removing the seam of least energy, remove the seam that

inserts the least energy to the image !

slide-67
SLIDE 67

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 76

Tracking Inserted Energy

§ Three possibilities when removing pixel Pi,j pi,j-1 pi,j pi,j+1 pi-1,j-1 pi-1,j pi,j pi-1,j+1 pi-1,j-1

slide-68
SLIDE 68

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 77

Pixel Pi,j : Left Seam

pi,j-1 pi,j pi,j+1 pi-1,j-1 pi-1,j pi,j pi-1,j+1 pi-1,j-1

slide-69
SLIDE 69

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 78

Pixel Pi,j : Right Seam

pi,j+1 pi,j pi,j-1 pi-1,j+1 pi-1,j pi,j pi-1,j-1 pi-1,j+1

slide-70
SLIDE 70

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 79

Pixel Pi,j : Vertical Seam

pi,j-1 pi,j pi,j+1 pi-1,j pi-1,j-1 pi,j pi-1,j+1 pi-1,j

slide-71
SLIDE 71

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 80

Old “Backward” Energy

E

slide-72
SLIDE 72

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 81

New Forward Looking Energy

E

slide-73
SLIDE 73

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 83

Results

Input Input Backward Forward Backward Forward

slide-74
SLIDE 74

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 84

Results

slide-75
SLIDE 75

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 85

Backward vs. Forward

Backward Forward

slide-76
SLIDE 76

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 86

Results

slide-77
SLIDE 77

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 87

From Images to Videos

  • In general, video processing is a much (much!) harder problem
  • 1. Cardinality

– Suppose 1min of video x 30 fps = 1800 frames – Say your algorithm processes an image in 1 minute à 30 hours !!

  • 2. Dimensionality/algorithmic

– Temporal coherency: human visual system is highly sensitive to motion!

slide-78
SLIDE 78

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 88

Seam-Carving Video?

  • Naive… frame by frame independently

Time

slide-79
SLIDE 79

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 89

Let’s check out this video

Frame-by-frame Seam-Carving

slide-80
SLIDE 80

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 90

From 2D to 3D

1D paths in images 2D manifolds in video cubes

slide-81
SLIDE 81

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 91

Example video retargeting

slide-82
SLIDE 82

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 92

Object Remove

slide-83
SLIDE 83

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 93

Object detection + seam carving

slide-84
SLIDE 84

Image Resizing

St Stanfor

  • rd University

15-Oct-2019 94

References

  • Seam Carving for Content-Aware Image Resizing – Avidan and Shamir 2007
  • Content-driven Video Retargeting – Wolf et al. 2007
  • Improved Seam Carving for Video Retargeting – Rubinstein et al. 2008
  • Optimized Scale-and-Stretch for Image Resizing – Wang et al. 2008
  • Summarizing Visual Data Using Bidirectional Similarity – Simakov et al. 2008
  • Multi-operator Media Retargeting – Rubinstein et al. 2009
  • Shift-Map Image Editing – Pritch et al. 2009
  • Energy-Based Image Deformation – Karni et al. 2009
  • Seam carving in Photoshop CS4:

http://help.adobe.com/en_US/Photoshop/11.0/WS6F81C45F-2AC0-4685-8FFD- DBA374BF21CD.html