Image Resizing Lecture: Image Resizing Juan Carlos Niebles and Ranjay Krishna 15-Oct-2019 Stanford Vision and Learning Lab 1 St Stanfor ord University
CS 131 Roadmap Image Resizing Pixels Segments Images Videos Web Neural networks Convolutions Recognition Resizing Motion Convolutional Edges Detection Segmentation Tracking neural networks Descriptors Machine learning Clustering 15-Oct-2019 2 St Stanfor ord University
Today’s agenda • Image resizing – Seam carving Image Resizing – Dynamic programming • Applications 15-Oct-2019 3 St Stanfor ord University
Display Devices Image Resizing 15-Oct-2019 4 St Stanfor ord University
Content Retargeting Image Resizing 15-Oct-2019 iPhone PC 5 St Stanfor ord University
Page Layout Image Resizing 15-Oct-2019 6 St Stanfor ord University
Simple Media Retargeting Operators Image Resizing Letterboxing Scaling 15-Oct-2019 ? ? 7 St Stanfor ord University
Content-aware Retargeting Operators “Important” content Image Resizing Content- aware 15-Oct-2019 Content- oblivious 8 St Stanfor ord University
Content-aware Retargeting Image Resizing Scale Crop Content-aware 15-Oct-2019 “less-Important” content Input 9 St Stanfor ord University
Image Retargeting • Problem statement: – Input Image 𝐽 nxm, and new size n’xm’ – Output Image 𝐽′ of size n’xm’ which will be “good representative” of Image Resizing the original image 𝐽 • To date, no agreed definition, or measure, as to what a good representative is in this context! 15-Oct-2019 10 St Stanfor ord University
Image/Video Retargeting • In large, we would expect retargeting to: 1. Adhere to the geometric constraints (display/aspect ratio) 2. Preserve the important content and structures Image Resizing 3. Limit artifacts • Very Ill-posed! – How do we define important? Is there a universal ground truth? 15-Oct-2019 – Would different viewers think the same about a retargeted image? – What about artistic impression in the original content? 11 St Stanfor ord University
Importance (Saliency) Measures • A function S: p à [0,1] Image Resizing Wang et al. 2008 • More sophisticated: attention models, eye tracking (gazing studies), face detectors, … 15-Oct-2019 Judd et al. ICCV09 Learning to predict where people look 12 St Stanfor ord University
General Retargeting Framework 1. Define an energy function E(I) 2. Use some operator(s) to change the image I (interest, importance, saliency) Image Resizing Setlur et al. [2005] Recompose Santella et 15-Oct-2019 al. [2005] Crop Gal et al. [2006] Warp 13 Stanfor St ord University
Previous Retargeting Approaches • Optimal Cropping Window Image Resizing • For videos: “Pan and scan” Still done manually in the movie industry 15-Oct-2019 Liu and Gleicher, Video Retargeting: Automating Pan and Scan (2006) 14 St Stanfor ord University
Cropping Image Resizing 15-Oct-2019 15 St Stanfor ord University
Seam Carving • Assume m x n à m x n’, n’<n (summarization) • Basic Idea: remove unimportant pixels from the image Image Resizing – Unimportant = pixels with less “energy” • Intuition for gradient-based energy: 15-Oct-2019 – Preserve strong contours – Human vision more sensitive to edges – so try remove content from smoother areas – Simple enough for producing some nice results 16 St Stanfor ord University
Pixel Removal Image Resizing 15-Oct-2019 Least-energy pixels Least-energy columns Optimal (per row) 17 St Stanfor ord University
A Seam • A connected path of pixels from top to bottom (or left to right). Exactly one in each row Image Resizing 15-Oct-2019 18 St Stanfor ord University
A Seam • A connected path of pixels from top to bottom (or left to right). Exactly one in each row Image Resizing 15-Oct-2019 19 St Stanfor ord University
Finding the Seam? Image Resizing 15-Oct-2019 20 St Stanfor ord University
The Optimal Seam Image Resizing 15-Oct-2019 ¶ ¶ = = + * s arg min E ( s ) E ( I ) | I | | I | ¶ ¶ x y S 21 St Stanfor ord University
The Optimal Seam • The recursion relation ( ) = + - - - - + M ( i , j ) E ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) Image Resizing • Can be solved efficiently using dynamic programming in × × O ( s n m ) (s=3 in the original algorithm) 15-Oct-2019 22 St Stanfor ord University
Dynamic Programming • Invariant property: – M(i,j) = minimal cost of a seam going through (i,j) (satisfying the seam properties) Image Resizing 5 8 12 3 5 8 12 3 4 2 3 9 7 3 4 2 9 2 5 5 7 8 15-Oct-2019 Energy - E(I,j) 23 St Stanfor ord University
Dynamic Programming ( ) = + - - - - + M ( i , j ) E ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) Image Resizing 5 8 12 3 5 8 12 3 4 2 3 9 7 3 4 2 9 2+5 5 5 7 8 15-Oct-2019 Energy - E(I,j) 24 Stanfor St ord University
Dynamic Programming ( ) = + - - - - + M ( i , j ) E ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) Image Resizing 5 8 12 3 5 8 12 3 4 2 3 9 7 3 4 2 9 7 3+3 5 5 7 8 15-Oct-2019 Energy - E(I,j) 25 Stanfor St ord University
Dynamic Programming ( ) = + - - - - + M ( i , j ) E ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) Image Resizing 5 8 12 3 5 8 12 3 4 2 3 9 7 3 4 2 9 7 6 12 5 5 7 8 14 9 10 8 15-Oct-2019 Energy - E(I,j) 14 14 15 8+8 26 St Stanfor ord University
Searching for Minimum • Backtrack (can store choices along the path, but do not have to) Image Resizing 5 8 12 3 9 7 6 12 14 9 10 8 15-Oct-2019 14 14 15 16 27 Stanfor St ord University
Backtracking the Seam Image Resizing 5 8 12 3 9 7 6 12 14 9 10 8 15-Oct-2019 14 14 15 16 28 Stanfor St ord University
Backtracking the Seam Image Resizing 5 8 12 3 9 7 6 12 14 9 10 8 15-Oct-2019 14 14 15 16 29 Stanfor St ord University
Backtracking the Seam Image Resizing 5 8 12 3 9 7 6 12 14 9 10 8 15-Oct-2019 14 14 15 16 30 Stanfor St ord University
H & V Cost Maps High cost Image Resizing Low cost 15-Oct-2019 Horizontal Cost Vertical Cost 31 St Stanfor ord University
Seam Carving Image Resizing 15-Oct-2019 32 St Stanfor ord University
The Seam-Carving Algorithm SEAM-CARVING(im, n’) // size(im) = m x n 1. Do (n-n’) times 2.1. E ß Compute energy map on im Image Resizing 2.2. s ß Find optimal seam in E 2.3. im ß Remove s from im 2. Return im • For vertical resize: transpose the image 15-Oct-2019 • Running time: 2.1 O(mn) 2.2 O(mn) 2.3 O(mn) è O(dmn) d=n-n’ 33 St Stanfor ord University
Changing Aspect Ratio Image Resizing 15-Oct-2019 34 St Stanfor ord University
Changing Aspect Ratio Image Resizing Seam Carving 15-Oct-2019 Original Scaling 35 St Stanfor ord University
Changing Aspect ratio Image Resizing 15-Oct-2019 Cropping Seams Scaling 36 St Stanfor ord University
Changing Aspect Ratio Image Resizing 15-Oct-2019 Scaling 37 St Stanfor ord University
Changing Aspect Ratio Image Resizing 15-Oct-2019 Scaling 38 St Stanfor ord University
Example seam carving Image Resizing 15-Oct-2019 39 St Stanfor ord University
Another example Image Resizing 15-Oct-2019 40 St Stanfor ord University
Questions? • Q: Will the result be the same if the image is flipped upside down? • A: Yes (up to numerical stability) Image Resizing • Q: Can we improve the running time? • A: Yes, by accounting for locality of operations 15-Oct-2019 43 St Stanfor ord University
A Local Operator Image Resizing 15-Oct-2019 44 St Stanfor ord University
Questions? • Q: Will the result be the same if the image is flipped upside down? • A: Yes (up to numerical stability) Image Resizing • Q: Can we improve the running time? • A: Yes, by accounting for locality of operations • Q: What happens to the overall energy in the image during seam carving? 15-Oct-2019 45 St Stanfor ord University
Preserved Energy Image Resizing Energy 15-Oct-2019 10% 30% 40% 75% While resizing: remove as many low energy pixels and as few high energy pixels! 46 St Stanfor ord University
Preserved Energy Image Resizing If we measure the average energy of pixels in the image after applying a resizing operator… …the average should increase! 15-Oct-2019 Average Pixel While resizing: remove as many low energy Energy pixels and as few high energy pixels! Image Reduction 47 St Stanfor ord University
Preserved Energy Image Resizing Average Pixel Energy Image Reduction 15-Oct-2019 pixel crop column seam optimal 48 St Stanfor ord University
Recommend
More recommend
