Image Retargeting Shai Avidan Tel Aviv University Bidirectional - PowerPoint PPT Presentation

Image Retargeting Shai Avidan Tel Aviv University

Bidirectional Similarity (Simakov et al. 2008) The Bidirectional (dis)similarity measure for images S and T: Goal: Given S, find T s.t. min d(S,T)

Bidirectional Similarity � The error term is the contributi on of the pixel q T to the global bidirectio nal error 1 1 � � � � � � � � � � � � � � � � � � n m � � 2 � � 2 ˆ Err T q S p T q S p T q i i � � N j 1 N i 1 S T � � � The error a pixel q T contribute s to d S , T : � � � complete The error a pixel q T contribute s to d S , T : cohere ˆ ˆ Let Q ,..., Q denote all patches in T that contain pixel q Let Q ,..., Q denote all patches in T that contain pixel q 1 n 1 m and serve as " the most similar patch" to some patches Let P ,..., P denote the correspond ing matches in S 1 m � � � � � � � � ˆ ˆ ˆ ˆ ˆ � � � � � � P ,..., P in S i.e., P S s.t. Q arg min D P , Q � i.e., P arg min D P , Q � � � 1 n j j j i i � � � � � � P S Q T Let p ,..., p be the pixels in P ,..., P correspond ing ˆ ˆ Let p ˆ ,..., p ˆ be the pixels in P ,..., P correspond ing 1 m 1 m 1 n 1 m to the position of pixel q in Q ,..., Q Then ˆ ˆ to the position of pixel q in Q ,..., Q Then 1 m 1 m 1 � � � � � � � � m � 2 1 S p T q � � � � � � � � n 2 � S p ˆ T q i N i 1 i N j 1 T S � is the contributi on of the color of pixel q T � is the contributi on of the color of pixel q T � � to the term d S , T � � to the term d S , T cohere complete � � � � � � Taking derivative of Err T q with respect to the unknown color T q and equating to zero, we get the update rule : 1 1 � � � � � � n m � S p ˆ S p i i N j � 1 N i � 1 � � � T q S T n m � N N S T

Shift-Map • Shift-Maps represent a mapping for each pixel in the output image into the input image � M(u,v) (t ,t ) x y • The color of the output pixel is copied from corresponding input pixel

Our Approach : Shift-Map � M(u,v) (t ,t ) x y � � � � R ( u , v ) I ( u t , v t ) I ( x , y ) x y � M(u,v) (t ,t ) x y • We use relative mapping coordinate (like in Optical Flow)

Our Approach : Shift-Map • • •

Geometric Editing as an Energy Minimization • We look for the optimal mapping - can be described as an Energy Minimization problem � � � � E ( M ) E ( M ( p )) E ( M ( p ), M ( q )) d s � � p R p , q N • •

The Smoothness Term • Assigns a penalty to a discontinuity introduced to the output image by a discontinuity in the Shift-Map This term will minimize editing artifacts and create good stitching in the output image • Discontinuities are computed based on color differences and gradient differences (preserve image structure)

The Smoothness Term R - Output Image I - Input Image p’ q’ p q No discontinuity in the shift-map � ! � M ( p ) M ( q ) E ( M ( p ), M ( q )) 0 s

The Smoothness Term R - Output Image I - Input Image n p’ p’ n q’ p q q’ D iscontinuity in the shift-map � ! � M ( p ) M ( q ) E ( M ( p ), M ( q )) 0 s # ! � M ( p ) M ( q ) E ( M ( p ), M ( q )) s � � � � � 2 2 ( I ( n ) I ( q ' )) ( I ( n ) I ( p ' )) p ' q ' " � " � " � " 2 2 ( I ( n ) I ( q ' )) ( I ( n ) I ( p ' )) p ' q '

The Data Term: Retargeting • Use picture borders • Can incorporate importance mask – Order constraint on mapping is applied to prevent duplications of important areas

Shift-Map as Graph Labeling • Minimal energy mapping can be represented as graph labeling where the Shift-Map value is the selected label for each output pixel • Label s: relative shift

Shift-Map as Graph Labeling • Minimal energy mapping can be represented as graph labeling where the Shift-Map value is the selected label for each output pixel • Label s: relative shift

✩ ✡ ✄ � ✁ Labels and Nodes Range # Nodes : number of pixels in the image # Labels : number of possible shifts • Retargeting: – Horizontal shifts: change in width – Vertical shifts: optional - add limited range External Constraint s : � � � � � � % � � � 0 u t x v t y � � � x y E M ( u , v ) � D $ otherwise � Saliency constraint s : Large S means low importance � � � � � E M ( u , v ) S ( u t , v t ) D x y Order Constraint s : � � � � � � � � � Let M ( u , v ) t , t and M ( u 1 , v ) t , t x y x y � & � 1 t t � � � � � � x x E M ( u , v ), M u 1 , v � S $ otherwise � Multiply by the smoothness term to prevent discontinu ity

Optimal Labeling Using Graph Cuts • Global minimization of the energy is NP- hard • We use approximate techniques : Graph Cuts based expansion moves Fast Approximate Energy Minimization via Graph Cuts, [Boykov, Veksler, Zabih, PAMI 2001]

Hierarchical Solution

Hierarchical Solution • This is an approximate solution –leading to local minimum –many of the theoretical guarantees of the “alpha expansion” algorithm are lost in practice we get good results Computation time is seconds

Results and comparison

Results and Comparison Input

Results and Comparison

Results and Comparison Input

Results and Comparison

Results and Comparison Input

Results and Comparison Optimized Scale Improved Video-Retargeting Shift-Maps and Stretch Seam Carving

Results and Comparison Input

Results and Comparison

When does it fail and why?

Forward Energy

Results

Results

Results