Image Retargeting Shai Avidan Tel Aviv University Bidirectional - - PowerPoint PPT Presentation

Image Retargeting

Shai Avidan

Tel Aviv University

Bidirectional Similarity (Simakov et al. 2008)

The Bidirectional (dis)similarity measure for images S and T: Goal: Given S, find T s.t. min d(S,T)

Bidirectional Similarity

• T

S d T q q T p S Q Q q P P p p Q P D P S P P q T Q Q T S d T q

cohere m i i m m m S P i i m m cohere

, term the to pixel

• f

color the

• f
• n

contributi the is N 1 Then ,..., in pixel

• f

position the to ing correspond ,..., in pixels the be ,..., Let , min arg i.e., in matches ing correspond the denote ,..., Let pixel contain that in patches all denote ,..., Let : , to s contribute pixel a error The

1 2 T 1 1 1 1 1

• T

S d T q q T p S Q Q q P P p p Q P D Q P S P P q T Q Q T S d T q

complete n j i m m n T Q j j j n n complete

, term the to pixel

• f

color the

• f
• n

contributi the is ˆ N 1 Then ˆ ,..., ˆ in pixel

• f

position the to ing correspond ˆ ,..., ˆ in pixels the be ˆ ,..., ˆ Let , ˆ min arg ˆ s.t. S ˆ i.e., in ˆ ,..., ˆ patches some to patch" similar most the " as serve and pixel contain that in patches all denote ˆ ,..., ˆ Let : , to s contribute pixel a error The

1 2 S 1 1 1 1 1

• m

i i n j i

q T p S q T p S q T Err T q

1 2 T 1 2 S

N 1 ˆ N 1 error nal bidirectio global the to pixel the

• f
• n

contributi the is error term The

• T

S 1 1 T S

N N N 1 ˆ N 1 : rule update get the we zero, to equating and color unknown the respect to with

• f

derivative Taking m n p S p S q T q T q T Err

n j m i i i

• Shift-Maps represent a mapping for each pixel in the output

image into the input image

• The color of the output pixel is copied from corresponding input

pixel

Shift-Map

) ,t (t M(u,v)

y x

• We use relative mapping coordinate (like in Optical Flow)

Our Approach : Shift-Map

) , ( ) , ( ) , ( y x I t v t u I v u R

y x

• )

,t (t M(u,v)

y x

• )

,t (t M(u,v)

y x

Our Approach : Shift-Map

• We look for the optimal mapping - can be

described as an Energy Minimization problem

Geometric Editing as an Energy Minimization

• N

q p s R p d

q M p M E p M E M E

,

)) ( ), ( ( )) ( ( ) (

The Smoothness Term

• Assigns a penalty to a discontinuity

introduced to the output image by a discontinuity in the Shift-Map

This term will minimize editing artifacts and create good stitching in the output image

• Discontinuities are computed based on

color differences and gradient differences

(preserve image structure)

The Smoothness Term

R - Output Image I - Input Image

)) ( ), ( ( ) ( ) (

• !
• q

M p M E q M p M

s

q’ p q No discontinuity in the shift-map

p’

2 ' 2 ' 2 ' 2 '

)) ' ( ) ( ( )) ' ( ) ( ( )) ' ( ) ( ( )) ' ( ) ( ( p I n I q I n I p I n I q I n I

q p q p

"

• "
• "
• "
• The Smoothness Term

q’

• !

# )) ( ), ( ( ) ( ) ( q M p M E q M p M

s

p’ np’ nq’ R - Output Image I - Input Image

)) ( ), ( ( ) ( ) (

• !
• q

M p M E q M p M

s

p q Discontinuity in the shift-map

• Use picture borders
• Can incorporate importance mask

– Order constraint on mapping is applied to prevent duplications of important areas

The Data Term: Retargeting

• Minimal energy mapping can be represented as

graph labeling where the Shift-Map value is the selected label for each output pixel

• Labels: relative shift

Shift-Map as Graph Labeling

• Minimal energy mapping can be represented as

graph labeling where the Shift-Map value is the selected label for each output pixel

• Labels: relative shift

Shift-Map as Graph Labeling

# Nodes : number of pixels in the image # Labels : number of possible shifts

• Retargeting:

– Horizontal shifts: change in width – Vertical shifts: optional - add limited range

Labels and Nodes Range

• ✁
• $
• %
• therwise

) , ( : s Constraint External y t v x t u v u M E

y x D

• )

, ( ) , ( importance low means S Large : s constraint Saliency

y x D

t v t u S v u M E

• ity

discontinu prevent to term smoothness by the Multiply

• therwise

1 , 1 ), , ( , ) , 1 ( and , ) , ( Let : s Constraint Order

• $

&

• x

x S y x y x

t t v u M v u M E t t v u M t t v u M

Optimal Labeling Using Graph Cuts

• Global minimization of the energy is NP-

hard

• We use approximate techniques :

Graph Cuts based expansion moves

Fast Approximate Energy Minimization via Graph Cuts, [Boykov, Veksler, Zabih, PAMI 2001]

Hierarchical Solution

• This is an approximate solution

–leading to local minimum –many of the theoretical guarantees of the “alpha expansion” algorithm are lost

in practice we get good results

Computation time is seconds

Hierarchical Solution

Results and comparison

Results and Comparison

Input

Results and Comparison

Results and Comparison

Input

Results and Comparison

Results and Comparison

Input

Results and Comparison

Video-Retargeting Optimized Scale and Stretch Improved Seam Carving Shift-Maps

Results and Comparison

Input

Results and Comparison

When does it fail and why?

Forward Energy

Results

Results

Results