Mobile Image Retargeting Daniel Graf Bachelor Thesis February 28, - - PowerPoint PPT Presentation

February 28, 2013

Mobile Image Retargeting

Daniel Graf · Bachelor Thesis

1

February 28, 2013 Daniel Graf #

Image Retargeting

2

February 28, 2013 Daniel Graf #

Image Retargeting

2

crop center

February 28, 2013 Daniel Graf #

Image Retargeting

2

crop center scaling

February 28, 2013 Daniel Graf #

Image Retargeting

2

crop center scaling letterbox

February 28, 2013 Daniel Graf #

Image Retargeting

3

retargeting

February 28, 2013 Daniel Graf #

Image Retargeting

3

retargeting

• content-aware
• objectives
• preserve

important structure and content

• no artifacts

February 28, 2013 Daniel Graf #

Image Retargeting

4

• content-aware
• objectives
• preserve

important structure and content

• no artifacts

saliency map

February 28, 2013 Daniel Graf #

Image Retargeting

5

• content-aware
• objectives
• preserve

important structure and content

• no artifacts

artifacts

February 28, 2013 Daniel Graf #

Goals

• bring image retargeting to a mobile device
• limited CPU, GPU and screen size
• intuitive user interface for small displays
• combine cropping and warping techniques

6

Related Work

February 28, 2013 Daniel Graf #

Seam Carving

• Seam Carving

8

„Seam carving for content-aware image resizing.“ Avidan and Shamir, SIGGRAPH 2007

repeatedly remove the least salient path

February 28, 2013 Daniel Graf #

Warping Techniques

• look for a continuous mapping

9

F : I → I0

p p0 I I0

H W W 0 H0

February 28, 2013 Daniel Graf #

Warping Techniques

• look for a continuous mapping

9

F : I → I0

p p0 I I0

H W W 0 H0

February 28, 2013 Daniel Graf #

Warping Techniques

• look for a continuous mapping
• modelled as energy minimization

10

F : I → I0

S(x, y) || JF (x, y) − ✓ 1 1 ◆ ||2

saliency map Jacobian of the mapping

February 28, 2013 Daniel Graf #

Warping Techniques

• look for a continuous mapping
• modelled as energy minimization

10

F : I → I0

E(F) = Z H

x=0

Z W

y=0

S(x, y) || JF (x, y) − ✓ 1 1 ◆ ||2 dxdy

saliency map Jacobian of the mapping

February 28, 2013 Daniel Graf #

Warping Techniques

• discretization on regular grid

11

• riginal grid

G G0 F

deformed grid

N columns M rows

February 28, 2013 Daniel Graf #

Possible Discontinuities

• impose hard constraints
• leads to a convex
• ptimization problem
• optimization is

expensive and only feasible on coarse grids

12

February 28, 2013 Daniel Graf #

Possible Discontinuities

• impose hard constraints
• leads to a convex
• ptimization problem
• optimization is

expensive and only feasible on coarse grids

12

Our Retargeting Operator

February 28, 2013 Daniel Graf #

Axis-Aligned Image Deformations

• linear number of variables
• fast optimization

14

G G0 F

scols

j+1

srows

i+1

s = (srows, scols)T

„Robust image retargeting via axis-aligned deformation.“ Panozzo et al., EG 2012

February 28, 2013 Daniel Graf #

ASAP Energy

• locally correspond to a similarity transform

15

important cells should not change their aspect ratio

February 28, 2013 Daniel Graf #

ASAP Energy

• locally correspond to a similarity transform

16

average saliency per grid cell change in aspect ratio

• we use a grid of 25 x 25 cells

✓ Ωi,j ✓M H srows

i

− N W scols

j

◆◆2

February 28, 2013 Daniel Graf #

ASAP Energy

• locally correspond to a similarity transform

16

EASAP =

M

X

i=1 N

X

j=1

✓ Ωi,j ✓M H srows

i

− N W scols

j

◆◆2

average saliency per grid cell change in aspect ratio

• we use a grid of 25 x 25 cells

February 28, 2013 Daniel Graf #

Constraints

• row and column sums are target image size

17

srow

1

+ srow

2

+ · · · + srow

M

= H0 scols

1

+ scols

2

+ · · · + scols

N

= W 0 I I0

H W W 0 H0

February 28, 2013 Daniel Graf #

Constraints

• no grid cell disappears: minimum cell size

18

scols

j

≥ ¯ W min

j

, j = 1, . . . , N srows

i

≥ ¯ Hmin

i

, i = 1, . . . , M

February 28, 2013 Daniel Graf #

Constraints

• no grid cell disappears: minimum cell size

18

scols

j

≥ ¯ W min

j

, j = 1, . . . , N srows

i

≥ ¯ Hmin

i

, i = 1, . . . , M

February 28, 2013 Daniel Graf #19

DEMO

Automatic Saliency Detection

February 28, 2013 Daniel Graf #

Automatic Saliency Detection

• What is important?
• Low-level stimuli driven approaches
• regions of high contrast
• we use gradient magnitude with dilation

21

February 28, 2013 Daniel Graf #

Automatic Saliency Detection

• What is important?
• Low-level stimuli driven approaches
• regions of high contrast
• we use gradient magnitude with dilation
• High-level object based saliency
• we use the iOS face detector

22

User Interface

February 28, 2013 Daniel Graf #

Problem

24

February 28, 2013 Daniel Graf #

Problem

24

• small screen does not allow split view

February 28, 2013 Daniel Graf #

Problem

24

• small screen does not allow split view
• combine saliency map and retargeted image

February 28, 2013 Daniel Graf #

Problem

24

• small screen does not allow split view
• combine saliency map and retargeted image
• updated warp moves image under the finger

I0

1

I0

2

p1 p2 p1

February 28, 2013 Daniel Graf #

Fixed-point stabilization

25

I0

1

I0

2

p2 p1

• move the image to fix the point at the finger

February 28, 2013 Daniel Graf #

Fixed-point stabilization

26

Cropping

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

• 1. optimize

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

• 1. optimize

✁ ✁

crop threshold

• 2. crop

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

• 1. optimize

✁ ✁

crop threshold

• 2. crop

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

• 1. optimize

✁ ✁

crop threshold

• 2. crop

3.optimize

February 28, 2013 Daniel Graf #

Threshold Range

29

✁ ✁ ✁ ✁ ✁

Results

February 28, 2013 Daniel Graf #

RetargetMe Benchmark

31

SV CR MULTIOP AAIR

• ur operator
• riginal

February 28, 2013 Daniel Graf #

Results

• interactive performance of 60 fps
• cropping operator
• greediness of threshold approach
• application
• picture gallery

32

February 28, 2013 Daniel Graf #

Results

• interactive performance of 60 fps
• cropping operator
• greediness of threshold approach
• application
• picture gallery

32

February 28, 2013 Daniel Graf #

Future Work

• more sophisticated cropping approaches
• application in a (mobile) web browser
• video retargeting

33

ETH Zurich

History

E T H w a s founded in 1854 by the Swiss Confederation and opened its doors in 1855 as a polytechnic i n s t i t u t e ( E i d g e n ö s s i s c h e Polytechnische Schule). It comprised in the beginning six departments: architecture, civil engineering, m e c h a n i c a l e n g i n e e r i n g , chemistry, forestry, and a catch-all department for mathematics, natural sciences, literature, and social and political sciences. its doors in 1855 as a polytechnic institute (Eidgenössische Polytechnische Schule). It comprised in the beginning six departments: architecture, civil engineering, mechanical engineering, chemistry, forestry, and a catch-all department for mathematics, natural sciences, literature, and social and political sciences. It is locally still known as Poly, derived from the original name Eidgenössisches Polytechnikum "Federal Polytechnic Institute".

ETH Zurich

History

ETH was founded in 1854 by the Swiss Confederation and opened

February 28, 2013 Daniel Graf #

Conclusion

• fast, fluid, fun to use application
• cropping operator is simple yet effective
• next steps
• make source code publicly available
• submit app to the iOS App Store

34

Questions

Thank you for your attention.