Mobile Image Retargeting Daniel Graf Bachelor Thesis February 28, - - PowerPoint PPT Presentation

mobile image retargeting
SMART_READER_LITE
LIVE PREVIEW

Mobile Image Retargeting Daniel Graf Bachelor Thesis February 28, - - PowerPoint PPT Presentation

Mobile Image Retargeting Daniel Graf Bachelor Thesis February 28, 2013 1 Image Retargeting February 28, 2013 Daniel Graf # 2 Image Retargeting crop center February 28, 2013 Daniel Graf # 2 Image Retargeting crop center scaling


slide-1
SLIDE 1

February 28, 2013

Mobile Image Retargeting

Daniel Graf · Bachelor Thesis

1

slide-2
SLIDE 2

February 28, 2013 Daniel Graf #

Image Retargeting

2

slide-3
SLIDE 3

February 28, 2013 Daniel Graf #

Image Retargeting

2

crop center

slide-4
SLIDE 4

February 28, 2013 Daniel Graf #

Image Retargeting

2

crop center scaling

slide-5
SLIDE 5

February 28, 2013 Daniel Graf #

Image Retargeting

2

crop center scaling letterbox

slide-6
SLIDE 6

February 28, 2013 Daniel Graf #

Image Retargeting

3

retargeting

slide-7
SLIDE 7

February 28, 2013 Daniel Graf #

Image Retargeting

3

retargeting

  • content-aware
  • objectives
  • preserve

important structure and content

  • no artifacts
slide-8
SLIDE 8

February 28, 2013 Daniel Graf #

Image Retargeting

4

  • content-aware
  • objectives
  • preserve

important structure and content

  • no artifacts

saliency map

slide-9
SLIDE 9

February 28, 2013 Daniel Graf #

Image Retargeting

5

  • content-aware
  • objectives
  • preserve

important structure and content

  • no artifacts

artifacts

slide-10
SLIDE 10

February 28, 2013 Daniel Graf #

Goals

  • bring image retargeting to a mobile device
  • limited CPU, GPU and screen size
  • intuitive user interface for small displays
  • combine cropping and warping techniques

6

slide-11
SLIDE 11

Related Work

slide-12
SLIDE 12

February 28, 2013 Daniel Graf #

Seam Carving

  • Seam Carving

8

„Seam carving for content-aware image resizing.“ Avidan and Shamir, SIGGRAPH 2007

repeatedly remove the least salient path

slide-13
SLIDE 13

February 28, 2013 Daniel Graf #

Warping Techniques

  • look for a continuous mapping

9

F : I → I0

p p0 I I0

H W W 0 H0

slide-14
SLIDE 14

February 28, 2013 Daniel Graf #

Warping Techniques

  • look for a continuous mapping

9

F : I → I0

p p0 I I0

H W W 0 H0

slide-15
SLIDE 15

February 28, 2013 Daniel Graf #

Warping Techniques

  • look for a continuous mapping
  • modelled as energy minimization

10

F : I → I0

S(x, y) || JF (x, y) − ✓ 1 1 ◆ ||2

saliency map Jacobian of the mapping

slide-16
SLIDE 16

February 28, 2013 Daniel Graf #

Warping Techniques

  • look for a continuous mapping
  • modelled as energy minimization

10

F : I → I0

E(F) = Z H

x=0

Z W

y=0

S(x, y) || JF (x, y) − ✓ 1 1 ◆ ||2 dxdy

saliency map Jacobian of the mapping

slide-17
SLIDE 17

February 28, 2013 Daniel Graf #

Warping Techniques

  • discretization on regular grid

11

  • riginal grid

G G0 F

deformed grid

N columns M rows

slide-18
SLIDE 18

February 28, 2013 Daniel Graf #

Possible Discontinuities

  • impose hard constraints
  • leads to a convex
  • ptimization problem
  • optimization is

expensive and only feasible on coarse grids

12

slide-19
SLIDE 19

February 28, 2013 Daniel Graf #

Possible Discontinuities

  • impose hard constraints
  • leads to a convex
  • ptimization problem
  • optimization is

expensive and only feasible on coarse grids

12

slide-20
SLIDE 20

Our Retargeting Operator

slide-21
SLIDE 21

February 28, 2013 Daniel Graf #

Axis-Aligned Image Deformations

  • linear number of variables
  • fast optimization

14

G G0 F

scols

j+1

srows

i+1

s = (srows, scols)T

„Robust image retargeting via axis-aligned deformation.“ Panozzo et al., EG 2012

slide-22
SLIDE 22

February 28, 2013 Daniel Graf #

ASAP Energy

  • locally correspond to a similarity transform

15

important cells should not change their aspect ratio

slide-23
SLIDE 23

February 28, 2013 Daniel Graf #

ASAP Energy

  • locally correspond to a similarity transform

16

average saliency per grid cell change in aspect ratio

  • we use a grid of 25 x 25 cells

✓ Ωi,j ✓M H srows

i

− N W scols

j

◆◆2

slide-24
SLIDE 24

February 28, 2013 Daniel Graf #

ASAP Energy

  • locally correspond to a similarity transform

16

EASAP =

M

X

i=1 N

X

j=1

✓ Ωi,j ✓M H srows

i

− N W scols

j

◆◆2

average saliency per grid cell change in aspect ratio

  • we use a grid of 25 x 25 cells
slide-25
SLIDE 25

February 28, 2013 Daniel Graf #

Constraints

  • row and column sums are target image size

17

srow

1

+ srow

2

+ · · · + srow

M

= H0 scols

1

+ scols

2

+ · · · + scols

N

= W 0 I I0

H W W 0 H0

slide-26
SLIDE 26

February 28, 2013 Daniel Graf #

Constraints

  • no grid cell disappears: minimum cell size

18

scols

j

≥ ¯ W min

j

, j = 1, . . . , N srows

i

≥ ¯ Hmin

i

, i = 1, . . . , M

slide-27
SLIDE 27

February 28, 2013 Daniel Graf #

Constraints

  • no grid cell disappears: minimum cell size

18

scols

j

≥ ¯ W min

j

, j = 1, . . . , N srows

i

≥ ¯ Hmin

i

, i = 1, . . . , M

slide-28
SLIDE 28

February 28, 2013 Daniel Graf #19

DEMO

slide-29
SLIDE 29

Automatic Saliency Detection

slide-30
SLIDE 30

February 28, 2013 Daniel Graf #

Automatic Saliency Detection

  • What is important?
  • Low-level stimuli driven approaches
  • regions of high contrast
  • we use gradient magnitude with dilation

21

slide-31
SLIDE 31

February 28, 2013 Daniel Graf #

Automatic Saliency Detection

  • What is important?
  • Low-level stimuli driven approaches
  • regions of high contrast
  • we use gradient magnitude with dilation
  • High-level object based saliency
  • we use the iOS face detector

22

slide-32
SLIDE 32

User Interface

slide-33
SLIDE 33

February 28, 2013 Daniel Graf #

Problem

24

slide-34
SLIDE 34

February 28, 2013 Daniel Graf #

Problem

24

  • small screen does not allow split view
slide-35
SLIDE 35

February 28, 2013 Daniel Graf #

Problem

24

  • small screen does not allow split view
  • combine saliency map and retargeted image
slide-36
SLIDE 36

February 28, 2013 Daniel Graf #

Problem

24

  • small screen does not allow split view
  • combine saliency map and retargeted image
  • updated warp moves image under the finger

I0

1

I0

2

p1 p2 p1

slide-37
SLIDE 37

February 28, 2013 Daniel Graf #

Fixed-point stabilization

25

I0

1

I0

2

p2 p1

  • move the image to fix the point at the finger
slide-38
SLIDE 38

February 28, 2013 Daniel Graf #

Fixed-point stabilization

26

slide-39
SLIDE 39

Cropping

slide-40
SLIDE 40

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

  • 1. optimize
slide-41
SLIDE 41

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

  • 1. optimize

✁ ✁

crop threshold

  • 2. crop
slide-42
SLIDE 42

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

  • 1. optimize

✁ ✁

crop threshold

  • 2. crop
slide-43
SLIDE 43

February 28, 2013 Daniel Graf #

Threshold-based Cropping

28

  • 1. optimize

✁ ✁

crop threshold

  • 2. crop

3.optimize

slide-44
SLIDE 44

February 28, 2013 Daniel Graf #

Threshold Range

29

✁ ✁ ✁ ✁ ✁

slide-45
SLIDE 45

Results

slide-46
SLIDE 46

February 28, 2013 Daniel Graf #

RetargetMe Benchmark

31

SV CR MULTIOP AAIR

  • ur operator
  • riginal
slide-47
SLIDE 47

February 28, 2013 Daniel Graf #

Results

  • interactive performance of 60 fps
  • cropping operator
  • greediness of threshold approach
  • application
  • picture gallery

32

slide-48
SLIDE 48

February 28, 2013 Daniel Graf #

Results

  • interactive performance of 60 fps
  • cropping operator
  • greediness of threshold approach
  • application
  • picture gallery

32

slide-49
SLIDE 49

February 28, 2013 Daniel Graf #

Future Work

  • more sophisticated cropping approaches
  • application in a (mobile) web browser
  • video retargeting

33

ETH Zurich

History

E T H w a s founded in 1854 by the Swiss Confederation and opened its doors in 1855 as a polytechnic i n s t i t u t e ( E i d g e n ö s s i s c h e Polytechnische Schule). It comprised in the beginning six departments: architecture, civil engineering, m e c h a n i c a l e n g i n e e r i n g , chemistry, forestry, and a catch-all department for mathematics, natural sciences, literature, and social and political sciences. its doors in 1855 as a polytechnic institute (Eidgenössische Polytechnische Schule). It comprised in the beginning six departments: architecture, civil engineering, mechanical engineering, chemistry, forestry, and a catch-all department for mathematics, natural sciences, literature, and social and political sciences. It is locally still known as Poly, derived from the original name Eidgenössisches Polytechnikum "Federal Polytechnic Institute".

ETH Zurich

History

ETH was founded in 1854 by the Swiss Confederation and opened

slide-50
SLIDE 50

February 28, 2013 Daniel Graf #

Conclusion

  • fast, fluid, fun to use application
  • cropping operator is simple yet effective
  • next steps
  • make source code publicly available
  • submit app to the iOS App Store

34

slide-51
SLIDE 51

Questions

slide-52
SLIDE 52

Thank you for your attention.