Rectangling Panoramic Images via Warping Kaiming He Huiwen Chang - - PowerPoint PPT Presentation

▶
rectangling panoramic images
SMART_READER_LITE
LIVE PREVIEW

Rectangling Panoramic Images via Warping Kaiming He Huiwen Chang - - PowerPoint PPT Presentation

May 30, 2023 267 likes •746 views

Rectangling Panoramic Images via Warping Kaiming He Huiwen Chang Jian Sun Microsoft Research Asia Tsinghua University Microsoft Research Asia Introduction Panoramas are irregular Introduction Panoramas are irregular Rectangles

slide-1
SLIDE 1

Rectangling Panoramic Images via Warping

Kaiming He

Microsoft Research Asia

Huiwen Chang

Tsinghua University

Jian Sun

Microsoft Research Asia

slide-2
SLIDE 2

Introduction

  • Panoramas are irregular
slide-3
SLIDE 3

Introduction

  • Panoramas are irregular
  • Rectangles are favored

panoramas in panoramas in

slide-4
SLIDE 4

Introduction

  • Panoramas are irregular
  • Rectangles are favored
  • “Rectangling” the panoramas
slide-5
SLIDE 5

Introduction

  • Panoramas are irregular
  • Rectangles are favored
  • “Rectangling” the panoramas

– Cropping

slide-6
SLIDE 6

Introduction

  • Panoramas are irregular
  • Rectangles are favored
  • “Rectangling” the panoramas

– Cropping – Inpainting

content-aware fill

slide-7
SLIDE 7

Introduction

  • Panoramas are irregular
  • Rectangles are favored
  • “Rectangling” the panoramas

– Cropping – Inpainting

content-aware fill

slide-8
SLIDE 8

Introduction

  • Panoramas are irregular
  • Rectangles are favored
  • “Rectangling” the panoramas

– Cropping – Inpainting – Warping new

  • ur warping
slide-9
SLIDE 9

Why Warping?

  • Panoramas are often distorted

distortion

slide-10
SLIDE 10

Why Warping?

  • Panoramas are often distorted
  • Warping can be unnoticeable
  • ur warping
slide-11
SLIDE 11

Why Warping?

  • Panoramas are often distorted
  • Warping can be unnoticeable
  • Warping is robust

– shape manipulation – image retargeting – image projection – video stabilization

[Igarashi et al, SIGGRAPH 05] … [Wang et al, SIGGRAPH Asia 08] … [Carroll et al, SIGGRAPH 09] … [Liu et al, SIGGRAPH 09] …

slide-12
SLIDE 12

Why Warping?

  • Panoramas are often distorted
  • Warping can be unnoticeable
  • Warping is robust
  • Rectangling via warping
slide-13
SLIDE 13

Challenges

  • Meshing

– irregular input – boundary conditions

?

slide-14
SLIDE 14

Challenges

  • Meshing

– irregular input – boundary conditions

  • Content-preserving

– boundary constraints – shapes – straight lines

?

slide-15
SLIDE 15

Solution: Local + Global

local warping mesh global warping warped back

slide-16
SLIDE 16

Local Warping

  • Mesh-free
slide-17
SLIDE 17

missing known pix

Local Warping

  • Mesh-free
  • Seam Carving [Avidan & Shamir 07]

longest missing boundary

slide-18
SLIDE 18

Local Warping

  • Mesh-free
  • Seam Carving [Avidan & Shamir 07]

– insert a seam – shift pixels

seam

shift

slide-19
SLIDE 19

Local Warping

  • Mesh-free
  • Seam Carving [Avidan & Shamir 07]

– insert a seam – shift pixels

  • Seam Carving = Warping

seam

shift

slide-20
SLIDE 20

Local Warping

  • Mesh-free
  • Seam Carving [Avidan & Shamir 07]

– insert a seam – shift pixels

  • Seam Carving = Warping

(A video was removed when converting this ppt to pdf.)

seam carving

slide-21
SLIDE 21

Local Warping

  • Mesh-free
  • Seam Carving [Avidan & Shamir 07]

– insert a seam – shift pixels

  • Seam Carving = Warping

grid mesh

slide-22
SLIDE 22

Local Warping

  • Mesh-free
  • Seam Carving [Avidan & Shamir 07]

– insert a seam – shift pixels

  • Seam Carving = Warping

warped back

slide-23
SLIDE 23

Global Warping

  • Mesh optimization

min 𝐹(𝑊)

𝑊: all vertexes

slide-24
SLIDE 24

Global Warping

  • Mesh optimization

– Boundary constraints

𝐹𝐶 𝑊 : hard data term

slide-25
SLIDE 25

Global Warping

  • Mesh optimization

– Boundary constraints – Shape preservation

as-similar-as- possible

[Igarashi et al, SIGGRAPH 05] [Liu et al, SIGGRAPH 09] [Wang et al, SIGGRAPH 10] …

𝐹𝑇 𝑊 = 𝑊𝑈𝑀𝑊

𝑀: Laplacian

smoothness term in warping

slide-26
SLIDE 26

input boundary + shape boundary + shape + line detected lines

[PAMI 10]

slide-27
SLIDE 27

Line Preservation

  • Lines in the same direction

are rotated by the same 𝜄

[Chang & Chuang, CVPR 12]

detected lines

in a block

slide-28
SLIDE 28

Line Preservation

  • Lines in the same direction

are rotated by the same 𝜄

[Chang & Chuang, CVPR 12]

quantized directions

(50 bins)

direction 𝑗 direction 𝑘

slide-29
SLIDE 29

Line Preservation

  • Lines in the same direction

are rotated by the same 𝜄

[Chang & Chuang, CVPR 12]

warped

direction 𝑗 direction 𝑘

𝜄𝑗 𝜄

𝑘

slide-30
SLIDE 30

Line Preservation

  • Lines in the same direction

are rotated by the same 𝜄

[Chang & Chuang, CVPR 12]

  • Bind lines to mesh

𝒗 𝒇

warp

𝒇 𝒗

rotate 𝜾

𝑾

bilinear

𝐹𝑀 𝑊, 𝜄 = 𝑊𝑈𝑀𝜄𝑊

𝑀𝜄: Laplacian

slide-31
SLIDE 31

Global Warping

  • Mesh optimization

– Boundary constraints – Shape preservation – Line preservation – Total energy

𝐹 𝑊, 𝜄 = 𝐹𝐶 + 𝐹𝑇 + 𝐹𝑀

fix 𝜄 update 𝑊 fix 𝑊 update 𝜄

slide-32
SLIDE 32

Global Warping

  • Target rectangle

input bounding box normalized scaling x : y ≈ 1:1

slide-33
SLIDE 33

Results

input

slide-34
SLIDE 34

Results

warp

slide-35
SLIDE 35

Results

input

slide-36
SLIDE 36

Results

warp

slide-37
SLIDE 37

Results

input

slide-38
SLIDE 38

Results

warp crop content-aware fill

slide-39
SLIDE 39

Results

input

slide-40
SLIDE 40

Results

warp

slide-41
SLIDE 41

Results

zoom-in output

slide-42
SLIDE 42

Results

zoom-in input

slide-43
SLIDE 43

Results

16-Mp CPU 1-core 2s

slide-44
SLIDE 44

Failure

input

slide-45
SLIDE 45

Failure

warp

slide-46
SLIDE 46

Conclusion

  • New concept - rectangling via warping
  • Unnoticeable, robust, and fast