Content-Based Projections for Panoramic Images and Panoramic - - PowerPoint PPT Presentation

content based projections for
SMART_READER_LITE
LIVE PREVIEW

Content-Based Projections for Panoramic Images and Panoramic - - PowerPoint PPT Presentation

Content- Based Projections for Content-Based Projections for Panoramic Images and Panoramic Images and Videos Videos Introduction Panoramic Leonardo Koller Sacht Images Optimizing Paulo Cezar Carvalho (advisor) Content- Preserving...


slide-1
SLIDE 1

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Content-Based Projections for Panoramic Images and Videos

Leonardo Koller Sacht Paulo Cezar Carvalho (advisor) Luiz Velho (co-advisor)

Visgraf - IMPA

April 5, 2010

Content-Based Projections for Panoramic Images and Videos

slide-2
SLIDE 2

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Motivation

Common cameras capture just a limited field of view of the scene, while our eyes see a much wider field of view with no obvious distortion; Representation of a scene; Extrapolation of the human perception perception; These motivations are even more clear for videos (interesting applications);

Content-Based Projections for Panoramic Images and Videos

slide-3
SLIDE 3

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Goals

Study and understand the problem of finding acceptable panoramic images; Deeply detail one reference on this topic; Propose extensions; Focus on mathematical aspects of the problem;

Content-Based Projections for Panoramic Images and Videos

slide-4
SLIDE 4

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Pipeline

A time line:

Figure: Structure of the presentation.

Content-Based Projections for Panoramic Images and Videos

slide-5
SLIDE 5

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

The viewing sphere

Each point has an associated color, the color that is seen when

  • ne looks toward this point.

Figure: A viewing sphere (looked from outside) that represents the visible information of some scene.

Content-Based Projections for Panoramic Images and Videos

slide-6
SLIDE 6

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Longitude/latitude representation

r : [−π, π] ×

  • −π

2 , π 2

S2 (λ, φ) → (cos(λ) cos(φ), sin(λ) cos(φ), sin(φ))

Figure: Longitude/latitude representation r.

Content-Based Projections for Panoramic Images and Videos

slide-7
SLIDE 7

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Equirectangular Images

Figure: “San Marco Plaza”, by Flickr user Veneboer, taken from [1].

Content-Based Projections for Panoramic Images and Videos

slide-8
SLIDE 8

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Equirectangular Images

Figure: “Reboot 8.0: Ianus demos Cabinet to Thomas’ kid”, by Flickr user Aldo, taken from [1].

Content-Based Projections for Panoramic Images and Videos

slide-9
SLIDE 9

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Problem Statement

We formulate the panoramic image problem as the one of finding a projection u : S ⊆ S2 → R2 (λ, φ) → (u, v) , with desirable properties.

Content-Based Projections for Panoramic Images and Videos

slide-10
SLIDE 10

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Perspective Projection

P :

  • −π

2 , π 2

  • ×
  • −π

2 , π 2

R2 (λ, φ) → (u, v) =

  • tan(λ), tan(φ)

cos(λ)

  • Figure: Left: 90 degree long./90 degree lat.; Right: 130/120.

Content-Based Projections for Panoramic Images and Videos

slide-11
SLIDE 11

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Stereographic Projection

S : (−π, π) ×

  • −π

2 , π 2

R2 (λ, φ) →

  • 2 sin(λ) cos(φ)

cos(λ) cos(φ)+1, 2 sin(φ) cos(λ) cos(φ)+1

  • Figure: Left: 180 degree long./180 degree lat.; Right: 180/180.

Content-Based Projections for Panoramic Images and Videos

slide-12
SLIDE 12

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Mercator Projection

M : [−π, π) ×

  • −π

2 , π 2

R2 (λ, φ) → (u, v) = (λ, log(sec(φ) + tan(φ)))

Figure: 360 degree longitude/150 degree latitude.

Content-Based Projections for Panoramic Images and Videos

slide-13
SLIDE 13

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Correction of Geometric Perceptual Distortions in Pictures (Zorin et. al, [2])

An optimization solution is proposed to compromise between preservation of lines and shapes of objects:

Figure: Correction with λ = 1

2.

Content-Based Projections for Panoramic Images and Videos

slide-14
SLIDE 14

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Squaring the Circle in Panoramas (Zelnik-Manor

  • et. al, [3])

Different perspective projections are combined in a way that fits the geometry of the scene

Figure: 180 degree longitude/90 degree longitude.

Content-Based Projections for Panoramic Images and Videos

slide-15
SLIDE 15

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Optimizing Content-Preserving Projections for Wide-Angle Images (Carroll et. al, [4])

Figure: A (cropped) result produced by the method described in this

  • chapter. FOV: 285 degree longitude/170 degree latitude.

Content-Based Projections for Panoramic Images and Videos

slide-16
SLIDE 16

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Desirable properties

Preserve straight lines; Preserve shape of objects; Vary scale and orientation smoothly; Depend on the scene content (without being restricted to scenes with particular structure); Mathematically formalize distortions; Handle wide fields of view...

Content-Based Projections for Panoramic Images and Videos

slide-17
SLIDE 17

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Discretization of the Viewing Sphere

λij = −π + j 2π n , φij = −π 2 + i π m, j = 0, . . . , n, i = 0, . . . m, uij = u(λij, φij) = (uij, vij), j = 0, . . . , n, i = 0, . . . m.

Figure: Discretization of the equirectangular domain.

Content-Based Projections for Panoramic Images and Videos

slide-18
SLIDE 18

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Preservation of Shapes ↔ Conformality

Definition 2.1: A dipheomorfism ϕ : S → S is a conformal mapping if for all p ∈ S and for all v1, v2 ∈ TpS holds dϕp(v1), dϕp(v2) = Θ2(p)v1, v2, where Θ2 is a differentiable function on S that never vanishes. Locally the mapping preserves angles and inner products (except for the stretching factor Θ2(p)). In our case, S = S2, S = R2 and ϕ = u.

Content-Based Projections for Panoramic Images and Videos

slide-19
SLIDE 19

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Differential vectors

Differential north vector: h = dup

  • ∂r

∂φ(p)

  • = dup

1

  • =

∂u

∂φ(p) ∂v ∂φ(p)

  • Differential east vector:

k = dup

  • ∂r

∂λ(p)

  • = dup
  • 1

cos(φ)

  • =

1 cos(φ) ∂u

∂λ(p) ∂v ∂λ(p)

  • Where

dup = ∂u

∂λ(p) ∂u ∂φ(p) ∂v ∂λ(p) ∂v ∂φ(p)

  • .

Content-Based Projections for Panoramic Images and Videos

slide-20
SLIDE 20

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Differential vectors

Lemma 2.1: u conformal ⇔ h = R90k or h = R−90k.

Figure: We exclude the second possibility above.

Content-Based Projections for Panoramic Images and Videos

slide-21
SLIDE 21

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Cauchy-Riemann Equations

Theorem 2.1 (Cauchy-Riemann Equations): Let p = (λ, φ) ∈ (−π, π) ×

  • −π

2 , π 2

  • . u : S2 → R2 is a

conformal mapping that preserves the orientation of the

  • rthonormal basis of TpS2 if and only if

∂u ∂φ(p) = − 1 cos(φ) ∂v ∂λ(p) and ∂v ∂φ(p) = 1 cos(φ) ∂u ∂λ(p). It is not difficult to show that Mercator and stereographic projections are conformal (satisfy the C-R equations).

Content-Based Projections for Panoramic Images and Videos

slide-22
SLIDE 22

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Finite differences:

∂u ∂φ(λij, φij) ≈

ui+1,j−uij

∆φ

  • ,

∂v ∂λ(λij, φij) ≈

vi,j+1−vij

∆λ

  • ,

∂v ∂φ(λij, φij) ≈

vi+1,j−vij

∆φ

  • ,

∂u ∂λ(λij, φij) ≈

ui,j+1−uij

∆λ

  • .

Content-Based Projections for Panoramic Images and Videos

slide-23
SLIDE 23

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Finite differences:

∂u ∂φ(λij, φij) ≈

ui+1,j−uij

∆φ

  • ,

∂v ∂λ(λij, φij) ≈

vi,j+1−vij

∆λ

  • ,

∂v ∂φ(λij, φij) ≈

vi+1,j−vij

∆φ

  • ,

∂u ∂λ(λij, φij) ≈

ui,j+1−uij

∆λ

  • .

Discretization of the C-R equations: ui+1,j − uij ∆φ + 1 cos φij vi,j+1 − vij ∆λ = 0 1 cos φij ui,j+1 − uij ∆λ − vi+1,j − vij ∆φ = 0.

Content-Based Projections for Panoramic Images and Videos

slide-24
SLIDE 24

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Finite differences:

∂u ∂φ(λij, φij) ≈

ui+1,j−uij

∆φ

  • ,

∂v ∂λ(λij, φij) ≈

vi,j+1−vij

∆λ

  • ,

∂v ∂φ(λij, φij) ≈

vi+1,j−vij

∆φ

  • ,

∂u ∂λ(λij, φij) ≈

ui,j+1−uij

∆λ

  • .

Discretization of the C-R equations: ui+1,j − uij ∆φ + 1 cos φij vi,j+1 − vij ∆λ = 0 1 cos φij ui,j+1 − uij ∆λ − vi+1,j − vij ∆φ = 0. We multiply the equations by spatially varying weights wij and the area of the quad (proportional to cos φij).

Content-Based Projections for Panoramic Images and Videos

slide-25
SLIDE 25

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Conformality Energy: Ec =

m−1

  • i=0

n−1

  • j=0

w2

ij

vi,j+1 − vij ∆λ + cos φij ui+1,j − uij ∆φ 2 + +

m−1

  • i=0

n−1

  • j=0

w2

ij

ui,j+1 − uij ∆λ − cos φij vi+1,j − vij ∆φ 2 .

  • Ec = Cx 2 .

Content-Based Projections for Panoramic Images and Videos

slide-26
SLIDE 26

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of Ec

Figure: A panoramic image obtained by minimizing Ec. Observe that it is the stereographic projection, which we already know to be conformal.

Content-Based Projections for Panoramic Images and Videos

slide-27
SLIDE 27

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

User Interface

Figure: Equirectangular image with lines marked by the user. Red lines stands for vertical lines, blue for horizontal ones and green for general orientation ones.

Content-Based Projections for Panoramic Images and Videos

slide-28
SLIDE 28

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Straight Lines

Figure: Distances should be zero.

Coefficient of the projection of ul

ij − ul start on the normal

direction: El =

  • qij∈Vl
  • (ul

ij − ul start)Tn(ul start, ul end)

2 , where n(ul

start, ul end) = R90

ul

end − ul start

ul

end − ul start .

Content-Based Projections for Panoramic Images and Videos

slide-29
SLIDE 29

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Straight Lines

Figure: Distances should be zero.

Norm of the difference between ul

ij − ul start and its

projection on the line connecting ul

start − ul end:

El =

  • qij∈Vl

(ul

ij−ul start)−s(ul ij, ul start, ul end)(ul end−ul start) 2,

where s(ul

ij, ul start, ul end) =

(ul

ij − ul start)T(ul end − ul start)

ul

end − ul start

.

Content-Based Projections for Panoramic Images and Videos

slide-30
SLIDE 30

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Avoiding Nonlinearities

Alternate between the following two steps: Fix the projections sij and minimize Eld =

  • qij∈Vl

(ul

ij − ul start) − sij(ul end − ul start) 2 .

Fix the normal vectors and minimize Elo =

  • qij∈Vl
  • (ul

ij − ul start)Tn

2 . Usually convergence is reached with three double iterations.

Content-Based Projections for Panoramic Images and Videos

slide-31
SLIDE 31

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy Terms

For lines with fixed orientation, we have Elo = LOx 2 . For lines with no specified orientation, we have Eld = LDAx 2, and Elo = LDOx 2 .

Content-Based Projections for Panoramic Images and Videos

slide-32
SLIDE 32

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of only line energies

Figure: The method tries to straighten the specified lines, but due to the discontinuous nature of the straight line constrains, the result becomes very unpleasant.

Content-Based Projections for Panoramic Images and Videos

slide-33
SLIDE 33

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Smoothness

Figure: The mapping changes too much to satisfy the line constrains. We want the solution to be smoother.

Content-Based Projections for Panoramic Images and Videos

slide-34
SLIDE 34

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Variation of north and east vectors

Figure: h and k varying too much for small variations of (λ, φ).

We impose smoothness on the north vector: ∂h ∂(λ, φ) =

  • ∂2u

∂φ∂λ ∂2u ∂φ2 ∂2v ∂φ∂λ ∂2v ∂φ2

  • =
  • .

Replacing derivatives by finite differences ⇒ Es = Sx 2 .

Content-Based Projections for Panoramic Images and Videos

slide-35
SLIDE 35

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of only Es

Figure: The solution for minimizing only Es.

Content-Based Projections for Panoramic Images and Videos

slide-36
SLIDE 36

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of ALL energies

Figure: Joining all the energies together leads to a smoother solution. Observe that the undesirable artifacts are corrected.

Content-Based Projections for Panoramic Images and Videos

slide-37
SLIDE 37

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Cropped Result

Figure: Cropped result.

Content-Based Projections for Panoramic Images and Videos

slide-38
SLIDE 38

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Spatially Varying Weights

Line endpoint weights(wL

ij ): Has higher values near line

endpoints, to avoid the stronger distortions introduced by the line constrains. Salience weights(wS

ij ): Has higher values in regions with

more details in the image. Face weights(wF

ij ): Has higher values in face regions. It

is important because we perceive much more distortions in faces. Total weights: wij = 2wL

ij + 2wS ij + 4wF ij + 1.

Content-Based Projections for Panoramic Images and Videos

slide-39
SLIDE 39

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Total energy

We alternate between minimizing Ed = w2

c Ec + w2 s Es + w2 l

  • l∈Lf

Elo + w2

l

  • l∈L\Lf

Eld and Eo = w2

c Ec + w2 s Es + w2 l

  • l∈Lf

Elo + w2

l

  • l∈L\Lf

Elo, where wc = 0.4, ws = 0.05 and wl = 1000.

Content-Based Projections for Panoramic Images and Videos

slide-40
SLIDE 40

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Total energy

By defining Ad =     wcC wsS wlLO wlLDA     and Ao =     wcC wsS wlLO wlLOA     , we can rewrite both energies as Ed = Adx2 and Eo = Aox2.

Content-Based Projections for Panoramic Images and Videos

slide-41
SLIDE 41

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization

Minimizing E(x) = Ax2 leads to finding the eigenvector

  • f ATA associated to the third eigenvalue.

Content-Based Projections for Panoramic Images and Videos

slide-42
SLIDE 42

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization

Minimizing E(x) = Ax2 leads to finding the eigenvector

  • f ATA associated to the third eigenvalue.

We replace E(x) by ˜ E(x) = E(x) + εx − y2 = Ax2 + εx − y2, where ε = 10−6 e y is the stereographic mapping.

Content-Based Projections for Panoramic Images and Videos

slide-43
SLIDE 43

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization

Minimizing E(x) = Ax2 leads to finding the eigenvector

  • f ATA associated to the third eigenvalue.

We replace E(x) by ˜ E(x) = E(x) + εx − y2 = Ax2 + εx − y2, where ε = 10−6 e y is the stereographic mapping. Statement 2.5: The minimizer of ˜ E in Rn is x = (ATA + εI)−1(εy).

Content-Based Projections for Panoramic Images and Videos

slide-44
SLIDE 44

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Input equirectangular image.

Content-Based Projections for Panoramic Images and Videos

slide-45
SLIDE 45

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Lines marked by the user.

Content-Based Projections for Panoramic Images and Videos

slide-46
SLIDE 46

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method. FOV: 210/140.

Content-Based Projections for Panoramic Images and Videos

slide-47
SLIDE 47

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method (cropped). FOV: 210/140.

Content-Based Projections for Panoramic Images and Videos

slide-48
SLIDE 48

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Input equirectangular image.

Content-Based Projections for Panoramic Images and Videos

slide-49
SLIDE 49

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Lines marked by the user.

Content-Based Projections for Panoramic Images and Videos

slide-50
SLIDE 50

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method in [3]. FOV: 180/100.

Content-Based Projections for Panoramic Images and Videos

slide-51
SLIDE 51

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method (cropped). FOV: 180/100.

Content-Based Projections for Panoramic Images and Videos

slide-52
SLIDE 52

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

3 double iterations are usually enough; It takes about 1 minute to perform all optimizations; Failure cases may happen when the lines are not well marked; All the results were produced using an application that we developed.

Content-Based Projections for Panoramic Images and Videos

slide-53
SLIDE 53

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Application Software

PLAY THE VIDEO

Content-Based Projections for Panoramic Images and Videos

slide-54
SLIDE 54

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Automatic Face Detection in Equirectangular Images

Steps of the method: Obtain the Mercator Projection; Process the Mercator image; Detect faces on the Mercator image; Map them back to the equirectangular domain.

Content-Based Projections for Panoramic Images and Videos

slide-55
SLIDE 55

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Result

Figure: 6 faces correctly detected and 1 false detection.

Content-Based Projections for Panoramic Images and Videos

slide-56
SLIDE 56

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Semiautomatic Line Detection in Equirectangular Images

Steps of the method: Obtain 6 perspective projections; Filter each perspective image with bilateral filter; Obtain the edges of the filtered images using Canny edge detector; Process the edge images with eigenvalue processing; Detect lines using the Hough transform; Map the detected lines back to the equirectangular domain.

Content-Based Projections for Panoramic Images and Videos

slide-57
SLIDE 57

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Result

Figure: 161 line segments detected.

Content-Based Projections for Panoramic Images and Videos

slide-58
SLIDE 58

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Panoramic Videos: the three cases

Stationary Viewpoint Stationary objects Moving objects Stationary FOV Trivial case Case 1 Moving FOV Case 2 Cases 1 + 2 Moving Viewpoint Case 3

Table: Separation of the panoramic video problem.

Content-Based Projections for Panoramic Images and Videos

slide-59
SLIDE 59

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Desirable properties

Per frame requirements: Each frame must be a good panoramic image; Moving objects must be well preserved; Temporal requirements: Temporal coherence of the scene; Temporal coherence of the objects.

Content-Based Projections for Panoramic Images and Videos

slide-60
SLIDE 60

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

The Temporal Viewing Sphere

Figure: An illustration of the temporal viewing sphere.

R : [−π, π] ×

  • −π

2 , π 2

  • × [0, t0]

→ R4 (λ, φ, t) → (cos(λ) cos(φ), sin(λ) cos(φ), sin(φ), t)

Content-Based Projections for Panoramic Images and Videos

slide-61
SLIDE 61

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Equirectangular Video

Figure: First frame of the video we use in this work.

Content-Based Projections for Panoramic Images and Videos

slide-62
SLIDE 62

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Problem Statement

We want to find a projection U : S ⊆ TS2 → R3 (λ, φ, t) → (U(λ, φ, t), V (λ, φ, t), t) , with desirable properties.

Content-Based Projections for Panoramic Images and Videos

slide-63
SLIDE 63

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Tangent Vectors

Figure: Projection U, tangent basis of the temporal viewing sphere and tangent basis of the final panoramic video.

Content-Based Projections for Panoramic Images and Videos

slide-64
SLIDE 64

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Transition Functions

Defined for objects and for the entire scene: ϕt1,t2 : S2 × {t1} → S2 × {t2} (λ, φ, t1) → (λt1,t2(λ, φ, t1), φt1,t2(λ, φ, t1), t2) .

Figure: Transition function from time t1 to time t2.

Content-Based Projections for Panoramic Images and Videos

slide-65
SLIDE 65

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 1

ϕsc

t1,t2(λ, φ, t1) = (λ, φ, t2) and ϕob t1,t2 is the function that

describes the movement of the object.

Content-Based Projections for Panoramic Images and Videos

slide-66
SLIDE 66

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 1

ϕsc

t1,t2(λ, φ, t1) = (λ, φ, t2) and ϕob t1,t2 is the function that

describes the movement of the object. Transport of the tangent basis: H(ϕsc

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕsc

t1,t2(λ, φ, t1))

= K(λ, φ, t1) . and H(ϕob

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕob

t1,t2(λ, φ, t1))

= K(λ, φ, t1) .

Content-Based Projections for Panoramic Images and Videos

slide-67
SLIDE 67

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 1

ϕsc

t1,t2(λ, φ, t1) = (λ, φ, t2) and ϕob t1,t2 is the function that

describes the movement of the object. Transport of the tangent basis: H(ϕsc

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕsc

t1,t2(λ, φ, t1))

= K(λ, φ, t1) . and H(ϕob

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕob

t1,t2(λ, φ, t1))

= K(λ, φ, t1) . We constrain only the north vector.

Content-Based Projections for Panoramic Images and Videos

slide-68
SLIDE 68

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Temporal Coherence equations (for case 1)

For the object:

  • ∂U

∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂V ∂φ (λ, φ, t1)

For the scene:

  • ∂U

∂φ (λ, φ, t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λ, φ, t2)

=

∂V ∂φ (λ, φ, t1) .

Content-Based Projections for Panoramic Images and Videos

slide-69
SLIDE 69

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Temporal Coherence equations (for case 1)

For the object:

  • ∂U

∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂V ∂φ (λ, φ, t1)

For the scene:

  • ∂U

∂φ (λ, φ, t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λ, φ, t2)

=

∂V ∂φ (λ, φ, t1) .

We apply the equations from one frame to the next, discretize them using finite differences and obtain energies Eob and Esc that are joined to the image energies.

Content-Based Projections for Panoramic Images and Videos

slide-70
SLIDE 70

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results - Videos in [5]

Figure: Marked lines for the example video in [5].

Content-Based Projections for Panoramic Images and Videos

slide-71
SLIDE 71

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 2

A very simple solution was adopted. Results in [5].

Content-Based Projections for Panoramic Images and Videos

slide-72
SLIDE 72

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Review

Bibliographic review; Deep and conclusive analysis of [4]; Development of the application software; Methods to detect features in equirectangular images; Panoramic Videos.

Content-Based Projections for Panoramic Images and Videos

slide-73
SLIDE 73

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Future Work

For images: Migrate all the Matlab code to C/C++; Apply the method to gigapixel images. For videos: Finish cases 1 and 2; Integrate the solutions for cases 1 and 2; Case 3; Real applications! Cinema, Sport broadcasting...

Content-Based Projections for Panoramic Images and Videos

slide-74
SLIDE 74

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Thanks!

For more: http://w3.impa.br/leo-ks/msc thesis

Content-Based Projections for Panoramic Images and Videos

slide-75
SLIDE 75

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

References

[1] “Flickr: Equirectangular.” URL: http://www.flickr.com/groups/equirectangular/. [2] D. Zorin and A. H. Barr, “Correction of geometric perceptual distortions in pictures,” in SIGGRAPH ’95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, (New York, NY, USA), pp. 257-264, ACM, 1995. [3] L. Zelnik-Manor, G. Peters, and P. Perona, “Squaring the circles in panoramas,” in ICCV ’05: Proceedings of the Tenth IEEE International Conference on Computer Vision, (Washington, DC, USA), pp. 1292-1299, IEEE Computer Society, 2005.

Content-Based Projections for Panoramic Images and Videos

slide-76
SLIDE 76

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

References

[4] R. Carroll, M. Agrawal, and A. Agarwala, “Optimizing content-preserving projections for wide-angle images,” ACM

  • Trans. Graph., vol. 28, no. 3, pp. 1-9, 2009.

[5]L. K. Sacht, “Content-based projections for panoramic images and videos.” URL: http://w3.impa.br/ leo-ks/msc thesis.

Content-Based Projections for Panoramic Images and Videos