Content-Based Projections for Panoramic Images and Panoramic - - PowerPoint PPT Presentation

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Content-Based Projections for Panoramic Images and Videos

Leonardo Koller Sacht Paulo Cezar Carvalho (advisor) Luiz Velho (co-advisor)

Visgraf - IMPA

April 5, 2010

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Motivation

Common cameras capture just a limited field of view of the scene, while our eyes see a much wider field of view with no obvious distortion; Representation of a scene; Extrapolation of the human perception perception; These motivations are even more clear for videos (interesting applications);

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Goals

Study and understand the problem of finding acceptable panoramic images; Deeply detail one reference on this topic; Propose extensions; Focus on mathematical aspects of the problem;

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Pipeline

A time line:

Figure: Structure of the presentation.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

The viewing sphere

Each point has an associated color, the color that is seen when

• ne looks toward this point.

Figure: A viewing sphere (looked from outside) that represents the visible information of some scene.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Longitude/latitude representation

r : [−π, π] ×

• −π

2 , π 2

• →

S2 (λ, φ) → (cos(λ) cos(φ), sin(λ) cos(φ), sin(φ))

Figure: Longitude/latitude representation r.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Equirectangular Images

Figure: “San Marco Plaza”, by Flickr user Veneboer, taken from [1].

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Equirectangular Images

Figure: “Reboot 8.0: Ianus demos Cabinet to Thomas’ kid”, by Flickr user Aldo, taken from [1].

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Problem Statement

We formulate the panoramic image problem as the one of finding a projection u : S ⊆ S2 → R2 (λ, φ) → (u, v) , with desirable properties.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Perspective Projection

P :

• −π

2 , π 2

• ×
• −π

2 , π 2

• →

R2 (λ, φ) → (u, v) =

• tan(λ), tan(φ)

cos(λ)

• Figure: Left: 90 degree long./90 degree lat.; Right: 130/120.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Stereographic Projection

S : (−π, π) ×

• −π

2 , π 2

• →

R2 (λ, φ) →

• 2 sin(λ) cos(φ)

cos(λ) cos(φ)+1, 2 sin(φ) cos(λ) cos(φ)+1

• Figure: Left: 180 degree long./180 degree lat.; Right: 180/180.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Mercator Projection

M : [−π, π) ×

• −π

2 , π 2

• →

R2 (λ, φ) → (u, v) = (λ, log(sec(φ) + tan(φ)))

Figure: 360 degree longitude/150 degree latitude.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Correction of Geometric Perceptual Distortions in Pictures (Zorin et. al, [2])

An optimization solution is proposed to compromise between preservation of lines and shapes of objects:

Figure: Correction with λ = 1

2.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Squaring the Circle in Panoramas (Zelnik-Manor

• et. al, [3])

Different perspective projections are combined in a way that fits the geometry of the scene

Figure: 180 degree longitude/90 degree longitude.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Optimizing Content-Preserving Projections for Wide-Angle Images (Carroll et. al, [4])

Figure: A (cropped) result produced by the method described in this

• chapter. FOV: 285 degree longitude/170 degree latitude.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Desirable properties

Preserve straight lines; Preserve shape of objects; Vary scale and orientation smoothly; Depend on the scene content (without being restricted to scenes with particular structure); Mathematically formalize distortions; Handle wide fields of view...

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Discretization of the Viewing Sphere

λij = −π + j 2π n , φij = −π 2 + i π m, j = 0, . . . , n, i = 0, . . . m, uij = u(λij, φij) = (uij, vij), j = 0, . . . , n, i = 0, . . . m.

Figure: Discretization of the equirectangular domain.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Preservation of Shapes ↔ Conformality

Definition 2.1: A dipheomorfism ϕ : S → S is a conformal mapping if for all p ∈ S and for all v1, v2 ∈ TpS holds dϕp(v1), dϕp(v2) = Θ2(p)v1, v2, where Θ2 is a differentiable function on S that never vanishes. Locally the mapping preserves angles and inner products (except for the stretching factor Θ2(p)). In our case, S = S2, S = R2 and ϕ = u.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Differential vectors

Differential north vector: h = dup

• ∂r

∂φ(p)

• = dup

1

• =

∂u

∂φ(p) ∂v ∂φ(p)

• Differential east vector:

k = dup

• ∂r

∂λ(p)

• = dup
• 1

cos(φ)

• =

1 cos(φ) ∂u

∂λ(p) ∂v ∂λ(p)

• Where

dup = ∂u

∂λ(p) ∂u ∂φ(p) ∂v ∂λ(p) ∂v ∂φ(p)

• .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Differential vectors

Lemma 2.1: u conformal ⇔ h = R90k or h = R−90k.

Figure: We exclude the second possibility above.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Cauchy-Riemann Equations

Theorem 2.1 (Cauchy-Riemann Equations): Let p = (λ, φ) ∈ (−π, π) ×

• −π

2 , π 2

• . u : S2 → R2 is a

conformal mapping that preserves the orientation of the

• rthonormal basis of TpS2 if and only if

∂u ∂φ(p) = − 1 cos(φ) ∂v ∂λ(p) and ∂v ∂φ(p) = 1 cos(φ) ∂u ∂λ(p). It is not difficult to show that Mercator and stereographic projections are conformal (satisfy the C-R equations).

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Finite differences:

∂u ∂φ(λij, φij) ≈

ui+1,j−uij

∆φ

• ,

∂v ∂λ(λij, φij) ≈

vi,j+1−vij

∆λ

• ,

∂v ∂φ(λij, φij) ≈

vi+1,j−vij

∆φ

• ,

∂u ∂λ(λij, φij) ≈

ui,j+1−uij

∆λ

• .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Finite differences:

∂u ∂φ(λij, φij) ≈

ui+1,j−uij

∆φ

• ,

∂v ∂λ(λij, φij) ≈

vi,j+1−vij

∆λ

• ,

∂v ∂φ(λij, φij) ≈

vi+1,j−vij

∆φ

• ,

∂u ∂λ(λij, φij) ≈

ui,j+1−uij

∆λ

• .

Discretization of the C-R equations: ui+1,j − uij ∆φ + 1 cos φij vi,j+1 − vij ∆λ = 0 1 cos φij ui,j+1 − uij ∆λ − vi+1,j − vij ∆φ = 0.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Finite differences:

∂u ∂φ(λij, φij) ≈

ui+1,j−uij

∆φ

• ,

∂v ∂λ(λij, φij) ≈

vi,j+1−vij

∆λ

• ,

∂v ∂φ(λij, φij) ≈

vi+1,j−vij

∆φ

• ,

∂u ∂λ(λij, φij) ≈

ui,j+1−uij

∆λ

• .

Discretization of the C-R equations: ui+1,j − uij ∆φ + 1 cos φij vi,j+1 − vij ∆λ = 0 1 cos φij ui,j+1 − uij ∆λ − vi+1,j − vij ∆φ = 0. We multiply the equations by spatially varying weights wij and the area of the quad (proportional to cos φij).

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy term

Conformality Energy: Ec =

m−1

• i=0

n−1

• j=0

w2

ij

vi,j+1 − vij ∆λ + cos φij ui+1,j − uij ∆φ 2 + +

m−1

• i=0

n−1

• j=0

w2

ij

ui,j+1 − uij ∆λ − cos φij vi+1,j − vij ∆φ 2 .

• Ec = Cx 2 .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of Ec

Figure: A panoramic image obtained by minimizing Ec. Observe that it is the stereographic projection, which we already know to be conformal.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

User Interface

Figure: Equirectangular image with lines marked by the user. Red lines stands for vertical lines, blue for horizontal ones and green for general orientation ones.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Straight Lines

Figure: Distances should be zero.

Coefficient of the projection of ul

ij − ul start on the normal

direction: El =

• qij∈Vl
• (ul

ij − ul start)Tn(ul start, ul end)

2 , where n(ul

start, ul end) = R90

ul

end − ul start

ul

end − ul start .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Straight Lines

Figure: Distances should be zero.

Norm of the difference between ul

ij − ul start and its

projection on the line connecting ul

start − ul end:

El =

• qij∈Vl

(ul

ij−ul start)−s(ul ij, ul start, ul end)(ul end−ul start) 2,

where s(ul

ij, ul start, ul end) =

(ul

ij − ul start)T(ul end − ul start)

ul

end − ul start

.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Avoiding Nonlinearities

Alternate between the following two steps: Fix the projections sij and minimize Eld =

• qij∈Vl

(ul

ij − ul start) − sij(ul end − ul start) 2 .

Fix the normal vectors and minimize Elo =

• qij∈Vl
• (ul

ij − ul start)Tn

2 . Usually convergence is reached with three double iterations.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Energy Terms

For lines with fixed orientation, we have Elo = LOx 2 . For lines with no specified orientation, we have Eld = LDAx 2, and Elo = LDOx 2 .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of only line energies

Figure: The method tries to straighten the specified lines, but due to the discontinuous nature of the straight line constrains, the result becomes very unpleasant.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Smoothness

Figure: The mapping changes too much to satisfy the line constrains. We want the solution to be smoother.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Variation of north and east vectors

Figure: h and k varying too much for small variations of (λ, φ).

We impose smoothness on the north vector: ∂h ∂(λ, φ) =

• ∂2u

∂φ∂λ ∂2u ∂φ2 ∂2v ∂φ∂λ ∂2v ∂φ2

• =
• .

Replacing derivatives by finite differences ⇒ Es = Sx 2 .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of only Es

Figure: The solution for minimizing only Es.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization of ALL energies

Figure: Joining all the energies together leads to a smoother solution. Observe that the undesirable artifacts are corrected.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Cropped Result

Figure: Cropped result.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Spatially Varying Weights

Line endpoint weights(wL

ij ): Has higher values near line

endpoints, to avoid the stronger distortions introduced by the line constrains. Salience weights(wS

ij ): Has higher values in regions with

more details in the image. Face weights(wF

ij ): Has higher values in face regions. It

is important because we perceive much more distortions in faces. Total weights: wij = 2wL

ij + 2wS ij + 4wF ij + 1.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Total energy

We alternate between minimizing Ed = w2

c Ec + w2 s Es + w2 l

• l∈Lf

Elo + w2

l

• l∈L\Lf

Eld and Eo = w2

c Ec + w2 s Es + w2 l

• l∈Lf

Elo + w2

l

• l∈L\Lf

Elo, where wc = 0.4, ws = 0.05 and wl = 1000.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Total energy

By defining Ad =     wcC wsS wlLO wlLDA     and Ao =     wcC wsS wlLO wlLOA     , we can rewrite both energies as Ed = Adx2 and Eo = Aox2.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization

Minimizing E(x) = Ax2 leads to finding the eigenvector

• f ATA associated to the third eigenvalue.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization

Minimizing E(x) = Ax2 leads to finding the eigenvector

• f ATA associated to the third eigenvalue.

We replace E(x) by ˜ E(x) = E(x) + εx − y2 = Ax2 + εx − y2, where ε = 10−6 e y is the stereographic mapping.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Minimization

Minimizing E(x) = Ax2 leads to finding the eigenvector

• f ATA associated to the third eigenvalue.

We replace E(x) by ˜ E(x) = E(x) + εx − y2 = Ax2 + εx − y2, where ε = 10−6 e y is the stereographic mapping. Statement 2.5: The minimizer of ˜ E in Rn is x = (ATA + εI)−1(εy).

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Input equirectangular image.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Lines marked by the user.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method. FOV: 210/140.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method (cropped). FOV: 210/140.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Input equirectangular image.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Lines marked by the user.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method in [3]. FOV: 180/100.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

Figure: Result produced by the method (cropped). FOV: 180/100.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results

3 double iterations are usually enough; It takes about 1 minute to perform all optimizations; Failure cases may happen when the lines are not well marked; All the results were produced using an application that we developed.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Application Software

PLAY THE VIDEO

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Automatic Face Detection in Equirectangular Images

Steps of the method: Obtain the Mercator Projection; Process the Mercator image; Detect faces on the Mercator image; Map them back to the equirectangular domain.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Result

Figure: 6 faces correctly detected and 1 false detection.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Semiautomatic Line Detection in Equirectangular Images

Steps of the method: Obtain 6 perspective projections; Filter each perspective image with bilateral filter; Obtain the edges of the filtered images using Canny edge detector; Process the edge images with eigenvalue processing; Detect lines using the Hough transform; Map the detected lines back to the equirectangular domain.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Result

Figure: 161 line segments detected.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Panoramic Videos: the three cases

Stationary Viewpoint Stationary objects Moving objects Stationary FOV Trivial case Case 1 Moving FOV Case 2 Cases 1 + 2 Moving Viewpoint Case 3

Table: Separation of the panoramic video problem.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Desirable properties

Per frame requirements: Each frame must be a good panoramic image; Moving objects must be well preserved; Temporal requirements: Temporal coherence of the scene; Temporal coherence of the objects.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

The Temporal Viewing Sphere

Figure: An illustration of the temporal viewing sphere.

R : [−π, π] ×

• −π

2 , π 2

• × [0, t0]

→ R4 (λ, φ, t) → (cos(λ) cos(φ), sin(λ) cos(φ), sin(φ), t)

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Equirectangular Video

Figure: First frame of the video we use in this work.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Problem Statement

We want to find a projection U : S ⊆ TS2 → R3 (λ, φ, t) → (U(λ, φ, t), V (λ, φ, t), t) , with desirable properties.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Tangent Vectors

Figure: Projection U, tangent basis of the temporal viewing sphere and tangent basis of the final panoramic video.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Transition Functions

Defined for objects and for the entire scene: ϕt1,t2 : S2 × {t1} → S2 × {t2} (λ, φ, t1) → (λt1,t2(λ, φ, t1), φt1,t2(λ, φ, t1), t2) .

Figure: Transition function from time t1 to time t2.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 1

ϕsc

t1,t2(λ, φ, t1) = (λ, φ, t2) and ϕob t1,t2 is the function that

describes the movement of the object.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 1

ϕsc

t1,t2(λ, φ, t1) = (λ, φ, t2) and ϕob t1,t2 is the function that

describes the movement of the object. Transport of the tangent basis: H(ϕsc

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕsc

t1,t2(λ, φ, t1))

= K(λ, φ, t1) . and H(ϕob

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕob

t1,t2(λ, φ, t1))

= K(λ, φ, t1) .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 1

ϕsc

t1,t2(λ, φ, t1) = (λ, φ, t2) and ϕob t1,t2 is the function that

describes the movement of the object. Transport of the tangent basis: H(ϕsc

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕsc

t1,t2(λ, φ, t1))

= K(λ, φ, t1) . and H(ϕob

t1,t2(λ, φ, t1))

= H(λ, φ, t1) K(ϕob

t1,t2(λ, φ, t1))

= K(λ, φ, t1) . We constrain only the north vector.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Temporal Coherence equations (for case 1)

For the object:

• ∂U

∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂V ∂φ (λ, φ, t1)

For the scene:

• ∂U

∂φ (λ, φ, t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λ, φ, t2)

=

∂V ∂φ (λ, φ, t1) .

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Temporal Coherence equations (for case 1)

For the object:

• ∂U

∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λob t1,t2(λ, φ, t1), φob t1,t2(λ, φ, t1), t2)

=

∂V ∂φ (λ, φ, t1)

For the scene:

• ∂U

∂φ (λ, φ, t2)

=

∂U ∂φ (λ, φ, t1) ∂V ∂φ (λ, φ, t2)

=

∂V ∂φ (λ, φ, t1) .

We apply the equations from one frame to the next, discretize them using finite differences and obtain energies Eob and Esc that are joined to the image energies.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Results - Videos in [5]

Figure: Marked lines for the example video in [5].

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Case 2

A very simple solution was adopted. Results in [5].

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Review

Bibliographic review; Deep and conclusive analysis of [4]; Development of the application software; Methods to detect features in equirectangular images; Panoramic Videos.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Future Work

For images: Migrate all the Matlab code to C/C++; Apply the method to gigapixel images. For videos: Finish cases 1 and 2; Integrate the solutions for cases 1 and 2; Case 3; Real applications! Cinema, Sport broadcasting...

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

Thanks!

For more: http://w3.impa.br/leo-ks/msc thesis

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

References

[1] “Flickr: Equirectangular.” URL: http://www.flickr.com/groups/equirectangular/. [2] D. Zorin and A. H. Barr, “Correction of geometric perceptual distortions in pictures,” in SIGGRAPH ’95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, (New York, NY, USA), pp. 257-264, ACM, 1995. [3] L. Zelnik-Manor, G. Peters, and P. Perona, “Squaring the circles in panoramas,” in ICCV ’05: Proceedings of the Tenth IEEE International Conference on Computer Vision, (Washington, DC, USA), pp. 1292-1299, IEEE Computer Society, 2005.

Content-Based Projections for Panoramic Images and Videos

Content- Based Projections for Panoramic Images and Videos Introduction Panoramic Images Optimizing Content- Preserving... Results Feature Detection Panoramic Videos Conclusion

References

[4] R. Carroll, M. Agrawal, and A. Agarwala, “Optimizing content-preserving projections for wide-angle images,” ACM

• Trans. Graph., vol. 28, no. 3, pp. 1-9, 2009.

[5]L. K. Sacht, “Content-based projections for panoramic images and videos.” URL: http://w3.impa.br/ leo-ks/msc thesis.

Content-Based Projections for Panoramic Images and Videos