SLIDE 1
Depth Camera for Mobile Devices
Instructor - Simon Lucey
16-423 - Designing Computer Vision Apps
Today
- Stereo Cameras
- Structured Light Cameras
- Time of Flight (ToF) Camera
Depth Camera for Mobile Devices Instructor - Simon Lucey 16-423 - - - PowerPoint PPT Presentation
Depth Camera for Mobile Devices Instructor - Simon Lucey 16-423 - - - PowerPoint PPT Presentation
Depth Camera for Mobile Devices Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps Today Stereo Cameras Structured Light Cameras Time of Flight (ToF) Camera Inferring 3D Points Given we have prior knowledge of
SLIDE 2
SLIDE 3
Inferring 3D Points
- Given we have prior knowledge of the,
- Intrinsics parameters,
- Extrinsic parameters,
- Corresponding points,
- Question is how to estimate the 3D point ?
{Ωj, τ j}J
j=1
{Λj}J
j=1
{xj}J
j=1
w
SLIDE 4
Inferring 3D Points
ˆ w = min
w J
X
j=1
η{xj − pinhole[w, Λj, Ωj, τ j]}
e.g. η{x} = ||x||2
2
SLIDE 5
Inferring 3D Points
- Optimization problem is inherently non-linear due to the
pinhole camera function.
- Can be made linear using homogeneous coordinates.
SLIDE 6
Inferring 3D Points
- Write j-th out the pinhole camera in homogenous
coordinates,
- Pre-multiply with inverse of the intrinsics matrix,
SLIDE 7
Inferring 3D Points
- Last equation gives,
- Substituting back into the other two equations,
- Re-arranging gives the following system of equations,
SLIDE 8
Inferring 3D Points
- Last equation gives,
- Substituting back into the other two equations,
- Re-arranging gives the following system of equations,
What is the minimum number of cameras (J)?
SLIDE 9
Stereo Camera
SLIDE 10
Stereo Camera
SLIDE 11
Stereo Camera
6.35 cm
SLIDE 12
Stereo Camera
6.35 cm
What is better wide or narrow baseline?
SLIDE 13
Stereo Camera
SLIDE 14
Stereo Camera
SLIDE 15
Examples in Mobile
“Amazon Fire Phone”
SLIDE 16
Examples in Mobile
“Amazon Fire Phone”
Why 4 cameras?
SLIDE 17
A(∆x)
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25Limitations - Texture
- Approach only works if an image patch has texture!!
12
A(∆x) = X
xk∈N (x)
||I(xk) − I(xk + ∆x)||2
SLIDE 18
A(∆x)
5 10 15 20 25 5 10 15 20 25 5 10 15 20 25 5 10 15 20 25Limitations - Texture
- Approach only works if an image patch has texture!!
12
A(∆x) = X
xk∈N (x)
||I(xk) − I(xk + ∆x)||2
SLIDE 19
Today
- Stereo Cameras
- Structured Light Cameras
- Time of Flight (ToF) Camera
SLIDE 20
Projector vs.Camera
14
SLIDE 21
Projector vs.Camera
14
“Camera”
SLIDE 22
Projector vs. Camera
15
SLIDE 23
Projector vs. Camera
15
“Projector”
SLIDE 24
Depth from Structured Light
16
SLIDE 25
Depth from Structured Light
16
How can we get away with one camera?
SLIDE 26
Depth from Structure Light
17
SLIDE 27
Depth from Structured Light
18
SLIDE 28
Prime Sense - Kinect 1.0 Camera
19
(Primesensor)
m). aprox.
How pattern looks like?
- First Region: Allows to obtain a
high accurate depth surface for near objects aprox. (0.8 – 1.2 m)
- Second Region: Allows to obtain
medium accurate depth surface
- aprox. (1.2 – 2.0 m).
- Third Region: Allows to obtain a
low accurate depth surface in far
- bjects aprox. (2.0 – 3.5 m).
SLIDE 29
Examples in Mobile
20
SLIDE 30
ItSeez - App
SLIDE 31
ItSeez - App
SLIDE 32
Limitations - Range
22
SLIDE 33
Limitations - DeFocus
23
(a) Scene (b) Disparity Map
SLIDE 34
Limitations - Ambient Light
- A sunny day on Earth can reach up to 1120Wm-2
- Tabletop projector releases on average 10W of light.
24
500 1000 1500 2000 2500 3000 3500 4000 0.5 1 1.5 2 2.5 Wavelength (in nm) Spectral Irradiance (in Wm−2nm−1) Extraterrestrial Radiation Direct + Circumsolar Irradiance
SLIDE 35
Today
- Stereo Cameras
- Structured Light Cameras
- Time of Flight (ToF) Camera
SLIDE 36
Time of Flight Cameras
- Light travels at approximately a constant speed c =
3x10
8 ms
- 1.
- Measuring the time it takes for light to travel over a
distance once can infer distance.
- Can be categorized into two types:-
- 1. Direct TOF - switch laser on and off rapidly.
- 2. Indirect TOF - send out modulated light, then
measure phase difference to infer depth.
SLIDE 37
Direct - TOF
- Light Detection And Ranging (LiDAR)
probably best example in computer vision and robotics.
- High-energy light pulses limit influence
- f background illumination.
- However, difficulty to generate short
light pulses with fast rise and fall times.
- High-accuracy time measurement
required.
- Prone to motion blur.
- Expensive.
SLIDE 38
Direct - TOF
- Light Detection And Ranging (LiDAR)
probably best example in computer vision and robotics.
- High-energy light pulses limit influence
- f background illumination.
- However, difficulty to generate short
light pulses with fast rise and fall times.
- High-accuracy time measurement
required.
- Prone to motion blur.
- Expensive.
SLIDE 39
Direct TOF - Zebedee CSIRO
SLIDE 40
Direct TOF - Zebedee CSIRO
SLIDE 41
Indirect - TOF
- Continuous light waves instead of short light pulses.
- Modulation in terms of frequency of sinusoidal waves.
- Detected wave after reflection has shifted phase.
- Phase shift proportional to distance from reflecting surface.
Emitter Detector
continuous wave 3D Surface
... ...
20 MHz
... ...
Phase Meter phase shift
SLIDE 42
Indirect - TOF
SLIDE 43
Indirect TOF
SLIDE 44
Examples - Mobile
SLIDE 45
REAL3
TM Image Sensor
SLIDE 46
The Future
SLIDE 47