ROBOTICS 01PEEQW Basilio Bona DAUIN Politecnico di Torino Mobile - - PowerPoint PPT Presentation

ROBOTICS 01PEEQW

Basilio Bona DAUIN – Politecnico di Torino

Mobile & Service Robotics Sensors for Robotics – 4

Vision

Vision is the most important sense in humans and is becoming important also in robotics

not expensive rich of information

Vision includes three steps

Data recording and transformation in the retina Data transmission through the

• ptical nerves

Data elaboration by the brain

3 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

CMOS (Complementary Metal Oxide Semiconductor technology)

Vision sensors

4

CCD (Coupled Charge Device, light-sensitive, discharging capacitors)

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

CCD sensors

A CCD sensor consists in a range of capacitors, each accumulating a charge proportional to the quantity of light that is hitting it. The charge in each capacitor is then turned into a numerical value by the camera inner system to produce a picture. Advantages/Features of CCD Sensors

• Conversion takes place in the chip without distortion
• CCDs have very high uniformity → Good for HD quality images (not videos)
• These sensors are more sensitive → Produce Better Images in Low Light
• CCD sensors produce cleaner and less grainy Images → Low-noise images
• CCD sensors has been produced for a longer period of time

Disadvantages of CCD Sensors

• CCD sensors consume much more power
• CCDs are interlaced
• Inferior HD videos → Less pixel rates
• CCDs are expensive as they require special manufacturing

Basilio Bona 5 ROBOTICS 01PEEQW - 2015/2016

CMOS sensors

• CMOS sensors can directly make the charge conversion on the generation photosite thanks to

their pixel amplifier. Several transistors at each pixel amplify and move the charge using more traditional wires. Charge is then turned into a numericalcal value which correspond to the image. This characteristic give them the ability to avoid several transfers and to increase the processing speed

• CMOS do not require any special manufacturing. Most of the digital cameras these days use

CMOS Sensor as it reduces cost

• Advantages/ Features Of CMOS Sensor
• CMOS consumes less power (100 times less than CCD)
• CMOS sensors are cheaper
• Each pixel can be individually addressed. High reading rate
• They produce better HD videos
• Disadvantages Of CMOS Sensor
• CMOS sensors are also more susceptible – sometimes images are grainy
• CMOS sensors need more light for better image
• Despite some disadvantages CMOS sensor is widely used in Mobile Phones, Tablets, PDAs and
• n most of the digital cameras.
• CCD cameras produce better images but CMOS sensors are catching up fast with its low power

consumption.

Basilio Bona 6 ROBOTICS 01PEEQW - 2015/2016

Parallel lines Converging lines

Artificial vision issues

Projection from a 3D world on a 2D plane: perspective projection (transformation matrices) Discretization effects due to pixels (CCD or CMOS) Misalignment errors (hardware)

7

Pixel discretization

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Camera models

Pinhole camera (aka perspective camera)

8 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Pinhole camera

9 ROBOTICS 01PEEQW - 2015/2016

A B image planes

Decreasing the image plane distance or the hole diameter makes the point images sharper Increasing the hole diameter makes the point images brighter Infinite depth-of-field Infinite depth-of-focus

point images point images

hole diameter

the image is reversed

Basilio Bona

Camera models

Thin lens camera: the lens has a thickness d that is negligible compared to the radii of curvature of the lens surfaces R1, R2 Rays are refracted as they go through the lens (refraction index n)

10 ROBOTICS 01PEEQW - 2015/2016

1 2

1 1 1 ( 1) ; 0 if convex

i

n R f R R     ≈ − − >    

Thin lens equation

Basilio Bona

Thin lens camera

Thin lens camera is reversible Rays parallel to the optical axis pass through the focus and viceversa Rays through the lens center are not refracted There are two symmetrical foci True lens shows aberration phenomena

11 ROBOTICS 01PEEQW - 2015/2016

f f

• ptical axis

lens center

Basilio Bona

Aberration

12 ROBOTICS 01PEEQW - 2015/2016

Spherical

Basilio Bona

Image formation

13 ROBOTICS 01PEEQW - 2015/2016

Optical axis Principal image plane Focal Plane Reversed image plane

π π′

F

π

3D object

Thin lens approximation ≈ Pinhole camera

Basilio Bona

Image formation and equations

14 ROBOTICS 01PEEQW - 2015/2016

image plane real object

p

image distance

• bject distance

q f

focal distance focal plane

( ) 1 1 1 ( ) ( ) p f f pq f p q f q f p q f − = ⇒ = + ⇒ + = − ( ) pf q p f = − if then p f q f ≈ ≫

the image plane is approx in the focal plane Lens equation

f

field of view angle

Basilio Bona

Image formation

15 ROBOTICS 01PEEQW - 2015/2016

f

c

C

c

k

c

i

z

p

x

p

i

x π

F

π

i

P

i i i i

x P y     ⇒ =       p P

x y z

p P p p       ⇒ =         p

Basilio Bona

Transformations

Coordinate transformation between the world frame and the camera frame Projection of 3D point coordinates onto 2D image plane coordinates Coordinate transformation between possible choices of image coordinate frame

16 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Transformations

17

R

c

R

i

R

pix

R

π′

ROBOTICS 01PEEQW - 2015/2016

Image plane World frame Camera frame

1

c c c

    =       R t T 0 T

i pix

T 1 1 1

c i

f         =           T

Rescaling Optical correction

Basilio Bona

Reference frames

18

R

c

p

c

C

c

R

f

Optical axis Focal plane Image plane

i

i

i

R

i

j

pix

R

i

O π v u

F

π

ROBOTICS 01PEEQW - 2015/2016

c

k

c

i

c

j

World frame

P

i

P

c

′ p

pix

p

i

p

c i pix

  →  → ←  ← 

translation translation+scale

R R R p

Basilio Bona

Vector notation

19 ROBOTICS 01PEEQW - 2015/2016

i i i i pix pix

x y u v   ⇔ =       ⇔ =     p p

T T

R R

in pixel units

in 3D in 2D

c c c c c c c c c c

x y x x y x x y x   ⇔ =       ⇔ =       ′ ′ ′ ′ ⇔ =     p p p

T T T

R R R

Basilio Bona

Camera projections Perspective projection Orthographic projection

20 ROBOTICS 01PEEQW - 2015/2016

x i x i z z

p x p x f p f p = → =

z i x

p x p α ≈ ⇒ = if const

large compared to the distance from the camera small compared to the distance from the camera

The pixel height of similar subject is different if the distance from the camera varies a lot. On the left the persons have different pixel height while on the right they have approximately similar heights, since their distance from the camera is high and does not vary much

Basilio Bona

Projections

Basilio Bona 21 ROBOTICS 01PEEQW - 2015/2016

Perspective projection

22

f

i

P

P

c

C

c

k

c

i

z

p

x

p

i

x π′

F

π π

i

C

i

P

x i x z z i

p x p p f p f x = ⇒ = − −

ROBOTICS 01PEEQW - 2015/2016

Usually the negative sign is avoided considering the reversed image plane

All points give the same image P’

f

Basilio Bona

Perspective projection

23 ROBOTICS 01PEEQW - 2015/2016

x c y z

p P p p       ⇒ =         p 1 1

i i c z i c c c z

f f p f p         = ⇒ = =             p p p p P p

x z y z

p p i p c i p z z

f x f P f y p p f             ′ ′   ⇒ = =                 p

i

p

perspective projection

0 1 1

i i c c c c

f f f f λ             = = =                   p p p P p

arbitrary positive constant

Basilio Bona

Perspective projection

24 ROBOTICS 01PEEQW - 2015/2016

Homogeneous coordinates

1

c x y z

p p p   =     p ɶ

T

1

i i i

x y   =     p ɶ

T

1 1 1

i i i z i i i z x i y i i c z i z i c c c z i c i c

x x f p f y y p p x f p p p y f p λ         = ⇒ =             ⇓                     = = = =                           ⇓ = p p p p T p p T p ɶ ɶ ɶ ɶ Π Π Π

Homogeneous perspective/projection matrix

Basilio Bona

Perspective projection

25 ROBOTICS 01PEEQW - 2015/2016

0 1 0 0 1 1 1 0 1

c c i c c

f f                 =                       R t T 0 T Π

Canonical projection matrix

this is the ideal case

Basilio Bona

Perspective projection

PP is studied by projective geometry PP preserves linearity; lines in 3D correspond to lines in 2D and viceversa PP does not preserve parallelism The intersection points in 2D of parallel lines in 3D define vanishing points (points infinitely far away)

26 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Camera parameters

Intrinsic parameters: the parameters that link the pixel coordinates of an image point to the corresponding (metric) coordinates in the camera reference frames Extrinsic parameters: the parameter that define the location and

• rientation of the camera reference frame with respect to a

known world reference frame Camera calibration: the procedure to estimate these parameters

27 ROBOTICS 01PEEQW - 2015/2016

1

W W c c W c

    = ⇒       R t T 0 T 6 parameters

Basilio Bona

Camera intrinsic parameters

28 ROBOTICS 01PEEQW - 2015/2016

n columns m rows

x

s

x y

s s

i

R

pix

R

pix

i

i

i

i

j

pix

j u v 6 8 u v = = ( , )

aspect ratio

u v ,

x pix y

• 

   =     c

in pixel units camera center in pixel units

x y

s s pix , in m

y

s

pix

c

Basilio Bona

Camera intrinsic parameters

Focal length f Transformation between pixel coordinates and camera coordinates Geometric distortion introduced by the optical lens systems

29 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

http://slideplayer.com/slide/7472373/

Camera intrinsic parameters

Transformation between pixel coordinates and camera coordinates (scaling + translation) in homogeneous coordinates

30 ROBOTICS 01PEEQW - 2015/2016

x i x y i y

u s x

• v

s y

• 

= +     = +   1 1 1

i x x px pi i y y py pi i

x s s o x y s s o y       −             = −                   ⇓ p ɶ

Basilio Bona

Lens distorsion

31

Types

Barrel distortion Pincushion distortion

Radial distortion is modelled by a function D(r) that affects each point v in the projected plane relative to the principal point p, where D(r) is normally a non-linear scalar function and p is close to the midpoint

• f the projected image.

Barrel projections are characterized by a positive gradient of the distortion function, whereas pincushion by a negative gradient

( )

d

v D v p v p = − +

Radial distortion Non radial distortion (tangential)

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Lens distortion

32 ROBOTICS 01PEEQW - 2015/2016

( ) ( )

2 4 6 1 2 3 2 4 6 1 2 3

1 1

i id i id

x x k r k r k r y y k r k r k r = + + + = + + +

where

id id

x y ,

are the coordinates of the distorted image points

2 2 2 id id

r x y = +

is the square of the distance from the camera image center

1 2 3

k k k , ,

are intrinsic parameters Radial distortion is approximated by

Basilio Bona

Other image sensor errors

33

Errors are due to the imperfect orthogonality of pixel elements in CCD or CMOS sensors

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Optical sensors

Optical distance sensors

• Depth from focus
• Stereo vision
• ToF cameras

Motion and optical flow

34 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Depth from focus

35

The method consists in measuring the distance of objects in a scene evaluating from two or more images the focal length adjustment necessary to bring objects in focus Short distance focused Medium distance focused Far distance focused

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Depth from focus

Basilio Bona 36 ROBOTICS 01PEEQW - 2015/2016

Depth from focus

Near focusing Far focusing

37 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Depth from focus

1 1 1 f D e = +

D f e δ focal plane image plane ( , , ) x y z ( , )

i i

x y

D

L ( ) 1 1 1 ( ) 2 ( ) ( )

D

L d e b x f d e s x + = − − + blur radius shape

38 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Stereo Cameras

Basilio Bona 39 ROBOTICS 01PEEQW - 2015/2016

Stereo disparity

40

x z

( )

,

r r

x y

( )

, x y

ℓ ℓ

( )

, , x y z f b

left lens image plane right lens baseline (known)

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Stereo disparity

Basilio Bona 41 ROBOTICS 01PEEQW - 2015/2016

Stereo disparity

42

( ) ( )

/ 2 / 2 , / 2 / 2

r r r r r r r

x x x b x b f z f z x x b f z x x x b x x y y y b y y f z b x x + − = = − = + = − + = − = −

ℓ ℓ ℓ ℓ ℓ ℓ ℓ

Idealized camera geometry for stereo vision

Disparity between two images → Depth computation

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Stereo vision

Distance is inversely proportional to disparity

closer objects can be measured more accurately

Disparity is proportional to baseline

For a given disparity error, the accuracy of the depth estimate increases with increasing baseline b However, as b is increased, some objects may appear in one camera, but not in the other

A point visible from both cameras produces a conjugate pair

Conjugate pairs lie on epipolar line (parallel to the x-axis for the arrangement in the figure above)

43 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Stereo vision

Basilio Bona 44 ROBOTICS 01PEEQW - 2015/2016

These two points are corresponding: how do you find them in the two images?

Stereo points correspondence

45

Left image Right image Disparity Right Left

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Epipolar lines

46

P

1

C

2

C π

1

q

2

q

1

e

2

e , R t

1

τ

2

τ

1

ℓ

2

ℓ

corresponding points stay on the epipolar lines epipolar lines these two points are known and fixed (they are called epipoles)

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Stereo vision

Depth calculation

The key problem in stereo vision is how to optimally solve the correspondence problem Corresponding points lie on the epipolar lines

Gray-Level Matching

Match gray-level features on corresponding epipolar lines Zero-crossing of Laplacian of Gaussians is a widely used approach for identifying the same feature in the left and right images “Brightness” = image irradiance or intensity I(x,y) is computed and used as shown below

47 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Depth images

48 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Time of Flight (ToF) Cameras

Range imaging systems based on the known speed of light They measure the time-of-flight, i.e., the time from the emission to the return of the signal The measurement is performed for each point of the image (different from Lidars) The distance resolution is 1 cm (larger than Lidars’) The simplest version of a time-of-flight camera uses light pulses

49 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Time of Flight (ToF) Cameras

Modulated light is emitted by a

• transmitter. This light is reflected by the
• bject to be detected. The returning light

is sampled by an on-chip photosensitive TOF CCD array. The receiver compares the phase difference between the emitted and the received light and computes the time difference of the "Time-of-Flight" individually per pixel. This value multiplied by the speed of light (ca. 300'000km/sec) and divided by 2 corresponds directly linearly to the distance.

Basilio Bona 50 ROBOTICS 01PEEQW - 2015/2016

Time of Flight (ToF) Cameras

The single pixel consists of a photo diode that converts the incoming light into a current In analog solutions fast switches are connected to the photo diode, which sends the current to one of two memory elements (capacitors) that act as summation elements In digital solutions a time counter, running at several gigahertz, is connected to each pixel and stops counting when light is sensed

51 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Time of Flight (ToF) Cameras

Pros

Simplicity

In contrast to stereo vision or triangulation systems, the whole system is very compact: the illumination is placed just next to the lens, whereas the other systems need a certain minimum base line. In contrast to laser scanning systems, no mechanical moving parts are needed

Efficient distance algorithm

It is very easy to extract the distance information out of the output signals of the TOF sensor, therefore this task uses only a small amount of processing power, again in contrast to stereo vision, where complex correlation algorithms have to be

• implemented. After the distance data has been extracted, object detection, for

example, is also easy to carry out because the algorithms are not disturbed by patterns on the object

Speed

Time-of-flight cameras are able to measure the distances within a complete scene with one shot. As the cameras reach up to 160 frames per second, they are ideally suited to be used in real-time applications

52 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Time of Flight (ToF) Cameras

Cons

Background light

When using CMOS or other integrating detectors or sensors that use visible or near visible light (400 nm - 700 nm), although most of the background light coming from artificial lighting or the sun is suppressed, the pixel still has to provide a high dynamic

• range. The background light also generates electrons, which have to be stored. For

example, the illumination units in many of today's TOF cameras can provide an illumination level of about 1 watt. The Sun has an illumination power of about 50 watts per square meter after the optical band-pass filter. Therefore, if the illuminated scene has a size of 1 square meter, the light from the sun is 50 times stronger than the modulated signal. For non-integrating TOF sensors that do not integrate light over time and are using near-infrared detectors (InGaAs) to capture the short laser pulse, direct viewing of the sun is a non-issue. Such TOF sensors are used in space applications and in consideration for automotive applications

53 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Time of Flight (ToF) Cameras

Cons

Interference

In certain types of TOF devices, if several time-of-flight cameras are running at the same time, the TOF cameras may disturb each other's measurements.

Multiple reflections

In contrast to laser scanning systems, where only a single point is illuminated at

• nce, the time-of-flight cameras illuminate a whole scene. On a phase difference

device, due to multiple reflections, the light may reach the objects along several paths and therefore, the measured distance may be greater than the true

• distance. Direct TOF imagers are vulnerable if the light is reflecting from a specular
• surface. There are published papers that outline the strengths and weaknesses of

the various TOF devices and approaches

54 ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Optical flow

55

Optical flow is the pattern of apparent motion of objects, surfaces, and edges in successive scenes caused by the relative motion between the camera and the scene Optical flow techniques

motion detection

• bject segmentation

time-to-collision motion compensated encoding stereo disparity measurement

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Optical flow

Basilio Bona 56 ROBOTICS 01PEEQW - 2015/2016

Optical flow

57

The optical flow methods try to calculate the motion between two image frames which are taken at times t and t + δt at every voxel position. These methods are called differential since they are based on local Taylor series approximations of the image signal, i.e., they use partial derivatives with respect to the spatial and temporal coordinates

A voxel (volume + pixel) is a volume element representing a value on a regular grid in 3D space ( , , ) ( , , ) ( , , ) ... I x y t I x x y y t t I I I I x y t x y t x y t δ δ δ δ δ δ = + + + ∂ ∂ ∂ = + + + + ∂ ∂ ∂ I I I x y t x y t δ δ δ ∂ ∂ ∂ + + = ∂ ∂ ∂

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Assuming the movement to be small

Optical flow

58

T t x y

V V V I I I x y t I I ∂ ∂ ∂ + + = ∂ ∂ ∂ ∇ ⋅ = −

This problem is known as the aperture problem of the

• ptical flow algorithms

There is only one equation in two unknowns and therefore cannot be solved To find the optical flow another set of equations is needed, given by some additional constraint. All optical flow methods introduce additional conditions for estimating the actual flow.

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona

Optical flow

Lucas–Kanade Optical Flow Method

A two-frame differential methods for motion estimation

1 1 2 2

1 1 1 1 2 2 2 2 1

( )

n n

x x y y t t x y x x y y t t x y x y xn yn xn x yn y t t T T

I V I V I I I I I V I V I I I I V V I I I V I V I I

−

      + = −              + = −         ⇒ = −                        + = −            = ⇒ = Ax b x A A A b ⋯ ⋯ ⋯ ⋯

The additional constraints needed for the estimation of the flow are introduced in this method by assuming that the flow is constant in a small window of size with which is centered at pixel Numbering the pixels as a set of equations can be found

59

,

x y

V V 1 m > ( , ) x y

2

1, ,n m = …

ROBOTICS 01PEEQW - 2015/2016 Basilio Bona