Mobile & Service Robotics Mobile & Service Robotics Sensors - - PowerPoint PPT Presentation

Mobile & Service Robotics Mobile & Service Robotics

Sensors for Robotics Sensors for Robotics – 3 Sensors for Robotics Sensors for Robotics 3

Laser sensors

Rays are transmitted and received coaxially Rays are transmitted and received coaxially The target is illuminated by collimated rays The receiver measures the time‐of‐flight (back and forth) It is possible to change the rays direction (2D or 3D measurements)

D D′

Transmitter R i

L

Receiver

( ) 2 ( ) c f L D D L D θ λ λ ′ ′ = + + = + + ( ) 2 ( ) 2 c f L D D L D λ λ π = + + = + +

2 Basilio Bona ROBOTICA 03CFIOR

Laser sensors

λ

plitude Amp

θ

Transmitted Phase Reflected

3 Basilio Bona ROBOTICA 03CFIOR

Laser sensors

METHODS Pulsed laser: direct measurement of time‐of‐flight: one shall be Pulsed laser: direct measurement of time of flight: one shall be able to measure intervals in the picoseconds range Beat frequency between a modulating wave and the reflected Beat frequency between a modulating wave and the reflected wave Ph d l Phase delay

It is the easiest implementable method

4 Basilio Bona ROBOTICA 03CFIOR

Laser sensors

; 2 2 c D L D L f θ λ λ π ′ = = + = + c = speed of light f D = ′ = frequency of the moduling wave total distance 5 60 D f λ total distance MHz; m The confidence on distance estimation is inversely proportional to 5 60 f λ = = MHz; m The confidence on distance estimation is inversely proportional to the square value of the received signal amplitude

5 Basilio Bona ROBOTICA 03CFIOR

Laser sensors

A typical image from a rotating mirror laser scanner. S t l th ti l t th t t i t Segment lengths are proportional to the measurement uncertainty

6 Basilio Bona ROBOTICA 03CFIOR

Triangulation

i l i i h f d i i h l i f Triangulation is the process of determining the location of an

• bject by measuring angles from known points to the object at

either end of a fixed known baseline either end of a fixed known baseline The point can be chosen as the third point of a triangle with one known side and two known angles In practice: Light sheets (or other patterns) are projected on the target R fl t d li ht i t d b li 2D t i li ht Reflected light is captured by a linear or 2D matrix light sensor Simple trigonometric relations are used to compute the distance

7 Basilio Bona ROBOTICA 03CFIOR

Triangulation

Triangulation concepts baseline

; tan tan 1 1 d d l l d α β = + ⇒ = +

8

tan tan β α β +

Basilio Bona ROBOTICA 03CFIOR

Triangulation

sin sin sin BC AC AB α β γ = = sin sin ; sin sin BC AC AB AB AB AC BC β α γ γ ⋅ ⋅ = = sin sin sin RC AC γ γ α = ⋅ sin RC BC β = ⋅ sin sin sin AB RC α β γ ⋅ ⋅ = sin sin sin( ) AB RC α β α β ⋅ ⋅ = +

9

sin( ) α β +

Basilio Bona ROBOTICA 03CFIOR

Triangulation

D f

Transmitter

D f

Transmitter

L

L D f =

x

D f x =

f

10

f

Basilio Bona ROBOTICA 03CFIOR

Structured light

11 Basilio Bona ROBOTICA 03CFIOR

Structured light

tan H D α =

12 Basilio Bona ROBOTICA 03CFIOR

Structured light

• di

i l Monodimensional case cot f u α −

D

D

f

u cot Du x f u α = −

α

cot Df z f u α = − f

13 Basilio Bona ROBOTICA 03CFIOR

Vision

Vision is the most important sense in humans Vision includes three steps Vision includes three steps

Data recording and transformation in the retina Data transmission through the Data transmission through the

• ptical nerves

Data elaboration by the brain

14 Basilio Bona ROBOTICA 03CFIOR

Natural vision

R ti

15

Retina

Basilio Bona ROBOTICA 03CFIOR

Natural vision

fMRI shows the brain areas Optic chiasm interested by neural activity associated to vision

16 Basilio Bona ROBOTICA 03CFIOR

Artificial vision

C ti Camera = retina Frame grabber = nerves CPU = brain

17 Basilio Bona ROBOTICA 03CFIOR

Vision sensors: hardware

CCD (Co pled Charge De ice light sensiti e discharging capacitors of 5 to CCD (Coupled Charge Device, light‐sensitive, discharging capacitors of 5 to 25 micron) CMOS (Complementary Metal Oxide Semiconductor technology)

18 Basilio Bona ROBOTICA 03CFIOR

Artificial vision

Projection from a 3D world on a 2D plane perspective Projection from a 3D world on a 2D plane: perspective projection (transform matrix) Discretization effects due to transducer pixels (CCD or Discretization effects due to transducer pixels (CCD or CMOS) Misalignment errors Misalignment errors

Pixel discretization Parallel lines Converging lines

19 Basilio Bona ROBOTICA 03CFIOR

Artificial vision

π′

F

π π

3D object Optical axis Principal image plane Focal Plane Reversed image plane Principal image plane

20 Basilio Bona ROBOTICA 03CFIOR

Artificial vision

P′ Geometric parameters

m

R

m

O x

i′

x ′ i

m

f Optical axis

i

x

i

i

i′

R

O

m

x

c

x

c

C

c

R

f

i′

j

c

t

i

O

i

i P

c

R

i

O j P Focal plane Image plane

i

R

π

i

j

F

π

21

Focal plane

Basilio Bona ROBOTICA 03CFIOR

Artificial vision

A

T

c

R

Several rigid and perspective transformations are involved

R P

c

are involved

m

R P

B

T ′ R R

π′

Rescaling Optical correction i

R

i

R

22 Basilio Bona ROBOTICA 03CFIOR

Artificial vision

′

c i c c c i

x x x z f z f x ′ = ⇒ = ′

c

i z P ′

c i c

C

c

k

c

z

i

x ′ A P P ′′

c

x

i

C f

P

c

π′

F

π π f ′ f

P

f

23 Basilio Bona ROBOTICA 03CFIOR

Artificial vision x R x

i′

R

i

R y y

24 Basilio Bona ROBOTICA 03CFIOR

Artificial vision

Image parameters

x

p

O

i′

j

y

p

i

O

i

i

i′

i

c

t

i

C j

25

i

j

Basilio Bona ROBOTICA 03CFIOR

Artificial vision Ab i Aberration types

B l di i Pi hi di i Barrel distortion Pincushion distortion Radial distortion Non radial distortion (tangential)

Radial distortion is modelled by a function D(r) that affects each point v in the projected plane relative to the principal point p, where D(r) is normally a non‐linear scalar function and p is close to the midpoint of the projected image. Barrel projections are characterized by a positive gradient of the distortion function , whereas pincushion by a negative gradient

( )

d

v D v p v p = − +

26

( )

d

p p +

Basilio Bona ROBOTICA 03CFIOR

Artificial vision

Image errors

Errors are due to the imperfect alignment of pixel elements

27 Basilio Bona ROBOTICA 03CFIOR

Vision sensors

Distance sensors

• Depth from focus
• Stereo vision

Stereo vision

Motion and optical flow

28 Basilio Bona ROBOTICA 03CFIOR

Depth from focus

The method consists in measuring the distance of an object evaluating the focal length adjustment necessary to bring it in focus the focal length adjustment necessary to bring it in focus Short distance focus Medium distance focus Far distance focus

29 Basilio Bona ROBOTICA 03CFIOR

Depth from focus

1 1 1 f D = +

D f

f D e

D f L i l ( , , ) x y z

D

L image plane ( , )

i i

x y f l l ( ) 1 1 1 L d e + e δ focal plane ( ) 1 1 1 ( ) 2 ( ) ( )

D

L d e b x f d e s x + = − − + blur radius shape

30 Basilio Bona ROBOTICA 03CFIOR

Depth from focus

Near focusing Far focusing

31 Basilio Bona ROBOTICA 03CFIOR

Stereo disparity

( )

, , x y z x z f

left lens right lens

( )

,

r r

x y

( )

, x y

• image plane

b

baseline (known)

32 Basilio Bona ROBOTICA 03CFIOR

Stereo disparity

/ 2 / 2 x x x b x b + − / 2 / 2 ,

r

x x x b x b f z f z x x b + = = −

• (

)/ 2

r

x x b f z x x = +

• Idealized camera

f

( ) ( )

/ 2

r r

x x x b x x + = −

• geometry for

stereo vision

( )/ 2

r r

y y y b y y + = −

• r

y y f z b x x = −

• r

x x

• Disparity between two

images → Depth

33

g p computation

Basilio Bona ROBOTICA 03CFIOR

Stereo vision

Distance is inversely proportional to disparity

closer objects can be measured more accurately closer objects can be measured more accurately

Disparity is proportional to baseline

For a given disparity error, the accuracy of the depth estimate increases with increasing baseline b However, as b is increased, some objects may appear in one camera, but not in the other ,

A point visible from both cameras produces a conjugate pair conjugate pair

Conjugate pairs lie on epipolar line (parallel to the x‐axis for the arrangement in the figure above)

34 Basilio Bona ROBOTICA 03CFIOR

Th i di

Stereo points correspondence

These two points are corresponding: how do you find them in the two images? Left image Right image Disparity Disparity Ri ht

35

Right Left

Basilio Bona ROBOTICA 03CFIOR

Epipolar lines

P

corresponding points stay on the epipolar lines

π τ τ

1

τ

2

τ

1

q

2

q

1

• C

1

e

2

e

1 2

• 1

C

2

C

2

, R t

epipolar lines

36

these two points are known and fixed (they are called epipoles)

Basilio Bona ROBOTICA 03CFIOR

Stereo vision

Depth calculation

The key problem in stereo vision is how to optimally solve the correspondence problem Corresponding points lie on the epipolar lines

Gray Level Matching Gray‐Level Matching

Match gray‐level features on corresponding epipolar lines Zero‐crossing of Laplacian of Gaussians is a widely used approach for identifying the same feature in the left and right images “Brightness” = image irradiance or intensity I(x,y) is computed and used as shown below

37 Basilio Bona ROBOTICA 03CFIOR

Laplacian

The Laplacian is a 2D isotropic measure of the 2‐nd spatial derivative of an image The Laplacian of an image highlights regions of rapid intensity change and is often used for edge detection change and is often used for edge detection The Laplacian is often applied to an image that has first been h d i h hi i i G i smoothed with something approximating a Gaussian smoothing filter, in order to reduce its sensitivity to noise The operator normally takes a single gray level image as input and produces another gray level image as output

38 Basilio Bona ROBOTICA 03CFIOR

Laplacian

Th L l i L( ) f i ith i l i t it l I( ) i The Laplacian L(x,y) of an image with pixel intensity values I(x,y) is given by:

2 2 2 2 2 2

( , ) I I L x y x y ∂ ∂ = + ∂ ∂ L P I = ⊗

1 ⎡ ⎤ ⎢ ⎥ ⎢ ⎥ 1 2 1 1 ⎡ ⎤ ⎢ ⎥ ⎢ ⎥

1

1 4 1 1 P ⎢ ⎥ = − ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ 1 2 4 2 16 1 2 1 G ⎢ ⎥ = ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦

Convolution

• perator

⎣ ⎦ ⎣ ⎦ 1 1 1 1 8 1 P ⎡ ⎤ − − − ⎢ ⎥ ⎢ ⎥ = ⎢ ⎥

2

1 8 1 1 1 1 P = − − ⎢ ⎥ ⎢ ⎥ − − − ⎢ ⎥ ⎣ ⎦

39 Basilio Bona ROBOTICA 03CFIOR

Convolution

Convolution is a simple mathematical operation which is fundamental to many image processing operators Convolution “multiplies together” two arrays of numbers, generally of different sizes, but of the same dimensionality, to produce a third array of numbers of the same dimensionality This can be used in image processing to implement operators whose output pixel values are simple linear combinations of certain input pixel values In an image processing, one of the input arrays is normally just the gray level image. The second array is usually much smaller, and is also two‐dimensional (although it may be just a single pixel thick), and is known as the kernel

40 Basilio Bona ROBOTICA 03CFIOR

Convolution

41 Basilio Bona ROBOTICA 03CFIOR

Convolution matrix

( ) I i j

11 12 13 14 15 16 17 18 19 21 22 23 24 25 26 27 28 29 31 32 33 34 35 36 37 38 39

( , ) I i j

31 32 33 34 35 36 37 38 39 41 42 43 44 45 46 47 48 49 51 52 53 54 55 56 57 58 59 61 62 63 64 65 66 67 68 69 IMAGE 11 12 13

( , ) K i j

21 22 23 KERNEL If the image has M rows and N columns, and the kernel has m rows and n columns, then the size of the output image will have

M - m + 1 rows, and N - n + 1 columns M m 1 rows, and N n 1 columns

(6 2 1) (9 3 1) 5 7 − + × − + = ×

42 Basilio Bona ROBOTICA 03CFIOR

Convolution product

k11 k12 k13 k21 k22 k23 k11 k12 k13 k21 k22 k23

( ) O i j ( , ) O i j

( , ) ( 1, 1) ( , )

m n

O i j I i k j l K k l = + − + −

∑∑

1 1

1, ,( 1); 1, ,( 1)

k l

i M m j N n

= =

= − + = − + … …

43 Basilio Bona ROBOTICA 03CFIOR

Stereo vision

Z i f L l i f G i Zero crossing of Laplacian of Gaussian Identification of features that are stable and match well Identification of features that are stable and match well Laplacian of intensity image Laplacian of intensity image Step/edge detection of noisy image: filter through Gaussian p/ g y g g smoothing

44 Basilio Bona ROBOTICA 03CFIOR

Edge detection

45 Basilio Bona ROBOTICA 03CFIOR

Stereo vision

L R VERTI CAL FI LTERED I MAGES Confidence im age Depth im age

46 Basilio Bona ROBOTICA 03CFIOR

Optical flow

Optical flow or optic flow is the pattern of apparent motion of

• bjects surfaces and edges in a
• bjects, surfaces, and edges in a

visual scene caused by the relative motion between an b ( )

• bserver (an eye or a camera)

and the scene Optical flow techniques such as Optical flow techniques such as motion detection, object segmentation, time‐to‐collision and focus of expansion calculations, motion compensated encoding, and compensated encoding, and stereo disparity measurement utilize this motion of the objects surfaces and edges

47

surfaces, and edges

Basilio Bona ROBOTICA 03CFIOR

Optical flow

The optical flow methods try to calculate the motion between two image frames which are taken at times t and t + δt at every voxel position. These methods are called differential i th b d l l T l i i ti f th i i l i th since they are based on local Taylor series approximations of the image signal, i.e., they use partial derivatives with respect to the spatial and temporal coordinates

( , , ) ( , , ) ( ) I x y t I x x y y t t I I I I x y t x y t δ δ δ δ δ δ = + + + ∂ ∂ ∂ = + + + + ( , , ) ... I x y t x y t x y t δ δ δ = + + + + ∂ ∂ ∂ I I I x y t x y t δ δ δ ∂ ∂ ∂ + + = ∂ ∂ ∂

A voxel (volumetric + pixel) is a volume element representing a value on

x y t ∂ ∂ ∂

48

( p ) p g a regular grid in 3D space

Basilio Bona ROBOTICA 03CFIOR

Optical flow

V V I I I ∂ ∂ ∂ + + = 0

T t x y

V V V x y t I I + + ∂ ∂ ∂ ∇ ⋅ = − t V I I ∇

Thi bl i k th t bl f th ti l fl l ith This problem is known as the aperture problem of the optical flow algorithms There is only one equation in two unknowns and therefore cannot be solved To find the optical flo anothe set of eq ations is needed gi en b some To find the optical flow another set of equations is needed, given by some additional constraint. All optical flow methods introduce additional conditions for estimating the actual flow.

49 Basilio Bona ROBOTICA 03CFIOR

Optical flow

Lucas–Kanade Optical Flow Method

A tw o-fram e differential m ethods for m otion estim ation A tw o-fram e differential m ethods for m otion estim ation

The additional constraints needed for the estimation of the flow are introduced in this method by assuming that the flow is constant

V V

introduced in this method by assuming that the flow is constant in a small window of size with which is centered at pixel Numbering the pixels as a set of equations can be found

,

x y

V V 1 m > ( , ) x y

2

1, ,n m = …

1 1

1 1 1 1 x x y y t t x y

I V I V I I I I ⎫ ⎡ ⎤ ⎡ ⎤ ⎪ + = − ⎪ ⎢ ⎥ ⎢ ⎥ ⎪ ⎢ ⎥

1 1 2 2

2 2 2 2 y x x y y t t x y x y

I V I V I I I I V V ⎢ ⎥ ⎪ ⎢ ⎥ ⎢ ⎥ ⎪ ⎡ ⎤ + = − ⎪ ⎢ ⎥ ⎪ ⎢ ⎥ ⎢ ⎥ ⇒ = − ⎬ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎪ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎪ ⎣ ⎦

• 1

( )

n n

y xn yn xn x yn y t t T T

I I I V I V I I ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎪ ⎣ ⎦ ⎪ ⎢ ⎥ ⎢ ⎥ ⎪ + = − ⎪ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎪ ⎭ ⎣ ⎦ A b A A A b

1

( )

T T −

= ⇒ = Ax b x A A A b

50 Basilio Bona ROBOTICA 03CFIOR