Fusing Camera and LiDAR to Detect and Recognize Motion Lukas Wendt - - PowerPoint PPT Presentation

▶
fusing camera and lidar to detect and recognize motion
SMART_READER_LITE
LIVE PREVIEW

Fusing Camera and LiDAR to Detect and Recognize Motion Lukas Wendt - - PowerPoint PPT Presentation

May 25, 2023 209 likes •497 views

MIN-Fakultt Fachbereich Informatik Fusing Camera and LiDAR to Detect and Recognize Motion Lukas Wendt Universitt Hamburg Fakultt fr Mathematik, Informatik und Naturwissenschaften Fachbereich Informatik Technische Aspekte Multimodaler

slide-1
SLIDE 1

MIN-Fakultät Fachbereich Informatik

Fusing Camera and LiDAR to Detect and Recognize Motion

Lukas Wendt

Universität Hamburg Fakultät für Mathematik, Informatik und Naturwissenschaften Fachbereich Informatik Technische Aspekte Multimodaler Systeme

  • 12. November 2018
  • L. Wendt – Fusion Camera/LiDAR

1 / 27

slide-2
SLIDE 2

Content

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

  • 1. Introduction

Camera LiDAR LiDAR Calculation Motion Detection & Recognition

  • 2. Sensor Fusion

Competitive Complementary Cooperative Demo Fusion

  • 3. Motion

Movement Relative to Absolute

  • 4. Conclusion & Future Works
  • 5. References
  • L. Wendt – Fusion Camera/LiDAR

2 / 27

slide-3
SLIDE 3

Introduction / Motivation

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Autonomous Robots/Cars ◮ Multiple Sensors

◮ Camera ◮ LiDAR ◮ ...

◮ Sensor Fusion

◮ Complete View of the World ◮ combining advantages of dif. Sensors ◮ Motion Planing

  • L. Wendt – Fusion Camera/LiDAR

3 / 27

slide-4
SLIDE 4

Camera

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ advantage

◮ cheap ◮ passive ◮ small ◮ high resolution

◮ disadvantage

◮ no depth Informations ◮ fog / rain ◮ intensive Computing

  • L. Wendt – Fusion Camera/LiDAR

4 / 27

slide-5
SLIDE 5

LiDAR

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ LiDAR (Light detection and Ranging) ◮ Laser ◮ Sensor ◮ advantage

◮ high distance ◮ depth Information ◮ density Information

◮ disadvantage

◮ expensive ◮ no color Informations ◮ active

  • L. Wendt – Fusion Camera/LiDAR

5 / 27

slide-6
SLIDE 6

LiDAR Calculation

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Calculation Distance

LiDAR Distance

◮ d = c·t 2

  • L. Wendt – Fusion Camera/LiDAR

6 / 27

slide-7
SLIDE 7

LiDAR Calculation (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Calculation Density

LiDAR Density

◮ A = Amplitude ◮ higher Amplitude = higher Density

  • L. Wendt – Fusion Camera/LiDAR

7 / 27

slide-8
SLIDE 8

Motion Detection & Recognition

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Motion Detection

◮ Detects if an object is moving

◮ Motion Recognition

◮ Detects which moving a Object doing

◮ Gesture Example

  • L. Wendt – Fusion Camera/LiDAR

8 / 27

slide-9
SLIDE 9

Sensor Fusion

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Combination of Sensor Data ◮ Redundancy ◮ Same or different Sensors ◮ Frequency or Resolution loose

Definition

„is the combining of sensory data ... so the resulting information is in some sense better than would be possible when these sources were used individually“a

aElmenreich 2001.

  • L. Wendt – Fusion Camera/LiDAR

9 / 27

slide-10
SLIDE 10

Competitive

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Sensor Fusion Competitive

◮ Airplane →Redundancy

  • L. Wendt – Fusion Camera/LiDAR

10 / 27

slide-11
SLIDE 11

Complementary

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Sensor Fusion Complementary

◮ Camera in the Front/Back

  • L. Wendt – Fusion Camera/LiDAR

11 / 27

slide-12
SLIDE 12

Cooperative

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Sensor Fusion Cooperative

◮ LiDAR Cameras Systems in Cars/Robots ◮ 2 Sensors to increase accuracy

  • L. Wendt – Fusion Camera/LiDAR

12 / 27

slide-13
SLIDE 13

A Short Video Demo

  • L. Wendt – Fusion Camera/LiDAR

13 / 27

slide-14
SLIDE 14

Fusion

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Lidar View Camera View Fused View

  • L. Wendt – Fusion Camera/LiDAR

14 / 27

slide-15
SLIDE 15

Fusion (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ First we need to calibrate

Side view of the sensor setup1 1Silva, Roche und Kondoz 2018b.

  • L. Wendt – Fusion Camera/LiDAR

15 / 27

slide-16
SLIDE 16

Movement

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

First Picture Second Picture

  • L. Wendt – Fusion Camera/LiDAR

16 / 27

slide-17
SLIDE 17

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Colliding Picture

◮ Possible Direction ◮ Maybe complete Wrong ◮ adding LiDAR Data

  • L. Wendt – Fusion Camera/LiDAR

17 / 27

slide-18
SLIDE 18

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Colliding Picture

◮ represented with a color transition

◮ green = 0 Speed, blue = negative Speed , red = positive Speed

◮ speed on Axis ◮ Movement between object

  • L. Wendt – Fusion Camera/LiDAR

18 / 27

slide-19
SLIDE 19

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ trigonometric functions ◮ Arithmetic

  • L. Wendt – Fusion Camera/LiDAR

19 / 27

slide-20
SLIDE 20

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Angle calculation Sphere

  • L. Wendt – Fusion Camera/LiDAR

20 / 27

slide-21
SLIDE 21

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Angle calculation Sphere Second

  • L. Wendt – Fusion Camera/LiDAR

21 / 27

slide-22
SLIDE 22

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Calcualte Points in Cordinate System

  • L. Wendt – Fusion Camera/LiDAR

22 / 27

slide-23
SLIDE 23

Movement (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Plot in a Grid ◮ (0/0) is the Lidar/Camera Position

Movement between Picture 1 and Picture 2

  • L. Wendt – Fusion Camera/LiDAR

23 / 27

slide-24
SLIDE 24

Conclusion

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ More complete picture of the environment ◮ Problems with different Frequencies, Resolutions .. ◮ LiDARs are expensive ◮ We can build a 3D environment ◮ Possibly Something data lost ◮ Accurate the Interpolation with higher Polynoms ◮ Tests with a real LiDAR ◮ exact Positions of detected Object can calculate ◮ calculation are Simple trigonometric and normal arithmetic

  • L. Wendt – Fusion Camera/LiDAR

24 / 27

slide-25
SLIDE 25

Questions

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

◮ Thanks for your attention ◮ Questions ?

  • L. Wendt – Fusion Camera/LiDAR

25 / 27

slide-26
SLIDE 26

References

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Cho, H. u. a. (2014). “A multi-sensor fusion system for moving

  • bject detection and tracking in urban driving environments”.

In: 2014 IEEE International Conference on Robotics and Automation (ICRA), S. 1836–1843. doi: 10.1109/ICRA.2014.6907100. Elmenreich, Wilfried (2001). An Introduction to Sensor Fusion. Research Report 47/2001. Treitlstr. 1-3/182-1, 1040 Vienna, Austria: Technische Universität Wien, Institut für Technische

  • Informatik. (Besucht am 06. 11. 2018).
  • uster (2018a). https://www.ouster.io/. (Besucht am
  • 06. 11. 2018).

– (2018b). https://www.youtube.com/watch?v=X7TljH5x2kE. (Besucht am 06. 11. 2018).

  • L. Wendt – Fusion Camera/LiDAR

26 / 27

slide-27
SLIDE 27

References (cont.)

Introduction Sensor Fusion Motion Conclusion & Future Works References Literatur

Silva, Varuna De, Jamie Roche und Ahmet M. Kondoz (2018a). “Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots”. In: Sensors. (Besucht am

  • 06. 11. 2018).

– (2018b). “Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots”. In: Sensors. (Besucht am 06. 11. 2018).

  • L. Wendt – Fusion Camera/LiDAR

27 / 27