StructSLAM: Visual SLAM with Building Structure Lines Danping Zou - - PowerPoint PPT Presentation

▶
structslam visual slam with
SMART_READER_LITE
LIVE PREVIEW

StructSLAM: Visual SLAM with Building Structure Lines Danping Zou - - PowerPoint PPT Presentation

Sep 10, 2023 290 likes •780 views

StructSLAM: Visual SLAM with Building Structure Lines Danping Zou Assistant Professor Key Laboratory of Navigation and Location-based Services Shanghai Jiao Tong University dpzou@sjtu.edu.cn http://drone.sjtu.edu.cn/dpzou Shanghai Jiao Tong

slide-1
SLIDE 1

StructSLAM: Visual SLAM with Building Structure Lines

Danping Zou Assistant Professor Key Laboratory of Navigation and Location-based Services Shanghai Jiao Tong University dpzou@sjtu.edu.cn http://drone.sjtu.edu.cn/dpzou

slide-2
SLIDE 2

Shanghai Jiao Tong University

  • 2 -

2016/3/30

What’s SLAM?

  • SLAM :Simultaneous Localization And Mapping.
  • Autonomous navigation
  • motion planning

Constructing a map of an unknown environment while simultaneously keeping track of the robot’s location.

slide-3
SLIDE 3

Shanghai Jiao Tong University

  • 3 -

Active areas for SLAM research

Robotics

  • UAVs， Service robots

2016/3/30

Augmented Reality (AR)

slide-4
SLIDE 4

Shanghai Jiao Tong University

  • 4 -

2016/3/30

SLAM with different kinds of sensors

  • Active sensing
  • Heavy and bulky
  • Active sensing
  • Energy consuming

2D laser rangefinder RGB-D camera 3D liDar

  • Passive sensing
  • Light & compact
  • Energy saving
  • Ubiquity

Since 2005, there has been intense research into VSLAM (Visual SLAM) using primarily visual (camera) sensors.

slide-5
SLIDE 5

Shanghai Jiao Tong University

  • 5 -

List of Visual SLAM methods

MonoSLAM FastSLAM GraphSLAM PTAM ORB-SLAM CoSLAM* DTAM LSD-SLAM

2016/3/30

Extended Kalman filter Structure-from-motion Dense / Semi dense Sparse feature points *Zou, Danping, and Ping Tan. "CoSLAM: Collaborative visual slam in dynamic environments." Pattern Analysis and Machine Intelligence, IEEE Transactions on 35.2 (2013): 354-366.

slide-6
SLIDE 6

Shanghai Jiao Tong University

  • 6 -

Knowing issues for Visual SLAM

Scale ambiguity Lack of Illumination Accumulated error (Drift error) Lack of Texture

2016/3/30

Vision +Ｘ

GNSS（GPS、Beidou） Inertial Units（ Gyro+Acc+Compass） Wireless（Wifi，Bluetooth） Map / Floor plan

slide-7
SLIDE 7

Shanghai Jiao Tong University

  • 7 -

What about if we consider one special case : indoor scenes

2016/3/30

Natural scenes Man-made scenes

slide-8
SLIDE 8

Shanghai Jiao Tong University

  • 8 -

StructSLAM

A novel visual SLAM method aided with structure lines

  • Structure lines : The line feature aligned with dominant orientations

2016/3/30

Zhou, Huizhong, Danping, Zou, et al. "StructSLAM: Visual SLAM with building structure lines." Vehicular Technology, IEEE Transactions on 64.4 (2015): 1364-1375.

  • Special session for indoor localization
slide-9
SLIDE 9

Shanghai Jiao Tong University

  • 9 -

Indoor scenes

Motivations:

  • Lines are better landmarks in texture-less scenes (like many indoor

scenes with only white walls) than points.

  • Some lines (structure lines) encode the global orientation information.

2016/3/30

Less accumulated error

slide-10
SLIDE 10

Shanghai Jiao Tong University

  • 10 -

2016/3/30

http://drone.sjtu.edu.cn/dpzou/project/structslam.php

slide-11
SLIDE 11

Shanghai Jiao Tong University

  • 11 -

EKF pipeline Feature management

Pipeline of StructSLAM

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

Extended Kalman filter:

  • Simple to be implemented
  • Easily fused with other

sensors (IMU, odometers)

  • More robust than structure-

from-motion pipeline.

slide-12
SLIDE 12

Shanghai Jiao Tong University

  • 12 -

Dominant directions

The man-made world are generally dominated by several directions.

  • Vertical direction (always points to the

sky)

  • Horizontal directions (are usually

perpendicular to each other, although not always)

2016/3/30

slide-13
SLIDE 13

Shanghai Jiao Tong University

  • 13 -

Parameter planes

Parameter plane is one of xy,yz,xz planes of the world frame. A structure line is represented by a point on the parameter plane. The parameter plane is selected so as to make sure it is the most perpendicular to the dominant direction.

2016/3/30

Parameter plane Structure line x y z

slide-14
SLIDE 14

Shanghai Jiao Tong University

  • 14 -

Structure line

Each structure line is represented by a point

  • n the parameter plane, denoted by a 4x1

vector. It is in fact a 2D inverse depth representation*

2016/3/30

Projection of camera center Direction Inverse depth

* Montiel, J. M. M., Javier Civera, and Andrew J. Davison. "Unified inverse depth parametrization for monocular SLAM." analysis 9 (2006): 1.

slide-15
SLIDE 15

Shanghai Jiao Tong University

  • 15 -

StructSLAM – State representation

State vector and covariance matrix (MonoSLAM)

2016/3/30

Camera pose + Points + Structure Lines

slide-16
SLIDE 16

Shanghai Jiao Tong University

  • 16 -

Initialization

Step 1:

  • Use LSD line detector* to detect line segments
  • n the image.

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

* Von Gioi, Rafael Grompone, et al. "LSD: A fast line segment detector with a false detection control." IEEE Transactions on Pattern Analysis & Machine Intelligence 4 (2008): 722-732.

slide-17
SLIDE 17

Shanghai Jiao Tong University

  • 17 -

Initialization

Step 2:

  • Apply J-linkage* to classify parallel line segments into

groups and detect vanishing points

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features *Toldo, Roberto, and Andrea Fusiello. "Robust multiple structures estimation with j-linkage." Computer Vision– ECCV 2008. Springer Berlin Heidelberg, 2008. 537-547.

slide-18
SLIDE 18

Shanghai Jiao Tong University

  • 18 -

Initialization

Step 3:

  • Estimate the dominant direction from the

vanishing points.

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

  • Dominant direction
  • Vanishing point

: Camera intrinsic : Rotation from the world frame to the camera frame

slide-19
SLIDE 19

Shanghai Jiao Tong University

  • 19 -

Initialization

Step 4:

  • Refine the dominant directions by non-linear least square optimization
  • Initialize new lines (See the feature management section)

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

slide-20
SLIDE 20

Shanghai Jiao Tong University

  • 20 -

State prediction

Camera – constant velocity or odometer data (if available) Landmarks – static

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

Odometer velocity

slide-21
SLIDE 21

Shanghai Jiao Tong University

  • 21 -

State prediction

Covariance propagation

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

slide-22
SLIDE 22

Shanghai Jiao Tong University

  • 22 -

Data association

Find the line segments corresponding to the structure line

2016/3/30

Structure line

slide-23
SLIDE 23

Shanghai Jiao Tong University

  • 23 -

Data association

Find the line segments corresponding to the structure line

2016/3/30

Structure line

  • One-to-multiple matching
  • There could be false matchings (outliers).
slide-24
SLIDE 24

Shanghai Jiao Tong University

  • 24 -

Data association

Step1 : Get candidate matching by

2016/3/30

: residual vector :Jacobian of observation function

slide-25
SLIDE 25

Shanghai Jiao Tong University

  • 25 -

Data association

Step 2: Comparing appearance by ZNCC (zero mean normalized cross-correlation)

2016/3/30

ZNCC >０.8 Previous frame Current frame

slide-26
SLIDE 26

Shanghai Jiao Tong University

  • 26 -

Data association

Step3 : One-feature RANSAC to eliminate false matchings (outliers):

  • Randomly sample a candidate matching
  • Run a tentative EKF update using the sampled matching and check the

number of inliers

  • Keep the inlier set with the maximum number
  • Repeat the above steps
  • Use the inlier set to run EKF update.

2016/3/30

slide-27
SLIDE 27

Shanghai Jiao Tong University

  • 27 -

State update

Observation model:

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

Structure line

slide-28
SLIDE 28

Shanghai Jiao Tong University

  • 28 -

State update

Project a structure line onto the image

2016/3/30

Structure line World frame Camera frame Image Connect with vanishing point Projection Vanishing point Dominant direction

slide-29
SLIDE 29

Shanghai Jiao Tong University

  • 29 -

State update

Observation model

  • Observation function

(measurement function):

  • Since the desired distance

is zero, the residual is computed as :

2016/3/30

Large residual Small residual

slide-30
SLIDE 30

Shanghai Jiao Tong University

  • 30 -

State update

Standard Extended Kalman Filter:

2016/3/30

:Jacobian of observation function Predicted state and covariance: : Observation noise - Uncertainty of detected line segments

slide-31
SLIDE 31

Shanghai Jiao Tong University

  • 31 -

State update

Multiple-pass EKF update (Point + Structure lines)

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features State update using point observation State update using structure line observation

slide-32
SLIDE 32

Shanghai Jiao Tong University

  • 32 -

Feature management

Initialize new structure lines

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

:Dominant direction Line though the point: Intersection with parameter plane: Expressed in parameter plane (xy-plane): Point in world frame:

slide-33
SLIDE 33

Shanghai Jiao Tong University

  • 33 -

Feature management

Initialize new structure lines

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

slide-34
SLIDE 34

Shanghai Jiao Tong University

  • 34 -

Feature management

Initialize new structure lines

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

slide-35
SLIDE 35

Shanghai Jiao Tong University

  • 35 -

Feature management

The number of features is limited in the state. For each dominant direction, we keep a maximum number of structure lines. Old features are removed according to the number of matching failure (NOF)

2016/3/30

Initialization Data Association Initialize new features State prediction State update Remove old features

slide-36
SLIDE 36

Shanghai Jiao Tong University

  • 36 -

Results

Simulated case

2016/3/30

slide-37
SLIDE 37

Shanghai Jiao Tong University

  • 37 -

Results

Simulated case

2016/3/30

Lines Structure lines

Lines V.S. structure lines

slide-38
SLIDE 38

Shanghai Jiao Tong University

  • 38 -

Results

Real-world case (Using one video camera)

  • Indoor texture-less scenes

2016/3/30

slide-39
SLIDE 39

Shanghai Jiao Tong University

  • 39 -

Results of real-word indoor scenes

Real world case (Using one video camera)

  • Outdoor texture rich scenes

2016/3/30

slide-40
SLIDE 40

Shanghai Jiao Tong University

  • 40 -

Benchmark datasets

Rawseeds datasets (all methods fused the odometer information)

  • Bicacco-02-25b : A 774m trajectory in the indoor scene

2016/3/30

slide-41
SLIDE 41

Shanghai Jiao Tong University

  • 41 -

Benchmark results

2016/3/30

slide-42
SLIDE 42

Shanghai Jiao Tong University

  • 42 -

Benchmark datasets

Bicacco-02-27a :

2016/3/30

slide-43
SLIDE 43

Shanghai Jiao Tong University

  • 43 -

Benchmark datasets

Comparision

2016/3/30

StructSLAM : Without any loop-closing algorithms being applied!

slide-44
SLIDE 44

Shanghai Jiao Tong University

  • 44 -

Running time efficiency

A common PC with i7 4-core cpu (2.7GHz) (c++ implementation)

2016/3/30

Average running time: 25.8 ms Peak running time: 62.9 ms

slide-45
SLIDE 45

Shanghai Jiao Tong University

  • 45 -

Conclusion & Discussion

StructSLAM is more robust in texture-less indoor scenes than conventional SLAM methods With global orientation information encoded in structure lines, StructSLAM produces much less drift error. It is well fit for robotic and augmented reality (AR) applications in indoor scenes.

2016/3/30

slide-46
SLIDE 46

Shanghai Jiao Tong University

  • 46 -

Conclusion & Discussion

There are also some problems:

  • Line are not distinguishable as points (ZNCC does not work well for large

view angle changes)

  • Dominant directions should be captured in the image at initialization

stage.

  • Dominant directions is fixed after initialization

2016/3/30

slide-47
SLIDE 47

Shanghai Jiao Tong University

  • 47 -

The Third UAV competition in SJTU

AI technology (computer vision, learning, autonomous exploration) on micro drones

2016/3/30

Coming competition (New URL) : http://drone.sjtu.edu.cn Past competition (Old URL) : http://mediosoc.sjtu.edu.cn/wordpress

slide-48
SLIDE 48

Shanghai Jiao Tong University

  • 48 -

2016/3/30

Thanks!