See Through Smoke : Robust Indoor Mapping with Low-cost mmWave Radar - - PowerPoint PPT Presentation

▶
see through smoke robust indoor mapping with low cost
SMART_READER_LITE
LIVE PREVIEW

See Through Smoke : Robust Indoor Mapping with Low-cost mmWave Radar - - PowerPoint PPT Presentation

Nov 11, 2022 239 likes •667 views

Cyber Pysical Systems Group See Through Smoke : Robust Indoor Mapping with Low-cost mmWave Radar (Chris) Xiaoxuan Lu * , Stefano Rosa * , Peijun Zhao * , Bing Wang * , Changhao Chen * , John.A.Stankovic + , Niki Trigoni * , Andrew Markham * *

slide-1
SLIDE 1

ACM MobiSys 2020 Toronto, Canada

1

See Through Smoke: Robust Indoor Mapping with Low-cost mmWave Radar

1

(Chris) Xiaoxuan Lu*, Stefano Rosa*, Peijun Zhao*, Bing Wang*, Changhao Chen*, John.A.Stankovic+, Niki Trigoni*, Andrew Markham*

*University of Oxford, UK

+University of Virginia, USA

Cyber Pysical Systems Group

slide-2
SLIDE 2

ACM MobiSys 2020 Toronto, Canada

2

Motivation

slide-3
SLIDE 3

ACM MobiSys 2020 Toronto, Canada

3

Spatial Awareness

✧ Why - lack of spatial awareness

  • Spatial awareness: An as-comprehensive-as-possible map
  • How about: employ a mobile robot to fast map the env. first?

‘Blind’: Is it an exit?? Use HANDs to perceive obstacles Mobile robot to survey first!

slide-4
SLIDE 4

ACM MobiSys 2020 Toronto, Canada

4

✧ Optical Sensors cannot see

through airborne particles

  • RGB camera
  • Depth Imaging
  • Lidar

Limitation of traditional sensors

Smoke Dust Fog

slide-5
SLIDE 5

ACM MobiSys 2020 Toronto, Canada

5

Single-Chip CMOS mmWave radar

Automobile Manufactory

slide-6
SLIDE 6

ACM MobiSys 2020 Toronto, Canada

6

Primer

✧ Working Principles

  • A transceiver device operating

in the spectrum between 30 GHz - 300 GHz

  • Use a frequency modulated

continuous wave (FMCW) approach to measure object distance and orientation

Object mmWave Radar mmWave Radar Object

slide-7
SLIDE 7

ACM MobiSys 2020 Toronto, Canada

7

Primer

✧ Pros

  • Sub-mm range accuracy
  • Impervious to

environmental conditions, e.g., fog, smoke dust …

  • Small footprint
  • Cheap
slide-8
SLIDE 8

ACM MobiSys 2020 Toronto, Canada

8

milliMap

Use a mobile-mounted single-chip mmWave radar for metric and semantic indoor mapping

slide-9
SLIDE 9

ACM MobiSys 2020 Toronto, Canada

9

✧ Very sparse point cloud

  • Fundamental specularity of

mmWave signals

  • 4 x 3 antennas for cost

reason

  • CFAR (Constant False Alarm

Rate) on-chip pre-processing

Challenge I

< 100 points per scan, 100-fold sparser than a lidar

slide-10
SLIDE 10

ACM MobiSys 2020 Toronto, Canada

10

Multi-path noise

  • Reflected signals arriving at a receiver antenna can from two or more paths
  • Leading to `ghost points' in a mmWave point cloud

Challenge II

Points outside the black line (i.e. walls) are ghost points

slide-11
SLIDE 11

ACM MobiSys 2020 Toronto, Canada

11

Formulate metric mapping to a reconstruction problem

  • Basic Model: conditional Generative Adversarial Network (cGAN)
  • Cross-modal Supervision: a co-located lidar providing labels for mmWave

Metric Map Reconstruction

Lidar works fine in benigh (non-smoke) situations!

slide-12
SLIDE 12

ACM MobiSys 2020 Toronto, Canada

12

Formulate metric mapping to a reconstruction problem

  • Basic Model: conditional Generative Adversarial Network (cGAN)
  • Online inference: independently predict a good map without the help of lidar

Metric Map Reconstruction

Independently work for online inference!

slide-13
SLIDE 13

ACM MobiSys 2020 Toronto, Canada

13

✧ Network Input

➡ Single-frame SCAN? ➡ Too much information absence ➡ Probably Overfitting if learn in

brute force

Metric Map Reconstruction

Single mmWave scan misses lots of information! Almost learn sth. from nothing…

Lidar Scan mmWave Scan

slide-14
SLIDE 14

ACM MobiSys 2020 Toronto, Canada

14

✧ Network Input

✓ Stitch scans into a patch

assisted by odometry

✓ Use patches as inputs ✓ KEY: Odometry drifts in

short-term is negligible

Metric Map Reconstruction

mmWave Patch lidar patch Series of Scans Odom

slide-15
SLIDE 15

ACM MobiSys 2020 Toronto, Canada

15

✧ Network Input

✓ Stitch scans into a patch

assisted by odometry

✓ Use patches as inputs ✓ KEY: Odometry drifts in

short-term is negligible

Metric Map Reconstruction

What loss?

slide-16
SLIDE 16

ACM MobiSys 2020 Toronto, Canada

16

✧ Bayes Perspective

✓ Goal: maximize posterior ✓ Likelihood (e.g., Pix2PixHD)

  • cGAN loss (appearance)
  • Feature Matching
  • Low-level geometry

✓ Prior ➡ Map of structure (lines etc.)

Metric Map Reconstruction

likelihood prior

slide-17
SLIDE 17

ACM MobiSys 2020 Toronto, Canada

17

✧ Map Prior (MP) Loss

✓ ‘Manhattan World’ Model ✓ Geometric regularities in indoor

environment, e.g., following rectilinear outlines

Metric Map Reconstruction

Indoor Floor Plan

slide-18
SLIDE 18

ACM MobiSys 2020 Toronto, Canada

18

✧ Map Prior (MP) Loss

✓ ‘Manhattan World’ Constraints ✓ Geometric regularities in indoor

environment, e.g., following rectilinear outlines

✓ Realised through shape detector

  • conv. masks (e.g., line detector)

Metric Map Reconstruction

Four line detection kernels which respond maximally to horizontal, vertical and oblique

slide-19
SLIDE 19

ACM MobiSys 2020 Toronto, Canada

19

milliMap

Use a mobile-mounted single-chip mmWave radar for metric and semantic indoor mapping

slide-20
SLIDE 20

ACM MobiSys 2020 Toronto, Canada

20

4 Key Access Objects (AO)

Horizontal AO - Door Vertical AO - Lift Alternative AO - Window Non AO - Wall

slide-21
SLIDE 21

ACM MobiSys 2020 Toronto, Canada

21

Complex interior construction objects

  • Indoor construction objects are made by different layers of materials

➡ Multiple reflections from internal layers, diffusion of mmWave on rough surfaces

Challenge III

Interior Wall made by multiple layers Complicated multi-path effects

Hard to Model!

slide-22
SLIDE 22

ACM MobiSys 2020 Toronto, Canada

22

Key Observation

  • Range FFT profile can capture the object-related mmWave propagation patterns
  • A Segment of Interest (SOI) is decided by profile peak point and its neighbours

Semantic Recogniser

Robot rotates get a perpendicular

  • bs. angle for mmWave radar
slide-23
SLIDE 23

ACM MobiSys 2020 Toronto, Canada

23

In a example of 5-point SOI feature

  • ‘Average’ SOI of three key objects aggregated from 27, 952 training samples
  • Distinct shape patterns observed for different objects

Semantic Recogniser

SOI - Door SOI - Lift SOI - Glass

slide-24
SLIDE 24

ACM MobiSys 2020 Toronto, Canada

24

Semantic Mapping

  • Employ a NN classifier to output softmax probability logit
  • Use the probability distribution to determine semantic label and alien obj.

Semantic Recogniser

slide-25
SLIDE 25

ACM MobiSys 2020 Toronto, Canada

25

✧ Multi-modal Robotic

Sensing Platform

  • OS: ROS Melodic
  • Robot Platform: Turtle Bot 2
  • mmWave radar: TI AWR1443
  • Lidar: Velodyne VLP-16
  • Odometry provision: wheel
  • dom + XSENS IMU

Implementation

Lidar Turtlebot 2 mmWave radar XSENS IMU

slide-26
SLIDE 26

ACM MobiSys 2020 Toronto, Canada

26

Experiment Sites

  • Training: 1st, 2nd and 3rd floor of Building A

Evaluation

1st floor, Building A 2nd floor, Building A 3rd floor, Building A

slide-27
SLIDE 27

ACM MobiSys 2020 Toronto, Canada

27

Experiment Sites

  • Cross-site Test: 4th floor of Building A, Building B and smoke-filled arcade

Evaluation

4th floor, Building A Building B Arcade (smoke)

slide-28
SLIDE 28

ACM MobiSys 2020 Toronto, Canada

28

✧ Metrics

✓ Mean absolute error (L1) ✓ Mean Intersection of Union

(IoU)

✓ NOTE: sometimes, manual

qualitative inspection is also needed in our context

Evaluation

IoU = |X ∩ Y| |X ∪ Y|

L1 = 1 N ∑

p∈P

|X(p) − Y(p)|

X - prediction Y - truth p - pixel index

slide-29
SLIDE 29

ACM MobiSys 2020 Toronto, Canada

29

✧ Order of Densification

➡ Fix the model by using two

established baseline generators

✓ Stitching-first consistently

yields smaller mean L1

Evaluation

1 2 3 4 Cross-floor Cross-Build.

w.o. stitch stitch

1 2 3 4 Cross-floor Cross-Build.

w.o. stitch stitch

Pix2Pix Pix2PixHD

slide-30
SLIDE 30

ACM MobiSys 2020 Toronto, Canada

30

✧ Order of Densification

➡ Fix the model by using two

established baseline generators

✓ Stitching-first consistently

yields larger mean IoU

Evaluation

0.1 0.2 0.3 0.4 Cross-floor Cross-Build.

w.o. stitch stitch

0.1 0.2 0.3 0.4 Cross-floor Cross-Build.

w.o. stitch stitch

Pix2Pix Pix2PixHD

slide-31
SLIDE 31

ACM MobiSys 2020 Toronto, Canada

31

Method comparison

  • Outperform 5 grid map reconstruction methods in both L1 and IoU

Evaluation

2 4 6 Cross-floor Cross-Build.

Linefitting CVAE BiCycGAN Pix2Pix Pix2PixHD Ours

IoU

0.0 0.2 0.3 0.5 Cross-floor Cross-Build.

Linefitting CVAE BiCycGAN Pix2Pix Pix2PixHD Ours

Mean L1 Mean IoU

slide-32
SLIDE 32

ACM MobiSys 2020 Toronto, Canada

32

Method comparison

  • ‘Ghost’ area mis-generated.

Evaluation

slide-33
SLIDE 33

ACM MobiSys 2020 Toronto, Canada

33

Where is `ghost' area coming from?

  • Incorrect lidar supervision due to presence of glass objects in training data

Evaluation

slide-34
SLIDE 34

ACM MobiSys 2020 Toronto, Canada

34

Effectiveness for downstream navigation

  • Cross-building: 0.285 m trans. error and 0.142 rad orientation error
  • Cross-building: 0.178 m trans. error and 0.140 rad orientation error

Evaluation

Translation Orientation

slide-35
SLIDE 35

ACM MobiSys 2020 Toronto, Canada

35

Generalise to handheld case

  • Imperfect yet odom i can largely recover the basic shape

Evaluation

raw mmWave Generated

slide-36
SLIDE 36

ACM MobiSys 2020 Toronto, Canada

36

Semantic Recognition

  • Over 0.9 F1 score for cross-floor, ~ 0.88 F1 score for cross-building
  • ‘Alien’ objects outside target classes

Evaluation

Cross-floor Cross-build.

slide-37
SLIDE 37

ACM MobiSys 2020 Toronto, Canada

37

Semantic Recognition

  • Impact of SOI length: 6 points (~20cm) yields best performance

Evaluation

Cross-floor Cross-build.

slide-38
SLIDE 38

ACM MobiSys 2020 Toronto, Canada

38

Demo

slide-39
SLIDE 39

ACM MobiSys 2020 Toronto, Canada

39

Smoke-filled Test

slide-40
SLIDE 40

ACM MobiSys 2020 Toronto, Canada

40

✧ Limitation & Next

  • More diverse and different

places for testing

  • Odometry drifts in long run
  • > mmWave Odom (our

milliEgo on arxiv already)

  • Aerial drones
  • Real disaster situation

Conclusion

slide-41
SLIDE 41

ACM MobiSys 2020 Toronto, Canada

41

Chris Xiaoxuan Lu (milliMap is open-sourced) https://christopherlu.github.io/

Thanks!