Fast field survey with a smartphone A. Masiero F. Fissore, F. - - PowerPoint PPT Presentation

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy 1

Fast field survey with a smartphone

• A. Masiero
• F. Fissore, F. Pirotti, A. Guarnieri, A. Vettore

CIRGEO – Interdept. Research Center of Geomatics University of Padova – Italy cirgeo@unipd.it

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

2

2

Use of embedded sensors:

• Camera is used as imaging sensor 3D reconstruction of

the observed environment via photogrammetry (e.g. SfM)

• Device position estimated by integrating information

provided by the embedded sensors spatial referring

• Low cost, fast w.r. to other techniques (e.g. TLS)
• Limited resources: stringent restrictions on the

computational power, limited battery life...

• Goal: exploit information provided by the navigation

system to improve the reconstruction procedure

Introduction

Mobile Mapping with Smartphones

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

3

3

Positioning achieved by integrating information:

• GNSS
• inertial sensors (embedded in the device, they

provide good local estimates of position variations but drift in long time intervals if used alone)

• WiFi signal strength
• Barometer
• Geometry of the environment

Nonlinear filtering

Positioning

Navigation system

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

4

4

Information fusion of PDR (Pedestrian Dead Reckoning), WiFi, building map... Particle filtering

• Device position is expressed as average position of N

particles

• Dynamic equation of each particle:

it exploits measured step length and heading direction

qt+i=qt+st[ sinαt cosαt]

Positioning

u v

Particle filtering

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

5

5

• Advantage: simple to introduce non-linear constraints (and to

deal with multiple hypothesis) in position estimation

• High accuracy for large N, but computational burden issues!
• [Masiero 2014] proposed a revised version of [Widyawan

2012] in order to increase accuracy for small N (N≈100) and uncalibrated sensors

• For further accuracy improvement:
• good sensor calibration
• exploiting landmarks

Positioning

Neglegted, and resampled

Particle filtering

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

6

6

• Information fusion of PDR (Pedestrian Dead Reckoning), WiFi,

building map... Particle filtering

• [Widyawan 2012]: Particle filter for PDR
• [Masiero 2014]: revised version of the particle filter in

[Widyawan 2012] in order to increase accuracy for small N (number of particles N≈100) and uncalibrated sensors

• Magnetometer & accelerometer simultaneous calibration

[Masiero MMT2015]

• Barometer altitude variation
• linear model to describe the relation between pressure

and altitude variations (precision ≈ 0.2m).

Positioning

Particle filtering

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

7

7

Reconstruction outline

• Compute feature locations (e.g. Harris feature detector)
• Compute feature descriptors (e.g. SIFT)
• Feature matching (Best Bin First – Kd tree search)
• Remove outliers (epipolar geometry, RANSAC or its variants)
• Bundle adjustment (optimize parameter values)

Projective reconstruction Control points are used to obtain Euclidean reconstruction and for georeferencing

3D reconstruction

3D photogrammetric reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

8

8

Reconstruction outline

• Compute feature locations (e.g. Harris feature detector)
• Compute feature descriptors
• Feature matching (Best Bin First – Kd tree search)
• Remove outliers (epipolar geometry, RANSAC or its variants)
• Bundle adjustment (optimize parameter values)

Projective reconstruction Control points are used to obtain Euclidean reconstruction and for georeferencing

3D reconstruction

3D photogrammetric reconstruction

take into account of affine transformations

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

9

9

Feature matching

• Typically done by using SIFT (Scale-invariant feature

transform, [Lowe 1999]) matchings [Vedaldi 2008]

• SIFT deals well with rotations with respect to rotations

along the optical axis

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

10

10

Feature matching

• However, issues can occur when considering other

rotations (as typical with generic changes of the point of view)

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

11

11

• ASIFT [Morel 2011] increases SIFT robustness with

respect to such rotations by modelling their effect by means of affine transformations.

• However, in ASIFT 32 affine transformations of each

feature are computed 322≈1000 comparisons between each couple of features in two different images.

• Goal: reducing computational complexity of ASIFT while

ensuring increase of matchings with respect to SIFT in the critical cases (e.g. previously described changes of the point of view...)

Feature matching

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

12

12

Feature matching

• Appearance of a feature seen by camera j depends on the

point of view and on the “spatial orientation” of the feature

• Information by the navigation system change of the point
• f view transformation (translation + rotation) approximately

known

• Uncertainty in the “spatial orientation” of the feature

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

13

13

Image plane Image plane

Feature matching

Surface of the real object

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

14

14

• Appearance of a feature seen by camera j depends on the

point of view and on the “spatial orientation” of the feature

• Information by the navigation system change of the point
• f view transformation (translation + rotation) approximately

known

• Uncertainty in the “spatial orientation” of the feature
• Compensate for this uncertainty by simulating the effect of 20

possible orientations (on a semi-sphere...)

• Thanks to information provided by the navigation system:
• ASIFT: 322≈1000 comparisons (per feature couple)
• Our approach: 20 comparisons (per feature couple)

Feature matching

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

15

15

Matches with SIFT

Images of this example available from the internet [Lhuillier and Quan, 2005]

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

16

16

Matches with the proposed method

Images of this example available from the internet [Lhuillier and Quan, 2005]

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

17

17

Number of correct matches vs (difference of) observation angle

SIFT: Blue x-marks Our approach: red circles

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

18

18

Reconstruction outline

• Compute feature locations (e.g. Harris feature detector)
• Compute feature descriptors
• Feature matching
• Remove outliers (epipolar geometry, RANSAC or its variants)
• Bundle adjustment (optimize parameter values)

Projective reconstruction Control points are used to obtain Euclidean reconstruction and for georeferencing

3D reconstruction

3D photogrammetric reconstruction

take into account of affine transformations Use approximate epipolar constraints to discard false matchings

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

19

19

Reconstruction outline

• Compute feature locations (e.g. Harris feature detector)
• Compute feature descriptors
• Feature matching
• Remove outliers (epipolar geometry, RANSAC or its variants)
• Bundle adjustment (optimize parameter values)

Control points

3D reconstruction

3D photogrammetric reconstruction

take into account of affine transformations Use approximate epipolar constraints to discard false matchings Use positions provided by the navigation system and calibrated camera to ease/improve Euclidian reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

20

20

• Hol, 2011. Sensor Fusion and Calibration of Inertial Sensors, Vision, Ultra-Wideband and GPS.
• PhD. Thesis, Linkoping University, The Institute of Technology.
• Masiero et al, 2014. A particle filter for smartphone-based indoor pedestrian navigation.

Micromachines 5(4), pp. 1012–1033

• Masiero and Vettore, 2015. Towards mobile mapping with smartphones. MMT 2015.
• Morel and Yu, 2011. Is SIFT scale invariant? Inverse Problems and Imaging 5(1), pp. 115–136.
• Lhuillier and Quan, 2005. A quasi-dense approach to surface reconstruction from uncalibrated
• images. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27(3), 418-433.
• Liu et al, 2014. Novel calibration algorithm for a three-axis strapdown magnetometer. Sensors

14(5), pp. 8485–8504.

• Lowe, 1999. Object recognition from local scale-invariant features. Proceedings of the

Seventh IEEE International Conference on Computer Vision (ICCV), Vol. 2, pp. 1150–1157 vol.2.

• Vedaldi and Fulkerson, 2008. VLFeat: An open and portable library of computer vision

algorithms.

• Widyawan et al, 2012. Virtual lifeline: Multimodal sensor data fusion for robust navigation in

unknown environments. Pervasive and Mobile Computing 8(3), pp. 388–401.

Bibliography

Conclusions

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

21

21

3D reconstruction

1) Estimation of the structure of the scene: camera positions and 3D positions of certain features 1a) More robust estimation results by exploiting “a priori” information provided by the navigation system

• n camera positions (and orientations)

1b) Feature matching for estimating geometry of the scene: feature matching issues when points of view are quite different. Goal: provides more robust feature matching (and more feature matches) 2) Dense reconstruction: (usually) greed algorithm for increasing the point cloud based on local matches

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

22

22

Selection of orientation based on information by the navigation system

3D reconstruction

Introduction Positioning 3D reconstruction Conclusions Masiero – Fast field survey with a smartphone – 19 Feb 2016, Trieste, Italy

23

23 Introduction Calibration with bundle adjustment Convex optimization methods Conclusions

3D reconstruction based on photogrammetry

• Feature extraction
• Feature matching
• Reconstruction via triangulation (Bundle adjustment)

From CRCSI webpage