Impiego di 3DF Zephyr nel rilievo da APR Andrea Fusiello - - PowerPoint PPT Presentation

▶
impiego di 3df zephyr nel rilievo da apr
SMART_READER_LITE
LIVE PREVIEW

Impiego di 3DF Zephyr nel rilievo da APR Andrea Fusiello - - PowerPoint PPT Presentation

Jan 09, 2023 242 likes •715 views

Impiego di 3DF Zephyr nel rilievo da APR Andrea Fusiello (andrea.fusiello@uniud.it) -- DIEGM University of Udine Roberto Toldo, Filippo Fantini, Simone Fantoni, Luca Giona -- 3Dflow srl www.3dflow.net 3DF Zephyr Software per la ricostruzione

slide-1
SLIDE 1

Impiego di 3DF Zephyr nel rilievo da APR

Andrea Fusiello (andrea.fusiello@uniud.it) -- DIEGM University of Udine Roberto Toldo, Filippo Fantini, Simone Fantoni, Luca Giona -- 3Dflow srl

slide-2
SLIDE 2

3DF Zephyr

Software per la ricostruzione 3D da fotografie Sviluppato da 3Dflow srl spinoff UniUD (ex UniVR) Gestisce blocchi non strutturati Gestisce camere eterogenee con calibrazione ignota Adatto sia per la ricostruzione di piccoli oggetti che per rilievi fotogrammetrici

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

www.3dflow.net

slide-3
SLIDE 3

Image acquisition

3DF Zephyr pipeline

3d photo-consistency from images 3d surface from 3d photo-consistency Image

  • rientation

3DF Samantha (structure and motion) 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006]

Images from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-4
SLIDE 4

Image acquisition

3DF Zephyr pipeline

3d photo-consistency from images 3d surface from 3d photo-consistency Image

  • rientation

3DF Samantha (structure from motion) 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006]

Images from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-5
SLIDE 5

Structure from motion

Credits to R. Toldo (3Dflow s.r.l.) , R. Gherardi and M. Farenzena (U. Verona)

slide-6
SLIDE 6

2-­‑d ¡%e-­‑points ¡ matching ¡

  • bject ¡3-­‑d ¡points ¡

Structure from motion

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-7
SLIDE 7

Structure from motion

Motion = exterior orientation

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-8
SLIDE 8

Orientation of a block

¤ Block: set of overlapping photographs ¤ Independent models block adjustment

¤ Create independent stereo-models with relative orientation for each pair of overlapping photographs ¤ Simultaneous transform the models into the ground coordinate system with absolute orientation (given known GCP).

¤ Bundle block adjustment

¤ Compute directly the relations between image coordinates and object coordinates, without introducing model coordinates as intermediate step.

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-9
SLIDE 9

Bundle block adjustment

  • UAV / RPAS in Italia - Modena 20-21 Febbraio 2014
slide-10
SLIDE 10

Resection-intersection cycle

¤ 1. Initialize:

¤ (a) Select two suitable photographs; ¤ (b) Solve relative orientation and form the stereo-model (intersection)

¤ 2. For every additional photo (following a suitable order),

¤ (a) Solve exterior orientation (resection) ¤ (c) Refine the existing model (intersection); ¤ 3. Transform the final model into the ground coordinate system with absolute orientation (given known GCP). ¤ This is also known in CV as sequential structure from motion (à la Bundler)

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-11
SLIDE 11

Hierarchical variation

¤ The previous method can be generalized by organizing the photographs on a tree instead of a chain. ¤ The tree is produced by hierarchical clustering the photographs according to their overlap (the overlapping relationship is indeed not a chain)

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-12
SLIDE 12

Hierarchical variation (Samantha)

¤ 1. Form many independent stereomodels from photo pairs at the leaves of the tree ¤ 2. Traverse the tree; in each node one of these operations takes place:

¤ Grow one model by adding one photo with resection followed by intersection; ¤ Merge two independent model with absolute orientation;

¤ 3. Transform the final model into the ground coordinate system with absolute orientation (given known GCP). ¤ Note:

¤ If the tree reduces to a chain, the algorithm is a sequential SfM; ¤ If the tree is perfectly balanced, only the Merge step is taken, and the resulting procedure resembles the IMBA.

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-13
SLIDE 13

Tree traversal

Relative Orientation Exterior Orientation Absolute Orientation

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-14
SLIDE 14

Comments

¤ CV methods are designed for irregular and unknown block

  • structure. Recovering block structure and tie-points matching is

fundamental (see forward) ¤ CV methods do not make use of GCP but at the end (free solution) because they are compulsory. ¤ The gold standard is bundle adjustment, the other methods are seen as tools to get an approximate solution. ¤ The hierarchical method is more effective than the sequential one in drift containment.

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-15
SLIDE 15

Pre-processing summary

¤ Keypoint extraction ¤ Matching - broad phase: select O(n) views to be matched ¤ Matching – narrow phase: match keypoints between pair ¤ Clustering: determine processing order (can be on-line)

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-16
SLIDE 16

Keypoints

¤ Detector: scale-space extrema of the scale-normalized Laplacian [T. Lindeberg, 1994]. SIFT is an variation on this theme. ¤ We use a 8-level scale-space and in each level the Laplacian is computed by convolution (in CUDA) with a 3 × 3 kernel. ¤ Key: multiresolution pyramid based on derivative operator. ¤ Descriptor: 128- dimensional radial descriptor based on the accumulated response of steerable derivative filters. ¤ Key: derivatives, directions histogram,

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-17
SLIDE 17

Which images are to be matched?

¤ Recover the image graph, i.e., the graph that tells which image

  • verlaps (or can be matched) with which other.

¤ For each key-point descriptor its approximate k nearest neighbours in feature space are computed (via ANN) ¤ A 2D histogram is then built: increment bin(i,j) whenever a keypoint of image i has a keypoint

  • f image j in its k-neighbourhood;

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-18
SLIDE 18

Which images are to be matched?

¤ Image graph G = (V,E) where V are views and the weighted adjacency matrix is the 2D histogram. ¤ This graph has | V | = O(n2). The objective is to extract a subgraph G’ with a number of edges that is linear in n. ¤ In graph theory, a graph is k-edge-connected if it remains connected whenever fewer than k edges are removed. ¤ We devised a strategy that builds a subgraph G’ of G which is k-edge-connected by construction and has has (n-1)k = O(n) edges

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-19
SLIDE 19

Matching

¤ Match keypoints (images connected in the graph) ¤ Validate matching with relative orientation (RANSAC) ¤ Non linear refinement of relative orientation

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-20
SLIDE 20

Tracks

¤ From pairwise matches to tracks

¤ just a matter of bookkeeping…

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-21
SLIDE 21

Clustering

¤ Agglomerative, hierarchical clustering problem ¤ Simple (complete, average, ward’s) linkage ¤ Distance needed:

¤ Common matches ¤ Good coverage

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-22
SLIDE 22

Image acquisition

3DF Zephyr pipeline

3d photo-consistency from images 3d surface from 3d photo-consistency Image

  • rientation

3DF Samantha (structure and motion) 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006]

Images from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-23
SLIDE 23

Multi-view stereo

Credits to R. Toldo (3Dflow s.r.l.), and G. Vogiatzis (Aston U.)

slide-24
SLIDE 24

Depth-map fusion

1. Compute depth hypotheses 2. Volumetrically fuse the depth-maps 3. Extract a 3d surface

Images from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-25
SLIDE 25

Computing depth maps

¤ For each pixel in the current image

¤ March along the back-projected ray of the pixel inside the bounding volume ¤ For each depth value d, project the 3D point in all the neighbouring views and compute NCC ¤ Equivalently, we work in image space on rectified images (easy to code on GPU) ¤ The mapping between epipolar lines is a 1-d homography

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-26
SLIDE 26

Computing depth maps

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-27
SLIDE 27

Aggregating NCC scores

¤ Several NCC profiles must be aggregated to extract the depth of the given pixel. ¤ All local maxima could be good hypotheses. ¤ How do we pick right one?

Images from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-28
SLIDE 28

MRF estimate

¤ Voting: each local peak casts a vote (weighted by its score value) on a discrete histogram along the optical ray. ¤ Winner take-all: the most voted value is the depth of the point à can do better ¤ MRF relaxation:

¤ the k bins of the histograms with the highest score are as the candidate depths for the pixel. ¤ minimization selects the best depth among several hypotesis taking neighbourhoods into account.

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-29
SLIDE 29

MRF result

Winner-take-all Depth map after MRF

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-30
SLIDE 30

Merge depth maps

¤ Depth maps are lifted in 3D space to produce a photoconsistency volume, represented by an

  • ctree that accumulates the scores coming from

each depth map.

Image from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-31
SLIDE 31

Photo-consistency of a 3d point

A B

A is photoconsistent B is NOT photoconsistent

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-32
SLIDE 32

Visibility

¤ The photoconsistency volume at this stage contains spurious points, which do not belong to a real surface. ¤ They are characterized by two features:

¤ their photoconsistency is generally lower than actual surface points ¤ they usually occludes actual surface points.

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-33
SLIDE 33

Visibility

¤ This observation leads to an iterative strategy where the photoconsistency of an occlusor is decreased by a fraction of the photoconsistency of the occluded point. ¤ Points with negative photoconsistency are eventually removed.

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-34
SLIDE 34

Image acquisition

3DF Zephyr pipeline

3d photo-consistency from images 3d surface from 3d photo-consistency Image

  • rientation

3DF Samantha (structure and motion) 3DF Stasia (multiple view stereo) Poisson [Kazhdan, Bolitho, and Hoppe, 2006]

Images from Vogiatzis, 2010

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-35
SLIDE 35

3D surface from points

¤ Compute an indicator function from oriented points by solving a Poisson problem [Kazhdan et. al]

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

slide-36
SLIDE 36

Alcuni risultati: 3DF Zephyr con foto da APR

Dati gentilmente forniti da F. Remondino, FBK

slide-37
SLIDE 37

Nettuno

Fotografie nadirali (15) e oblique (28), con due fotocamere diverse (compatte). Tutto il blocco di 43 fotografie viene orientato da Zephyr. Punti di legame: 13K Reference variance: 1.58 pix

slide-38
SLIDE 38

Nettuno

Dopo multiple-view stereo si ottengono 4.498M punti (vertici mesh). [Video]

slide-39
SLIDE 39

Cerere

Fotografie nadirali (52), oblique (24) e terrestri interno/esterno (212), con tre fotocamere diverse (compatta su APR, reflex digitale a terra). Tutto il blocco di 288 fotografie viene orientato da Zephyr. Punti di legame: 127K Reference variance: 0.73 pix

slide-40
SLIDE 40

Cerere

Dopo multiple-view stereo si ottengono 3.521M punti (vertici mesh) [Video]

slide-41
SLIDE 41

Ventimiglia

Fotografie nadirali (253) e oblique (224), con stessa fotocamera (reflex digitale). Tutto il blocco di 447 fotografie viene orientato da Zephyr. Punti di legame: 329K Reference variance: 0.59 pix

slide-42
SLIDE 42

Ventimiglia

Dopo multiple-view stereo si ottengono 2.715M punti (vertici mesh) [Video]

slide-43
SLIDE 43

Ventimiglia

Due vedute della maglia poligonale risultante

slide-44
SLIDE 44

Ventimiglia

Ortofoto Impronta del pixel (GSD): 0.0018m (quota inferiore)

slide-45
SLIDE 45

Accuracy analysis

2 4 3 5 6 7 8 9 1011121314151617181921222324 0.02 0.04 0.06 [m] Distance to ground control points 2 4 3 5 6 7 8 9 1011121314151617181921222324 −0.02 0.02 0.04 x [m] Difference with ground control points on each dimension 2 4 3 5 6 7 8 9 1011121314151617181921222324 −0.04 −0.02 0.02 y [m] 2 4 3 5 6 7 8 9 1011121314151617181921222324 −0.1 −0.05 0.05 0.1 z [m]

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Cross validation mean distance: 0.0358

slide-46
SLIDE 46

UAV / RPAS in Italia - Modena 20-21 Febbraio 2014

Grazie per l’attenzione.