A Survey of Urban Reconstruction Przem yslaw Musialski Peter W - - PowerPoint PPT Presentation

▶
a survey of urban reconstruction
SMART_READER_LITE
LIVE PREVIEW

A Survey of Urban Reconstruction Przem yslaw Musialski Peter W - - PowerPoint PPT Presentation

Dec 28, 2023 340 likes •1.49k views

Eurographics 2012, Cagliari, Italy A Survey of Urban Reconstruction Przem yslaw Musialski Peter W onka Daniel G. Aliaga Michael W im m er Luc van Gool W erner Purgathofer Eurographics 2012, Cagliari, Italy STAR: A Survey of Urban

slide-1
SLIDE 1

Eurographics 2012, Cagliari, Italy

A Survey of Urban Reconstruction

Przem yslaw Musialski Peter W onka Daniel G. Aliaga Michael W im m er Luc van Gool W erner Purgathofer

slide-2
SLIDE 2

Eurographics 2012, Cagliari, Italy 2 STAR: A Survey of Urban Reconstruction

W ho are the Authors?

  • Przem yslaw Musialski:

– Postdoc (TU-Wien/ ASU), formerly researcher at VRVis – Field: Graphics, Image Processing

  • Peter W onka

– Associate Prof (ASU/ KAUST) – Field: Graphics, Image Processing

  • Daniel Aliaga

– Associate Prof (Purdue University) – Field: Vision, Graphics

  • Michael W im m er

– Associate Prof (TU-Wien) – Field: Graphics

  • Luc van Gool

– Full Prof (ETH Zurich & KU Leuven) – Field: Vision, Photogrammetry & Remote Sensing

  • W erner Purgathofer

– Full Prof (TU-Wien) and Scientific Director (VRVis) – Field: Graphics

2

slide-3
SLIDE 3

Eurographics 2012, Cagliari, Italy

W hat is Urban Reconstruction?

  • Creating digital m odels of real cities
  • Cities are large collections
  • f m an-m ade objects

at m any LODs

3 STAR: A Survey of Urban Reconstruction

slide-4
SLIDE 4

Eurographics 2012, Cagliari, Italy

Possible Applications

  • Cyber-Tourism
  • Com puter Gam es
  • Movie-I ndustry and

Entertainm ent I ndustry

  • Digital Maps and Routing
  • City-Planers and Architects
  • Archeological Research
  • More Sciences ( Sociology,…)

4 STAR: A Survey of Urban Reconstruction

slide-5
SLIDE 5

Eurographics 2012, Cagliari, Italy

Scope

  • W e cover geom etric reconstruction

– Graphics, Vision and some Photogrammetry & Remote Sens. – Different Levels of Detail – Interactive and Automatic Methods

  • W e do NOT cover

– Manual Reconstruction (CAD-Modeling) – Procedural Modeling – Mobile- and Mapping-Technology – Geo-Sciences – Architecture & Civil Engineering – Hardware, Sensors, Electrical Engineering

5 STAR: A Survey of Urban Reconstruction

slide-6
SLIDE 6

Eurographics 2012, Cagliari, Italy

Contributions from Different Fields:

  • Com puter Graphics

– Usually Interactive Modeling – Inverse Procedural Modeling – (Procedural Modeling)

  • Com puter Vision

– Automatic Reconstruction – Inverse Procedural Modeling

  • Photogram m etry and

Rem ote Sensing

– Measuring and Documenting the Earth

6 STAR: A Survey of Urban Reconstruction

slide-7
SLIDE 7

Eurographics 2012, Cagliari, Italy

I nput Data

8 STAR: A Survey of Urban Reconstruction

slide-8
SLIDE 8

Eurographics 2012, Cagliari, Italy

Challenges

  • Full Autom ation

– The Chicken-Or-Egg Dilemma – Top-Down versus Bottom-Up

  • Quality and Scalability

– User-Interaction does not scale well – Fully-automatic systems lack production quality

  • Acquisition Constraints

– Real buildings are often not easy to capture – Occlusions, Reflections and other obstacles

9 STAR: A Survey of Urban Reconstruction

slide-9
SLIDE 9

Eurographics 2012, Cagliari, Italy

Overview

10 STAR: A Survey of Urban Reconstruction

slide-10
SLIDE 10

Eurographics 2012, Cagliari, Italy

Overview

  • A. Point Clouds & Cam eras

– Fundamentals of Stereo – Structure from Motion – Multiview Stereo

  • B. Buildings & Sem antics

– Image Based Modeling (IMB) – LiDAR-Based Modeling – Inverse-Procedural Modeling (IPM)

  • C. Façades & I m ages

– Façade Image Processing – Façade Parsing – Façade Modeling

  • D. Blocks & Cities

– Ground Based Reconstruction – Aerial Reconstruction – Massive City Reconstruction

11 STAR: A Survey of Urban Reconstruction

slide-11
SLIDE 11

Eurographics 2012, Cagliari, Italy

Overview

  • A. Point Clouds & Cam eras

– Fundamentals of Stereo – Structure from Motion – Multiview Stereo

  • B. Buildings & Sem antics

– Image Based Modeling (IMB) – LiDAR-Based Modeling – Inverse-Procedural Modeling (IPM)

  • C. Façades & I m ages

– Façade Image Processing – Façade Parsing – Façade Modeling

  • D. Blocks & Cities

– Ground Based Reconstruction – Aerial Reconstruction – Massive City Reconstruction

12 STAR: A Survey of Urban Reconstruction

slide-12
SLIDE 12

Eurographics 2012, Cagliari, Italy

Overview

  • A. Point Clouds & Cam eras

– Fundamentals of Stereo – Structure from Motion – Multiview Stereo

  • B. Buildings & Sem antics

– Image Based Modeling (IMB) – LiDAR-Based Modeling – Inverse-Procedural Modeling (IPM)

  • C. Façades & I m ages

– Façade Image Processing – Façade Parsing – Façade Modeling

  • D. Blocks & Cities

– Ground Based Reconstruction – Aerial Reconstruction – Massive City Reconstruction

13 STAR: A Survey of Urban Reconstruction

slide-13
SLIDE 13

Eurographics 2012, Cagliari, Italy

Overview

  • A. Point Clouds & Cam eras

– Fundamentals of Stereo – Structure from Motion – Multiview Stereo

  • B. Buildings & Sem antics

– Image Based Modeling (IMB) – LiDAR-Based Modeling – Inverse-Procedural Modeling (IPM)

  • C. Façades & I m ages

– Façade Image Processing – Façade Parsing – Façade Modeling

  • D. Blocks & Cities

– Ground Based Reconstruction – Aerial Reconstruction – Massive City Reconstruction

14 STAR: A Survey of Urban Reconstruction

slide-14
SLIDE 14

Eurographics 2012, Cagliari, Italy

Overview

15 STAR: A Survey of Urban Reconstruction

slide-15
SLIDE 15

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

16 STAR: A Survey of Urban Reconstruction

  • Cam era Model

– Central Projection – Pinhole Camera

slide-16
SLIDE 16

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

17 STAR: A Survey of Urban Reconstruction

  • W hat is Cam era Calibration?
  • Calibration m eans to obtain the param eters:

– Intrinsic Calibration:

  • Projection Parameters
  • (Focal Length, etc.)

– Using Markers we can infer intrinsic parameters

slide-17
SLIDE 17

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

18 STAR: A Survey of Urban Reconstruction

  • W hat is Cam era Calibration?
  • Calibration m eans to obtain the param eters:

– Intrinsic Calibration:

  • Projection Parameters
  • (Focal Length, etc.)

– Extrinsic Calibration (Pose Estimation)

  • Pose of the camera

in the world space

slide-18
SLIDE 18

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

19 STAR: A Survey of Urban Reconstruction

  • W hat is Cam era Calibration?
  • Calibration m eans to obtain the param eters:

– Intrinsic Calibration:

  • Projection Parameters
  • (Focal Length, etc.)

– Extrinsic Calibration (Pose Estimation)

  • Pose of the camera

in the world space

slide-19
SLIDE 19

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

20 STAR: A Survey of Urban Reconstruction

  • W hat is Cam era Calibration?
  • Calibration m eans to obtain the param eters:

– Intrinsic Calibration:

  • Projection Parameters
  • (Focal Length, etc.)

– Extrinsic Calibration (Pose Estimation)

  • Pose of the camera

in the world space

  • Can be determined from

5 (7) image correspondences

slide-20
SLIDE 20

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

  • Stereo Geom etry

– Given is the point x1 on the image – How to determine the 3D point X?

21 STAR: A Survey of Urban Reconstruction

slide-21
SLIDE 21

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

22 STAR: A Survey of Urban Reconstruction

  • Stereo Geom etry

– We need a second image with x2 corresponding to x1

slide-22
SLIDE 22

Eurographics 2012, Cagliari, Italy

A.1 Fundam entals of Stereo

  • Stereo Triangulation

23 STAR: A Survey of Urban Reconstruction

slide-23
SLIDE 23

Eurographics 2012, Cagliari, Italy

Overview

24 STAR: A Survey of Urban Reconstruction

slide-24
SLIDE 24

Eurographics 2012, Cagliari, Italy

  • Structure from Motion

– Only images as input – A high number of images can be registered – A high number of points can be triangulated – Since images where taken with a camera in motion  Structure from Motion (SFM)

A.2 Structure from Motion

25 STAR: A Survey of Urban Reconstruction

slide-25
SLIDE 25

Eurographics 2012, Cagliari, Italy

A.2 Structure from Motion

26 STAR: A Survey of Urban Reconstruction

  • Structure from Motion

– Input: set of images – Challenges:

  • Correspondence Problem
  • Structure Triangulation Problem
  • Additional product: Camera poses
slide-26
SLIDE 26

Eurographics 2012, Cagliari, Italy

A.2 Structure from Motion

  • Correspondence Problem
  • Feature Detection and Matching

– Mutual matching e.g. KD-Tree – Geometric verification: (RANSAC)

  • Fischler & Bolles [ FB81]

27 STAR: A Survey of Urban Reconstruction

slide-27
SLIDE 27

Eurographics 2012, Cagliari, Italy

A.2 Structure from Motion

  • I ncrem ental process

– Starting from initial image pair – Adding more images – Features and Camera Poses are determined – Image networks are generated

  • Bundle Adjustm ent

– Non-linear

  • ptimization
  • f the whole

network

29 STAR: A Survey of Urban Reconstruction

slide-28
SLIDE 28

Eurographics 2012, Cagliari, Italy

A.2 Structure from Motion

  • Photo Tourism

– Snavely et al. [ SSS06,SSS07,SGSS08, SSG* 10] – Use collections of images of sight seeing from the Internet – Generate sparse point clouds – Use image-blending in order to smoothly move from image to image

30 STAR: A Survey of Urban Reconstruction

slide-29
SLIDE 29

Eurographics 2012, Cagliari, Italy

A.2 Structure from Motion

31 STAR: A Survey of Urban Reconstruction

slide-30
SLIDE 30

Eurographics 2012, Cagliari, Italy

A.2 Structure from Motion

  • Building Rom e in a Day

– Agrawal et al. [ ASS* 09] – Optimization of the pipeline – Over 150 000 images of Rome – (250 000 from Venice) – Processed in parallel in a processor-cluster – Reconstructs sparse point clouds

32 STAR: A Survey of Urban Reconstruction

slide-31
SLIDE 31

Eurographics 2012, Cagliari, Italy

Overview

33 STAR: A Survey of Urban Reconstruction

slide-32
SLIDE 32

Eurographics 2012, Cagliari, Italy

A.3 Multiview Stereo

  • Dense Multiview Stereo

– Use sparse stereo and camera networks as input – Compute dense, possibly water-tight, reconstructions

35 STAR: A Survey of Urban Reconstruction

slide-33
SLIDE 33

Eurographics 2012, Cagliari, Italy

A.3 Multiview Stereo

  • Dense Matching System s

– Pollefeys et al. [ PvGV* 04,PNF* 08] – Vergauven and van Gool [ VvG06] – Akbarzadeh et al. [ AFM* 06] – Frahm et al. [ FFGG10] – Furukawa and Ponce [ FP07,PF9] – Agrawal et al. [ AFS* 11,FP09]

36 STAR: A Survey of Urban Reconstruction

slide-34
SLIDE 34

Eurographics 2012, Cagliari, Italy

[VvG06] [FP07] [MK10]

A.3 Multiview Stereo

  • Problem :

– Dense reconstructions are not perfectly flat

  • Solution: Planar Priors

– Manhattan World Priors

  • Furukawa et al. [ FCSS09]

– Piece-Wise Planar Priors

  • Micusic and Kosecka [ MK09,MK10]
  • Sinha et al. [ SSS09]
  • Chauve et al. [ CLP10]
  • Gallup et al. [ GLP10]

39 STAR: A Survey of Urban Reconstruction

slide-35
SLIDE 35

Eurographics 2012, Cagliari, Italy

  • A. Point-Clouds and Cam eras
  • Sum m ary

– Sparse MVS and SfM are mature and robust – Dense MVS deliver also quite impressive results – Systems are very generic – not only urban reconstruction – Scale well as shown by Frahm et al. [ FFGG10] :

  • 3 million images on one day on a single PC

– Downside: results are usually dense meshes, not segmented and semantic

  • bjects

40 STAR: A Survey of Urban Reconstruction

slide-36
SLIDE 36

Eurographics 2012, Cagliari, Italy

Overview

41 STAR: A Survey of Urban Reconstruction

slide-37
SLIDE 37

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Also referred to as Photogram m etric Modeling
  • Subcategories

– Interactive Multiview Modeling – Automatic Multiview Modeling – Interactive Singleview Modeling – Automatic Singleview Modeling

42 STAR: A Survey of Urban Reconstruction

slide-38
SLIDE 38

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Façade ( Debevec et al. [ DTM9 6 ] )

– Primitive polyhedral elements – Parallel and Orthogonal – Constrained to each other to reduce the parameter space

  • Good layer of abstraction

– Low-level features are difficult to deal with – Surface model is implicit

43 STAR: A Survey of Urban Reconstruction

slide-39
SLIDE 39

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Façade Modeling Process [ DTM9 6 ]

– Multiview Input – Automatic edge detection in images – User establishes corresponding edges in images interactively – System optimizes in background (non-realtime)

  • I terative m odeling

process

  • Finally projective

texturing from input im ages

44 STAR: A Survey of Urban Reconstruction

slide-40
SLIDE 40

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Photobuilder:

– Cipolla and Robertson [ CR99,CRB99] – Automatic edge detection – User interactively labels a few parallel and orthogonal edges – Camera parameters can be determined – System computed this model

45 STAR: A Survey of Urban Reconstruction

slide-41
SLIDE 41

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Photobuilder:

– Cipolla and Robertson [ CR99,CRB99] – Automatic edge detection – User interactively labels a few parallel and orthogonal edges – Camera parameters can be determined – System computed this model

46 STAR: A Survey of Urban Reconstruction

slide-42
SLIDE 42

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • I nteractive Modeling from Video ( VideoTrace)

– Van den Hengel et al. [ vdHDT* 06, vdHDT* 07] – Camera and point-cloud network from SFM as input – Hierarchy of primitive shapes as model – User-input to establish relations – Automatic optimization in background (near-realtime)

47 STAR: A Survey of Urban Reconstruction

slide-43
SLIDE 43

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • I nteractive Multiview Modeling from Unordered

Sets of Photographs

  • Sinha et al. [ SSS* 08]
  • Image-Network as input
  • Automatic detection of vanishing points
  • Simple interactions like rough sketching
  • Realtime interactive optimization in background

48 STAR: A Survey of Urban Reconstruction

slide-44
SLIDE 44

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Further m ethods and im provem ents

– Combination of ground and aerial imagery

  • Lee et al. [ LHN00, LJN02, LN03,…

]

– Database with reusable elements

  • El-Hakim et al. [ EhWGG05,EhWG05]

– Automatically snapping polygons

  • Arikan et al [ ASW* 12]

49 STAR: A Survey of Urban Reconstruction

slide-45
SLIDE 45

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Autom atic Multiview Modeling

– Buildings are well suited due to parallelism and orthogonality – Line features, contours and vanishing points can be found automatically – Using least-squares and robust estimation (RANSAC) planes can be fitted

  • Autom ation of the

I nteractive Modeling Approach

– Libowitz and Zisserman [ LZ99] – Coorg and Teller [ CT99] – Werner and Zisserman [ WZ02]

50 STAR: A Survey of Urban Reconstruction

slide-46
SLIDE 46

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Autom atic Multiview Modeling

– Dick et al. [ DTC00,DCT04] – Probabilistic model with predefined prior distributions – Parameters fitted from a set of images using MCMC – Semantically annotated objects

51 STAR: A Survey of Urban Reconstruction

slide-47
SLIDE 47

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Single I m age I nteractive Modeling
  • Utilize the symmetry of the building to reconstruct 3d structure

Jiang et al. 2009 [ JTC09]

  • Interactively determine a frustum
  • Determine camera pose (calibration)
  • Use mirror-symmetry for stereo-reconstruction

52 STAR: A Survey of Urban Reconstruction

slide-48
SLIDE 48

Eurographics 2012, Cagliari, Italy

B.1 I m age-Based Modeling ( I BM)

  • Sum m ary

– There is a large number of approaches – Some methods attempt automatic solutions – Nonetheless, the quality of fully- automatic systems is still below expected production standards – Due to the demand of high-quality models, interactive/ semi-manual modeling is still interesting

53 STAR: A Survey of Urban Reconstruction

slide-49
SLIDE 49

Eurographics 2012, Cagliari, Italy

Overview

54 STAR: A Survey of Urban Reconstruction

slide-50
SLIDE 50

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • LiDAR ( Light Detection and Ranging)
  • scans are w ell suited for reconstruction, but
  • Problem s:

– Point cloud contains holes due to occlusions – Especially in ground-based LiDAR

55 STAR: A Survey of Urban Reconstruction

slide-51
SLIDE 51

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • LiDAR scans are w ell suited for reconstruction,

but

  • Problem s:

– Oblique scanning angles – Laser energy attenuation on range – Especially in ground-based LiDAR

56 STAR: A Survey of Urban Reconstruction

slide-52
SLIDE 52

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • I nteractive Modeling from LiDAR ( Sm artBoxes)

– Nan et al. [ NSZ* 10] – User assembles small sets of “boxes” from primitive shapes – These are automatically fitted to the point cloud minimizing a sum of two energies:

  • Data: how well does each box fit to the local point cloud
  • Context: how well are the boxes synchronized

57 STAR: A Survey of Urban Reconstruction

slide-53
SLIDE 53

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • I nteractive Modeling from LiDAR ( Sm artBoxes)

– Nan et al. [ NSZ* 10]

58 STAR: A Survey of Urban Reconstruction

slide-54
SLIDE 54

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • Autom atic Modeling from terrestrial LiDAR

– Scans of buildings are well suited for automatic reconstruction

  • Stamos and Allen [ SA00,SA02]
  • Früh and Zakhor [ FZ03,FZ04]
  • Pu and Vosselman [ PV09]
  • Vanegas et al. [ VAB12]
  • and more

– Segmentation into planar regions

  • Clustering of Normals

– Plane Fitting

  • RANSAC
  • Least-Squares

– Fitting of Outline Polygons

59 STAR: A Survey of Urban Reconstruction

slide-55
SLIDE 55

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • Autom atic Model Fitting

– Manhattan-World assumption in order to improve the robustness of the fit

  • Vanegas et al. [ VAB12]

60 STAR: A Survey of Urban Reconstruction

slide-56
SLIDE 56

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • Autom atic Segm entation of LiDAR

– Recursive Heuristic Splitting using Symmetry

  • Shen et al. [ SHFA11]

61 STAR: A Survey of Urban Reconstruction

slide-57
SLIDE 57

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • Autom atic Modeling from aerial LiDAR

– 2.5D dual contouring (Zhou and Neumann [ ZN08,ZN10] ) – Detailed results

62 STAR: A Survey of Urban Reconstruction

slide-58
SLIDE 58

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • Autom atic Modeling from aerial LiDAR

– 2.5D dual contouring (Zhou and Neumann [ ZN08,ZN10] )

63 STAR: A Survey of Urban Reconstruction

slide-59
SLIDE 59

Eurographics 2012, Cagliari, Italy

B.2 LiDAR-Based Modeling

  • Sum m ary

– LiDAR is accessible for quite a while – Top-down fitting of buildings into the data delivers good results – The full potential of LiDAR- driven reconstruction is still not explored – More interesting methods are expected to appear in the near future

64 STAR: A Survey of Urban Reconstruction

slide-60
SLIDE 60

Eurographics 2012, Cagliari, Italy

Overview

65 STAR: A Survey of Urban Reconstruction

slide-61
SLIDE 61

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • Rather novel approach

– Related to Procedural Modeling – Idea: derive a grammar from the structure

  • I nfer from the input ( I m agery
  • r LiDAR)

– (1) A grammar – (2) Parameters of the grammar – Some methods predefine (1) and infer only (2)

  • I nteractive and Autom atic

approaches

66 STAR: A Survey of Urban Reconstruction

slide-62
SLIDE 62

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • I nteractive System s

– Aliaga et al. [ ARB07] – Model a geometric model interactively from a few photos – Segment the model interactively and assign grammar

67 STAR: A Survey of Urban Reconstruction

slide-63
SLIDE 63

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • I nteractive System s

– Aliaga et al. [ ARB07] – Use grammar to generate novel variations of the building

68 STAR: A Survey of Urban Reconstruction

slide-64
SLIDE 64

Eurographics 2012, Cagliari, Italy

  • I nteractive System s

– Aliaga et al. [ ARB07] – Use grammar to generate novel variations of the building

B.3 I nverse-Procedural Modeling ( I PM)

69 STAR: A Survey of Urban Reconstruction

slide-65
SLIDE 65

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • Autom atic Methods

– Simplification: predefine grammar and fit only the parameters – Vanegas et al. [ VAB10] – Using aerial imagery and GIS-data

70 STAR: A Survey of Urban Reconstruction

slide-66
SLIDE 66

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • Autom atic Methods

– Generate initial 3D building envelope

  • Use the footprint from GIS and extrude

– Divide the bounding box into floors

71 STAR: A Survey of Urban Reconstruction

slide-67
SLIDE 67

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • Autom atic Methods

– Generate initial 3D building envelope

  • Use the footprint from GIS and extrude

– Divide the bounding box into floors – Adjust each floor automatically from the information from images and the constraints of the grammar

72 STAR: A Survey of Urban Reconstruction

slide-68
SLIDE 68

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • Further m ethods

– Use partial symmetry to derive shape grammars

  • f 3D models
  • Bokeloh et al. [ BWS10]

– Generative Modeling Language (GML)

  • Havemann [ Hav05]
  • Hohmann et al.

[ HKHF09,HHKF10]

– Façade Image Segmentation

  • Coming in the next section!

73 STAR: A Survey of Urban Reconstruction

slide-69
SLIDE 69

Eurographics 2012, Cagliari, Italy

B.3 I nverse-Procedural Modeling ( I PM)

  • Sum m ary

– IPM is a quite new field – It enables a very compact description of the models – Very suitable for generation

  • f content

– Many further exciting papers to appear!

74 STAR: A Survey of Urban Reconstruction

slide-70
SLIDE 70

Eurographics 2012, Cagliari, Italy

Overview

75 STAR: A Survey of Urban Reconstruction

slide-71
SLIDE 71

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • I m agery is essential in Urban Reconstruction

– For a realistic look – As source for reconstruction

  • Applications

– Panoramas – Projective Textures – Source for 3D structure

76 STAR: A Survey of Urban Reconstruction

slide-72
SLIDE 72

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Strip-Panoram as

– Agrawala et al. [ AAC* 06]

77 STAR: A Survey of Urban Reconstruction

slide-73
SLIDE 73

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Multiview Projective Texturing
  • Aliaga et al. [ * 10]

78 STAR: A Survey of Urban Reconstruction

slide-74
SLIDE 74

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Multiview Projective Texturing
  • Musialski et al. [ MLS* 10]

79 STAR: A Survey of Urban Reconstruction

slide-75
SLIDE 75

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

80 STAR: A Survey of Urban Reconstruction

  • Multiview Projective Texturing
  • Musialski et al. [ MLS* 10]
slide-76
SLIDE 76

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Multiview Projective Texturing
  • Musialski et al. [ MLS* 10]

81 STAR: A Survey of Urban Reconstruction

slide-77
SLIDE 77

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Sym m etry-based façade im age repair
  • Musialski et al. [ MWR* 09]

82 STAR: A Survey of Urban Reconstruction

slide-78
SLIDE 78

Eurographics 2012, Cagliari, Italy 84 STAR: A Survey of Urban Reconstruction

slide-79
SLIDE 79

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Sym m etry-based façade im age repair

85 STAR: A Survey of Urban Reconstruction

slide-80
SLIDE 80

Eurographics 2012, Cagliari, Italy

C.1 Façade I m age Processing

  • Sum m ary

– Panoramas are a kind of reconstruction by themselves – Processing of urban imagery is quite well researched – There are still challenges

  • Automatic segmentation
  • Parsing and semantic extraction

86 STAR: A Survey of Urban Reconstruction

slide-81
SLIDE 81

Eurographics 2012, Cagliari, Italy

Overview

87 STAR: A Survey of Urban Reconstruction

slide-82
SLIDE 82

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Façade parsing

– Automatic semantic segmentation façade data

  • Images or Laser Scans

– Often use of higher-order models, like grammars

  • First step is low level processing

– Feature-, Edge-, Blob-Detection

88 STAR: A Survey of Urban Reconstruction

slide-83
SLIDE 83

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Façade parsing

– Automatic semantic segmentation of façade data

  • Teboul et al. [ TSKP10,TKS* 11]

89 STAR: A Survey of Urban Reconstruction

slide-84
SLIDE 84

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Façade parsing

– Automatic semantic segmentation of façade data

  • Teboul et al. [ TSKP10,TKS* 11]

90 STAR: A Survey of Urban Reconstruction

slide-85
SLIDE 85

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Further m ethods

– Predefined grammar based segmentations of images

  • Allegre and Dalleart [ AD04]

– Predefined grammar based segmentations of image and LiDAR

  • Brenner and Ripperda

[ BR06,RB07,RB09]

– Inference of both grammar and parameters from LiDAR

  • Becker and Haala [ BH07,NH09]

91 STAR: A Survey of Urban Reconstruction

slide-86
SLIDE 86

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Sym m etry Detection

– Another example of higher-order knowledge is symmetry – Number of methods detect symmetry in façades

  • In perspective images

– Wu et al. [ WFP10]

  • In ortho-rectified, occluded images

– Musialski et al. [ MRM* 10]

92 STAR: A Survey of Urban Reconstruction

slide-87
SLIDE 87

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Sym m etry detection in point clouds

– Pauly et al. [ PMW* 08] – The symmetries can be used to complete missing data

93 STAR: A Survey of Urban Reconstruction

slide-88
SLIDE 88

Eurographics 2012, Cagliari, Italy

C.2 Façade Parsing

  • Sum m ary

– Recent automatic methods provide quite stable results – The downside is the still quite low level-of-detail – Also, errors are often difficult to fix – This field is still in active research

94 STAR: A Survey of Urban Reconstruction

slide-89
SLIDE 89

Eurographics 2012, Cagliari, Italy

Overview

95 STAR: A Survey of Urban Reconstruction

slide-90
SLIDE 90

Eurographics 2012, Cagliari, Italy

C.3 Façade Modeling

  • I nteractive Modeling

– Pro: provides very good quality – Con: slower and does not scale very well

96 STAR: A Survey of Urban Reconstruction

slide-91
SLIDE 91

Eurographics 2012, Cagliari, Italy

C.3 Façade Modeling

  • Post-processing of autom atic m ethods
  • Xiao et al. [ XFT* 08]

– Use automatic heuristics to generate initial segmentation – User interactive post-processing to fix errors in the initial segmentation – Infer depth from multi-view setups – Post-process interactively to fix errors

97 STAR: A Survey of Urban Reconstruction

slide-92
SLIDE 92

Eurographics 2012, Cagliari, Italy

C.3 Façade Modeling

  • Post-processing of autom atic m ethods
  • Xiao et al. [ XFT* 08]

– Very good results – But a quite a time consuming task

98 STAR: A Survey of Urban Reconstruction

slide-93
SLIDE 93

Eurographics 2012, Cagliari, Italy

C.3 Façade Modeling

  • Coherence-Based I nteractive Modeling
  • Musialski et al. [ MWW12]

– Incorporate the user from the beginning

  • Let the user define high-level structure
  • Group coherent regions
  • Perform automatic splits on
  • verlapping groups
  • Combine these splits for final

segmentation

  • Add depth interactively

99 STAR: A Survey of Urban Reconstruction

slide-94
SLIDE 94

Eurographics 2012, Cagliari, Italy

C.3 Façade Modeling

  • Coherence-Based I nteractive Modeling

– Very good results – Better high-level structure – Still quite time-consuming

100 STAR: A Survey of Urban Reconstruction

slide-95
SLIDE 95

Eurographics 2012, Cagliari, Italy

C.3 Façade Modeling

  • Sum m ary

– Interactive Modeling is slow and does not scale well – Today's productions still rely mostly on interactive methods – Integration of user- interaction and automatism is still to improve

101 STAR: A Survey of Urban Reconstruction

slide-96
SLIDE 96

Eurographics 2012, Cagliari, Italy

Overview

102 STAR: A Survey of Urban Reconstruction

slide-97
SLIDE 97

Eurographics 2012, Cagliari, Italy

D.1 Ground-Based Reconstruction

  • Algorithm s w ork w ell w ith sm all data sets
  • Challenge: large scale

– Irschara et al. [ IZB07,IZB11] – Data acquisition problem: incorporate users to provide photos (Wiki-Principle)

103 STAR: A Survey of Urban Reconstruction

slide-98
SLIDE 98

Eurographics 2012, Cagliari, Italy

D.1 Ground-Based Reconstruction

  • Generate reconstructions during acquisition

– Cornelis et al. [ CLCvG08] – Use a vehicle to drive and acquire input images – Run reconstruction in “real-time”, during diving

104 STAR: A Survey of Urban Reconstruction

slide-99
SLIDE 99

Eurographics 2012, Cagliari, Italy

D.1 Ground-Based Reconstruction

  • Sum m ary

– Generally limited to smaller areas compared to aerial approaches – But the only way to provide high-detailed street level models

105 STAR: A Survey of Urban Reconstruction

slide-100
SLIDE 100

Eurographics 2012, Cagliari, Italy

Overview

106 STAR: A Survey of Urban Reconstruction

slide-101
SLIDE 101

Eurographics 2012, Cagliari, Italy

D.2 Aerial Reconstruction

  • Aerial data is very w ell suited
  • Good for docum enting and m easuring

107 STAR: A Survey of Urban Reconstruction

slide-102
SLIDE 102

Eurographics 2012, Cagliari, Italy

D.2 Aerial Reconstruction

  • Often, com bination of different inputs

– Digital Surface Model (DSM)

  • Surface with man-made objects

– Digital Terrain Models (DTM)

  • Pure terrain surface

108 STAR: A Survey of Urban Reconstruction

slide-103
SLIDE 103

Eurographics 2012, Cagliari, Italy

D.2 Aerial Reconstruction

  • Often, com bination of different inputs

– Lafarge et al. [ LDZPD11] – Extract buildings from DSM – Treat each building as a 3d parametric block of geometric primitves

109 STAR: A Survey of Urban Reconstruction

slide-104
SLIDE 104

Eurographics 2012, Cagliari, Italy

D.2 Aerial Reconstruction

  • Further m ethods

– Combine aerial and ground imagery

  • Wang et al. [ WYN07]
  • Stitch ground-images to panoramas
  • Detect footprints in aerial imagery
  • User interaction for fine tuning

– More automatic methods with DSM

  • Zebedin et al. [ ZBKB08]
  • Karantzalos and Paragios [ KP10]

110 STAR: A Survey of Urban Reconstruction

slide-105
SLIDE 105

Eurographics 2012, Cagliari, Italy

Overview

111 STAR: A Survey of Urban Reconstruction

slide-106
SLIDE 106

Eurographics 2012, Cagliari, Italy

D.3 Massive City Reconstruction

  • I m aged-Based ground dense reconstruction

– Frahm et al. [ FFGG* 10] – Tuning and optimization of existing algorithms – 3 Million input images – 1 single PC – 1 day of computing

112 STAR: A Survey of Urban Reconstruction

slide-107
SLIDE 107

Eurographics 2012, Cagliari, Italy

D.3 Massive City Reconstruction

  • W ater-Tide Polygonal Meshes from LiDAR

– Poullis and You [ PY09,PY11] – Areas of several thousands of buildings

113 STAR: A Survey of Urban Reconstruction

slide-108
SLIDE 108

Eurographics 2012, Cagliari, Italy

D.3 Massive City Reconstruction

  • Massive reconstruction from LiDAR

– Lafarge and Mallet [ LM11]

114 STAR: A Survey of Urban Reconstruction

slide-109
SLIDE 109

Eurographics 2012, Cagliari, Italy

D.3 Massive City Reconstruction

  • Massive reconstruction from LiDAR

– Lafarge and Mallet [ LM11] – Complete reconstructions:

  • Particular polygonal buildings
  • Vegetation
  • Terrain

– Generalized for any urban environments

115 STAR: A Survey of Urban Reconstruction

slide-110
SLIDE 110

Eurographics 2012, Cagliari, Italy

D.3 Massive City Reconstruction

116 STAR: A Survey of Urban Reconstruction

slide-111
SLIDE 111

Eurographics 2012, Cagliari, Italy

  • D. Blocks and Cities
  • Sum m ary

– Current results are impressive – Problems remain in

  • Processing of huge amounts of data
  • Scalable algorithms
  • Integration of different data types

117 STAR: A Survey of Urban Reconstruction

slide-112
SLIDE 112

Eurographics 2012, Cagliari, Italy

Conclusions and Outlook

  • Autom atic reconstructions

– often rely on assumptions which are not true in practice – Combination of user-interaction and automatic methods can be improved

  • Collaborative reconstruction

– Many projects incorporate Internet or user data – Simple methods could animate user to contribute to the reconstructions

  • More interdisciplinary w ork

– The borders between Graphics and Vision are thin – But the interdisciplinary cooperation between those and the Photogrammetry and Remote Sensing could be improved

118 STAR: A Survey of Urban Reconstruction

slide-113
SLIDE 113

Eurographics 2012, Cagliari, Italy

The End

  • Thank you!
  • Questions?

119 STAR: A Survey of Urban Reconstruction

slide-114
SLIDE 114

Eurographics 2012, Cagliari, Italy

The End

  • Thank you!
  • Questions?

120 STAR: A Survey of Urban Reconstruction