Exteriors of Residential Buildings Lubin Fan Peter Wonka - - PowerPoint PPT Presentation

A Probabilistic Model for Exteriors of Residential Buildings

Peter Wonka Lubin Fan

Motivation

2

A B D C E F A B C D E F

Goals

• Learning

The model can be learned from available remote sensing data.

• Sampling

The model should be generative.

• Hard constraints

The model can handle hard geometric constraints.

3

ℎ 𝑥

• Learning

The model can be learned from available remote sensing data.

• Sampling

The model should be generative.

• Hard constraints

The model can handle hard geometric constraints.

Goals

• Learning

The model can be learned from available remote sensing data.

• Sampling

The model should be generative.

• Hard constraints

The model can handle hard geometric constraints.

4

ℎ 𝑥

• Learning

The model can be learned from available remote sensing data.

• Sampling

The model should be generative.

• Hard constraints

The model can handle hard geometric constraints.

Model

• How to represent a building model?

Problem: each building has a different number of parameters.

• How to design a probabilistic model?

Problem: training the model from a small dataset

• How to generate a building model?

Problem: generating a 3D building model from a high dimensional feature vector

Challenges

5

Model

Building representation

• How to represent a building model?

Problem: each building has a different number of parameters.

• How to design a probabilistic model?

Problem: training the model from a small dataset

• How to generate a building model?

Problem: generating a 3D building model from a high dimensional feature vector

Related Work: Procedural Modeling

6

Metropolis procedural modeling [Talton et al. 2011] CGA [Müller et al. 2006] CGA++ [Schwarz and Müller 2015]

Learning Sampling Hard constraints

Related Work: Inverse Procedural Modeling

7

Symmetry maximization [Zhang et al. 2013] Inverse procedural modeling [Wu et al. 2014] Bayesian grammar learning [Martinović and Van Gool 2013] Bayesian grammar induction [Talton et al. 2011]

Learning Sampling Hard constraints Learning Sampling Hard constraints

Related Work: Probabilistic Models

8

Computer-generated building layouts [Merrel et al. 2010]

interior building model no label ambiguity

Assembly-based 3D modeling [Chaudhuri et al. 2011] Component-based shape synthesis [Kalogerakis et al. 2012]

Pipeline

9

Database

…

New buildings

Model

sampling training

Hierarchical graphical model Parametric representation

Pipeline

10

Database sampling training Attribute-based representation Attribute-based representation training sampling Building generation

Building Representations

A building model A parametric representation An attribute-based representation

12

A Parametric Building Representation

Right Back Left Top Front

ℎ

13

An Attribute-based Building Representation

𝑁𝑑 Mass model attributes

• Size of bounding box
• Building coverage ratio
• Boundary complexity
• Floor area ratio
• Area ratio of floors
• Garage attributes
• Porch attributes

𝑁𝐝,𝑐𝑒𝑠𝑧 𝑁𝐝,𝑐𝑑𝑠 𝑁𝐝,𝑏𝑠𝑔 𝑁𝐝,𝑕𝑏𝑠 𝑁𝐝,𝑞𝑝𝑠 𝑁𝐝,𝑔𝑏𝑠 𝑁𝐝,𝑐𝑐𝑝𝑦

𝐹𝑒 Element attributes

• Roof styles
• Window styles
• Special building elements

𝐹𝑠𝑝𝑝𝑔 𝐹𝑥𝑜𝑒 𝐹𝑡𝑞𝑓

𝐺

𝑑

Facade attributes

• Window-to-wall ratio
• Symmetry
• Alignment between facade pieces 𝐺

𝐝,𝑏𝑚𝑗𝑕𝑜

𝐺

𝐝,𝑥𝑥𝑠

𝐺

𝐝,𝑡𝑧𝑛

Probabilistic Modeling

Training data Hierarchical graphical model Samples

15

Design Choices

𝐷2 𝐷1 𝐷𝑗 𝐸

𝑘

𝐸2 𝐸1 𝑇 𝐷𝑗

𝐽

𝐸

𝑘 𝐾

• It requires large

training data.

• 𝑇 depends on both

continuous and discrete variables.

• Defining a distance

metric is difficult.

𝐼𝑗

𝐽

𝐷𝑗 𝑇 𝐸

𝑘 𝐾

16

Hierarchical Graphical Model

𝐽

𝑁𝐞,𝑗 𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿 𝐾

𝐺𝐞,𝑘 𝐺𝐝,𝑘 𝑇: the overall style of a building

Level 2: discrete variables Level 1: discrete variable Level 3: continuous variables

𝑁𝐞,𝑗 and 𝐺𝐞,𝑘: the styles of each attribute, respectively mass model facade variables element variables

𝐽

𝑁𝐞,𝑗 𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿 𝐾

𝐺𝐞,𝑘 𝐺𝐝,𝑘

𝑪 = {𝑇, 𝑁𝐞, 𝑁𝐝, 𝐺𝐞, 𝐺

𝐝, 𝐹𝐞}

𝑄 𝑪 = 𝑄 𝑇

𝑗

𝑄 𝑁𝐞,𝑗|𝑇 𝑄 𝑁𝐝,𝑗|𝑁𝐞,𝑗, 𝜌 𝑁𝐝,𝑗 ∙

𝑘

𝑄 𝐺𝐞,𝑘|𝑇 𝑄 𝐺𝐝,𝑘|𝐺𝐞,𝑘, 𝜌 𝐺𝐝,𝑘 ∙

𝑙

𝑄 𝐹𝐞,𝑙|𝜌 𝐹𝐞,𝑙 , 𝑇

17

Hierarchical Graphical Model

𝑁𝐝,𝑐𝑒𝑠𝑧 𝑁𝐝,𝑞𝑝𝑠 𝑁𝐝,𝑐𝑑𝑠 𝑁𝐝,𝑕𝑏𝑠 𝑁𝐝,𝑐𝑐𝑝𝑦 𝑁𝐝,𝑏𝑠𝑔 𝑁𝐝,𝑔𝑏𝑠

lateral edges

• Learning framework [Koller and Friedman 2009]

18

Learning

𝑁𝐞,𝑗

𝐽

𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿

𝐺𝐞,𝑘

𝐾

𝐺𝐝,𝑘

𝐷𝑇 𝐻 𝑃 = log 𝑄(𝐻) + log 𝑄(𝑃∗|𝐻) + log 𝑄(𝑃|𝐻, Θ𝐻) − log 𝑄(𝑃∗|𝐻, Θ𝐻)

Input: 𝑂 buildings 𝑃 = {𝑃1, 𝑃2, ⋯ , 𝑃𝑂} Output:

• the structure 𝐻
• the cardinality of the values of the hidden variables
• the lateral edges
• the parameters Θ
• Cheeseman-Stutz score [Cheeseman and Stutz 1996]

𝑁𝐞,𝑗

𝐽

𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿

𝐺𝐞,𝑘

𝐾

𝐺𝐝,𝑘

Learning

• Structure learning algorithm

19

𝐼𝑗

𝐽

𝐷𝑗 𝑇 𝐸

𝑘 𝐾

Step 1: 𝑡 and {ℎ𝑗} Step 2

𝐷1 𝐷2 𝐷3 𝐷4 𝐷𝐽

𝐽

𝑁𝐞,𝑗 𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿 𝐾

𝐺𝐞,𝑘 𝐺𝐝,𝑘

𝑁𝑑

∗

𝐺

𝑑 ∗

𝐹𝑒

∗

𝐶𝑗

Samples

20

Sampling

training sampling

𝑁𝑑

∗

𝐺

𝑑 ∗

𝐹𝑒

∗

𝐶𝑗 𝑁𝐝

∗

𝐺

𝐝 ∗

𝐹𝐞

∗

𝐶𝑗 𝑁𝐝 𝐺

𝐝

𝐹𝐞 𝑃𝑗

Training data

Building Generation

Mass model and roof generation Attribute-based representation Facade generation

22

Mass Model and Roof Generation

• Input: desired mass model attributes 𝑁𝐝

∗

• Output: building mass model 𝑁
• Mass model energy

𝐹𝑛𝑏𝑡𝑡 𝑁 =

𝑗

𝑁𝐝,𝑗 − 𝑁𝐝,𝑗

∗ 2

𝑁𝐝,𝑗

∗ 2

+ 𝐹𝑢𝑝𝑞𝑝 𝑁 + 𝐹𝑢ℎ𝑗𝑑𝑙 𝑁

Attributes term: 𝑁𝐝,𝑗 denotes the attributes of 𝑁 Topology term: All boxes should be connected.

Top

Thickness term: Mass models should not be too narrow.

Top

Mass Model and Roof Generation

• Algorithm: simulated annealing

23

Iterations Energy

Initialization Top views

A B C D E F A B C D E F

Mass Model and Roof Generation

24

F

25

Facade Generation

• Input: mass model 𝑁, desired facade attributes 𝐺

𝐝 ∗,

and the database (i.e., element set)

• Output: facades of the building 𝐺
• Facade layout energy

𝐹

𝑔𝑏𝑑 𝐺 = 𝑡 𝑘

𝐺

𝐝,𝑘,𝑡 − 𝐺 𝐝,𝑘,𝑡 ∗

𝐺

𝒅,𝑘,𝑡 ∗ 2 attributes of 𝐺 𝑡: right Facade pieces on the right side

Facade Generation

26

Results & Applications

Thumbnails of all buildings in our dataset.

Dataset

• 200 buildings: building footprints + photographs

28

Thumbnails of all buildings in our dataset.

Dataset

• 200 buildings: building footprints + photographs

29

front porch

back porch

garage

𝑁𝐝 𝐺𝐝 𝐹𝐞

Model Structure: Hidden Variables

30

𝑁𝐞,𝑗

𝐽

𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿

𝐺𝐞,𝑘

𝐾

𝐺𝐝,𝑘

Style 1 Style 2 Style 3 Style 4 Style 5 Style 6 Style 7

𝑁𝐞,𝑗

𝐽

𝑁𝐝,𝑗

𝑇

𝐹𝐞,𝑙

𝐿

𝐺𝐞,𝑘

𝐾

𝐺𝐝,𝑘

31

Model Structure: Lateral Edges

𝑁𝐝,𝑐𝑒𝑠𝑧 𝑁𝐝,𝑞𝑝𝑠 𝑁𝐝,𝑐𝑑𝑠 𝑁𝐝,𝑕𝑏𝑠 𝑁𝐝,𝑐𝑐𝑝𝑦 𝑁𝐝,𝑏𝑠𝑔 𝐹𝑠𝑝𝑝𝑔 𝐹𝑥𝑜𝑒

vs. vs.

32

Comparison with Other Models

• Holdout validation method
• a training set (80%)
• a test set (20%)
• Models
• Our model
• Model 1: without learned edges between variables
• Model 2: without hidden variables on the second level
• Model 3: a directed graphical model without hidden variables
• GMM: this model does not encode the relationship between

building attributes

Application I: Building Synthesis

33

Application II: Building Completion

34

Application II: Building Completion

35

Application II: Building Completion

36

Completion 1 Completion 2 Completion 3 Input

Incomplete building model Backside of each building model

Discussions

• Special building structures: garages and porches

37

garage porch parcel

Limitations

• Our model does not consider
• appearance of the building: color and texture
• context of the building: parcel shape, parcel slope, etc.
• No guarantee to find a global minimum

Limitations

• Our model does not consider
• appearance of the building: color and texture
• context of the building: parcel shape, parcel slope, etc.
• No guarantee to find a global minimum

38

Conclusions

• We propose a new framework to

model the exterior of residential buildings.

• We demonstrate our framework by
• synthesizing new building models
• completing observed building models

Building synthesis Building completion

Thank you!

39

More details about this project are available at Acknowledgements:

Anonymous reviewers

Award No. OCRF-2014-CRG3-62140401