Structure Completion of Facade Layouts Lubin Fan 1,2 , Przemyslaw - - PowerPoint PPT Presentation

β–Ά
structure completion of facade layouts
SMART_READER_LITE
LIVE PREVIEW

Structure Completion of Facade Layouts Lubin Fan 1,2 , Przemyslaw - - PowerPoint PPT Presentation

Feb 02, 2024 978 likes β€’1.34k views

Structure Completion of Facade Layouts Lubin Fan 1,2 , Przemyslaw Musialski 3 , Ligang Liu 4 , Peter Wonka 1,5 1 King Abdullah University of Science and Technology 2 Zhejiang University 3 Vienna University of Technology 4 University of Science and

slide-1
SLIDE 1

Structure Completion of Facade Layouts

Lubin Fan1,2, Przemyslaw Musialski3, Ligang Liu4, Peter Wonka1,5

1 King Abdullah University of Science and Technology 2 Zhejiang University 3 Vienna University of Technology 4 University of Science and Technology of China 5 Arizona State University

slide-2
SLIDE 2

Completing A Layout

?

2

slide-3
SLIDE 3

Challenges

  • We cannot only rely on observations.
  • We need additional information.
  • bservation

completion

3

slide-4
SLIDE 4

This Work

  • Two sources of information
  • bservation

database

  • A statistical model evaluates layouts.
  • A planning algorithm generates candidates.

4

slide-5
SLIDE 5

Related Work

  • Structural image inpainting

Structure propagation [Sun et al. 2005] Statistics patch offsets [He and Sun 2012] Texture synthesis [Dai et al. 2013]

They cannot complete facade with large missing regions.

Planar structure guidance [Huang et al. 2014]

5

slide-6
SLIDE 6

Related Work

  • Facade modeling

Metropolis procedural modeling [Talton et al. 2011] Single view reconstruction [Koutsourakis et al. 2011] Structure-preserving retargeting [Lin et al. 2011] Procedural facade variation [Bao et al. 2013] Tiled patterns [Yeh et al. 2013]

They cannot generate facade layouts consistent with given observations.

6

slide-7
SLIDE 7

Related Work

  • Facade analysis

Symmetry maximization [Zhang et al. 2013] Rank-one approximation [Yang et al. 2012] Procedural modeling [MΓΌller et al. 2007] Adaptive partitioning [Shen et al. 2011] Shape grammar parsing [Teboul et al. 2011] Inverse procedural modeling [Wu et al. 2014]

7

slide-8
SLIDE 8

Facade Representation

  • Grid layout - 𝐻

Parameters of 𝑓: (𝑓. 𝑦, 𝑓. 𝑧) 𝑓. π‘šπ‘π‘π‘“π‘š (𝑓. π‘₯, 𝑓. β„Ž) Parameters of 𝑕: 𝑕. 𝑦0 = 2.0; 𝑕. 𝑧0 = 3.0; 𝑕. 𝑦𝑗 = β‹― ; 𝑕. 𝑠𝑝π‘₯𝑑 = 2; 𝑕. π‘‘π‘π‘šπ‘£π‘›π‘œπ‘‘ = 4; 𝑕. π‘₯π‘—π‘’π‘’β„Ž = β‹― ;𝑕. β„Žπ‘“π‘—π‘•β„Žπ‘’ = β‹― ; Example Grid 𝑕:

8

slide-9
SLIDE 9

Facade Dataset

  • 100 facade images
  • Box abstraction
  • Statistics of elements

and grids

9

slide-10
SLIDE 10

Overview

Input Statistical Model Factor Graph Candidate Generation Planning Algorithm

10

slide-11
SLIDE 11

A Statistical Model for Facade Layouts

slide-12
SLIDE 12

A Good Completion

  • Criteria
  • It satisfies some constraints.
  • It is consistent with the observations

and the layouts in database.

  • Likelihood of a facade layout

𝑔

𝑏 𝐻 = ln π‘žπ‘(𝐻) 𝐻: grid layout 𝑄

𝑏: distribution of the grid attributes in the database

𝑔

𝑏 𝐻 = ln π‘žπ‘(𝐻)

12

slide-13
SLIDE 13

Unary Grid Functions

  • Element aspect ratio - 𝑔

𝑏𝑑(𝑕)

  • Element spacing - 𝑔

𝑓𝑒(𝑕)

  • Grid regularity - 𝑔

𝑕𝑠(𝑕)

  • Grid completeness - 𝑔

𝑕𝑑(𝑕)

𝑓 𝑓. π‘₯π‘—π‘’π‘’β„Ž 𝑓. β„Žπ‘“π‘—π‘•β„Žπ‘’ Element aspect ratio: 𝑒𝐼 π‘’π‘Š Element spacing: 𝑠𝑓𝑕𝐼(𝑕) π‘ π‘“π‘•π‘Š 𝑕 Grid regularity: π‘‘π‘π‘›π‘ž(𝑕) Grid completeness:

13

slide-14
SLIDE 14

Binary Grid Functions

  • Pattern of interleaved grids

‐ 𝑔

π‘•π‘ž(𝑕𝑗, π‘•π‘˜)

  • Grid alignment

‐ 𝑔

𝑕𝑏(𝑕𝑗, π‘•π‘˜) pattern: AB Pattern of interleaved grids: π‘•π‘π‘ž(𝑕𝑗, π‘•π‘˜) Grid alignment:

14

slide-15
SLIDE 15

Global Grid Functions

  • Element compatibility - 𝑔

𝑓𝑑(𝐻)

  • Grid coverage - 𝑔

𝑕𝑑(𝐻)

  • Facade border - 𝑔

𝑔𝑐(𝐻)

  • Facade symmetry - 𝑔

𝑔𝑑(𝐻)

Element compatibility: O𝑐𝑑𝑓𝑠𝑀𝑓𝑒 πΉπ‘šπ‘“π‘›π‘“π‘œπ‘’π‘‘ 𝑃𝐹 = { , , , } π»π‘šπ‘π‘π‘π‘š π·π‘π‘œπ‘‘π‘—π‘‘π‘’π‘“π‘œπ‘‘π‘§ 𝑇𝑓𝑒 (𝐻𝐷𝑇) Grid coverage: Facade border: Facade symmetry:

15

slide-16
SLIDE 16

Factor Graph

  • Factors

β„±

π‘£π‘œπ‘π‘ π‘§ 𝑕𝑗 = exp π‘₯𝑏𝑑𝑔 𝑏𝑑 𝑕𝑗 + π‘₯𝑓𝑒𝑔 𝑓𝑒 𝑕𝑗 + π‘₯𝑕𝑠𝑔 𝑕𝑠 𝑕𝑗 + π‘₯𝑕𝑑𝑔 𝑕𝑑 𝑕𝑗

β„±π‘π‘—π‘œπ‘π‘ π‘§ 𝑕𝑗, π‘•π‘˜ = exp π‘₯π‘•π‘žπ‘”

π‘•π‘ž 𝑕𝑗, π‘•π‘˜ + π‘₯𝑕𝑏𝑔 𝑕𝑏 𝑕𝑗, π‘•π‘˜

β„±π‘•π‘šπ‘π‘π‘π‘š(𝐻) = exp π‘₯𝑓𝑑𝑔

𝑓𝑑 𝐻 + π‘₯𝑕𝑑𝑔 𝑕𝑑 𝐻 + π‘₯𝑔𝑐𝑔 𝑔𝑐 𝐻 + π‘₯𝑔𝑑𝑔 𝑔𝑑 𝐻

16

slide-17
SLIDE 17

Factor Graph

  • The overall probability

π‘ž 𝐻 𝒙 = 1 π‘Ž(β„±, 𝒙)

β„±

β„± π‘‡π‘‘π‘π‘žπ‘“β„±(𝐻)

the partition function variables connected to factor β„±

  • Weight learning - 𝒙
  • Maximum likelihood parameter estimation

17

𝒙 𝒙

slide-18
SLIDE 18

Structure Candidate Generation

slide-19
SLIDE 19

Planning Algorithm

  • Value of state 𝑑 using Bellman’s equation

𝑑 𝑑’

𝑓′

𝑏 = 𝜌(𝑑, 𝝁)

reward of s transition probabilities

19

slide-20
SLIDE 20

Planning Algorithm

  • Optimal policy

s s’

𝑏 = 𝜌(𝑑, 𝝁)

  • Actions consist of adding one single element.

20

slide-21
SLIDE 21

Policy Design

  • Policy for adding an element: 𝜌(𝑑, 𝝁)

𝑓𝑑 𝑓′ 𝑒

  • Seed element (𝑓𝑑) selection
  • Extension direction
  • Extension spacing
  • Extension label
  • Other parameters
  • Snapping
  • Symmetric copying

𝝁 = { }

πœ‡0, πœ‡1, πœ‡2, πœ‡3, πœ‡6, πœ‡7, πœ‡8, πœ‡4, πœ‡5, πœ‡9, πœ‡10

21

slide-22
SLIDE 22

Policy Optimization

  • For each facade
  • Genetic algorithm
  • Initial policies are learnt from the database.

Mutation

𝝁 = {… , πœ‡π‘˜, … } πœ‡π‘˜ ⟡ πœ‡π‘˜ + 𝑒, 𝑒~π’ͺ(0, 𝜏)

Crossover

𝝁𝒃 = {… , πœ‡π‘—

𝑏, … }

𝝁𝒄 = {… , πœ‡π‘—

𝑐, … }

22

slide-23
SLIDE 23

Policy Optimization

  • bservation

a completion using policy optimization a completion with a fixed specified policy

23

slide-24
SLIDE 24

Results and Applications

slide-25
SLIDE 25

Results

  • Completion results influenced by the number of
  • bserved elements

stylized visualization ground truth completions

25

slide-26
SLIDE 26

Results

  • Completions of incoherent observations.

ground truth

  • bservation

completion

26

slide-27
SLIDE 27

An Application

27

slide-28
SLIDE 28

Evaluation I: Structure Completion

  • Completion ranking test

Which of two possible completions is more plausible?

A B

  • 1. A is more plausible.
  • 2. B is more plausible.
  • 3. They look the same.

28

slide-29
SLIDE 29

Evaluation I: Structure Completion

  • Ground truth data received 31.5%.
  • Our completion received 40.2%.
  • Both equally received 28.3% of all votes.

Ground truth is more plausible.

The completion is more plausible. They look the same.

29

slide-30
SLIDE 30

Evaluation II: Scoring functions

  • Leave-one-out test
  • bservation

all terms included

aspect ratio term excluded spacing term excluded regularity term excluded completeness term excluded pattern term excluded alignment term excluded compatibility term excluded coverage term excluded border term excluded

30

slide-31
SLIDE 31

Evaluation III: Comparison

  • Comparison to simulated annealing

ground truth

  • bservation

simulated annealing

  • ur completion

31

slide-32
SLIDE 32

32

Limitation

  • Our statistical model only considers simple pattern.

ground truth

  • bservation

completion

slide-33
SLIDE 33

Conclusions

  • A framework for structure completion of facade layouts
  • Large missing regions!
  • A statistical model to evaluate layouts
  • A planning algorithm to generate candidate layouts
  • An application in the area of urban reconstruction

33

slide-34
SLIDE 34

Acknowledgement

  • Anonymous reviewers
  • Research grants
  • Visual Computing Center of KAUST
  • Austrian Science Funds
  • National Natural Science Foundation of China
  • One Hundred Talent Project of the Chinese Academy of Sciences
  • U.S. National Science Foundation

34

slide-35
SLIDE 35

Thank you!

More details about this project are available at:

https://sites.google.com/site/lubinfan/publications/2014-facade-completion