Structure Completion of Facade Layouts Lubin Fan 1,2 , Przemyslaw Musialski 3 , Ligang Liu 4 , Peter Wonka 1,5 1 King Abdullah University of Science and Technology 2 Zhejiang University 3 Vienna University of Technology 4 University of Science and Technology of China 5 Arizona State University
Completing A Layout ? 2
Challenges • We cannot only rely on observations. • We need additional information. observation completion 3
This Work • Two sources of information observation database • A statistical model evaluates layouts. • A planning algorithm generates candidates. 4
Related Work • Structural image inpainting Structure propagation Statistics patch offsets [Sun et al. 2005] [He and Sun 2012] Texture synthesis Planar structure guidance [Dai et al. 2013] [Huang et al. 2014] They cannot complete facade with large missing regions. 5
Related Work • Facade modeling Metropolis procedural modeling Single view reconstruction [Talton et al. 2011] [Koutsourakis et al. 2011] Structure-preserving retargeting Procedural facade variation Tiled patterns [Lin et al. 2011] [Bao et al. 2013] [Yeh et al. 2013] They cannot generate facade layouts consistent with given observations. 6
Related Work • Facade analysis Procedural modeling Shape grammar parsing Adaptive partitioning [Müller et al. 2007] [Teboul et al. 2011] [Shen et al. 2011] Rank-one approximation Symmetry maximization Inverse procedural modeling [Yang et al. 2012] [Zhang et al. 2013] [Wu et al. 2014] 7
Facade Representation • Grid layout - 𝐻 Example Grid : Parameters of : Parameters of 𝑓 : (𝑓. 𝑦, 𝑓. 𝑧) . 𝑦 𝑗 = ⋯ ; . 𝑠𝑝𝑥𝑡 = 2; . 𝑑𝑝𝑚𝑣𝑛𝑜𝑡 = 4; . 𝑦 0 = 2.0; (𝑓. 𝑥, 𝑓. ℎ) . 𝑥𝑗𝑒𝑢ℎ = ⋯ ; . ℎ𝑓𝑗ℎ𝑢 = ⋯ ; . 𝑧 0 = 3.0; 𝑓. 𝑚𝑏𝑐𝑓𝑚 8
Facade Dataset • 100 facade images • Box abstraction • Statistics of elements and grids 9
Overview Statistical Model Candidate Generation Input Factor Graph Planning Algorithm 10
A Statistical Model for Facade Layouts
A Good Completion • Criteria • It satisfies some constraints. • It is consistent with the observations and the layouts in database. • Likelihood of a facade layout 𝑔 𝑔 𝑏 𝐻 = ln 𝑞 𝑏 (𝐻) 𝑏 𝐻 = ln 𝑞 𝑏 (𝐻) 𝑄 𝑏 : distribution of the grid attributes in the database 𝐻 : grid layout 12
Unary Grid Functions • Element aspect ratio - 𝑔 𝑏𝑡 () • Element spacing - 𝑔 𝑓𝑒 () • Grid regularity - 𝑔 𝑠 () • Grid completeness - 𝑔 𝑑 () Element aspect ratio: Grid completeness: Grid regularity: Element spacing: 𝑓. ℎ𝑓𝑗ℎ𝑢 𝑓 𝑠𝑓 𝑊 𝑒 𝑊 𝑓. 𝑥𝑗𝑒𝑢ℎ 𝑑𝑝𝑛𝑞() 𝑠𝑓 𝐼 () 𝑒 𝐼 13
Binary Grid Functions • Pattern of interleaved grids ‐ 𝑔 𝑞 ( 𝑗 , 𝑘 ) • Grid alignment ‐ 𝑔 𝑏 ( 𝑗 , 𝑘 ) Grid alignment: Pattern of interleaved grids: 𝑏𝑞( 𝑗 , 𝑘 ) pattern: AB 14
Global Grid Functions • Element compatibility - 𝑔 𝑓𝑑 (𝐻) • Grid coverage - 𝑔 𝑑 (𝐻) • Facade border - 𝑔 𝑔𝑐 (𝐻) • Facade symmetry - 𝑔 𝑔𝑡 (𝐻) Grid coverage: Element compatibility: Facade symmetry: Facade border: O 𝑐𝑡𝑓𝑠𝑤𝑓𝑒 𝐹𝑚𝑓𝑛𝑓𝑜𝑢𝑡 𝑃𝐹 = { , , , } 𝐻𝑚𝑝𝑐𝑏𝑚 𝐷𝑝𝑜𝑡𝑗𝑡𝑢𝑓𝑜𝑑𝑧 𝑇𝑓𝑢 (𝐻𝐷𝑇) 15
Factor Graph • Factors ℱ 𝑣𝑜𝑏𝑠𝑧 𝑗 = exp 𝑥 𝑏𝑡 𝑔 𝑏𝑡 𝑗 + 𝑥 𝑓𝑒 𝑔 𝑓𝑒 𝑗 + 𝑥 𝑠 𝑔 𝑠 𝑗 + 𝑥 𝑑 𝑔 𝑑 𝑗 ℱ 𝑐𝑗𝑜𝑏𝑠𝑧 𝑗 , 𝑘 = exp 𝑥 𝑞 𝑔 𝑞 𝑗 , 𝑘 + 𝑥 𝑏 𝑔 𝑏 𝑗 , 𝑘 ℱ 𝑚𝑝𝑐𝑏𝑚 (𝐻) = exp 𝑥 𝑓𝑑 𝑔 𝑓𝑑 𝐻 + 𝑥 𝑑 𝑔 𝑑 𝐻 + 𝑥 𝑔𝑐 𝑔 𝑔𝑐 𝐻 + 𝑥 𝑔𝑡 𝑔 𝑔𝑡 𝐻 16
Factor Graph • The overall probability 1 𝑞 𝐻 𝒙 = 𝒙 𝑎(ℱ, 𝒙) ℱ 𝑇𝑑𝑝𝑞𝑓 ℱ (𝐻) 𝒙 ℱ variables connected to factor ℱ the partition function • Weight learning - 𝒙 • Maximum likelihood parameter estimation 17
Structure Candidate Generation
Planning Algorithm • Value of state 𝑡 using Bellman’s equation reward of s transition probabilities 𝑓′ 𝑏 = 𝜌(𝑡, 𝝁) 𝑡 𝑡’ 19
Planning Algorithm • Optimal policy • Actions consist of adding one single element. 𝑏 = 𝜌(𝑡, 𝝁) s s ’ 20
Policy Design • Policy for adding an element: 𝜌(𝑡, 𝝁) 𝝁 = { } 𝜇 0 , 𝜇 1 , 𝜇 2 , 𝜇 3 , 𝜇 4 , 𝜇 5 , 𝜇 6 , 𝜇 7 , 𝜇 8 , 𝜇 9 , 𝜇 10 • Seed element ( 𝑓 𝑡 ) selection • Extension direction 𝑓 𝑡 • Extension spacing 𝑒 • Extension label 𝑓′ • Other parameters • Snapping • Symmetric copying 21
Policy Optimization • For each facade • Genetic algorithm • Initial policies are learnt from the database. Crossover Mutation 𝝁 𝒃 = {… , 𝜇 𝑗 𝑏 , … } 𝝁 = {… , 𝜇 𝑘 , … } 𝝁 𝒄 = {… , 𝜇 𝑗 𝑐 , … } 𝜇 𝑘 ⟵ 𝜇 𝑘 + 𝑒, 𝑒~𝒪(0, 𝜏) 22
Policy Optimization a completion with a a completion using observation fixed specified policy policy optimization 23
Results and Applications
Results • Completion results influenced by the number of observed elements stylized visualization ground truth completions 25
Results • Completions of incoherent observations. ground truth observation completion 26
An Application 27
Evaluation I: Structure Completion • Completion ranking test Which of two possible completions is more plausible? 1. A is more plausible. 2. B is more plausible. 3. They look the same. B A 28
Evaluation I: Structure Completion • Ground truth data received 31.5%. • Our completion received 40.2% . • Both equally received 28.3% of all votes. Ground truth is more plausible. The completion is more plausible. They look the same. 29
Evaluation II: Scoring functions • Leave-one-out test observation all terms included aspect ratio term excluded spacing term excluded regularity term excluded completeness term excluded pattern term excluded alignment term excluded 30 border term excluded compatibility term excluded coverage term excluded
Evaluation III: Comparison • Comparison to simulated annealing ground truth observation simulated our completion annealing 31
Limitation • Our statistical model only considers simple pattern. ground truth observation completion 32
Conclusions • A framework for structure completion of facade layouts • Large missing regions! • A statistical model to evaluate layouts • A planning algorithm to generate candidate layouts • An application in the area of urban reconstruction 33
Acknowledgement • Anonymous reviewers • Research grants • Visual Computing Center of KAUST • Austrian Science Funds • National Natural Science Foundation of China • One Hundred Talent Project of the Chinese Academy of Sciences • U.S. National Science Foundation 34
Thank you! More details about this project are available at: https://sites.google.com/site/lubinfan/publications/2014-facade-completion
Recommend
More recommend
