Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay Technical University of Munich Department of Mathematics Data Innovation Lab Garching, 18. February 2020
Agenda • Introduction & Objectives • Related work • Dataset • "Feature Vector" Approach • Solution Segmentation Network o Room Proposal Extraction o Algorithm for placing icons o • Results • Conclusion & limitations
Introduction & Objectives For a lot of simple architectural tasks a person is needed One reason: Used data in the form of rasterized floorplan images Living room Bed- room Kitchen Bath Sebastian Schlegel 3
Introduction & Objectives For a lot of simple architectural tasks a person is needed One reason: Used data in the form of rasterized floorplan images Living room Bed- room Goal of this Project: To place icons (furniture, facilities) on an "empty" floorplan Kitchen Bath Sebastian Schlegel 4
Related work – Liu et al. 2017 A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach: CNN Sebastian Schlegel 5
Related work – Liu et al. 2017 A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach: • Extracting geometric and semantic information independently • M geometries CNN semantics Sebastian Schlegel 6
Related work – Liu et al. 2017 A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach: • Extracting geometric and semantic information independently • Merging both rule-based using Integer Programming • M Integer Programming geometries CNN semantics Sebastian Schlegel 7
Related work – Liu et al. 2017 Geometric information is represented through "junction points" • 4 different junctions for "openings" (windows, doors): Sebastian Schlegel 8
Related work – Liu et al. 2017 Geometric information is represented through "junction points" • 4 different junctions for "openings" (windows, doors): • 4 different junctions for icons: Sebastian Schlegel 9
Related work – Liu et al. 2017 Geometric information is represented through "junction points" • 4 different junctions for "openings" (windows, doors): • 4 different junctions for icons: • 13 different junctions for walls: In total: 21 different junction types Sebastian Schlegel 10
Related work – Liu et al. 2017 Network (modified ResNet 152) output: Geometric information: • 21 heatmaps – one regressed for every junction type Sebastian Schlegel 11
Related work – Liu et al. 2017 Network (modified ResNet 152) output: Geometric information: • 21 heatmaps – one regressed for every junction type Semantic information: • One per-pixel classification for room types Sebastian Schlegel 12
Related work – Liu et al. 2017 Network (modified ResNet 152) output: Geometric information: • 21 heatmaps – one regressed for every junction type Semantic information: • One per-pixel classification for room types • One per-pixel classification for icons, windows & doors Sebastian Schlegel 13
Related work – Kalervo et al. 2019 ("CubiCasa") Modifying the approach of Liu et al. Differences: • Applying automatic weighting to the multi-task CNN of Liu et al. • Instead of Integer Programming a heuristic is used for merging geometries Heuristic CNN semantics Sebastian Schlegel 14
Related work – Kalervo et al. 2019 ("CubiCasa") Merging geometric and semantic information • Split floorplan in a grid of rectangular cells using triplets of Junction points Sebastian Schlegel 15
Related work – Kalervo et al. 2019 ("CubiCasa") Merging geometric and semantic information • Split floorplan in a grid of rectangular cells using triplets of Junction points • Label cells applying pixel-wise maximum voting Sebastian Schlegel 16
Related work – Kalervo et al. 2019 ("CubiCasa") Merging geometric and semantic information • Split floorplan in a grid of rectangular cells using triplets of Junction points • Label cells applying pixel-wise maximum voting • Merge cells with the same label if no separating wall is in between Sebastian Schlegel 17
Agenda • Introduction & Objectives • Related work • Dataset • "Feature Vector" Approach • Solution Segmentation Network o Room Proposal Extraction o Algorithm for placing icons o • Results • Conclusion & limitations
CubiCasa Dataset Model Vladimir Yugay 19
CubiCasa input Vladimir Yugay 20
CubiCasa model generalization Model Vladimir Yugay 21
CubiCasa model evaluation Vladimir Yugay 22
CubiCasa label Vladimir Yugay 23
Input generation Vladimir Yugay 24
Label generation Vladimir Yugay 25
Input generation Vladimir Yugay 26
Input generation Vladimir Yugay 27
Dataset randomization Each image has a random texture • Each text annotation has a random font size, weight and family • Each text annotation is rotated from 0 to 11 degrees • Vladimir Yugay 28
Dataset statistics Number of room types were reduced from 65 to 8 • Only rooms with kitchen and bathroom were kept • In total 1881 images split in 1281/300/300 as train/validation/test • Vladimir Yugay 29
Feature Vector Approach – Reusing Trained Model • Geometrical features (i.e. walls) are invariant to text removal • Detect Rooms, and Find their types using walls. Type IOU (Text) IOU (No Text) Wall 65.5% 64% Kitchen 63.9% 50.6% Living Room 73.1% 24.8% Marsil Zakour 30
Feature Vector Approach – Building a Graph Marsil Zakour 31
Feature Vector Approach – Features • What Set of nodes makes a room. • Wall Features: • Geometrical • Relational • Room Features: • Aggregation/summary of walls features. • Or better use Graph Convolutions Wall features height width # rooms # walls Marsil Zakour 32
Feature Vector Approach - Stop Why we Stopped: • Rooms are not only cycles • More support for segmentation than for GCNs Marsil Zakour 33
Agenda • Introduction & Objectives • Related work • Dataset • "Feature Vector" Approach • Solution Segmentation Network o Room Proposal Extraction o Algorithm for placing icons o • Results • Conclusion & limitations
Solution Room type segmentation Classified polygons Icon Room placement polygons Openings information Vladimir Yugay 35
Segmentation network details U-Net architecture • ResNet as a backbone • 25 millions of parameters • Dice loss • Vladimir Yugay 36
Segmentation network Segmentation Prediction Label Network tensor tensor Vladimir Yugay 37
Room type segmentation Predicted image Label image Input image Vladimir Yugay 38
Openings segmentation Input image Predicted image Label image Vladimir Yugay 39
Room Proposals – New Formulation Input Proposals Marsil Zakour 40
Room Proposals - Training Marsil Zakour 41
Room Proposals – Qualitative Results Labels Predictions Inputs Room Proposal Qualitative Results Marsil Zakour 42
Predictions Fusion - Input Room Proposal input Room Type Marsil Zakour 43
Predictions Fusion - Voting and Inpaint Align Vote Inpaint Marsil Zakour 44
Predictions Fusion - Extract Polygons Polygons types Extract Polygons Marsil Zakour 45
Predictions Fusion - Extract Polygon cont. (5,5) (0,0) - (5,5) 1,1,… 1 X > 3 X <= 3 2,2,… (3,0) - (5,5) (0,0) - (3,5) 1 y 2 y > 4 y <= 4 (3,0) - (3,4) (3,4) - (5,5) (0,0) x Marsil Zakour 46
Predictions Fusion - Example Marsil Zakour 47
Algorithm for placing icons The task of placing icons is rule-based : • "don´t place an icon in front of a door" Living room Bed- • "place a bath tube in a bath" room • … Kitchen Bath Sebastian Schlegel 48
Algorithm for placing icons The task of placing icons is rule-based : • "don´t place an icon in front of a door" Living room Bed- • "place a bath tube in a bath" room • … "creative" (inconclusive) : • There is not one optimal solution Kitchen • Multiple constellations are possible • "Best choice" can depend on "taste" of a person Bath Sebastian Schlegel 49
Algorithm for placing icons Information used for placing icons: door • Geometry of a room window • Type of a room • Location of windows and doors bedside table bed Sebastian Schlegel 50
Algorithm for placing icons Information used for placing icons: • Distance from wall • If pixel in corner • Dist. From window • Dist. from door Sebastian Schlegel 51
Recommend
More recommend
