Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay - - PowerPoint PPT Presentation

▶
sparkmap
SMART_READER_LITE
LIVE PREVIEW

Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay - - PowerPoint PPT Presentation

Feb 07, 2024 124 likes •924 views

Sparkmap Marsil Zakour, Sebastian Schlegel, Vladimir Yugay Technical University of Munich Department of Mathematics Data Innovation Lab Garching, 18. February 2020 Agenda Introduction & Objectives Related work Dataset

slide-1
SLIDE 1

Sparkmap

Marsil Zakour, Sebastian Schlegel, Vladimir Yugay Technical University of Munich Department of Mathematics Data Innovation Lab Garching, 18. February 2020

slide-2
SLIDE 2

Agenda

  • Introduction & Objectives
  • Related work
  • Dataset
  • "Feature Vector" Approach
  • Solution
  • Segmentation Network
  • Room Proposal Extraction
  • Algorithm for placing icons
  • Results
  • Conclusion & limitations
slide-3
SLIDE 3

Sebastian Schlegel

For a lot of simple architectural tasks a person is needed One reason: Used data in the form of rasterized floorplan images

Introduction & Objectives

3

Kitchen Living room Bed- room Bath

slide-4
SLIDE 4

Sebastian Schlegel

Introduction & Objectives

4

Kitchen Living room Bed- room Bath For a lot of simple architectural tasks a person is needed One reason: Used data in the form of rasterized floorplan images Goal of this Project: To place icons (furniture, facilities) on an "empty" floorplan

slide-5
SLIDE 5

Sebastian Schlegel

Related work – Liu et al. 2017

5

A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach:

CNN

slide-6
SLIDE 6

Sebastian Schlegel

Related work – Liu et al. 2017

6

A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach:

  • Extracting geometric and semantic information independently
  • M

geometries semantics

CNN

slide-7
SLIDE 7

Sebastian Schlegel

Related work – Liu et al. 2017

7

A multi-step process using a convolutional neural network (CNN) for automatically parsing rasterized floorplan images Approach:

  • Extracting geometric and semantic information independently
  • Merging both rule-based using Integer Programming
  • M

geometries semantics

CNN

Integer Programming

slide-8
SLIDE 8

Sebastian Schlegel 8

Geometric information is represented through "junction points"

  • 4 different junctions for "openings" (windows, doors):

Related work – Liu et al. 2017

slide-9
SLIDE 9

Sebastian Schlegel 9

Geometric information is represented through "junction points"

  • 4 different junctions for "openings" (windows, doors):
  • 4 different junctions for icons:

Related work – Liu et al. 2017

slide-10
SLIDE 10

Sebastian Schlegel 10

Geometric information is represented through "junction points"

  • 4 different junctions for "openings" (windows, doors):
  • 4 different junctions for icons:
  • 13 different junctions for walls:

In total: 21 different junction types

Related work – Liu et al. 2017

slide-11
SLIDE 11

Sebastian Schlegel 11

Network (modified ResNet 152) output: Geometric information:

  • 21 heatmaps – one regressed for every junction type

Related work – Liu et al. 2017

slide-12
SLIDE 12

Sebastian Schlegel 12

Related work – Liu et al. 2017

Network (modified ResNet 152) output: Geometric information:

  • 21 heatmaps – one regressed for every junction type

Semantic information:

  • One per-pixel classification for room types
slide-13
SLIDE 13

Sebastian Schlegel 13

Related work – Liu et al. 2017

Network (modified ResNet 152) output: Geometric information:

  • 21 heatmaps – one regressed for every junction type

Semantic information:

  • One per-pixel classification for room types
  • One per-pixel classification for icons, windows & doors
slide-14
SLIDE 14

Sebastian Schlegel

Related work – Kalervo et al. 2019 ("CubiCasa")

14

Modifying the approach of Liu et al. Differences:

  • Applying automatic weighting to the multi-task CNN of Liu et al.
  • Instead of Integer Programming a heuristic is used for merging

geometries semantics

CNN

Heuristic

slide-15
SLIDE 15

Sebastian Schlegel

Related work – Kalervo et al. 2019 ("CubiCasa")

15

Merging geometric and semantic information

  • Split floorplan in a grid of rectangular cells using triplets
  • f Junction points
slide-16
SLIDE 16

Sebastian Schlegel

Related work – Kalervo et al. 2019 ("CubiCasa")

16

Merging geometric and semantic information

  • Split floorplan in a grid of rectangular cells using triplets
  • f Junction points
  • Label cells applying pixel-wise maximum voting
slide-17
SLIDE 17

Sebastian Schlegel

Related work – Kalervo et al. 2019 ("CubiCasa")

17

Merging geometric and semantic information

  • Split floorplan in a grid of rectangular cells using triplets
  • f Junction points
  • Label cells applying pixel-wise maximum voting
  • Merge cells with the same label if no separating wall is

in between

slide-18
SLIDE 18

Agenda

  • Introduction & Objectives
  • Related work
  • Dataset
  • "Feature Vector" Approach
  • Solution
  • Segmentation Network
  • Room Proposal Extraction
  • Algorithm for placing icons
  • Results
  • Conclusion & limitations
slide-19
SLIDE 19

Vladimir Yugay

CubiCasa Dataset

19

Model

slide-20
SLIDE 20

Vladimir Yugay

CubiCasa input

20

slide-21
SLIDE 21

Vladimir Yugay

CubiCasa model generalization

21

Model

slide-22
SLIDE 22

Vladimir Yugay

CubiCasa model evaluation

22

slide-23
SLIDE 23

Vladimir Yugay

CubiCasa label

23

slide-24
SLIDE 24

Vladimir Yugay

Input generation

24

slide-25
SLIDE 25

Vladimir Yugay

Label generation

25

slide-26
SLIDE 26

Vladimir Yugay

Input generation

26

slide-27
SLIDE 27

Vladimir Yugay

Input generation

27

slide-28
SLIDE 28

Vladimir Yugay

Dataset randomization

28

  • Each image has a random texture
  • Each text annotation has a random font size, weight and family
  • Each text annotation is rotated from 0 to 11 degrees
slide-29
SLIDE 29

Vladimir Yugay

Dataset statistics

29

  • Number of room types were reduced from 65 to 8
  • Only rooms with kitchen and bathroom were kept
  • In total 1881 images split in 1281/300/300 as train/validation/test
slide-30
SLIDE 30

Marsil Zakour

  • Geometrical features (i.e. walls) are invariant to

text removal

  • Detect Rooms, and Find their types using walls.

Feature Vector Approach – Reusing Trained Model

30

Type IOU (Text) IOU (No Text) Wall 65.5% 64% Kitchen 63.9% 50.6% Living Room 73.1% 24.8%

slide-31
SLIDE 31

Marsil Zakour

Feature Vector Approach – Building a Graph

31

slide-32
SLIDE 32

Marsil Zakour

  • What Set of nodes makes a room.
  • Wall Features:
  • Geometrical
  • Relational
  • Room Features:
  • Aggregation/summary of walls features.
  • Or better use Graph Convolutions

Feature Vector Approach – Features

32

Wall features height width # rooms # walls

slide-33
SLIDE 33

Marsil Zakour

Why we Stopped:

  • Rooms are not only cycles
  • More support for segmentation than for GCNs

Feature Vector Approach - Stop

33

slide-34
SLIDE 34

Agenda

  • Introduction & Objectives
  • Related work
  • Dataset
  • "Feature Vector" Approach
  • Solution
  • Segmentation Network
  • Room Proposal Extraction
  • Algorithm for placing icons
  • Results
  • Conclusion & limitations
slide-35
SLIDE 35

Vladimir Yugay

Solution

35

Room type segmentation Room polygons Openings information Classified polygons Icon placement

slide-36
SLIDE 36

Vladimir Yugay

Segmentation network details

36

  • U-Net architecture
  • ResNet as a backbone
  • 25 millions of parameters
  • Dice loss
slide-37
SLIDE 37

Vladimir Yugay

Segmentation network

37 Segmentation Network Prediction tensor Label tensor

slide-38
SLIDE 38

Vladimir Yugay

Room type segmentation

38 Input image Predicted image Label image

slide-39
SLIDE 39

Vladimir Yugay

Openings segmentation

39 Input image Predicted image Label image

slide-40
SLIDE 40

Marsil Zakour

Room Proposals – New Formulation

40

Input Proposals

slide-41
SLIDE 41

Marsil Zakour

Room Proposals - Training

41

slide-42
SLIDE 42

Marsil Zakour

Room Proposals – Qualitative Results

42

Room Proposal Qualitative Results Labels Predictions Inputs

slide-43
SLIDE 43

Marsil Zakour

Predictions Fusion - Input

43

input Room Proposal Room Type

slide-44
SLIDE 44

Marsil Zakour

Predictions Fusion - Voting and Inpaint

44

Align Vote Inpaint

slide-45
SLIDE 45

Marsil Zakour

Predictions Fusion - Extract Polygons

45

Extract Polygons Polygons types

slide-46
SLIDE 46

Marsil Zakour

Predictions Fusion - Extract Polygon cont.

46

1 2 1 (0,0) - (5,5) y x (0,0) - (3,5) (3,0) - (5,5) (3,0) - (3,4) (3,4) - (5,5) X <= 3 X > 3 y <= 4 y > 4 1,1,… 2,2,… (0,0) (5,5)

slide-47
SLIDE 47

Marsil Zakour

Predictions Fusion - Example

47

slide-48
SLIDE 48

Sebastian Schlegel

Algorithm for placing icons

48

Kitchen Living room Bed- room Bath The task of placing icons is rule-based:

  • "don´t place an icon in front of a door"
  • "place a bath tube in a bath"
  • …
slide-49
SLIDE 49

Sebastian Schlegel

Algorithm for placing icons

49

Kitchen Living room Bed- room Bath The task of placing icons is rule-based:

  • "don´t place an icon in front of a door"
  • "place a bath tube in a bath"
  • …

"creative" (inconclusive):

  • There is not one optimal solution
  • Multiple constellations are possible
  • "Best choice" can depend on "taste" of a person
slide-50
SLIDE 50

Sebastian Schlegel

Algorithm for placing icons

50

Information used for placing icons:

  • Geometry of a room
  • Type of a room
  • Location of windows and doors

bed bedside table window door

slide-51
SLIDE 51

Sebastian Schlegel

Algorithm for placing icons

51

Information used for placing icons:

  • Distance from wall
  • If pixel in corner
  • Dist. From window
  • Dist. from door
slide-52
SLIDE 52

Sebastian Schlegel

Algorithm for placing icons

52

Information used for placing icons:

  • Distance from wall
  • If pixel in corner
  • Dist. from window
  • Dist. from door
  • Dist. from icon – long side
  • Dist. from icon – short side
  • eucl. Dist. from icon edge
slide-53
SLIDE 53

Sebastian Schlegel

Algorithm for placing icons

53

Information used for placing icons:

  • Distance from wall
  • If pixel in corner
  • Dist. from window
  • Dist. from door
  • Dist. from icon – long side
  • Dist. from icon – short side
  • eucl. Dist. from icon edge

One distance vector for every pixel

slide-54
SLIDE 54

Sebastian Schlegel

Algorithm for placing icons

54

Rules for placing icons: Implemented through a weighting vector

  • Weights can be positive (minimize dist.)
  • Or negative (maximize dist.)

weighting vector

slide-55
SLIDE 55

Sebastian Schlegel

Algorithm for placing icons

55

Rules for placing icons: Implemented through a weighting vector

  • Weights can be positive (minimize dist.)
  • Or negative (maximize dist.)

distance vector weighting vector

slide-56
SLIDE 56

Sebastian Schlegel

Algorithm for placing icons

56

Rules for placing icons: Implemented through a weighting vector

  • Weights can be positive (minimize dist.)
  • Or negative (maximize dist.)

distance vector weighting vector

Value for each pixel

slide-57
SLIDE 57

Sebastian Schlegel

Algorithm for placing icons

57

Rules for placing icons: Implemented through a weighting vector

  • Weights can be positive (minimize dist.)
  • Or negative (maximize dist.)
  • Additional conditions e.g.:
  • Placing "dummies" between icons allows for more

complex constellations

distance vector weighting vector

Value for each pixel

slide-58
SLIDE 58

Sebastian Schlegel

Algorithm for placing icons

58

Searching for best icon spots: Grid search over every possible icon spot

  • An area value is calculated for every spot
slide-59
SLIDE 59

Sebastian Schlegel

Algorithm for placing icons

59

Searching for best icon spots: Grid search over every possible icon spot

  • An area value is calculated for every spot
slide-60
SLIDE 60

Sebastian Schlegel

Algorithm for placing icons

60

Searching for best icon spots: Grid search over every possible icon spot

  • An area value is calculated for every spot
  • Repeated for icon being rotated by 90 degrees
slide-61
SLIDE 61

Sebastian Schlegel

Algorithm for placing icons

61

Searching for best icon spots: Grid search over every possible icon spot

  • An area value is calculated for every spot
  • Repeated for icon being rotated by 90 degrees
slide-62
SLIDE 62

Sebastian Schlegel

Algorithm for placing icons

62

Searching for best icon spots: Grid search over every possible icon spot

  • An area value is calculated for every spot
  • Repeated for icon being rotated by 90 degrees

Random factor for higher variability

  • Small random value for every area to randomly pick
  • ne of the optimal spots
slide-63
SLIDE 63

Sebastian Schlegel

Algorithm for placing icons

63

Searching for best icon spots: Grid search over every possible icon spot

  • An area value is calculated for every spot
  • Repeated for icon being rotated by 90 degrees

Random factor for higher variability

  • Small random value for every area to randomly pick
  • ne of the optimal spots
  • Order in which icons are placed is randomly, taking

rules into account (e.g. place chair after table)

slide-64
SLIDE 64

Sebastian Schlegel

Algorithm for placing icons

64

"Learning of rules":

  • Rules were first set due to common sense
  • Rules were fine-tuned according to feedback
  • Outlook: learn user preferences in a loop of human-

machine-interactions

slide-65
SLIDE 65

Sebastian Schlegel

Results

65

slide-66
SLIDE 66

Sebastian Schlegel

Conclusion & Limitations

66

Initial goals were met Limitations:

  • Room segmentation can´t tolerate text rotated by 90 degrees
  • Room proposal heavily depends on good proposals segmentation
  • Icon placing faces problems when dealing with complex room shapes
slide-67
SLIDE 67

Backup

67

slide-68
SLIDE 68
  • Dr. rer. nat. Erika Mustermann (TUM) | kann beliebig erweitert werden | Infos mit Strich trennen

Computer-based processing of floorplan images is well researched in pattern regocnition Early approaches mainly rely on low-level image processing heuristics:

  • Separating textual data from graphical
  • Detecting and grouping lines
  • Overcoming gaps through polygonal approximation or edge-linking

To increase performance, recent works applied deep convolutional neural networks (CNNs)

68

Related work – early approaches

slide-69
SLIDE 69
  • Dr. rer. nat. Erika Mustermann (TUM) | kann beliebig erweitert werden | Infos mit Strich trennen

69

Merging geometric and semantic information applying predefined constraints 5 different constraints

  • e.g. loop constraint

− High level: exterior boundary of a room must form a closed loop − Enforced locally: For each pair of walls the room type must be the same Constraints are applied using Integer programming Result: performance – especially for geometries – increases significantly

Related work – Liu et al. 2017

slide-70
SLIDE 70

Sebastian Schlegel

Performance

70

Related work – Liu et al.

slide-71
SLIDE 71

Sebastian

Performance

71

Related work – Kalervo et al.

slide-72
SLIDE 72

Marsil Zakour

Rotated Input

72

slide-73
SLIDE 73

Marsil Zakour

(image from https://datawarrior.wordpress.com/2018/08/08/graph-convolutional-neural- network-part-i/)

Graph Convolutional Networks (GCNs)

73

slide-74
SLIDE 74
  • Dr. rer. nat. Erika Mustermann (TUM) | kann beliebig erweitert werden | Infos mit Strich trennen

F1-loss

74

slide-75
SLIDE 75

Marsil Zakour

Voting

75

slide-76
SLIDE 76

Vladimir Yugay

U-Net

76

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.

slide-77
SLIDE 77

Vladimir Yugay

ResNet

77

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.

slide-78
SLIDE 78

Vladimir Yugay

Dice Loss

78

Dice loss