Learning to Group Discrete Graphical Patterns Zhaoliang Lun* a - - PowerPoint PPT Presentation

▶
learning to group discrete graphical patterns
SMART_READER_LITE
LIVE PREVIEW

Learning to Group Discrete Graphical Patterns Zhaoliang Lun* a - - PowerPoint PPT Presentation

Oct 30, 2022 313 likes •891 views

Learning to Group Discrete Graphical Patterns Zhaoliang Lun* a Changqing Zou* b Haibin Huang a Evangelos Kalogerakis a Ping Tan b Marie-Paule Cani c Hao Zhang b a UMASS Amherst b Simon Fraser University c Ecole Polytechnique Pattern Grouping Problem

slide-1
SLIDE 1

Zhaoliang Lun*a Changqing Zou*b Haibin Huang a Evangelos Kalogerakis a Ping Tan b Marie-Paule Cani c Hao Zhang b

Learning to Group Discrete Graphical Patterns

a UMASS Amherst

c Ecole Polytechnique b Simon Fraser University

slide-2
SLIDE 2

Pattern Grouping Problem: motivation

Pattern Grouping

slide-3
SLIDE 3

Pattern Grouping Problem: motivation

Pattern Grouping

Symmetry Similarity

Continuity & Proximity

slide-4
SLIDE 4

Input Pattern Symmetry rule wins Similarity rule wins

Challenges (1): conflicting grouping principles

slide-5
SLIDE 5

Challenges (2): various noises

slide-6
SLIDE 6

Challenges (2): various noises

Inaccurate Symmetry Loose Similarity

slide-7
SLIDE 7

Challenges (3): Rich Variations and Complexity

slide-8
SLIDE 8

q Pattern Editing

Applications of Pattern Grouping

Inverse Procedural Modeling by Automatic Generation of L-systems. O. Stava, et al. 2010

slide-9
SLIDE 9

q Pattern Editing q Pattern Exploration

Applications of Pattern Grouping

PATEX: Exploring Pattern Variations. P. Guerrero, et al. 2016

slide-10
SLIDE 10

q Pattern Editing q Pattern Exploration q Layout Optimization

Applications of Pattern Grouping

GACA: Group-Aware Command-based Arrangement of Graphic Elements. P. Xu, et al. 2015

slide-11
SLIDE 11

Related Work: Model & Rule Driven

q Gestalt-based pattern grouping

Ø Conjoining Gestalt Rules for Abstraction of Architectural Drawings. Nan et al. TOG, 2011. Ø Perceptual grouping by selection of a logically minimal model, Feldman, ICCV, 2003. Ø The whole is equal to the sum of its parts: A probabilistic model of grouping by proximity and similarity in regular patterns, Kubovy & Berg. Psychological Review, 2008.

q Symmetry-based pattern grouping

Ø Folding meshes: hierarchical mesh segmentation based on planar symmetry. Simari et al. SGP, 2006. Ø Co-Hierarchical Analysis of Shape Structures. O. Kaick et al. TOG, 2013. Ø Symmetry Hierarchy of Man-Made Objects. Wang et al. Computer Graphics Forum, CGF, 2011. Ø Layered Analysis of Irregular Facades via Symmetry Maximization. Zhang et al. TOG, 2013.

slide-12
SLIDE 12

Related Work: Gestalt-Based Pattern Grouping

Conjoining gestalt rules for abstraction of architectural drawings, Nan et al. TOG, 2011.

Group & Simplify

slide-13
SLIDE 13

Nan’s Strategy: model-driven

q Hand-engineering rules to quantify grouping models q Hand-tuning relative importance of rules Nan et al ’s strategy

slide-14
SLIDE 14

Our Work: First data-driven approach

q Learning to group discrete graphical patterns from human annotations q Loosely consider Gestalt principles q Learn relative importance of features, without hand-engineer rules q Robust noise handling thanks to learning approach Convey

slide-15
SLIDE 15

(b) Generalize

Our Strategy

q Learning to group discrete graphical patterns from human annotations q Loosely consider Gestalt principles q Learn relative importance of features, without hand-engineer rules q Robust noise handling thanks to learning approach

slide-16
SLIDE 16

Our Solution: learn features for clustering

q Learned feature descriptor for each elements q Clustering in learned feature space

slide-17
SLIDE 17

Our Solution: bottom-up strategy

q Learned feature descriptor for each elements q Clustering in learned feature space

slide-18
SLIDE 18

q Learned feature descriptor for each elements q Clustering in learned feature space q Not optimize the clustering algorithm itself q Learn a feature space suitable for clustering

Our Solution: learn features for clustering

slide-19
SLIDE 19

Our Solution

q Element -> learned feature descriptor q Clustering in learned feature space q Not optimize the clustering algorithm itself q Learn a feature space suitable for clustering

slide-20
SLIDE 20

Feature Learning: how do human group

slide-21
SLIDE 21

Feature Learning: how do human group

Similar & close-by

slide-22
SLIDE 22

Feature Learning: how do human group

Horizontal Alignment

slide-23
SLIDE 23

Feature Learning: how do human group

How can we migrate human experience into machine learning?

slide-24
SLIDE 24

Feature Learning: local Information

q Similarity q Continuity q Proximity

  • ---- Shape-Aware

Context-Aware Local Information

slide-25
SLIDE 25

q Similarity q Continuity q Proximity

  • ---- Shape-Aware

Context-Aware Local Information q Symmetry ----- Structure-Aware Global Information

Feature Learning: global Information

slide-26
SLIDE 26

Local feature: Atomic Element Encoder

slide-27
SLIDE 27

Local feature: Atomic Element Encoder

slide-28
SLIDE 28

Local feature: Atomic Element Encoder

slide-29
SLIDE 29

Local feature: Structure Encoder

slide-30
SLIDE 30

Global feature: Structure Encoder

slide-31
SLIDE 31

Atomic element Encoder Structure Encoder Location & Size 45 45 8M training pairs

Network Architecture

slide-32
SLIDE 32

Atomic element Encoder Structure Encoder Location & Size 45 45

Network Architecture

8M training pairs

slide-33
SLIDE 33

Data Collection: Lack of suitable patterns on the web

Black & White Color Gradient Lack Structural Variety

slide-34
SLIDE 34

Layout Templates Based Training Data Collection

Layout Template Pattern Examples

slide-35
SLIDE 35

Training Data: element collection

Basic atomic elements

deforming complexing adding noises

slide-36
SLIDE 36

Training Data Collection

q ~800 pattern layout templates q ~8K pattern images q 500 positive and 500 negative pairs of elements q ∼ 8M training pairs

slide-37
SLIDE 37

Results on synthesized patterns

slide-38
SLIDE 38

Grouping Results on synthesized patterns

Clover Arrow

slide-39
SLIDE 39

Results on synthesized patterns

Noise level increase

slide-40
SLIDE 40

Results on downloaded patterns

slide-41
SLIDE 41

Results on downloaded patterns

slide-42
SLIDE 42

Results on downloaded Challenging patterns

slide-43
SLIDE 43

Results on downloaded Challenging patterns

slide-44
SLIDE 44

Quantitative Results with various measures and alternatives

Greater score mean better grouping results

slide-45
SLIDE 45

Quantitative Results with various Clustering Strategies

slide-46
SLIDE 46

Results of User Study

slide-47
SLIDE 47

Limitation on model

No Semantic knowledge

Unreasonable grouping results Unreasonable grouping results

slide-48
SLIDE 48

Limitation on input data

Edge Regions

slide-49
SLIDE 49

Future work: Unified Framework for Various types of Input Data

slide-50
SLIDE 50

Other Future Directions: Hierarchical Grouping

The optimal result

slide-51
SLIDE 51

Other Future Directions: learn to rank grouping results

Which Grouping Results is better?

Input

Grouping (a) Grouping (b)

slide-52
SLIDE 52

Ranking order Changed

Input

Grouping (a) Grouping (b)

Other Future Directions: learn to rank grouping results

slide-53
SLIDE 53

Conclusion

q First (data-driven + deep CNN) for discrete 2D patterns. q Learned shape-, context-, and structure-aware descriptors for graphical elements. q A large, annotated dataset is provided online.

http://people.cs.umass.edu/~zlun/papers/PatternGrouping/ (source code + dataset)

slide-54
SLIDE 54

Conclusion

q First (data-driven + deep CNN) for discrete 2D patterns. q Learned shape-, context-, and structure-aware descriptors for graphical elements. q A large, annotated dataset is provided online.

http://people.cs.umass.edu/~zlun/papers/PatternGrouping/ (source code + dataset)

slide-55
SLIDE 55

Conclusion

q First (data-driven + deep CNN) for discrete 2D patterns. q Learned shape-, context-, and structure-aware descriptors for graphical elements. q A large, annotated dataset is provided online.

http://people.cs.umass.edu/~zlun/papers/PatternGrouping/ (source code + dataset)

slide-56
SLIDE 56

q Dr. Ke Li for the help on experimental data preparation. q The Science and Technology Plan Project of Hunan Province. q The Massachusetts Technology Collaborative grant for funding the UMASS GPU cluster. q NSERC Canada. q Gift funds from Adobe Research.

Acknowledgements

slide-57
SLIDE 57

Thanks! Q&A

http://people.cs.umass.edu/~zlun/papers/PatternGrouping/ (Source code & Dataset)