Robust Relational Layout Synthesis from Examples Pavol Bielik , - - PowerPoint PPT Presentation

Robust Relational Layout Synthesis from Examples

Pavol Bielik, Marc Fischer, Martin Vechev Department of Computer Science ETH Zurich

picture created by

User Interface Design Today

[Credits: Radu Cristian]

Designer Developer

Designer

Materialize

GroundworkCSS 2

Developer

User Interface Design Today

Designer Developer

Our Goal

Designer Developer Tool

Preventing Layout Errors

Nexus 4 Alcatel Pop 3

Preventing Layout Errors

Nexus 4 Alcatel Pop 3

Preventing Layout Errors

Centering Overlaying Views Out of Screen Asobimasu Amigo Candid

Improving Performance

Layout #1 Layout #2

[Credits: Sriram Ramani]

Layout #3

Slow Fast

Code Porting

Old / Deprecated

Linear Layout (horizontal) Linear Layout (vertical) Relative Layout Constraint Layout

>98% <2%

picture created by

[Credits: Radu Cristian]

Designer Developer

Automating End-User Design

Machine Learning Approach

✘ Syntactically Incorrect ✘ No Formal Guarantees ✘ Robustness ✘ User Feedback Design Specification

Layout

</> Training

Programming Languages Approach

Design Specification

Layout Formalization List of Devices Probabilistic Model User Feedback Robustness Properties

Layout

</>

Layout Synthesis

Relational Layout Constraints

View

top baseline bottom

B A Align left of B to right of A

margin

B.left = A.right + margin

View

left right

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B.left = A.right + 0

B A A B ✘

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A B A C Center B in between right of A and left of C A B ✘

B.left = A.right + margin

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A Center B in between right of A and left of C A B ✘

B.left = A.right + margin

B A C

margin

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A Center B in between right of A and left of C A B ✘

B.left = A.right + margin

B A C

mL bias = 0.25

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A Center B in between right of A and left of C A B ✘

B.left = A.right + margin

B A C

mL bias = 0.25

B A C

margin

Center B in between right of A and left of C (Dynamic Size)

Relational Layout Constraints

View

top baseline bottom

View

left right

Center B in between right of A and left of C B A C

mL bias = 0.25

(1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(C.left - mR)

A B

r

α

Align Center of A to Center of B at an Angle + Distance B A C

margin

Center B in between right of A and left of C (Dynamic Size)

B.left = A.right + mL ∧ B.right = A.left + mR

B.left + B.right = 2r*sin(α) + A.left + A.right

B.left + B.right = 2r*sin(α) + A.left + A.right

Relational Layout Constraints

View

top baseline bottom

View

left right

Center B in between right of A and left of C B A C

mL bias = 0.25

(1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(C.left - mR)

A B

r

α

Align Center of A to Center of B at an Angle + Distance B A C

margin

Center B in between right of A and left of C (Dynamic Size)

B.left = A.right + mL ∧ B.right = A.left + mR

Formalized 26 types of constraints used by the latest Android Constraint Layout

Overview

Layout Formalization Robustness Properties User Feedback List of Devices Probabilistic Model

Design Specification

Layout

</>

Layout Synthesis

Layout Solving

Goal Compute absolute position of all the views left right top bottom

screen: Target Device c: Horizontal Position Constraints

Layout Solving

B A

40

B.left = A.right + 40 Goal Compute absolute position of all the views A.left = screen.left + 20

20

screen.left = 0 ∧ screen.right = 240 B.left = ? B.right = ? A.left = ? A.right = ? v: Absolute Horizontal Positions

screen: Target Device c: Horizontal Position Constraints v: Absolute Horizontal Positions

Layout Solving

B A

40

B.left = A.right + 40 Goal Compute absolute position of all the views B.left = ? B.right = ? A.left = 20 A.right = ? A.left = 0 + 20

20

screen.left = 0 ∧ screen.right = 240

Layout Solving

B A

40

Goal Compute absolute position of all the views B.left = 140 B.right = 220 A.left = 20 A.right = 100

20

s: Horizontal Size Constraints screen: Target Device c: Horizontal Position Constraints B.left = A.right + 40 A.left = 0 + 20 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 B.right - B.left = 80 v ⊧ ψlayout(screen, c, s) v: Absolute Horizontal Positions

Layout Solving vs Layout Synthesis

B A

40

Goal Compute absolute position of all the views B.left = ? B.right = ? A.left = ? A.right = ?

20

B.left = A.right + 40 A.left = screen.left + 20 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 B.right - B.left = 80

B A

40

Goal Synthesize position and size constraints B.left = 120 B.right = 200 A.left = 0 A.right = 80

20

? ? screen.left = 0 ∧ screen.right = 240 ? ? v ⊧ ψlayout(screen, c, s) c ∧ s ⊧ ψsyn(screen, v)

Layout Solving vs Layout Synthesis

B A

40

Goal Compute absolute position of all the views B.left = ? B.right = ? A.left = ? A.right = ?

20

B.left = A.right + 40 A.left = screen.left + 20 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 B.right - B.left = 80

B A

40

Goal Synthesize position and size constraints B.left = 120 B.right = 200 A.left = 0 A.right = 80

20

? ? screen.left = 0 ∧ screen.right = 240 ? ? v ⊧ ψlayout(screen, c, s) c ∧ s ⊧ ψsyn(screen, v)

Approach

• 1. Encode the synthesis program as a logical

formula ψsyn(screen, v)

• 2. Find a satisfying assignment to

c ∧ s ⊧ ψsyn(screen, v)

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v) φposition ≝ screen.left = 0 ∧ screen.right = 240 ∧ A.left = 0 ∧ A.right = 80 ∧ B.left = 120 ∧ B.right = 200

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn}

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B A B.left = A.right + mL (c1)

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B A B.left = A.right + mL (c1) B A B.left = screen.right + mL (c2)

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B A B.left = A.right + mL (c1) B A B.left = screen.right + mL (c2) B A (1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(screen.right - mR) (c3)

Layout Synthesis

B A What are the possible constraints for view B? B A B A C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B.left = A.right + mL (c1) (1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(screen.right - mR) (c3) O(|v|2) B.left = screen.right + mL (c2)

Layout Synthesis

B A What are the possible constraints for view B? B A B A C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B.left = A.right + mL (c1) (1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(screen.right - mR) (c3) O(|v|2) Select exactly one constraint for each view B.left = screen.right + mL (c2)

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} g1 ⇒ (B.left = A.right + mL) (c1) g2 ⇒ (B.left = A.left + mL) (c2) … gn ⇒ ((1-b)*B.left + b*B.right = …) (cn) O(|v|2) Select exactly one constraint for each view Boolean variables denoting whether constraint was selected

View constraints

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} g1 ⇒ (B.left = A.right + mL) (c1) g2 ⇒ (B.left = A.left + mL) (c2) … gn ⇒ ((1-b)*B.left + b*B.right = …) (cn) O(|v|2) Boolean variables denoting whether constraint was selected

φconstraints(screen, v) ≝ gk ⇒ ⟦ck⟧ ∧ g0 + … + g|C(B, v, screen)| = 1

∧

k = 0 C(B, v, screen)

( )

Select exactly one constraint for each view

View constraints Select exactly one constraint for each view

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} g1 ⇒ (B.left = A.right + mL) (c1) g2 ⇒ (B.left = A.left + mL) (c2) … gn ⇒ ((1-b)*B.left + b*B.right = …) (cn) O(|v|2) Boolean variables denoting whether constraint was selected

φconstraints(screen, v) ≝ gk ⇒ ⟦ck⟧ ∧ g0 + … + g|C(v, v, screen)| = 1

∧

k = 0 C(v, v, screen)

∧

i = 0 |v|

( )

i i i i

i

i

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

B A

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v) φvalid(v) ≝ (mL ≥ 0) ∧ (mR ≥ 0) ∧ (0 ≤ b ≤ 1) ∧ (0 ≤ α < 360) ∧ (r ≥ 0) φacyclic(v) ≝

✘

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

c ∧ s ⊧ ψsyn(screen, v)

ψsyn(screen, v) Solver c ∧ s

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

c ∧ s ⊧ ψsyn(screen, v)

ψsyn(screen, v) c ∧ s Are We Done? Solver

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

c ∧ s ⊧ ψsyn(screen, v)

ψsyn(screen, v) c ∧ s Are We Done? No Solver

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Are We Done? No

Scalability max 4 views Generalization

13%

Considers only Single Device

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Are We Done? No

Generalize to Multiple Devices Probabilistic Model User Feedback Considers only Single Device

Natural Layouts

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v) synthesize any layout that satisfies the specification the most likely

log P(ci | screen, v)

∑

i = 0 |c|

max

Probability of each selected constraint according to learned probabilistic model

ψsyn(screen, v) c ∧ s Solver

Natural Layouts

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

log P(ci | screen, v)

∑

i = 0 |c|

max

Probability of each selected constraint according to learned probabilistic model

75%

Generalization

13%

Scalability max 4 views

12 views

Probabilistic Model

B A B A Learn a probabilistic model that predicts likelihood that a constraint is used

P(c | screen, v) ⟶ <0, 1>

B A Which constraint is most likely? ✔

Probabilistic Model

B A B A Learn a probabilistic model that predicts likelihood that a constraint is used

P(c | screen, v) ⟶ <0, 1>

B A B A B A B A ✔ ✔

Probabilistic Model

B A B A Learn a probabilistic model that predicts likelihood that a constraint is used

P(c | screen, v) ⟶ <0, 1>

B A B A B A B A B A B A B A ✔ ✔ ✔

Probabilistic Model

fm Margins fb Bias

<mL, mR> b

fd Distance B A fo Orientation B A fs Size B ft Constraint Type B A fi Intersection B A

All Trained using Maximum Likelihood Estimation

fc Complexity

b == 0 mL == 0, mR == 0 1 Z(screen, v) P(c | screen, v) = Pf (c | fk(c, screen, v))

∏

wk

Generalize to Multiple Devices

Synthesis

Layout

</>

• 1. Synthesize Layout Using Input Specification
• 2. Render on Multiple Devices
• 3. Check that Synthesized Layout Generalizes

Generalize to Multiple Devices

c ∧ s ⊧ ψsyn(screen, v)

Layout

</>

• 1. Synthesize Layout Using Input Specification
• 2. Render on Multiple Devices
• 3. Check that Synthesized Layout Generalizes

φpreserve_aspect_ratio(v, vd) ∧ φinside_screen(d, vd) ∧ φpixel_perfect(vd) ∧ φpreserve_order(v, vd) ∧ φpreserve_centering(v, vd) ∧ φpreserve_margins(v, vd)

Set of Robustness Properties

vd ⊧ ψlayout(d, v, c, s) All Steps Encoded as a Single Formula!

User Feedback

Layout

</>

User Feedback

Layout

</>

User Feedback φuser_feedback ≝ Bk.left = 120 ∧ Bk.right = 200

Evaluation

Top 500 Top 500

Evaluation

Synthesis Success Ratio / Runtime [2, 4) [4, 8) [8, 12) [12, 16) 100%

37 ms

100%

59 ms

100%

129 ms

100%

519 ms Number of Views

Single Device 99%

44 ms

93%

95 ms

73%

314 ms

56%

3 s

Multi Device

Success Ratio Average Synthesis Time

Layout

</>

View Generalization

13%

75%

+ Probabilistic Model

92%

+ Robustness Properties ψsingle_syn(screen, v) Percentage of Views That Correctly Generalize

Layout

</>

User Feedback

63%

no feedback

25%

1 feedback

8%

2 feedbacks

4%

3+ feedbacks Number of User Feedbacks Required

Layout Errors

¬φinside_screen ¬φpreserve_margins

24

(5%)

¬φpreserve_order

19

(4%)

21

(4%)

¬φpreserve_aspect_ratio ¬φpixel_perfect ¬φpreserve_centering

41

(8.5%)

4

(0.8%)

16

(3.3%)

Design Specification

Layout Formalization List of Devices Probabilistic Model User Feedback Robustness Properties

Layout

</>

Layout Synthesis