From Images to Layouts: Combining Deep Learning and Program - - PowerPoint PPT Presentation

From Images to Layouts:

Combining Deep Learning and Program Synthesis

Pavol Bielik, Marc Fischer, Martin Vechev Department of Computer Science ETH Zurich

Big Code @ ETH

Last 5 Years

jsnice.org

jsnice.org

JavaScript Android C/C++ Binaries

www.deepcode.ai/

www.deepcode.ai/

Big Code: last 5 years @ ETH

Learning Over Programs

Deobfuscation Code Review Writing Code

PHOG

Probabilistic Higher Order Grammar Inferring Crypto API Rules from Code Changes. PLDI’18 Neural Code Comprehension: A Learnable Representation of Code Semantics. NIPS’18 Program Synthesis for Character Level Language Modeling. ICLR’17 Probabilistic Model for Code with Decision Trees. OOPSLA’16 Learning Programs from Noisy Data. POPL’16 Code Completion With Statistical Language Models. PLDI’14 Predicting Program Properties from “Big Code”. POPL’15, CACM’18 DEBIN: Predicting Debug Information in Stripped Binaries. CCS’18 Statistical Deobfuscation of Android Applications. CCS’16

http://plml.ethz.ch

Big Code: last 5 years @ ETH

Learning Over Programs

Deobfuscation Code Review Writing Code

PHOG

Probabilistic Higher Order Grammar

Learning Programs

Static Analysis Solving SMT Formulas User Interface Layouts

function collect(val, idx, obj) { if (val >= this.threshold) { ... } } dat.filter( collect, ctx );

points-to

Martin Vechev Veselin Raychev Pavol Bielik Svetoslav Karaivanov Christine Zeller Pascal Roos Benjamin Bischel Benjamin Mularczyk Mateo Panzacchi Petar Tsankov Timon Gehr

Big Code: last 5 years @ ETH

Marc Fischer

picture created by

User Interface Design Today

[Credits: Radu Cristian]

Designer Developer

Designer

Materialize

GroundworkCSS 2

Cardinal

Developer

User Interface Design Today

Designer Developer

Our Goal

Designer Developer Tool

Preventing Layout Errors

Nexus 4 Alcatel Pop 3

Preventing Layout Errors

Nexus 4 Alcatel Pop 3

Preventing Layout Errors

Centering Overlaying Views Out of Screen Asobimasu Amigo Candid

Improving Performance

Layout #1 Layout #2

[Credits: Sriram Ramani]

Layout #3

Slow Fast

Code Porting

Old / Deprecated

Linear Layout (horizontal) Linear Layout (vertical) Relative Layout Constraint Layout

>98% <2%

picture created by

[Credits: Radu Cristian]

Designer Developer

Automating End-User Design

Button Text Text Image

Component Detection

Button

Button Button

SIGN IN

Font Family Font Size Corner Radius Background Color Roboto 12 pt Shadow 0 dp Border Width 0 dp ...

Attribute Prediction

Attributes

Image Image Image Button Button Button Text Text

Layout Synthesis

Layout

Button Text Text Image

Component Detection

Button

Button Button

SIGN IN

Font Family Font Size Corner Radius Background Color Roboto 12 pt Shadow 0 dp Border Width 0 dp ...

Attribute Prediction

Attributes

Image Image Image Button Button Button Text Text

Layout Synthesis

Layout

Machine Learning Approach

✘ Syntactically Incorrect ✘ No Formal Guarantees ✘ Robustness ✘ User Feedback Design Specification

Layout

</> Training

Programming Languages Approach

Design Specification

Layout Formalization List of Devices Probabilistic Model User Feedback Robustness Properties

Layout

</>

Layout Synthesis

Relational Layout Constraints

View

top baseline bottom

B A Align left of B to right of A

margin

B.left = A.right + margin

View

left right

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B.left = A.right + 0

B A A B ✘

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A B A C Center B in between right of A and left of C A B ✘

B.left = A.right + margin

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A Center B in between right of A and left of C A B ✘

B.left = A.right + margin

B A C

margin

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A Center B in between right of A and left of C A B ✘

B.left = A.right + margin

B A C

mL bias = 0.25

Relational Layout Constraints

View

top baseline bottom

View

left right

B A Align left of B to right of A

margin

B A Center B in between right of A and left of C A B ✘

B.left = A.right + margin

B A C

mL bias = 0.25

B A C

margin

Center B in between right of A and left of C (Dynamic Size)

Relational Layout Constraints

View

top baseline bottom

View

left right

Center B in between right of A and left of C B A C

mL bias = 0.25

(1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(C.left - mR)

A B

r

α

Align Center of A to Center of B at an Angle + Distance B A C

margin

Center B in between right of A and left of C (Dynamic Size)

B.left = A.right + mL ∧ B.right = A.left + mR

B.left + B.right = 2r*sin(α) + A.left + A.right

B.left + B.right = 2r*sin(α) + A.left + A.right

Relational Layout Constraints

View

top baseline bottom

View

left right

Center B in between right of A and left of C B A C

mL bias = 0.25

(1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(C.left - mR)

A B

r

α

Align Center of A to Center of B at an Angle + Distance B A C

margin

Center B in between right of A and left of C (Dynamic Size)

B.left = A.right + mL ∧ B.right = A.left + mR

Formalized 26 types of constraints used by the latest Android Constraint Layout

Overview

Layout Formalization Robustness Properties User Feedback List of Devices Probabilistic Model

Design Specification

Layout

</>

Layout Synthesis

Layout Solving

Goal Compute absolute position of all the views left right top bottom

screen: Target Device c: Horizontal Position Constraints

Layout Solving

B A

40

B.left = A.right + 40 Goal Compute absolute position of all the views A.left = screen.left + 20

20

screen.left = 0 ∧ screen.right = 240 B.left = ? B.right = ? A.left = ? A.right = ? v: Absolute Horizontal Positions

screen: Target Device c: Horizontal Position Constraints v: Absolute Horizontal Positions

Layout Solving

B A

40

B.left = A.right + 40 Goal Compute absolute position of all the views B.left = ? B.right = ? A.left = 20 A.right = ? A.left = 0 + 20

20

screen.left = 0 ∧ screen.right = 240

Layout Solving

B A

40

Goal Compute absolute position of all the views B.left = 140 B.right = 220 A.left = 20 A.right = 100

20

s: Horizontal Size Constraints screen: Target Device c: Horizontal Position Constraints B.left = A.right + 40 A.left = 0 + 20 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 B.right - B.left = 80 v ⊧ ψlayout(screen, c, s) v: Absolute Horizontal Positions

Layout Solving vs Layout Synthesis

B A

40

Goal Compute absolute position of all the views B.left = ? B.right = ? A.left = ? A.right = ?

20

B.left = A.right + 40 A.left = screen.left + 20 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 B.right - B.left = 80

B A

40

Goal Synthesize position and size constraints B.left = 120 B.right = 200 A.left = 0 A.right = 80

20

? ? screen.left = 0 ∧ screen.right = 240 ? ? v ⊧ ψlayout(screen, c, s) c ∧ s ⊧ ψsyn(screen, v)

Layout Solving vs Layout Synthesis

B A

40

Goal Compute absolute position of all the views B.left = ? B.right = ? A.left = ? A.right = ?

20

B.left = A.right + 40 A.left = screen.left + 20 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 B.right - B.left = 80

B A

40

Goal Synthesize position and size constraints B.left = 120 B.right = 200 A.left = 0 A.right = 80

20

? ? screen.left = 0 ∧ screen.right = 240 ? ? v ⊧ ψlayout(screen, c, s) c ∧ s ⊧ ψsyn(screen, v)

Approach

• 1. Encode the synthesis program as a logical

formula ψsyn(screen, v)

• 2. Find a satisfying assignment to

c ∧ s ⊧ ψsyn(screen, v)

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v) φposition ≝ screen.left = 0 ∧ screen.right = 240 ∧ A.left = 0 ∧ A.right = 80 ∧ B.left = 120 ∧ B.right = 200

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn}

Layout Synthesis

B A What are the possible constraints for view B? B A B A C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B.left = A.right + mL (c1) (1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(screen.right - mR) (c3) O(|v|2) B.left = screen.right + mL (c2)

Layout Synthesis

B A What are the possible constraints for view B? B A B A C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} B.left = A.right + mL (c1) (1-b)*B.left + b*B.right = (1-b)*(A.right + mL) + b*(screen.right - mR) (c3) O(|v|2) Select exactly one constraint for each view B.left = screen.right + mL (c2)

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} g1 ⇒ (B.left = A.right + mL) (c1) g2 ⇒ (B.left = A.left + mL) (c2) … gn ⇒ ((1-b)*B.left + b*B.right = …) (cn) O(|v|2) Select exactly one constraint for each view Boolean variables denoting whether constraint was selected

View constraints

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} g1 ⇒ (B.left = A.right + mL) (c1) g2 ⇒ (B.left = A.left + mL) (c2) … gn ⇒ ((1-b)*B.left + b*B.right = …) (cn) O(|v|2) Boolean variables denoting whether constraint was selected

φconstraints(screen, v) ≝ gk ⇒ ⟦ck⟧ ∧ g0 + … + g|C(B, v, screen)| = 1

∧

k = 0 C(B, v, screen)

( )

Select exactly one constraint for each view

View constraints Select exactly one constraint for each view

Layout Synthesis

What are the possible constraints for view B? C(B, v, screen) = {cB, cB, cB, … , cB} C(B, v, screen) = {c1, c2, c3, … , cn} g1 ⇒ (B.left = A.right + mL) (c1) g2 ⇒ (B.left = A.left + mL) (c2) … gn ⇒ ((1-b)*B.left + b*B.right = …) (cn) O(|v|2) Boolean variables denoting whether constraint was selected

φconstraints(screen, v) ≝ gk ⇒ ⟦ck⟧ ∧ g0 + … + g|C(v, v, screen)| = 1

∧

k = 0 C(v, v, screen)

∧

i = 0 |v|

( )

i i i i

i

i

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

B A

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v) φvalid(v) ≝ (mL ≥ 0) ∧ (mR ≥ 0) ∧ (0 ≤ b ≤ 1) ∧ (0 ≤ α < 360) ∧ (r ≥ 0) φacyclic(v) ≝

✘

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

c ∧ s ⊧ ψsyn(screen, v)

ψsyn(screen, v) Solver c ∧ s

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

c ∧ s ⊧ ψsyn(screen, v)

ψsyn(screen, v) c ∧ s Are We Done? No Solver

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Are We Done? No

Scalability max 4 views Generalization

13%

Considers only Single Device

Ensure Constraints are Valid Constraints Encoding Input Specification

Layout Synthesis

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

Are We Done? No

Generalize to Multiple Devices Natural Layouts User Feedback Considers only Single Device

Natural Layouts

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v) synthesize any layout that satisfies the specification the most likely

log P(ci | screen, v)

∑

i = 0 |c|

max

Probability of each selected constraint according to learned probabilistic model

ψsyn(screen, v) c ∧ s Solver

Natural Layouts

ψsyn(screen, v) = φposition(screen, v) ∧ φconstraints(v) ∧ φvalid(v) ∧ φacyclic(v)

log P(ci | screen, v)

∑

i = 0 |c|

max

Probability of each selected constraint according to learned probabilistic model

75%

Generalization

13%

Scalability max 4 views

12 views

Probabilistic Model

B A B A Learn a probabilistic model that predicts likelihood that a constraint is used

P(c | screen, v) ⟶ <0, 1>

B A Which constraint is most likely? ✔

Probabilistic Model

B A B A Learn a probabilistic model that predicts likelihood that a constraint is used

P(c | screen, v) ⟶ <0, 1>

B A B A B A B A ✔ ✔

Probabilistic Model

B A B A Learn a probabilistic model that predicts likelihood that a constraint is used

P(c | screen, v) ⟶ <0, 1>

B A B A B A B A B A B A B A ✔ ✔ ✔

Probabilistic Model

fm Margins fb Bias

<mL, mR> b

fd Distance B A fo Orientation B A fs Size B ft Constraint Type B A fi Intersection B A

All Trained using Maximum Likelihood Estimation

fc Complexity

b == 0 mL == 0, mR == 0 1 Z(screen, v) P(c | screen, v) = Pf (c | fk(c, screen, v))

∏

wk

Generalize to Multiple Devices

Synthesis

Layout

</>

• 1. Synthesize Layout Using Input Specification
• 2. Render on Multiple Devices
• 3. Check that Synthesized Layout Generalizes

Generalize to Multiple Devices

c ∧ s ⊧ ψsyn(screen, v)

Layout

</>

• 1. Synthesize Layout Using Input Specification
• 2. Render on Multiple Devices
• 3. Check that Synthesized Layout Generalizes

φpreserve_aspect_ratio(v, vd) ∧ φinside_screen(d, vd) ∧ φpixel_perfect(vd) ∧ φpreserve_order(v, vd) ∧ φpreserve_centering(v, vd) ∧ φpreserve_margins(v, vd)

Set of Robustness Properties

vd ⊧ ψlayout(d, v, c, s) All Steps Encoded as a Single Formula!

User Feedback

Layout

</>

User Feedback

Layout

</>

User Feedback φuser_feedback ≝ Bk.left = 120 ∧ Bk.right = 200

Layout Synthesis

Design Specification

Layout Formalization List of Devices Probabilistic Model User Feedback Robustness Properties

Layout

</>

Layout Synthesis

Evaluation

Top 500 Top 500

Evaluation

Synthesis Success Ratio / Runtime [2, 4) [4, 8) [8, 12) [12, 16) 100%

37 ms

100%

59 ms

100%

129 ms

100%

519 ms Number of Views

Single Device 99%

44 ms

93%

95 ms

73%

314 ms

56%

3 s

Multi Device

Success Ratio Average Synthesis Time

Layout

</>

View Generalization

13%

75%

+ Probabilistic Model

92%

+ Robustness Properties ψsingle_syn(screen, v) Percentage of Views That Correctly Generalize

Layout

</>

User Feedback

63%

no feedback

25%

1 feedback

8%

2 feedbacks

4%

3+ feedbacks Number of User Feedbacks Required

Layout Errors

¬φpreserve_aspect_ratio ¬φinside_screen ¬φpixel_perfect ¬φpreserve_order ¬φpreserve_centering ¬φpreserve_margins

21

(4%)

41

(8.5%)

4

(0.8%)

19

(4%)

24

(5%)

16

(3.3%)

Button Text Text Image

Component Detection

Button

Image Image Image Button Button Button Text Text

Layout Synthesis

Layout

Component Detection Button Object Detection Dog Ball Person Button Button TextView TextView Image

Component Detection

? ? ? ? ? ? ? ? ?

Image Classification

Image Classification

Dog Ball Person ... 85% 12% 1% Low level Medium level High level Feature Visualization [Zeiler & Fergus 2013]

Feature Extraction

Dog Ball Person ... 85% 12% 1% Trainable Classifier Image Features

Component Detection

Features Object Class classification TextView EditView Button Image ... 71% 18% 9% 2% 360 50 200 40 Object Position regression x y width height

Component Detection

How to detect regions containing objects? Check all of the possible locations on the screen

Sliding window Aspect Ratio & Size

a lot

Region Proposal Network

Object? Object Position classification regression x y width height Yes No 71% 29% 360 50 200 40 Features Filtering & Merging

Component Detection: Putting it Together

End-to-end Trainable Neural Network

Faster R-CNN. Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun. NIPS’15

Region Proposal Network Object Class Object Position Features

Evaluation

8 000

Screenshots

100 000

Views

15

Classes

4 days

Training

0.23 s

Query

Evaluation

8 000

Screenshots

Ads Web View / Images Screen Overlay

Button

Obfuscation

aa bc x

100 000

Views

15

Classes Significant part of the dataset was removed due to:

REMAUI: Reverse Engineering Mobile Application User Interfaces

Tuan Anh Nguyen, Christoph Csallner. ASE’15

Input Edges Dilated Edges Contours Container Hierarchy OCR Invalid Words Merge Words Components

Evaluation

Pixel Precision Pixel Recall Ground Truth Prediction Ambiguity

Component Our Work REMAUI TextView Precision 89% 75% Recall 86% 26% Image Precision 94% 30% Recall 95% 10%

Button Text Text Image

Component Detection

Button

Button Button

SIGN IN

Font Family Font Size Corner Radius Background Color Roboto 12 pt Shadow 0 dp Border Width 0 dp ...

Attribute Prediction

Attributes

Image Image Image Button Button Button Text Text

Layout Synthesis

Layout

Button Attributes

Font Family Font Size Text Gravity Lobster Cav

11pt

14pt left right Border Color Border Width Border Radius purple gray 0 dp 4 dp 15 dp 6 dp Background Color Padding Shadow purple blue bottom right 0 dp 10 dp

Attribute Prediction

Border Radius 4 dp Border Width Background Color Shadow 0 dp #F87541 0 dp Font Family Font Size Text Gravity Roboto 6 dp center Input Image

SIGN IN

Rendered Image

Future Designs

Training Dataset

Brand Designs Common Designs

Design Over the Years

2016 2014 2009 2010 2011 2012 2013 2015

[credits: Wojciech Dobry]

Synthetic Dataset

Font Family Font Size Border Width Border Radius Border Color Background Color Text Color ... Cav Roboto Lobster

10 pt

14 pt

12 pt

SIGN IN

Sample Rescaling Position

SIGN IN SIGN IN SIGN IN SIGN IN SIGN IN SIGN IN

Evaluation

Border Width Border Radius Synthetic Dataset Real Dataset 0.1 px 0.63 px 1.37 px 2.84 px Average Error in Pixels

Less than 3px typically not noticable

Training on Synthetic Datasets Leads to Good Generalization

Font Family Font Size Border Radius Background Color Roboto 12 pt

Button Text Text Image

Component Detection

Button

Button Button

SIGN IN

Font Family Font Size Corner Radius Background Color Roboto 12 pt Shadow 0 dp Border Width 0 dp ...

Attribute Prediction

Attributes

Image Image Button Button Button Text Text

Layout Synthesis

Layout