From Images to Layouts: Combining Deep Learning and Program Synthesis Pavol Bielik , Marc Fischer, Martin Vechev Department of Computer Science ETH Zurich
Big Code @ ETH Last 5 Years
jsnice.org
JavaScript Android C/C++ Binaries jsnice.org
www.deepcode.ai/
www.deepcode.ai/
Big Code: last 5 years @ ETH Learning Over Programs http://plml.ethz.ch Deobfuscation Predicting Program Properties from “Big Code”. POPL’15, CACM’18 DEBIN: Predicting Debug Information in Stripped Binaries. CCS’18 Statistical Deobfuscation of Android Applications. CCS’16 Code Review Inferring Crypto API Rules from Code Changes. PLDI’18 Neural Code Comprehension: A Learnable Representation of Code Semantics. NIPS’18 Writing Code Program Synthesis for Character Level Language Modeling. ICLR’17 Probabilistic Model for Code with Decision Trees. OOPSLA’16 PHOG Learning Programs from Noisy Data. POPL’16 Probabilistic Higher Order Grammar Code Completion With Statistical Language Models. PLDI’14
Big Code: last 5 years @ ETH Learning Over Programs Learning Programs Deobfuscation Static Analysis function collect(val, idx, obj) { if (val >= this .threshold) { ... } } points-to dat.filter( collect, ctx ); Code Review Solving SMT Formulas Writing Code User Interface Layouts PHOG Probabilistic Higher Order Grammar
Big Code: last 5 years @ ETH Martin Pavol Veselin Petar Timon Vechev Bielik Raychev Tsankov Gehr Christine Pascal Benjamin Svetoslav Benjamin Mateo Marc Zeller Roos Bischel Karaivanov Mularczyk Panzacchi Fischer
User Interface Design Today Designer Developer picture created by [Credits: Radu Cristian]
Designer
Cardinal Materialize GroundworkCSS 2 Developer
User Interface Design Today Designer Developer
Our Goal Tool Designer Developer
Preventing Layout Errors Nexus 4 Alcatel Pop 3
Preventing Layout Errors Nexus 4 Alcatel Pop 3
Preventing Layout Errors Centering Asobimasu Overlaying Views Amigo Out of Screen Candid
Improving Performance Slow Layout #1 Layout #2 Fast Layout #3 [Credits: Sriram Ramani]
Code Porting Linear Layout (horizontal) Linear Layout (vertical) Constraint Layout Relative Layout >98% <2% Old / Deprecated
Automating End-User Design Designer Developer picture created by [Credits: Radu Cristian]
Component Detection Layout Synthesis Attribute Prediction Text Text Font Family Roboto Text Text Font Size 12 pt Corner Radius Border Width 0 dp Background Color Image Image Image Shadow 0 dp Image ... Button Button Button Button Button Button Button SIGN IN Layout Attributes
Component Detection Layout Synthesis Attribute Prediction Text Text Font Family Roboto Text Text Font Size 12 pt Corner Radius Border Width 0 dp Background Color Image Image Image Shadow 0 dp Image ... Button Button Button Button Button Button Button SIGN IN Layout Attributes
Machine Learning Approach Training </> Layout ✘ Syntactically Incorrect ✘ No Formal Guarantees ✘ Robustness ✘ User Feedback Design Specification
Programming Languages Approach Layout Robustness User Formalization Properties Feedback </> Layout Layout Synthesis Probabilistic List of Devices Model Design Specification
Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A A B margin B.left = A.right + margin
Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A A B margin B.left = A.right + 0 A B A B ✘
Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C margin B.left = A.right + margin A B A B ✘
Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C margin margin B.left = A.right + margin A B A B ✘
Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C m L margin bias = 0.25 B.left = A.right + margin A B A B ✘
Relational Layout Constraints left right top View View baseline bottom Align left of B to right of A Center B in between right of A and left of C A B A B C m L margin bias = 0.25 B.left = A.right + margin Center B in between right of A and left of C (Dynamic Size) A B A B C margin A B ✘
Relational Layout Constraints left right top View View baseline bottom Align Center of A to Center of B Center B in between right of A and left of C at an Angle + Distance A B C m L A r bias = 0.25 (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(C.left - m R ) α B Center B in between right of A and left of C (Dynamic Size) A B C margin B.left + B.right = 2r*sin( α ) + A.left + A.right B.left = A.right + m L ∧ B.right = A.left + m R
Relational Layout Constraints left right top View View baseline bottom Align Center of A to Center of B Center B in between right of A and left of C Formalized 26 types of constraints used by the at an Angle + Distance latest Android Constraint Layout A B C m L A r bias = 0.25 (1-b)*B.left + b*B.right = (1-b)*(A.right + m L ) + b*(C.left - m R ) α B Center B in between right of A and left of C (Dynamic Size) A B C margin B.left + B.right = 2r*sin( α ) + A.left + A.right B.left = A.right + m L ∧ B.right = A.left + m R
Overview Layout Robustness User Formalization Properties Feedback </> Layout Layout Synthesis Probabilistic List of Devices Model Design Specification
Layout Solving Goal Compute absolute position of all the views left right top bottom
Layout Solving Goal Compute absolute position of all the views A.left = ? B.left = ? v : Absolute Horizontal Positions A.right = ? B.right = ? A B 20 40 B.left = A.right + 40 c : Horizontal Position Constraints A.left = screen.left + 20 screen: Target Device screen.left = 0 ∧ screen.right = 240
Layout Solving Goal Compute absolute position of all the views A.left = 20 B.left = ? v : Absolute Horizontal Positions A.right = ? B.right = ? A B 20 40 B.left = A.right + 40 c : Horizontal Position Constraints A.left = 0 + 20 screen: Target Device screen.left = 0 ∧ screen.right = 240
Layout Solving Goal Compute absolute position of all the views A.left = 20 B.left = 140 v : Absolute Horizontal Positions A.right = 100 B.right = 220 A B 20 40 B.left = A.right + 40 c : Horizontal Position Constraints A.left = 0 + 20 screen: Target Device screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 s : Horizontal Size Constraints B.right - B.left = 80 v ⊧ ψ layout (screen, c , s )
Layout Solving vs Layout Synthesis Goal Goal Compute absolute position of all the views Synthesize position and size constraints A.left = ? B.left = ? A.left = 0 B.left = 120 A.right = ? B.right = ? A.right = 80 B.right = 200 A B A B 20 40 20 40 B.left = A.right + 40 ? A.left = screen.left + 20 ? screen.left = 0 ∧ screen.right = 240 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 ? B.right - B.left = 80 ? v ⊧ ψ layout (screen, c , s ) c ∧ s ⊧ ψ syn (screen, v )
Layout Solving vs Layout Synthesis Goal Goal Compute absolute position of all the views Synthesize position and size constraints Approach A.left = ? B.left = ? A.left = 0 B.left = 120 A.right = ? B.right = ? A.right = 80 B.right = 200 1. Encode the synthesis program as a logical formula ψ syn (screen, v ) A B A B 20 40 20 40 2. Find a satisfying assignment to B.left = A.right + 40 ? A.left = screen.left + 20 ? c ∧ s ⊧ ψ syn (screen, v ) screen.left = 0 ∧ screen.right = 240 screen.left = 0 ∧ screen.right = 240 A.right - A.left = 80 ? B.right - B.left = 80 ? v ⊧ ψ layout (screen, c , s ) c ∧ s ⊧ ψ syn (screen, v )
Layout Synthesis ψ syn (screen, v ) = φ position (screen, v ) ∧ φ constraints ( v ) ∧ φ valid ( v ) ∧ φ acyclic ( v ) Input Constraints Ensure Constraints Specification Encoding are Valid
Layout Synthesis ψ syn (screen, v ) = φ position (screen, v ) ∧ φ constraints ( v ) ∧ φ valid ( v ) ∧ φ acyclic ( v ) Input Constraints Ensure Constraints Specification Encoding are Valid screen.left = 0 ∧ φ position ≝ screen.right = 240 ∧ A.left = 0 ∧ A.right = 80 ∧ B.left = 120 ∧ B.right = 200
Layout Synthesis ψ syn (screen, v ) = φ position (screen, v ) ∧ φ constraints ( v ) ∧ φ valid ( v ) ∧ φ acyclic ( v ) Input Constraints Ensure Constraints Specification Encoding are Valid
Layout Synthesis What are the possible constraints for view B? C (B, v, screen) = {c B , c B , c B , … , c B } C (B, v, screen) = {c 1 , c 2 , c 3 , … , c n }
Recommend
More recommend
