SLIDE 1
PROGRAMMATIC AND DIRECT MANIPULATION
TOGETHER AT LAST
Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob Albers
Program
2
TOGETHER AT LAST Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob - - PowerPoint PPT Presentation
TOGETHER AT LAST Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob - - PowerPoint PPT Presentation
PROGRAMMATIC AND DIRECT MANIPULATION TOGETHER AT LAST Ravi Chugh, Brian Hempel, Mitchell Spradlin, Jacob Albers Program 2 Program 2 Changed Program Program 2 Changed Program Program 2 Changed Changed Program Program Program
SLIDE 2
SLIDE 3
Program
2
SLIDE 4
Program
2
Changed Program
SLIDE 5
Program
2
Changed Program
SLIDE 6
Program
2
Changed Program Changed Program (again)
SLIDE 7
Program
2
Changed Program Changed Program (again)
SLIDE 8
Program
2
Changed Program Changed Program (again) Changed Program
(again again)
SLIDE 9
Program
2
Changed Program Changed Program (again) Changed Program
(again again)
SLIDE 10
Program
3
SLIDE 11
Program
3
SLIDE 12
Program
3
SLIDE 13
Program
3
Changed Program
SLIDE 14
Program
3
Changed Program
GOAL: Live Sync
SLIDE 15
Program
3
Changed Program
Fast + Intuitive + Automatic
GOAL: Live Sync
SLIDE 16
SIMPLE EXAMPLE
4
SLIDE 17
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
5
SLIDE 18
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
6
SLIDE 19
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
7
SLIDE 20
x2 = 110
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
8
SLIDE 21
x2 = 110 x2’ = 155
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
8
SLIDE 22
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155 x2 = 110
GOAL
Change program so
xi = 155
when i = 2
9
SLIDE 23
x2’ = 155
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
10
SLIDE 24
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = if i=2 then 155 else x0+i*sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155
11
SLIDE 25
let (x0, y0, w, h, sep, n) = (95, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155
12
SLIDE 26
let (x0, y0, w, h, sep, n) = (95, 120, 20, 90, 52.5, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155
13
SLIDE 27
let (x0, y0, w, h, sep, n) = (65, 120, 20, 90, 55, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155
14
SLIDE 28
let (x0, y0, w, h, sep, n) = (75, 120, 20, 90, 40, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155
15
SLIDE 29
let (x0, y0, w, h, sep, n) = (95, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2’ = 155
16
SLIDE 30
155 = x0 + 2*sep
xi ↦ if i=2 then 155 else x0 + i * sep x0 ↦ 75, sep ↦ 40 x0 ↦ 55, sep ↦ 50 x0 ↦ …, sep ↦ … x0 ↦ 95 sep ↦ 52.5
x2’ = 155
17
SLIDE 31
xi ↦ if i=2 then 155 else x0 + i * sep
155 = x0 + 2*sep
x0 ↦ 75, sep ↦ 40 x0 ↦ 55, sep ↦ 50 x0 ↦ …, sep ↦ … x0 ↦ 95 sep ↦ 52.5
x2’ = 155
GOAL
Intuitive Changes
18
SLIDE 32
155 = x0 + 2*sep
x0 ↦ 75, sep ↦ 40 x0 ↦ 55, sep ↦ 50 x0 ↦ …, sep ↦ … x0 ↦ 95 sep ↦ 52.5
x2’ = 155
xi ↦ if i=2 then 155 else x0 + i * sep
19
SLIDE 33
155 = x0 + 2*sep
xi ↦ if i=2 then 155 else x0 + i * sep x0 ↦ 75, sep ↦ 40 x0 ↦ 55, sep ↦ 50 x0 ↦ …, sep ↦ … x0 ↦ 95 sep ↦ 52.5
x2’ = 155
20
SLIDE 34
155 = x0 + 2*sep
xi ↦ if i=2 then 155 else x0 + i * sep x0 ↦ 75, sep ↦ 40 x0 ↦ 55, sep ↦ 50 x0 ↦ …, sep ↦ … x0 ↦ 95 sep ↦ 52.5
x2’ = 155
21
SLIDE 35
155 = x0 + 2*sep
x0 ↦ 95 sep ↦ 52.5
x2’ = 155
“Small Updates”
22
SLIDE 36
x0 ↦ 95 sep ↦ 52.5
x2’ = 155
?
23
SLIDE 37
x0 ↦ 95 sep ↦ 52.5
x2’ = 155
?
GOAL
Automatic
24
SLIDE 38
x2’ = 155
A HEURISTIC
x0 ↦ 95 sep ↦ 52.5
25
SLIDE 39
x2’ = 155
A HEURISTIC
x0 ↦ 95 sep ↦ 52.5
26
SLIDE 40
A HEURISTIC
27
SLIDE 41
A HEURISTIC
x0 ↦ … sep ↦ …
28
SLIDE 42
A HEURISTIC
x0 ↦ … sep ↦ …
29
SLIDE 43
A HEURISTIC
x0 ↦ … sep ↦ …
30
SLIDE 44
A HEURISTIC
x0 ↦ … sep ↦ …
31
SLIDE 45
A HEURISTIC
x0 ↦ … sep ↦ …
32
SLIDE 46
A HEURISTIC
x0 ↦ … sep ↦ …
33
SLIDE 47
A HEURISTIC
x0 ↦ … sep ↦ …
34
SLIDE 48
A HEURISTIC
x0 ↦ … sep ↦ …
35
SLIDE 49
A HEURISTIC
36
sep x0 x0 x0 sep
SLIDE 50
37
Small Updates
!
Heuristics
!
Traces
!
Simple Solver
SLIDE 51
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2 = 110
38
SLIDE 52
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
x2 = 110
38
SLIDE 53
TRACES
39
let a = 3 in let b = 5 in a + b
⇓
8
SLIDE 54
TRACES
let a = 3 in let b = 5 in a + b
⇓
8
40
SLIDE 55
TRACES
let a = 3a in let b = 5b in a + b
⇓
8
41
SLIDE 56
TRACES
let a = 3a in let b = 5b in a + b
⇓
8a+b
42
SLIDE 57
x0 + 2*sep
x2 =
43
110x0 + 2*sep 110
x0 x0 sep
SLIDE 58
x0 + 2*sep
x2 =
44
110x0 + 2*sep 110
x0 x0 sep
SLIDE 59
45
x0 x0 sep
=
x0 + 2*sep
x2 = 110x0 + 2*sep 110
SLIDE 60
46
x0 x0 sep
=
sep x0 + 2*
SLIDE 61
47
x0 x0 sep
=
sep
155
x0 + 2*
SLIDE 62
=
sep
155
+ 2*
x0
48
x0 x0 sep
SLIDE 63
= 155
+ 2*30
x0
49
x0 x0 sep
SLIDE 64
= 155
+ 2*30
x0
50
95
x0 x0 sep
SLIDE 65
51
= 155
+ 2*30
x0
95
SLIDE 66
TRACE LANGUAGE
e ⇓ nt
52
SLIDE 67
TRACE LANGUAGE
e ⇓ nt
t ::= x | t1 + t2 | t1 * t2 | sin t | pow t1 t2
...
52
SLIDE 68
TRACE LANGUAGE
e ⇓ nt
t ::= x
n’ = t
| t1 + t2 | t1 * t2 | sin t | pow t1 t2
...
52
SLIDE 69
TRACES
let a = 3a in let b = 5b in a + b
!
53
SLIDE 70
TRACES
let a = 3a in let b = (true ? 5b1 : 6b2) in a + b
54
SLIDE 71
TRACES
let a = 3a in let b = (true ? 5b1 : 6b2) in a + b
⇓
8a+b1
54
SLIDE 72
t ::= x | t1 + t2 | t1 * t2 | sin t | pow t1 t2
...
55
SLIDE 73
SOLVER
56
5*sin(x) = n x*(y + y2) = n x2 = n
✓ ✓ ✓
Can solve for x if x only occurs once.
SLIDE 74
SOLVER
56
5*sin(x) = n x*(y + y2) = n x*(x + y2) = n x2 = n x*x = n
✓ ✓ ✗ ✓ ✗
Can solve for x if x only occurs once.
SLIDE 75
SOLVER
56
5*sin(x) = n x*(y + y2) = n x*(x + y2) = n x2 = n x*x = n
✓ ✓ ✗ ✓ ✗
Can solve for x if x only occurs once. Good for 80% of equations.
SLIDE 76
57
Small Updates
!
Heuristics
!
Traces
!
Simple Solver
SLIDE 77
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
58
SLIDE 78
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 1. Evaluate
with Traces
58
SLIDE 79
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 1. Evaluate
with Traces
58
SLIDE 80
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 1. Evaluate
with Traces
?
58
x0 sep
SLIDE 81
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 1. Evaluate
with Traces
x0 ?58
x0 sep
x0 SLIDE 82
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 1. Evaluate
with Traces
x0 sep?58
x0 sep
x0 sep SLIDE 83
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 1. Evaluate
with Traces
x0 sep x058
x0 sep x0 SLIDE 84
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
- 1. Evaluate
with Traces
x0 sep x058
x0 sep x0 SLIDE 85
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
- 1. Evaluate
with Traces
x0 sep x0User Changes Output;
58
x0 sep x0 SLIDE 86
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
- 1. Evaluate
with Traces
x0 sep x0User Changes Output;
58
= 155
+ 2*30
x0
x0 sep x0 SLIDE 87
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
Solve Trace Equation;
- 1. Evaluate
with Traces
x0 sep x0User Changes Output;
58
= 155
+ 2*30
x0
x0 sep x0 SLIDE 88
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
Solve Trace Equation;
- 1. Evaluate
with Traces
x0 sep x0User Changes Output;
58
= 155
+ 2*30
x0
95
x0 sep x0 SLIDE 89
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
Solve Trace Equation; Apply Small Update;
- 1. Evaluate
with Traces
x0 sep x0User Changes Output;
58
95
x0 sep x0 SLIDE 90
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
Solve Trace Equation; Apply Small Update;
- 1. Evaluate
with Traces
x0 sep x0Re-run and Render User Changes Output;
58
95
x0 sep x0 SLIDE 91
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
Solve Trace Equation; Apply Small Update;
- 1. Evaluate
with Traces
x0 sep x0Re-run and Render User Changes Output;
58
95
SLIDE 92
Sketch-n-Sketch
59
SLIDE 93
- 60
SLIDE 94
- 61
SLIDE 95
- 62
SLIDE 96
- 63
SLIDE 97
- 64
SLIDE 98
- 65
SLIDE 99
- 66
SLIDE 100
- 67
SLIDE 101
- 68
SLIDE 102
- 69
SLIDE 103
- 70
SLIDE 104
- 71
SLIDE 105
- 72
SLIDE 106
- 73
SLIDE 107
- 74
SLIDE 108
- 75
SLIDE 109
- 76
SLIDE 110
- 77
SLIDE 111
- 78
SLIDE 112
- 79
SLIDE 113
- 80
SLIDE 114
- 81
SLIDE 115
- 82
SLIDE 116
- 83
SLIDE 117
- 84
SLIDE 118
- 85
SLIDE 119
- 86
SLIDE 120
Custom User Interface Widgets
87
SLIDE 121
- 88
SLIDE 122
- 89
SLIDE 123
- 90
SLIDE 124
- 91
SLIDE 125
- 92
SLIDE 126
- 93
SLIDE 127
- 94
SLIDE 128
- 95
SLIDE 129
- 96
SLIDE 130
- 97
SLIDE 131
- 98
SLIDE 132
- 99
SLIDE 133
- 100
SLIDE 134
- 101
SLIDE 135
- 102
SLIDE 136
DEMOS
103
SLIDE 137
Just google “sketch n sketch”
Demos on YouTube
Sketch-n-Sketch
https://ravichugh.github.io/sketch-n-sketch
104
SLIDE 138
FUTURE WORK
105
SLIDE 139
106
- •
- Traces
Heuristics Solver
SLIDE 140
107
Program
SLIDE 141
107
Program
SLIDE 142
108
SLIDE 143
108
SLIDE 144
108
Program
SLIDE 145
108
Program
SLIDE 146
109
Program
SLIDE 147
110
Program
PROGRAMMATIC AND DIRECT MANIPULATION
IN OTHER DOMAINS
SLIDE 148
RELATED WORK
111
SLIDE 149
Old Program: Constants: User Action: “Sketch” “Holes” “Example”
PROGRAM SYNTHESIS
112
SLIDE 150
LIVE SYNC
Numbers + Heuristics
PROGRAM ← OUTPUT
String Origin Analysis for PHP by Wang et al.
String Literals
113
SLIDE 151
LIVE SYNC
constraints in synthesis only
CONSTRAINTS
Sketchpad, ThingLab, Babelsberg, … constraints in language
114
SLIDE 152
BX PROGRAMMING
Boomerang, biXid, Beanbag…
LIVE SYNC
DSL
View Data
115
SLIDE 153
LIVE SYNC
λ-calc + DM
- PROG. BY MANIPULATION
Victor, Apparatus, PBM for Layout, …
DM + DSL
116
SLIDE 154
Program
117
SLIDE 155
Program
117
SLIDE 156
Program
117
SLIDE 157
Program
117
Changed Program
SLIDE 158
Program
117
Changed Program
Live Sync
SLIDE 159
Program
117
Changed Program
Fast + Intuitive + Automatic
Live Sync
SLIDE 160
Program
117
Changed Program
Fast + Intuitive + Automatic
Live Sync
Thank You!
SLIDE 161
let (x0, y0, w, h, sep, n) = (50, 120, 20, 90, 30, 3)
!
let boxi i = let xi = x0 + i * sep in rect 'lightblue' xi y0 w h
!
map boxi (zeroTo n)
- 2. Prepare for
User Actions with Heuristics
- 3. Live Sync
Solve Trace Equation; Apply Small Update;
- 1. Evaluate
with Traces
Re-run and Render User Changes Output;
118
= 155
+ 2*30
x0
95
x0 sep
SLIDE 162
Extra Slides
119
SLIDE 163
USER STUDY
120
Before After Histograms Figure 9. The goal of each task (Ferris Wheel, Keyboard,
SLIDE 164
TIMINGS
121
Operation Min Med Avg Max Parse 9 ms 53 ms 77 ms 520 ms Eval <1 ms 5 ms 12 ms 165 ms Prepare 1 ms 13 ms 200 ms 6,789 ms Solve <1 ms <1 ms <1 ms 14 ms
SLIDE 165
SOLVABILITY
122
Unique Pre-Equations 4,574 Outside Fragment 919 20% Inside Fragment 3,655 No Solution for d = 1 194 4% Solution for d = 1 3,461 No Solution for d = 100 438 10% Solution for d = 100 3,023 66%
SLIDE 166
AMBIGUITY
123