Visualization By Example Chenglong Wang 1 , Yu Feng 2 , Alvin Cheung - PowerPoint PPT Presentation

Visualization By Example Chenglong Wang 1 , Yu Feng 2 , Alvin Cheung 3 , Ras Bodik 1 , Isil Dillig 4 1 University of Washington 2 University of California, Santa Barbara 3 University of California, Berkeley 4 University of Texas, Austin Read the paper! 1

Visualizations Product price in different region Net cash flow in a year Survey result 1. How to formalize visualizations? 2. How to synthesize visualizations to bring it to the masses? Performance rating distribution for each department Housing price in different region 2

Formalizing Visualizations “Transformation of the symbolic into the geometric” [McCormick et al. 1987] Visual Program ggplot2, Matplotlib, ϕ V Excel, Vega-Lite V T Table A set of geometric objects A → bar.x A B B → bar.height bar(x=1, h=1) 1 1 bar(x=2, h=4) 2 4 bar(x=3, h=9) 3 9 bar(x=4, h=16) 4 16 bar(x=5, h=25) 5 25 3

Visualization in Practice Visual program alone is often insufficient. Visual Program ϕ V V T Table A set of geometric objects ? X A B C bar(x=1, h=1, color=A) 1 1 4 3 bar(x=1, h=4, color=B) bar(x=1, h=3, color=C) 2 2 3 2 bar(x=2, h=2, color=A) 3 5 2 1 …… 4 3 6 1 expects a certain shape ϕ V of the input table Expects 3 columns that map to bar.x, bar.height, bar.color 4

Visualization in Practice Visual Program Data Adapter ϕ T ϕ V T V T ′ gather( X → bar.x T, X Key Val Val → bar.height X A B C X, bar(x=1, h=1, color=A) 1 A 1 Key → bar.color 1 1 4 3 [A,B,C]) bar(x=1, h=4, color=B) 1 B 4 bar(x=1, h=3, color=C) 2 2 3 2 1 C 3 bar(x=2, h=2, color=A) 3 5 2 1 2 A 2 …… 4 3 6 1 … … … Gather (1) turn A,B,C into the Key column Data adapter prepares the (2) move values in A,B,C columns input table to match the shape into the Val column expected by ϕ V 5

Visualization Challenges 1. Users need to master both data prep libraries and visualization libraries. ggplot2, Vega-Lite, matplotlib … Python, R, … ϕ T ϕ V V T T ′ 2. Reshaping and aggregation of data 3. Change of visualization designs requires requires deep data transformation frequent change of data adapters. insight. [Feng et al. 2018] 6

Visualization Challenges [Gatto 2015] Data Adapter Visual Program ϕ T ϕ V V T T ′ 2. Limited Software Knowledge 1. Diverse data shapes in practice 3. Limited knowledge about visualization concepts Visualization Synthesis (1) handle diverse data shapes Visualization Synthesis! (2) expressive and ϕ T ϕ V (3) user specification requires little visualization concepts Gatto, Malu AC. "Making research useful: Current challenges and good practices in data visualization." (2015). 7

How to specify visualization? How would you explain intent of this visualization? Partial visualization A subset of geometric objects of the final visualization 8

Visualization by Example ϕ T ϕ V T V T ′ 9

Visualization by Example Input Input V partial ⊆ ϕ T ϕ V T V T ′ 10

Visualization by Example Input Input V partial ⊆ ϕ T ϕ V T V T ′ ( is a set of V Synthesize geometric objects) Given , , synthesize , such that ( ( )) T V partial ϕ T , ϕ V ϕ V ϕ T T ⊇ V partial 11

Visualization by Example 3. Weak Specification: “ ” instead of “=" ⊆ 2. Potentially large input table V partial e.g. 3000 x 10 bar(x=1, h=1, color=A) bar(x=1, h=4, color=B) ϕ T ⊆ ϕ V gather( T X → bar.x T, X Key Val Val → bar.height id=X, X A B C bar(x=1, h=1, color=A) 1 A 1 Key → bar.color key=[A,B,C]) 1 1 4 3 bar(x=1, h=4, color=B) 1 B 4 2 2 3 2 bar(x=1, h=3, color=C) 1 C 3 3 5 2 1 bar(x=2, h=2, color=A) 2 A 2 4 3 6 1 …… … … … 1. Compositional Synthesis: and ϕ T ∈ ℒ T ϕ V ∈ ℒ V Requirement: ( ( )) ϕ V ϕ T T ⊇ V partial 12

Visualization synthesis V partial bar(x=1, h=1, color=A) bar(x=1, h=4, color=B) ⊆ T X A B C ϕ V ϕ T 1 1 4 3 V T ′ 2 2 3 2 3 5 2 1 4 3 6 1 Requirement: ( ( )) ϕ V ϕ T T ⊇ V partial 13

Visualization synthesis Step 1: decompile visualization s.t., ( )= ϕ V T sketch V partial V partial T sketch bar(x=1, h=1, color=A) X Key Val bar(x=1, h=4, color=B) 1 A 1 1 B 4 ϕ V ⊆ T ⊆ X → bar.x Val → bar.height X A B C ϕ T Key → bar.color 1 1 4 3 V T ′ 2 2 3 2 3 5 2 1 4 3 6 1 Requirement: ( ( )) ϕ V ϕ T T ⊇ V partial 14

Visualization synthesis Step 1: decompile visualization s.t., ( )= ϕ V T sketch V partial Step 2: Synthesize data adapter s.t., T sketch ⊆ ϕ T ( T ) V partial T sketch bar(x=1, h=1, color=A) X Key Val bar(x=1, h=4, color=B) 1 A 1 ϕ T 1 B 4 ϕ V gather( ⊆ T ⊆ X → bar.x T, Val → bar.height id=X, X A B C Key → bar.color 1 1 4 3 key=[A,B,C]) V T ′ 2 2 3 2 3 5 2 1 4 3 6 1 Key: push the containment requirement from visualization to data adapter. Requirement: ( ( )) ϕ V ϕ T T ⊇ V partial 15

Step 1: Decompile Visualization V partial T sketch ⊆ ⊆ ϕ V T V T ′ Requirement: ( )= ϕ V T sketch V partial What data generates ? V partial bar(x=1, h=1, color=A) What mapping bar(x=1, h=4, color=B) generates ? V partial T sketch ϕ − 1 C1 → bar.x V C1 C2 C3 ϕ V C2 → bar.height 1 A 1 C3 → bar.color 1 B 4 (other alternatives …) Key: formalize visualization as mappings (and leave the challenges for tables) 16

Step 2: Data Adapter Synthesis C1 C2 C3 bar(x=1, h=1, color=A) bar(x=1, h=4, color=B) 1 A 1 1 B 4 V partial T sketch ⊆ ⊆ ϕ V ϕ T V T T ′ C1 → bar.x C2 → bar.height C3 → bar.color Requirement: ( ) ϕ T T ⊇ T sketch 17

Step 2: Data Adapter Synthesis V partial T sketch ⊆ ⊆ ϕ T ϕ V X A B C V T T ′ X Key Val 1 1 4 3 1 A 1 2 2 3 2 3 5 2 1 1 B 4 4 3 6 1 Requirement: ( ) ϕ T T ⊇ T sketch filter(T, A > 0) ❌ Holes “ ▢ ” are uninstantiated …… parameters of the partial program …… filter(T, A == 1) ❌ X Key Val 1 A 1 …… 1 B 4 filter(T, ⃞ ) 1 C 3 gather(T, key=C, val= ⃞ ) 2 A 2 …… … … … Start gather(T, key=A, val= ⃞ ) gather(T, key= ⃞ , val= ⃞ ) …… gather(T, key=X, val=[A, B, C]) ✓ …… gather(T, key=X, val=[A, B]) gather(T, key=X, val= ⃞ ) ❌ …… spread(T, id= ⃞ , key= ▢ , val= ▢ ) gather(T, key=X, val=[A]) ❌ …… [Wang PLDI17, Feng PLDI17, Feng PLDI18] 18

Step 2: Data Adapter Synthesis V partial T sketch ⊆ ⊆ ϕ T ϕ V X A B C V T T ′ X Key Val 1 1 4 3 1 A 1 2 2 3 2 3 5 2 1 1 B 4 4 3 6 1 For any predicate, we have Requirement: ( ) Contradicts : ϕ T T ⊇ T sketch filter(T, A > 0) X A B C X A B C 1 1 4 2 1 1 4 2 1 A 1 …… 4 ⊈ ϕ T ( T ) ⊆ 2 2 3 2 2 2 3 2 1 B 3 5 2 1 3 5 2 1 …… filter(T, A == 1) 4 3 6 1 4 3 6 1 X Y Label 1 1 A 1 4 B …… filter(T, ⃞ ) 1 3 C ❌ gather(T, key=C, val= ⃞ ) 2 2 A …… … … … Start gather(T, key=A, val= ⃞ ) gather(T, key= ⃞ , val= ⃞ ) …… Forward reasoning gather(T, key=X, val=[A, B, C]) …… Given and partial , T ϕ T gather(T, key=X, val=[A, B]) gather(T, key=X, val= ⃞ ) what’s the property of the output ? ϕ T ( T ) …… spread(T, id= ⃞ , key= ▢ , val= ▢ ) gather(T, key=X, val=[A]) …… [Wang PLDI17, Feng PLDI18] 19

Step 2: Data Adapter Synthesis V partial T sketch ⊆ ⊆ ϕ T ϕ V X A B C V T T ′ X Key Val 1 1 4 3 1 A 1 2 2 3 2 3 5 2 1 1 B 4 4 3 6 1 c1 c2 c3 Contradicts: = T sketch 1 A 1 Requirement: ( ) ϕ T T ⊇ T sketch X A B C 1 B 4 1 1 4 2 C A B 4 ⊈ 2 2 3 2 C A B 1 1 ⊆ T in Thus, 3 5 2 1 1 1 4 4 3 6 1 …… X Y Label 1 1 A 1 4 B …… filter(T, ⃞ ) 1 3 C ❌ gather(T, key=C, val= ⃞ ) ❌ 2 2 A …… … … … Start gather(T, key=A, val= ⃞ ) gather(T, key= ⃞ , val= ⃞ ) …… Backward reasoning : gather(T, key=X, val=[A, B, C]) …… Given property and partial , ϕ T ( T ) ⊇ T sketch ϕ T gather(T, key=X, val=[A, B]) gather(T, key=X, val= ⃞ ) what’s the property of ? T …… spread(T, id= ⃞ , key= ▢ , val= ▢ ) gather(T, key=X, val=[A]) 20

Step 2: Data Adapter Synthesis V partial T sketch ⊆ ⊆ ϕ T ϕ V X A B C V T T ′ X Key Val 1 1 4 3 1 A 1 2 2 3 2 3 5 2 1 1 B 4 4 3 6 1 Contribution: Bidirectional reasoning Requirement: ( ) ϕ T T ⊇ T sketch Inductively defined for all operators in ℒ T filter(T, ⃞ ) ❌ gather(T, key=C, val= ⃞ ) ❌ Start gather(T, key=A, val= ⃞ ) gather(T, key= ⃞ , val= ⃞ ) ❌ …… gather(T, key=X, val=[A, B, C]) ✓ …… gather(T, key=X, val= ⃞ ) gather(T, key=X, val=[A, B]) …… spread(T, id= ⃞ , key= ▢ , val= ▢ ) ❌ gather(T, key=X, val=[A]) 21

Visualization by Example ( ( )) ϕ V ϕ T T ⊇ V partial Potentially multiple pairs can satisfy ( ϕ T , ϕ V ) V partial the specification. bar(x=1, h=1, color=A) bar(x=1, h=4, color=B) ϕ T ⊆ ϕ V gather( T X → bar.x T, X Key Val Val → bar.height id=X, X A B C bar(x=1, h=1, color=A) 1 A 1 Key → bar.color key=[A,B,C]) 1 1 4 3 bar(x=1, h=4, color=B) 1 B 4 2 2 3 2 bar(x=1, h=3, color=C) 1 C 3 3 5 2 1 bar(x=2, h=2, color=A) 2 A 2 4 3 6 1 …… … … … 22