SLIDE 1
people people name name id id age age - - PowerPoint PPT Presentation
people people name name id id age age - - PowerPoint PPT Presentation
people people name name id id age age stephanie stephanie 1 1 19 19 Query 1 dylan dylan 2 2 26 26 people.filter{p => p.age 18} mary kate mary kate 3 3 17 17 pets
SLIDE 2
SLIDE 3
- ○
○
- ○
○
- ○
SLIDE 4
- ○
SLIDE 5
SLIDE 6
Query 1 people.filter{p => p.age 18}
people name id age stephanie 1 19 dylan 2 26 mary kate 3 17
Query 2
people.join(pets, "id === owner") .filter(people.age 18)
pets name
- wner
catsidy 2 gigi 3 people name id age stephanie 1 19 dylan 2 26 mary kate 3 17
SLIDE 7
Physical Planning Cache Substitution Optimization
people.filter(age 18)
filter { p => p.age > 18 } table people filter { p => p.age > 18 } table people filter { p => p.age > 18 } FileScan people Cache filter { p => p.age > 18 } table people
SLIDE 8
Physical Planning Cache Substitution Optimization
people.join(pets, "id === owner") .filter(people.age 18)
Cache filter { p => p.age > 18 } table people filter people.age > 18 join (owner, id) table people table pets select * filter people.age > 18 table people table pets select * join (owner, id) filter people.age > 18 filescan people filescan pets select * hashjoin (owner, id)
SLIDE 9
Physical Planning Cache Substitution Optimization Cache filter { p => p.age > 18 } table people filter people.age > 18 join (owner, id) table people table pets select * filter people.age > 18 table people table pets select * join (owner, id) filter people.age > 18 filescan people filescan pets select * hashjoin (owner, id)
SLIDE 10
○ ○ ○ ○
SLIDE 11
Cache
Current Pipeline
Optimization Cache Physical Planning Optimization Physical Planning
Optimization-first pipeline
SLIDE 12
SLIDE 13
- ○
○
- ○
○ ○
SLIDE 14
Cache
Current Pipeline
Optimization Cache Physical Planning Optimization Physical Planning Partial Optimization Cache Optimization
Insight: not all optimizations help caching!
Physical Planning
Optimization-first pipeline (slow!)
SLIDE 15
Boolean Simplification Constant Propagation ID Reassignment Filter Pruning Object Elimination Custom Rules ...
SLIDE 16
SLIDE 17
SLIDE 18
○ ○ ○ ○
SLIDE 19
UDFs are blackboxes that hide caching opportunities
select * where age > 18 table people select * table people { p => p.age > 18 }
SLIDE 20
Program Synthesis User Annotation Froid Acorn
SLIDE 21
Program Synthesis User Annotation Froid Acorn Correct
✓ ✓ ✓ ✓
SLIDE 22
Program Synthesis User Annotation Froid Acorn Correct
✓ ✓ ✓ ✓
Transparent
✓ X ✓ ✓
SLIDE 23
Program Synthesis User Annotation Froid Acorn Correct
✓ ✓ ✓ ✓
Transparent
✓ X ✓ ✓
General
(Java, Scala)
✓ X X ✓
SLIDE 24
Program Synthesis User Annotation Froid Acorn Correct
✓ ✓ ✓ ✓
Transparent
✓ X ✓ ✓
General
(Java, Scala)
✓ X X ✓
Fast
X ✓ ✓ ✓
SLIDE 25
- Scala
Native Spark
SLIDE 26
person.filter(p => p.age > 18) 1 aload_1 2 invokeinterface 3 dload_1 4 ldc2_w 5 dcmpg 6 ifge 18 7 iconst_1 8 goto 10 9 iconst_0 10 aload_0 11 aload_1 1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 < $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
SLIDE 27
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this
SLIDE 28
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this d0 double Attribute("age")
SLIDE 29
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this d0 double Attribute("age") d1 int Literal(18)
SLIDE 30
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this d0 double Attribute("age") d1 int Literal(18) If GreaterThan(Attribute("age"), Literal(18))
SLIDE 31
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this d0 double Attribute("age") d1 int Literal(18) If GreaterThan(Attribute("age"), Literal(18)) cast (0) as boolean
SLIDE 32
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this d0 double Attribute("age") d1 int Literal(18) If GreaterThan(Attribute("age"), Literal(18)) cast(1) as boolean
SLIDE 33
1 Person r1 := @param0 2 double $d0 = r1.age() 3 int $d1 = 18 4 if $d0 > $d1 5 goto 8 6 boolean $zo = 1 7 goto 9 8 $zo = 0 9 return $zo
Name Type Expression r1 class[Person] this d0 double Attribute("age") d1 int Literal(18) If GreaterThan(Attribute("age"), Literal(18)) cast (1) as boolean cast (0) as boolean
SLIDE 34
IF GreaterThan(Attribute("age"), Literal(18) cast (1) as boolean cast(0) as boolean
person.filter(p => p.age > 18)
select age table people filterUDF{ p => p.age > 18 }
SLIDE 35
IF GreaterThan(Attribute("age"), Literal(18)) cast (1) as boolean cast(0) as boolean
person.filter(p => p.age > 18)
select age table people filter(If(GreaterThan("age", 18), cast 0 as boolean, cast 1 as boolean))
SLIDE 36
person.filter(p => p.age > 18)
select * table people filter(If(GreaterThan("age", 18), cast 0 as boolean, cast 1 as boolean)) Partial Optimizer select * table people filter "age" > 18
person.filter(age > 18)
SLIDE 37
SLIDE 38
SLIDE 39
SLIDE 40
SLIDE 41
SLIDE 42
SLIDE 43
- ○