How Far Can Machines Go? Hila Peleg Technion Laziness Laziness - - PowerPoint PPT Presentation

▶
how far can machines go
SMART_READER_LITE
LIVE PREVIEW

How Far Can Machines Go? Hila Peleg Technion Laziness Laziness - - PowerPoint PPT Presentation

Feb 10, 2024 277 likes •1.01k views

Automatic Programming: How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness Impatience Hubris Automatic Programming Clean up my spreadsheet! Right away, boss! Program synthesis to the rescue Program

slide-1
SLIDE 1

Automatic Programming: How Far Can Machines Go?

Hila Peleg Technion

slide-2
SLIDE 2
slide-3
SLIDE 3

Laziness

slide-4
SLIDE 4

Laziness Impatience

slide-5
SLIDE 5

Laziness Impatience Hubris

slide-6
SLIDE 6

Automatic Programming

Clean up my spreadsheet! Right away, boss!

slide-7
SLIDE 7

Program synthesis to the rescue

slide-8
SLIDE 8

Program Synthesis

⇒ ⇒

slide-9
SLIDE 9

What we really want

⇒ ⇒

This is what I want

slide-10
SLIDE 10

What if we have automatic programming?

  • Hyper-intelligent program generation for your every need
slide-11
SLIDE 11

What if we have automatic programming?

  • Hyper-intelligent program generation for your every need
  • Self-aware, self-augmenting AI
slide-12
SLIDE 12

What if we have automatic programming?

  • Hyper-intelligent program generation for your every need
  • Self-aware, self-augmenting AI
  • The singularity
slide-13
SLIDE 13

What if we have automatic programming?

  • Hyper-intelligent program generation for your every need
  • Self-aware, self-augmenting AI
  • The singularity
  • ???
slide-14
SLIDE 14

What if we have automatic programming?

  • Hyper-intelligent program generation for your every need
  • Self-aware, self-augmenting AI
  • The singularity
  • ???
  • THE ROBOT APOCALYPSE
slide-15
SLIDE 15

The Robot Apocalypse

slide-16
SLIDE 16

But it’s ok!

slide-17
SLIDE 17

But it’s ok!

Understand users Build a program

slide-18
SLIDE 18

But it’s ok!

Generalizing partial intent is hard Understand users Build a program

slide-19
SLIDE 19

But it’s ok!

Generalizing partial intent is hard Solving HALT is hard* Understand users Build a program

slide-20
SLIDE 20
slide-21
SLIDE 21

User intent is hard

Clean up my spreadsheet! Right away, boss!

slide-22
SLIDE 22

User intent is hard

Clean up my spreadsheet! Right away, boss! What’s “clean”? Is this right? What am I doing with my life?

slide-23
SLIDE 23

Intent via examples

slide-24
SLIDE 24

Intent via examples

slide-25
SLIDE 25

Intent via examples

slide-26
SLIDE 26

Intent via examples

slide-27
SLIDE 27

Query knowledge about (some kind of) code

I want to apply foo() to requests to my http server

slide-28
SLIDE 28

Query knowledge about (some kind of) code

I want to apply foo() to requests to my http server

val routes : Route = ??? val bindingFuture = Http(). bindAndHandle(routes, "localhost", 8080)

slide-29
SLIDE 29

Commit Strip said it best…

http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/

slide-30
SLIDE 30

Commit Strip said it best…

http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/

slide-31
SLIDE 31

Commit Strip said it best…

http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/

slide-32
SLIDE 32

Commit Strip said it best…

http://www.commitstrip.com/en/2016/08/25/a-very-comprehensive-and-precise-spec/

slide-33
SLIDE 33

Building a program is also hard*

Get me a program that takes a program and an input and tells me if that program stops on that input.

slide-34
SLIDE 34

Building a program is also hard*

Get me a program that takes a program and an input and tells me if that program stops on that input. (i.e., the halting problem)

slide-35
SLIDE 35

Adjusting our expectations

Here’s a grammar of 20 functions and 10 constants, get me a program that I’m certain is in this space.

slide-36
SLIDE 36

Adjusting our expectations

Here’s a grammar of 20 functions and 10 constants, get me a program that I’m certain is in this space.

slide-37
SLIDE 37

Adjusting our expectations

Here’s a grammar of 20 functions and 10 constants, get me a program that I’m certain is in this space.

slide-38
SLIDE 38

What does this all mean?

slide-39
SLIDE 39

What does this all mean?

  • Generally, “find me a program that—” cannot be solved
slide-40
SLIDE 40

What does this all mean?

  • Generally, “find me a program that—” cannot be solved
  • Still, we’re not giving up
slide-41
SLIDE 41

What does this all mean?

  • Generally, “find me a program that—” cannot be solved
  • Still, we’re not giving up
  • Realistic expectations for realistic program synthesis
slide-42
SLIDE 42

Realistic expectations for realistic synthesis

We still want

  • Partial specifications
  • To not have to know

everything

  • A result!

But we’ll have to live without

  • Checking every possible

program

  • Fully automatic solution
  • Single-step solution
slide-43
SLIDE 43

Realistic expectations for realistic synthesis

We still want

  • Partial specifications
  • To not have to know

everything

  • A result!

But we’ll have to live without

  • Checking every possible

program

  • Fully automatic solution
  • Single-step solution

Synthesis Engine Interaction Model

slide-44
SLIDE 44

The synthesis engine

  • Predicts code for intent
  • Draws its understanding

from language syntax

  • and/or crowd wisdom
  • and/or semantic

specifications

  • Reduce the number of

programs seen

slide-45
SLIDE 45

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

slide-46
SLIDE 46

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

  • But how do we know they’re the

same?

slide-47
SLIDE 47

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

  • But how do we know they’re the

same?

  • 1. Heuristics
slide-48
SLIDE 48

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

  • But how do we know they’re the

same?

  • 1. Heuristics
  • 2. Solvers (e.g., Z3)
slide-49
SLIDE 49

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

  • But how do we know they’re the

same?

  • 1. Heuristics
  • 2. Solvers (e.g., Z3)
  • 3. Observational Equivalence
slide-50
SLIDE 50

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

  • But how do we know they’re the

same?

  • 1. Heuristics
  • 2. Solvers (e.g., Z3)
  • 3. Observational Equivalence

Equivalence: 𝑞1 ≡ 𝑞2 i.f.f. for every possible input 𝑗 ever, 𝑞1 𝑗 = 𝑞2 (𝑗)

slide-51
SLIDE 51

Reducing equivalent programs

  • We’ve seen x+y, so we don’t

want y+x

  • But how do we know they’re the

same?

  • 1. Heuristics
  • 2. Solvers (e.g., Z3)
  • 3. Observational Equivalence

Observational equivalence: 𝑞1 ≡𝑃𝐹 𝑞2 i.f.f. for every input 𝑗 the user cares about, 𝑞1 𝑗 = 𝑞2 (𝑗) Equivalence: 𝑞1 ≡ 𝑞2 i.f.f. for every possible input 𝑗 ever, 𝑞1 𝑗 = 𝑞2 (𝑗)

slide-52
SLIDE 52

Trying a different strategy altogether

  • Use knowledge bases instead of

the language grammar

  • They no longer contain every

program (neither limited grammars)

  • Searchable via graph algorithms
  • r probability equations
slide-53
SLIDE 53

The Interaction Model

  • Aimed at programmers
  • Specify intent
  • Express yourself
  • Think like a programmer
slide-54
SLIDE 54

Specifying (and re-specifying) intent

Task: find the median of a list User: examples!

  • 1. [1,2,3]→2
  • 2. [7,8,7,3]→7

Synthesis engine:

input[input.length/2]

User:

slide-55
SLIDE 55

Specifying (and re-specifying) intent

Task: find the median of a list User: examples!

  • 1. [1,2,3]→2
  • 2. [7,8,7,3]→7

Synthesis engine:

input[input.length/2]

User:

It managed to find a single formula, let’s make a counterexample

slide-56
SLIDE 56

Specifying (and re-specifying) intent

Task: find the median of a list User: examples!

  • 1. [1,2,3]→2
  • 2. [7,8,7,3]→7

Synthesis engine:

input[input.length/2]

User:

It managed to find a single formula, let’s make a counterexample

Principle #1: Cost of communicating intent + consuming result << cost of manually performing the task

slide-57
SLIDE 57

Programming Not Only by Example

input[input.length/2] [] input / . 2 input length

  • A programmer can talk at

the level of the program

  • Read debug info
  • Reason about subtrees or

sequences of methods

  • Even rewrite the program
  • But also give examples, if

those happen to be easier

slide-58
SLIDE 58

Programming Not Only by Example

input[input.length/2] [] input / . 2 input length

  • A programmer can talk at

the level of the program

  • Read debug info
  • Reason about subtrees or

sequences of methods

  • Even rewrite the program
  • But also give examples, if

those happen to be easier

Retain Exclude

slide-59
SLIDE 59

Programming Not Only by Example

input[input.length/2] [] input / . 2 input length

  • A programmer can talk at

the level of the program

  • Read debug info
  • Reason about subtrees or

sequences of methods

  • Even rewrite the program
  • But also give examples, if

those happen to be easier Principle #2: Let developers be developers

Retain Exclude

slide-60
SLIDE 60

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l.

slide-61
SLIDE 61

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l. groupBy( map( fold(

slide-62
SLIDE 62

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l.groupBy( identity) x => x.length) x => x[0])

slide-63
SLIDE 63

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l.groupBy(identity) . map(x => x._1 -> x._2 x => x.length) x => x[0])

slide-64
SLIDE 64

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l.groupBy(identity) . map(x => x._1 -> x._2. filter(y => y.startsWith( x._2.length x._2[0])

slide-65
SLIDE 65

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l.groupBy(identity) . map(x => x._1 -> x._2. filter(y => y.startsWith(

slide-66
SLIDE 66

When models “out-think” the programmer

def counts(l : List[String]) : Map[String,Int]= l.groupBy(identity) . map(x => x._1 -> x._2. filter(y => y.startsWith(

slide-67
SLIDE 67

Keep understanding what’s going on

??

slide-68
SLIDE 68

Keep understanding what’s going on

??

slide-69
SLIDE 69

Keep understanding what’s going on

??

Principle #3: Results must be explainable

slide-70
SLIDE 70

Programmers aren’t as good as they think

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

histogram

  • no. lines with text

most frequent word

% correct answer

PBE Only new operations All operations

slide-71
SLIDE 71

Read-Eval-Synth Loops

slide-72
SLIDE 72

What if we have automatic programming?

  • Hyper-intelligent program generation for your every need
  • Self-aware, self-augmenting AI
  • The singularity
  • ???
  • THE ROBOT APOCALYPSE