SLIDE 1
GI++ == Focused Auto Programming? Robert Feldt Chalmers University - - PowerPoint PPT Presentation
GI++ == Focused Auto Programming? Robert Feldt Chalmers University - - PowerPoint PPT Presentation
GI++ == Focused Auto Programming? Robert Feldt Chalmers University of Technology, Sweden at the COW-50, UCL, London, 2017-01-31 @drfeldt One view of SBSE: Ever-expanding Success! A contrarian view of SBSE: Not quite there yet A contrarian
SLIDE 2
SLIDE 3
A contrarian view of SBSE: Not quite there yet…
SLIDE 4
A contrarian view of SBSE: Not quite there yet…
SLIDE 5
A contrarian view of SBSE: Not quite there yet…
SLIDE 6
A contrarian view of SBSE: Not quite there yet…
"Evolution is the natural way to program” - Tom Ray
SLIDE 7
A contrarian view of SBSE: Not quite there yet…
"I would rather fly on a plane running software evolved by a program like this, than fly on a plane running software I wrote myself," says Hillis, programmer extraordinaire.
SLIDE 8
Of course it all started much earlier (with Turing)… ;)
[Koza2010] in GPEM Anniversary issue
SLIDE 9
Some common GP/SBSE “cop outs”
SLIDE 10
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
SLIDE 11
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
SLIDE 12
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
SLIDE 13
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
- Focus on non-mainstream/obscure languages /
processing formalisms where humans (currently) have less experience
SLIDE 14
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
- Focus on non-mainstream/obscure languages /
processing formalisms where humans (currently) have less experience
- Evolve test data rather than programs
SLIDE 15
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
- Focus on non-mainstream/obscure languages /
processing formalisms where humans (currently) have less experience
- Evolve test data rather than programs
- Evolve test cases and not programs
SLIDE 16
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
- Focus on non-mainstream/obscure languages /
processing formalisms where humans (currently) have less experience
- Evolve test data rather than programs
- Evolve test cases and not programs
- Requiring lots and lots of example Input/Outputs
SLIDE 17
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
- Focus on non-mainstream/obscure languages /
processing formalisms where humans (currently) have less experience
- Evolve test data rather than programs
- Evolve test cases and not programs
- Requiring lots and lots of example Input/Outputs
- …
SLIDE 18
Some common GP/SBSE “cop outs”
- Tune only constants/numbers in fixed program
- Delete/remix existing code
- Focus on (minimal) interfaces between existing codes
- Focus on non-mainstream/obscure languages /
processing formalisms where humans (currently) have less experience
- Evolve test data rather than programs
- Evolve test cases and not programs
- Requiring lots and lots of example Input/Outputs
- …
Clear goal, small search space, less/short structure
SLIDE 19
A continuum of Automated Programming
SLIDE 20
A continuum of Automated Programming
Complexity Time
SLIDE 21
A continuum of Automated Programming
Complexity Time
GP
SLIDE 22
A continuum of Automated Programming
Complexity Time
GP AP?
SLIDE 23
A continuum of Automated Programming
Complexity Time
GP AP? AP!
SLIDE 24
A continuum of Automated Programming
Complexity Time
GP AP? AP! GI!?
SLIDE 25
A continuum of Automated Programming
Complexity Time
GP AP? AP! GI!?
Focused AP!?
SLIDE 26
Focused Automated Programming
SLIDE 27
Focused Automated Programming
- I propose we should study FAP! aka…
SLIDE 28
Focused Automated Programming
- I propose we should study FAP! aka…
- Domain-specific Automated Programming (DAP)
SLIDE 29
Focused Automated Programming
- I propose we should study FAP! aka…
- Domain-specific Automated Programming (DAP)
- Task-specific Automated Programming (TAP)
SLIDE 30
Focused Automated Programming
- I propose we should study FAP! aka…
- Domain-specific Automated Programming (DAP)
- Task-specific Automated Programming (TAP)
- Defined as: “Focused application of search and
- ptimisation to create/adapt/tune (parts of) program
code during its development, setup and/or execution”
SLIDE 31
Focused Automated Programming
- I propose we should study FAP! aka…
- Domain-specific Automated Programming (DAP)
- Task-specific Automated Programming (TAP)
- Defined as: “Focused application of search and
- ptimisation to create/adapt/tune (parts of) program
code during its development, setup and/or execution”
- Focused here essentially means “human-guided”, i.e.
it is a hybrid/interactive development philosophy
SLIDE 32
Focused Automated Programming
- I propose we should study FAP! aka…
- Domain-specific Automated Programming (DAP)
- Task-specific Automated Programming (TAP)
- Defined as: “Focused application of search and
- ptimisation to create/adapt/tune (parts of) program
code during its development, setup and/or execution”
- Focused here essentially means “human-guided”, i.e.
it is a hybrid/interactive development philosophy
- => we need ideas, intuition and methods/processes
for how to use search/optimisation more actively in the software development process
SLIDE 33
Example: Web extraction library
SLIDE 34
Example: Web extraction library
SLIDE 35
Example: Web extraction library
{ “name”: “V Basili”, “citations”: 33501, “h-index”: 82 }
SLIDE 36
Web extraction, traditional solution vs AdaptiLib
SLIDE 37
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
SLIDE 38
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
SLIDE 39
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib
SLIDE 40
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib Regex Lib
SLIDE 41
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib Regex Lib
+
SLIDE 42
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib Regex Lib
+
Custom code
SLIDE 43
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib Regex Lib
+
Custom code AWE Lib
SLIDE 44
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib Regex Lib
+
Custom code AWE Lib
+
SLIDE 45
Web extraction, traditional solution vs AdaptiLib
WebGet Lib
+
XML Parser Lib Regex Lib
+
Custom code AWE Lib
+
Examples
SLIDE 46
Adaptive Libraries
SLIDE 47
Adaptive Libraries
- A normal library (lib):
SLIDE 48
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
SLIDE 49
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
SLIDE 50
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
SLIDE 51
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
SLIDE 52
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
- But only 1 above is directly useable without a human
SLIDE 53
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
- But only 1 above is directly useable without a human
- 2-4 requires a human to assemble solution based on text
SLIDE 54
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
- But only 1 above is directly useable without a human
- 2-4 requires a human to assemble solution based on text
- Adaptive libraries (AdaptiLibs):
SLIDE 55
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
- But only 1 above is directly useable without a human
- 2-4 requires a human to assemble solution based on text
- Adaptive libraries (AdaptiLibs):
- 1. Still has basic “atoms” = functions to be called
SLIDE 56
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
- But only 1 above is directly useable without a human
- 2-4 requires a human to assemble solution based on text
- Adaptive libraries (AdaptiLibs):
- 1. Still has basic “atoms” = functions to be called
- (2a) But also executable examples that uses atoms to
perform specific, named sequences
SLIDE 57
Adaptive Libraries
- A normal library (lib):
- 1. has a number of functions that can be called
- 2. to solve specific tasks
- 3. has documentation to describe the functions
- 4. and examples to understand API & how to put together
- But only 1 above is directly useable without a human
- 2-4 requires a human to assemble solution based on text
- Adaptive libraries (AdaptiLibs):
- 1. Still has basic “atoms” = functions to be called
- (2a) But also executable examples that uses atoms to
perform specific, named sequences
- (2b) And allow fuzzy mapping of user needs to tasks
SLIDE 58
Example: Adaptive Web Extraction (AWE!) library, in practice
SLIDE 59
Example: Adaptive Web Extraction (AWE!) library, in practice
examples = [ (“scholar.google.se/citations?user=B3C4aY8AAAAJ&hl=en”, {“name”: “V Basili”, “citations”: 33501, “h-index”: 82}), (“scholar.google.se/citations?user=Zj897NoAAAAJ&hl=en”, {“name”: “Lionel Briand”, “citations”: 21505, “h-index”: 69})]
SLIDE 60
Example: Adaptive Web Extraction (AWE!) library, in practice
examples = [ (“scholar.google.se/citations?user=B3C4aY8AAAAJ&hl=en”, {“name”: “V Basili”, “citations”: 33501, “h-index”: 82}), (“scholar.google.se/citations?user=Zj897NoAAAAJ&hl=en”, {“name”: “Lionel Briand”, “citations”: 21505, “h-index”: 69})] gscholar_ex = create_extractor(examples)
SLIDE 61
Example: Adaptive Web Extraction (AWE!) library, in practice
examples = [ (“scholar.google.se/citations?user=B3C4aY8AAAAJ&hl=en”, {“name”: “V Basili”, “citations”: 33501, “h-index”: 82}), (“scholar.google.se/citations?user=Zj897NoAAAAJ&hl=en”, {“name”: “Lionel Briand”, “citations”: 21505, “h-index”: 69})] gscholar_ex = create_extractor(examples) extract(gscholar_ex, “scholar.google.se/citations? user=CQDOm2gAAAAJ&hl=en”)
SLIDE 62
Example: Adaptive Web Extraction (AWE!) library, in practice
examples = [ (“scholar.google.se/citations?user=B3C4aY8AAAAJ&hl=en”, {“name”: “V Basili”, “citations”: 33501, “h-index”: 82}), (“scholar.google.se/citations?user=Zj897NoAAAAJ&hl=en”, {“name”: “Lionel Briand”, “citations”: 21505, “h-index”: 69})] gscholar_ex = create_extractor(examples) extract(gscholar_ex, “scholar.google.se/citations? user=CQDOm2gAAAAJ&hl=en”) # returns: # {“name”: “Barbara Ann Kitchenham”, # “citations”: 63, # “h-index”: 154})]
SLIDE 63
Big benefits with semantically similar task
{ “name”: “V Basili”, “citations”: 33501, “h-index”: 82 }
SLIDE 64
Big benefits with semantically similar task
{ “name”: “V Basili”, “citations”: 33501, “h-index”: 82 }
SLIDE 65
Big benefits with semantically similar task
{ “name”: “Victor R. Basili”, “citations”: 36839, “influential”: 322 }
Only change 2 I/O examples & re-adapt!
SLIDE 66
GI would not help: Only semantic, not syntactic similarity
“...>Citations</a></td><td class="gsc_rsb_std">33501</ td><td class=“gsc_rsb_std”>9054</td>..." “...:{“hIndex”:51,”estimatedTotalCitationCount”:{“min": 31675,"value":36839,"max":42905,...”
SLIDE 67
Design Rules for AdaptiLibs (so far…)
SLIDE 68
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
SLIDE 69
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
SLIDE 70
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
SLIDE 71
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
SLIDE 72
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
SLIDE 73
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
SLIDE 74
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
SLIDE 75
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
- Feldt’s Law for Designing Lib incl. Search, consider in order:
SLIDE 76
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
- Feldt’s Law for Designing Lib incl. Search, consider in order:
- 1. Deterministic / Exact (fastest, most efficient)
SLIDE 77
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
- Feldt’s Law for Designing Lib incl. Search, consider in order:
- 1. Deterministic / Exact (fastest, most efficient)
- 2. Heuristics / Approximations (order by applicability)
SLIDE 78
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
- Feldt’s Law for Designing Lib incl. Search, consider in order:
- 1. Deterministic / Exact (fastest, most efficient)
- 2. Heuristics / Approximations (order by applicability)
- 3. Focused Search (part of solution only, then aggregate)
SLIDE 79
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
- Feldt’s Law for Designing Lib incl. Search, consider in order:
- 1. Deterministic / Exact (fastest, most efficient)
- 2. Heuristics / Approximations (order by applicability)
- 3. Focused Search (part of solution only, then aggregate)
- 4. Interact / Ask Developer (in adapt step)
SLIDE 80
Design Rules for AdaptiLibs (so far…)
- Start by defining basic “atomic” operations
- Type conversion operations: parseToInt, parseToFloat
- Data transformation: uppercase, lowercase, leadingcase
- Basic data access: get_url
- Matching: matchregexp, matchregexp_ignorecase
- Go through concrete task from example & note how a
human solves it in as atomic steps as possible
- Extend with atoms, and possibly (complex) atom seq.
- Feldt’s Law for Designing Lib incl. Search, consider in order:
- 1. Deterministic / Exact (fastest, most efficient)
- 2. Heuristics / Approximations (order by applicability)
- 3. Focused Search (part of solution only, then aggregate)
- 4. Interact / Ask Developer (in adapt step)
- 5. Full/free search (search from atoms & up, warn dev)
SLIDE 81
Conclusions
SLIDE 82
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
SLIDE 83
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
SLIDE 84
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
- As techniques and processing power increase we will see
more practical AP
SLIDE 85
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
- As techniques and processing power increase we will see
more practical AP
- But semantic similarity does not imply syntactic similarity
=> less opportunity for detailed code reuse
SLIDE 86
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
- As techniques and processing power increase we will see
more practical AP
- But semantic similarity does not imply syntactic similarity
=> less opportunity for detailed code reuse
- But we can also deliver practical AP now by hybridising it
with human intelligence and guidance
SLIDE 87
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
- As techniques and processing power increase we will see
more practical AP
- But semantic similarity does not imply syntactic similarity
=> less opportunity for detailed code reuse
- But we can also deliver practical AP now by hybridising it
with human intelligence and guidance
- We are developing AdaptiLibs, general libraries that adapt to
I/O examples of users/developers
SLIDE 88
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
- As techniques and processing power increase we will see
more practical AP
- But semantic similarity does not imply syntactic similarity
=> less opportunity for detailed code reuse
- But we can also deliver practical AP now by hybridising it
with human intelligence and guidance
- We are developing AdaptiLibs, general libraries that adapt to
I/O examples of users/developers
- Combines task-driven design & experience of humans
SLIDE 89
Conclusions
- Despite many promises of GP & SBSE it has under
delivered on practical Automated Programming
- Compared to other SBSE, GI comes closer to AP
- As techniques and processing power increase we will see
more practical AP
- But semantic similarity does not imply syntactic similarity
=> less opportunity for detailed code reuse
- But we can also deliver practical AP now by hybridising it
with human intelligence and guidance
- We are developing AdaptiLibs, general libraries that adapt to
I/O examples of users/developers
- Combines task-driven design & experience of humans
- with brute force and flexibility of search, only wh. needed
SLIDE 90
Thank you!
robert.feldt@chalmers.se
@drfeldt
SLIDE 91
But what about Bartoli et al?!
SLIDE 92