Searching for Diverse Software Engineering Solutions Robert Feldt, - - PowerPoint PPT Presentation

▶
searching for diverse software engineering solutions
SMART_READER_LITE
LIVE PREVIEW

Searching for Diverse Software Engineering Solutions Robert Feldt, - - PowerPoint PPT Presentation

Mar 03, 2024 233 likes •685 views

Searching for Diverse Software Engineering Solutions Robert Feldt, robert.feldt@chalmers.se 23rd of March 2012, COW18, London HOSE Lab (Human-fOcused SE) Division of Software Engineering Chalmers Univ of Tech, Sweden fredag den 13 april 2012

slide-1
SLIDE 1

Searching for Diverse Software Engineering Solutions

Robert Feldt, robert.feldt@chalmers.se

23rd of March 2012, COW18, London

Division of Software Engineering

HOSE Lab (Human-fOcused SE)

Chalmers Univ of Tech, Sweden

fredag den 13 april 2012

slide-2
SLIDE 2

How make software less brittle / more robust?

fredag den 13 april 2012

slide-3
SLIDE 3

How make software less brittle / more robust?

Engineered Systems

  • ften brittle!

fredag den 13 april 2012

slide-4
SLIDE 4

How make software less brittle / more robust?

Engineered Systems

  • ften brittle!

Biological Systems sometimes robust!

fredag den 13 april 2012

slide-5
SLIDE 5

How make software less brittle / more robust?

Engineered Systems

  • ften brittle!

Biological Systems sometimes robust!

?

fredag den 13 april 2012

slide-6
SLIDE 6

N-Version Programming

1

fredag den 13 april 2012

slide-7
SLIDE 7

N-Version Programming

1

fredag den 13 april 2012

slide-8
SLIDE 8

N-Version Programming

1 2 3

fredag den 13 april 2012

slide-9
SLIDE 9

N-Version Programming

1 2 3 Vote

fredag den 13 april 2012

slide-10
SLIDE 10

N-Version Programming

1 2 3 Vote Design Diversity

fredag den 13 april 2012

slide-11
SLIDE 11

N-Version Programming

1 2 3 Vote Design Diversity

fredag den 13 april 2012

slide-12
SLIDE 12

N-Version Programming

1 2 3 Vote Design Diversity

X

fredag den 13 april 2012

slide-13
SLIDE 13

N-Version Programming

1 2 3 Vote Design Diversity

X

fredag den 13 april 2012

slide-14
SLIDE 14

N-Version Programming

1 2 3 Vote Design Diversity

X

GP

fredag den 13 april 2012

slide-15
SLIDE 15

N-Version Programming

1 2 3 Vote Design Diversity

X

GP

fredag den 13 april 2012

slide-16
SLIDE 16

fredag den 13 april 2012

slide-17
SLIDE 17

fredag den 13 april 2012

slide-18
SLIDE 18

fredag den 13 april 2012

slide-19
SLIDE 19

# test cases # failing programs 22 79 24 78

Failure Diversity Structural Diversity

Size Statistic Value Max 459 Average 100.2 Min 17

fredag den 13 april 2012

slide-20
SLIDE 20

Diversity at different levels

fredag den 13 april 2012

slide-21
SLIDE 21

Diversity at different levels

Statistic All Top 10 Min correlation

  • 0.21

0.55 Max Distinct failures 98.9% 59.7%

Pair-wise Failure Diversity between Programs

fredag den 13 april 2012

slide-22
SLIDE 22

Diversity at different levels

Statistic All Top 10 Min correlation

  • 0.21

0.55 Max Distinct failures 98.9% 59.7%

Pair-wise Failure Diversity between Programs Pair-wise Failure Diversity between Methods Most are high, 11 below 0.20, and 2 negative (but they are worst)

fredag den 13 april 2012

slide-23
SLIDE 23

Diversity at different levels

Statistic All Top 10 Min correlation

  • 0.21

0.55 Max Distinct failures 98.9% 59.7%

Pair-wise Failure Diversity between Programs Pair-wise Failure Diversity between Methods Most are high, 11 below 0.20, and 2 negative (but they are worst) Failure Diversity Inter-Method vs Intra-Method Inter-Method Diversity higher (p<10^-10)

fredag den 13 april 2012

slide-24
SLIDE 24

Diversity at different levels

Statistic All Top 10 Min correlation

  • 0.21

0.55 Max Distinct failures 98.9% 59.7%

Pair-wise Failure Diversity between Programs Pair-wise Failure Diversity between Methods Most are high, 11 below 0.20, and 2 negative (but they are worst) Failure Diversity Inter-Method vs Intra-Method Inter-Method Diversity higher (p<10^-10) 34% of all 3VP voters (worst case) improved, best was -20%

fredag den 13 april 2012

slide-25
SLIDE 25

Explanations?

fredag den 13 april 2012

slide-26
SLIDE 26

Explanations?

Crowd error = Average Individual Error - Diversity among Individuals Diversity Prediction Theorem [Page2007]:

fredag den 13 april 2012

slide-27
SLIDE 27

Explanations?

Crowd error = Average Individual Error - Diversity among Individuals Diversity Prediction Theorem [Page2007]: Diversity trumps Ability [Hong&Page2004]: Random group of solvers often outperform a group of best solvers

fredag den 13 april 2012

slide-28
SLIDE 28

Explanations?

Crowd error = Average Individual Error - Diversity among Individuals Diversity Prediction Theorem [Page2007]: Diversity trumps Ability [Hong&Page2004]: Random group of solvers often outperform a group of best solvers

Logic: Ideal group = High ability & Diverse But: Larger pool of solvers => Best solvers more similar => Highest-ability solvers are not diverse Group size matters: Too small => Random solvers overlap, Too large => Best solvers can differ

fredag den 13 april 2012

slide-29
SLIDE 29

Explanations?

Crowd error = Average Individual Error - Diversity among Individuals Diversity Prediction Theorem [Page2007]: Diversity trumps Ability [Hong&Page2004]: Random group of solvers often outperform a group of best solvers

Logic: Ideal group = High ability & Diverse But: Larger pool of solvers => Best solvers more similar => Highest-ability solvers are not diverse Group size matters: Too small => Random solvers overlap, Too large => Best solvers can differ

BUT, disregards Communication and Learning

fredag den 13 april 2012

slide-30
SLIDE 30

But Diversity is not a simple concept...

“Bolt Effect”

[The Indepedent, March 23 2012]

fredag den 13 april 2012

slide-31
SLIDE 31

Analysis of Failures of GP programs

fredag den 13 april 2012

slide-32
SLIDE 32

Analysis of Failures of GP programs

Main difficulty: High energies!

fredag den 13 april 2012

slide-33
SLIDE 33

Analysis of Failures of GP programs

Too High Retardation; but only slightly

fredag den 13 april 2012

slide-34
SLIDE 34

Analysis of Failures of GP programs

Slightly “under” spec points; continuous spec instead?

fredag den 13 april 2012

slide-35
SLIDE 35

Analysis of Failures of GP programs

Cable force never violated; use weaker cable!?

fredag den 13 april 2012

slide-36
SLIDE 36

Analysis of Failures of GP programs

Clustering the programs showed fundamental tradeoff: Retardation or

  • verrun

fredag den 13 april 2012

slide-37
SLIDE 37

Many Limitations

fredag den 13 april 2012

slide-38
SLIDE 38

Many Limitations

Small target application

fredag den 13 april 2012

slide-39
SLIDE 39

Many Limitations

Small target application Few requirements

fredag den 13 april 2012

slide-40
SLIDE 40

Many Limitations

Small target application Few requirements Low-dimensional input space

fredag den 13 april 2012

slide-41
SLIDE 41

Many Limitations

Small target application Few requirements Low-dimensional input space Existing simulator; typically not available in early phases

fredag den 13 april 2012

slide-42
SLIDE 42

Many Limitations

Small target application Few requirements Low-dimensional input space Existing simulator; typically not available in early phases Fundamental assumption: SB AutoProgramming fail in similar ways to human programmers

fredag den 13 april 2012

slide-43
SLIDE 43

Generalization: Search-Based SW Prg Exploration

fredag den 13 april 2012

slide-44
SLIDE 44

http://www.cse.chalmers.se/~feldt/

Questions?

robert.feldt@chalmers.se

fredag den 13 april 2012