Evolving Reasons for Tests or Can we gain something from Directly - - PowerPoint PPT Presentation

▶
evolving reasons for tests or
SMART_READER_LITE
LIVE PREVIEW

Evolving Reasons for Tests or Can we gain something from Directly - - PowerPoint PPT Presentation

Sep 05, 2023 119 likes •644 views

Evolving Reasons for Tests or Can we gain something from Directly Searching SE Decision Spaces? Robert Feldt CREST Open Workshop, UCL, London, 2011-05-12 HOSE Lab (Human-fOcused SE) Chalmers University of Technology Sweden onsdag den 18 maj

slide-1
SLIDE 1

Evolving Reasons for Tests or

Can we gain something from Directly Searching SE Decision Spaces?

Robert Feldt

CREST Open Workshop, UCL, London, 2011-05-12

HOSE Lab (Human-fOcused SE)

Chalmers University of Technology Sweden

  • nsdag den 18 maj 2011
slide-2
SLIDE 2

What is Engineering?

  • nsdag den 18 maj 2011
slide-3
SLIDE 3

What is Engineering?

“Finding good-enough (technical) solutions to real-world problems”

  • nsdag den 18 maj 2011
slide-4
SLIDE 4

What is Engineering?

“Finding good-enough (technical) solutions to real-world problems” “Making trade-offs & Balancing competing constraints”

  • nsdag den 18 maj 2011
slide-5
SLIDE 5

What is Engineering?

  • nsdag den 18 maj 2011
slide-6
SLIDE 6

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions

  • nsdag den 18 maj 2011
slide-7
SLIDE 7

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out

  • nsdag den 18 maj 2011
slide-8
SLIDE 8

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out (and Setting up for the Next Round of the game...)

[Cockburn2006]

  • nsdag den 18 maj 2011
slide-9
SLIDE 9

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out (and Setting up for the Next Round of the game...)

[Cockburn2006]

Preparing for Decision

  • nsdag den 18 maj 2011
slide-10
SLIDE 10

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out (and Setting up for the Next Round of the game...)

[Cockburn2006]

Preparing for Decision Making Decisions

  • nsdag den 18 maj 2011
slide-11
SLIDE 11

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out (and Setting up for the Next Round of the game...)

[Cockburn2006]

Preparing for Decision Making Decisions Explaining/Solidifying Decisions

  • nsdag den 18 maj 2011
slide-12
SLIDE 12

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out (and Setting up for the Next Round of the game...)

[Cockburn2006]

Preparing for Decision Making Decisions Explaining/Solidifying Decisions Justifying Decisions

  • nsdag den 18 maj 2011
slide-13
SLIDE 13

What is Engineering?

Engineering is a Co-operative Game to Explore a Space of possibilities and Making Justified Decisions and then Spelling them out (and Setting up for the Next Round of the game...)

[Cockburn2006]

Preparing for Decision Making Decisions Explaining/Solidifying Decisions Justifying Decisions Changing Decisions

  • nsdag den 18 maj 2011
slide-14
SLIDE 14

SBSE motivation

  • Fits search/optimization like a glove so we argue that SBSE:
  • Easily Applicable (“We already have many spaces!” & “Requires

relatively little expertise!”)

  • Generic (“Same idea/search in many spaces!”)
  • Robust (“Works even if info incomplete, fuzzy, ...”)
  • Realistic (“Caters for multiple objectives”)
  • Insight-rich (“We can learn about spaces we search manually”)
  • Scalable (“CPU’s gets cheaper and faster!”)
  • Less Biased (“Fewer assumptions, that might be wrong!”)
  • Than Humans & Than other Engineering Disciplines
  • nsdag den 18 maj 2011
slide-15
SLIDE 15
  • nsdag den 18 maj 2011
slide-16
SLIDE 16

Solution Space

  • nsdag den 18 maj 2011
slide-17
SLIDE 17

Solution Space Problem Space

  • nsdag den 18 maj 2011
slide-18
SLIDE 18

Solution Space Problem Space Process Space

  • nsdag den 18 maj 2011
slide-19
SLIDE 19

Solution Space Problem Space It Executes! Process Space

  • nsdag den 18 maj 2011
slide-20
SLIDE 20

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space

  • nsdag den 18 maj 2011
slide-21
SLIDE 21

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Searches while Executing!

  • nsdag den 18 maj 2011
slide-22
SLIDE 22

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Searches while Executing!

  • nsdag den 18 maj 2011
slide-23
SLIDE 23

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Searches while Executing! It Helps with How to ManuCreate!

  • nsdag den 18 maj 2011
slide-24
SLIDE 24

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Promotes Understanding! It Searches while Executing! It Helps with How to ManuCreate!

  • nsdag den 18 maj 2011
slide-25
SLIDE 25

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Promotes Understanding! It Searches while Executing! It Helps with How to ManuCreate!

  • nsdag den 18 maj 2011
slide-26
SLIDE 26

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Promotes Understanding! It Searches while Executing! It Helps with How to ManuCreate!

  • nsdag den 18 maj 2011
slide-27
SLIDE 27

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Promotes Understanding! It Searches while Executing! It Helps with How to ManuCreate!

Avoid or Avoid Spelling

  • ut Decision
  • nsdag den 18 maj 2011
slide-28
SLIDE 28

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Promotes Understanding! It Searches while Executing! It Helps with How to ManuCreate!

Avoid or Avoid Spelling

  • ut Decision

Select/Prio Decisions

  • nsdag den 18 maj 2011
slide-29
SLIDE 29

Solution Space Problem Space It Executes! It AutoCreates what Executes! Process Space It Helps ManuCreate what Executes! It Promotes Understanding! It Searches while Executing! It Helps with How to ManuCreate!

Avoid or Avoid Spelling

  • ut Decision

Select/Prio Decisions “Preparing” for Decision

  • nsdag den 18 maj 2011
slide-30
SLIDE 30

Solution Space Problem Space Process Space Decision Space

  • nsdag den 18 maj 2011
slide-31
SLIDE 31

Engineering Decisions

  • Reasoning and justification involved is often:
  • Ill-defined
  • Ill-structured
  • Incomplete
  • Use Inconsistent and contradicting Information
  • Support not only logic, facts and probability but hunches, gut

feelings, strange ideas

  • nsdag den 18 maj 2011
slide-32
SLIDE 32

Engineering Decisions

  • Reasoning and justification involved is often:
  • Ill-defined
  • Ill-structured
  • Incomplete
  • Use Inconsistent and contradicting Information
  • Support not only logic, facts and probability but hunches, gut

feelings, strange ideas

Suited to Search and Optimization!

  • nsdag den 18 maj 2011
slide-33
SLIDE 33

Engineering Decisions

  • Reasoning and justification involved is often:
  • Ill-defined
  • Ill-structured
  • Incomplete
  • Use Inconsistent and contradicting Information
  • Support not only logic, facts and probability but hunches, gut

feelings, strange ideas

Suited to Search and Optimization! => Possibilistic Reasoning

  • nsdag den 18 maj 2011
slide-34
SLIDE 34

What would it mean in reality?

  • nsdag den 18 maj 2011
slide-35
SLIDE 35

What would it mean in reality?

Real-world feedback from SW Engineers: “Great that we can generate tests automatically but there are too many and we find too many bugs we do not want to bother with”

  • nsdag den 18 maj 2011
slide-36
SLIDE 36

What would it mean in reality?

Real-world feedback from SW Engineers: “Great that we can generate tests automatically but there are too many and we find too many bugs we do not want to bother with” Can we directly justify the tests we generate?

  • nsdag den 18 maj 2011
slide-37
SLIDE 37

What would it mean in reality?

Real-world feedback from SW Engineers: “Great that we can generate tests automatically but there are too many and we find too many bugs we do not want to bother with” Can we directly justify the tests we generate? Which decisions/justification space can we search?

  • nsdag den 18 maj 2011
slide-38
SLIDE 38
  • nsdag den 18 maj 2011
slide-39
SLIDE 39

Reasons for triangle(3,3,3)?

  • nsdag den 18 maj 2011
slide-40
SLIDE 40

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

  • nsdag den 18 maj 2011
slide-41
SLIDE 41

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

I want a high quality system

  • nsdag den 18 maj 2011
slide-42
SLIDE 42

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

I want a high quality system For a high quality system, I want to test all methods

  • nsdag den 18 maj 2011
slide-43
SLIDE 43

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

I want a high quality system For a high quality system, I want to test all methods triangle is a method

  • nsdag den 18 maj 2011
slide-44
SLIDE 44

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

I want a high quality system For a high quality system, I want to test all methods triangle is a method “VERB all Ts” => “VERB to a T”

  • nsdag den 18 maj 2011
slide-45
SLIDE 45

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

I want a high quality system For a high quality system, I want to test all methods triangle is a method “VERB all Ts” => “VERB to a T” to test is a VERB

  • nsdag den 18 maj 2011
slide-46
SLIDE 46

Reasons for triangle(3,3,3)?

Specification: “triangle is a method and takes 3 integer arguments that are the length of its sides. Output is ‘equilateral’ if all sides are the same. ...”

I want a high quality system For a high quality system, I want to test all methods triangle is a method “VERB all Ts” => “VERB to a T” to test is a VERB I want to test triangle

  • nsdag den 18 maj 2011
slide-47
SLIDE 47

Possibilistic Reasoning

Input as text Chain Searcher “Sugar” & Language rules Translate rules Blackboard with Hypothesis Action rules

Test Justification

+

  • nsdag den 18 maj 2011
slide-48
SLIDE 48

So, now what?

  • Consider not only:
  • Which artefact am I searching for? and
  • Which activity do I support? but
  • Which engineering decision am I supporting? and
  • Can I more directly support that decision?
  • Since benefits are:
  • Help engineers explore not only artefacts and info
  • Less assumptions means less missed (as long as we can make

some progress) opportunities and “errors”

  • nsdag den 18 maj 2011
slide-49
SLIDE 49

Extra

  • nsdag den 18 maj 2011
slide-50
SLIDE 50

Normalized Compression Distance

  • Conditional Kolmogorov Complexity K(X|Y)
  • Cilibrasi: Use a compression algorithm, C!
  • Non-negative number 0<=NCD<=1+e,

where e depends on how good C approximates K

  • nsdag den 18 maj 2011