for Test Factoring David Saff, Michael D. Ernst MIT CSAIL PASTE, - - PowerPoint PPT Presentation

▶
for test factoring
SMART_READER_LITE
LIVE PREVIEW

for Test Factoring David Saff, Michael D. Ernst MIT CSAIL PASTE, - - PowerPoint PPT Presentation

Feb 29, 2024 231 likes •545 views

Mock Object Creation for Test Factoring David Saff, Michael D. Ernst MIT CSAIL PASTE, 2004 June 2/24 Motivation Continuous testing plug-in for the Eclipse IDE* Test suite: 30 secs 1 sec / plug-in test Set up Eclipse Problem:

slide-1
SLIDE 1

Mock Object Creation for Test Factoring

David Saff, Michael D. Ernst MIT CSAIL PASTE, 2004 June

slide-2
SLIDE 2

2/24

Motivation

  • Continuous testing plug-in for the Eclipse

IDE*

  • Test suite:
  • Problem: find out about errors faster
  • Solution: mock objects to replace Eclipse

framework

Set up Eclipse 30 secs 1 sec / plug-in test * Saff, Ernst, ETX 2004: Continuous testing in Eclipse

slide-3
SLIDE 3

3/24

Outline

  • Mock objects introduced
  • Test factoring introduced
  • Mock object creation for test factoring
  • Conclusion
slide-4
SLIDE 4

4/24

Outline

  • Mock objects introduced
  • Test factoring introduced
  • Mock object creation for test factoring
  • Conclusion
slide-5
SLIDE 5

5/24

Plug-in

Unit test for plug-in

Provided Checked

slide-6
SLIDE 6

6/24

Plug-in

System Test for plug-in

Eclipse Provided Checked

slide-7
SLIDE 7

7/24

Unit Test with Mock Object

Mock Objects replacing Eclipse Plug-in Provided Checked Checked Checked Checked Provided Provided Provided A mock object:

  • provides part of the functionality of the original object(s)
  • is focused on allowing the test to proceed
slide-8
SLIDE 8

8/24

Mock objects for our example

  • Using a debugger, determined:

– 147 static calls from plug-in to framework

  • Defined on 49 classes

– 8 callbacks from framework to plug-in

  • Substantial work to define mock objects.
  • How well can we automate this process

without additional manual effort?

slide-9
SLIDE 9

9/24

Outline

  • Mock objects introduced
  • Test factoring introduced
  • Mock object creation for test factoring
  • Conclusion
slide-10
SLIDE 10

10/24

What is a factored test?

  • Split a system test into several smaller

factored tests that

– exercise less code than system test – can be added to the suite and prioritized

  • Find out about errors faster

– embody assumptions about future code changes

slide-11
SLIDE 11

11/24

Pros and cons of factored tests

  • Pro: factored test should be faster if system test

– is slow – requires an expensive resource or human interaction

  • Pro: isolates bugs in subsystems
  • Con: if assumptions about how developer will

change the code are violated, can lead to:

– false negatives: OK, some delay – false positives: bad, distract developer

slide-12
SLIDE 12

12/24

Change language

  • When change language is violated, factored test

must be discarded and re-created

– Can detect violation through analysis, or incorrect result.

db.insertRecord(“alice”, “617”); db.insertRecord(“bob”, “314”); db.insertRecord(“bob”, “314”); db.insertRecord(“alice”, “617”); Change method order? db.insertRecords( “alice: 617, bob: 314” ); Replace with equivalent call?

Change language: the set of tolerated changes

slide-13
SLIDE 13

13/24

A small catalog of test factorings

  • Like refactorings, test factorings can be

catalogued, reasoned about, and automated

Separate Sequential Code:

Also “Unroll Loop”, “Inline Method”, etc. to produce sequential code

slide-14
SLIDE 14

14/24

A small catalog of test factorings

Original test Mocked Eclipse Plug-in Mocked Plug-in Eclipse

Introduce Mock:

slide-15
SLIDE 15

15/24

Binkley ’97 (static) This work (static + dynamic)

Related work

Developer makes change Produce factored tests Slice based on change

Run factored tests Run factored tests

Run

  • riginal

tests Early warning if assumptions hold Correct test results

slide-16
SLIDE 16

16/24

Outline

  • Mock objects introduced
  • Test factoring introduced
  • Mock object creation for test factoring
  • Conclusion
slide-17
SLIDE 17

17/24

Basic Procedure: Trace Capture

Tested Realm Mocked Realm Params, Returns, Callbacks MockExpectations “Boundary”

slide-18
SLIDE 18

18/24

Basic Procedure: code generation

  • MockExpectations encodes a state

machine:

MockExpectations 1 2 3 DebugPlugin.getDefault() → [object id 347] [object id 347].getLaunchManager() → [object id 78] [object id 78].uniqueLaunchNameFrom(“a”) → “a134”

slide-19
SLIDE 19

19/24

Expanding the change language

  • Current tolerated change language

includes:

– Extract method – Inline method

  • Using static analysis on mocked code,

improve the procedure to include:

– Reorder calls to independent objects – Add or remove calls to pure methods

slide-20
SLIDE 20

20/24

Reorder calls to independent

  • bjects
  • Group objects that share state into state

sets

  • One MockExpectations per state set:

MockExpectations A 1 2 3 MockExpectations C 1 2 3 MockExpectations B 1 2 3

slide-21
SLIDE 21

21/24

Add or remove pure method calls

  • Allow reordering, addition, removal of calls

to pure methods:

MockExpectations 1 DebugPlugin.getDefault() → [object id 347] [object id 347].getLaunchManager() → [object id 78] [object id 78].uniqueLaunchNameFrom(“a”) → “a134” [object id 78].removeLaunch(“a134”) → NEXT STATE

slide-22
SLIDE 22

22/24

Outline

  • Mock objects introduced
  • Test factoring introduced
  • Mock object creation for test factoring
  • Conclusion
slide-23
SLIDE 23

23/24

Future work

  • Develop a framework for test factoring
  • Implement the “Implement Mock” factoring
  • Analytic evaluation of framework

– Capture real-project change data* – Measure notification time, false positives

  • Case studies of test factoring in practice

– How do developers feel about the feedback they receive?

* Saff, Ernst, ISSRE 2003: Reducing wasted development time via continuous testing

slide-24
SLIDE 24

24/24

Conclusion

  • Test factoring can indicate errors earlier
  • “Introduce Mock” is an important test

factoring for complicated systems

  • We propose:

– Dynamic analysis for building mock objects – Static analysis for increasing the change language

  • Mail: saff@mit.edu
slide-25
SLIDE 25

25/24

slide-26
SLIDE 26

26/24

A small catalog of test factorings

  • Separate Sequential Test:

– [graphic]

  • Unroll Loop:

– [graphic]

  • Introduce Mock:

– [graphic]

slide-27
SLIDE 27

27/24

  • Frequent automatic testing in

continuous testing.

Frequent testing is good:

  • Frequent manual testing in

agile methodologies

  • A testing framework should minimize

the cost of frequent testing

– Suite completes rapidly – First failing test completes rapidly

slide-28
SLIDE 28

28/24

Getting faster to the first failing test

  • Default:
  • Test selection:
  • Test prioritization:
  • Test factoring:

B1 B2 A1 A2 A3 A1 A2 A3 B2 A1 A2 A3 B1 B1’ B2’ A1’ A2’ A2 A3’ B2 A1 B1 …

slide-29
SLIDE 29

29/24

Dynamic, change-independent test factoring

  • Dynamic: instrument and run the original

test

  • Change-independent: factoring happens

before any changes are made.

– Requires a hypothesized change language

  • Binkley ’97: Static, change-dependent test

factoring

slide-30
SLIDE 30

30/24

Automatic test factoring: change-dependence

  • Change-dependent test factoring:

– After tested code is changed, generate new tests with same result as old tests for that change.

  • Change-independent test factoring:

– Before tested code is changed, generate new tests that have the same result as old tests for some set of changes.

Better

slide-31
SLIDE 31

31/24

Automatic test factoring: static vs. dynamic analysis

  • Static analysis (Binkley ’97)

– Analyze code to determine mock object behavior – Well-suited for change-dependent factoring – May fail

  • without source
  • when dependent on file system or user interaction

– Guaranteed change language may be restrictive

  • Dynamic analysis (this work)

– Instrument and run the original test, gather logs – May run original test after factored test fails