Model-based Testing Without a Model: Assessing Portability in the - - PowerPoint PPT Presentation

▶

model based testing without a model assessing portability

Model-based Testing Without a Model: Assessing Portability in the - - PowerPoint PPT Presentation

Feb 15, 2023 303 likes •430 views

Model-based Testing Without a Model: Assessing Portability in the Seattle Testbed Justin Cappos and Jonathan Jacky University of Washington Seattle, Washington USA justinc@cs.washington.edu, jon@u.washington.edu with special thanks to Jeff

slide-1

SLIDE 1

Model-based Testing Without a Model: Assessing Portability in the Seattle Testbed

Justin Cappos and Jonathan Jacky

University of Washington Seattle, Washington USA justinc@cs.washington.edu, jon@u.washington.edu with special thanks to Jeff Rasley

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-2

SLIDE 2

Seattle Testbed

Seattle distributed computing testbed https://seattle.cs.washington.edu/ Seattle programs — Coded in a Python subset Use a special API Run in a “sandbox”, a safe execution environment Run on many platforms: Windows, Linux, Mac OS X, BSD, various mobile . . .

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-3

SLIDE 3

Portability

Seattle programs should be portable: behave the same on all platforms. Portability has proved to be a problem — On other systems (Java, POSIX, . . . ) On Seattle itself (file system, network . . . )

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-4

SLIDE 4

Seattle API

Simple API: 32 functions File I/O: 6 functions Locks: 3 Debugging: 2 Threads: 3 Code verification and evaluation: 2 Network DNS, UDP, TCP: 13 Accounting: 2 Random bytes (crypto): 1

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-5

SLIDE 5

Testing the Seattle API

Unit tests are not enough. We need — Many interleavings (to check different orders) Handle nondeterminism (for network API) We chose model-based testing.

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-6

SLIDE 6

Traces

Traces are samples of behavior: (listfiles_start, ()), (listfiles_finish, (["junk.testfile"],)), (removefile_start, ("junk.testfile",)), (removefile_finish, ()), (openfile_start, ("junk.testfile", True)), (openfile_finish, (fileobject0)), (filewriteat_start, (fileobject0, "hello world!!!", 0)), (filewriteat_finish, ()), (filewriteat_start, (fileobject0, "ked", 9)), (filewriteat_finish, ()), (filereadat_start, (fileobject0, None, 0)), (filereadat_finish, ("hello worked!!"), (fileclose_start, (fileobject0)), (fileclose_finish, ())

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-7

SLIDE 7

Model-based testing

Model Program Offline Test Generator Traces Traces Offline Test Runner Test Harness Implementation Traces Model Program On-the-fly Test Runner Test Harness Implementation Traces

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-8

SLIDE 8

Model-based testing with trace capture / replay

Model Program Offline Test Generator Traces Reference Implementation Interposition Traces Traces Offline Test Runner Test Harness Implementation Traces Model Program On-the-fly Test Runner Test Harness Implementation Traces

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-9

SLIDE 9

Experience

Technology demonstration: File system API for next Seattle version Capture traces (via interposition) on Windows, Mac, Linux Captured traces while executing unit tests for file system API Replay traces on Windows, Mac, Linux Discovered, fixed several portability problems involving filenames

Justin Cappos and Jonathan Jacky Model-based testing without a model

slide-10

SLIDE 10

Future work

Test with traces captured “in the wild” Test entire Seattle API Compare to offline testing, on-the-fly testing Consider testing other systems with portability requirements Seattle distributed computing testbed https://seattle.cs.washington.edu/ PyModel model-based testing framework http://staff.washington.edu/jon/pymodel/www/

Justin Cappos and Jonathan Jacky Model-based testing without a model