Testing (or how I learned to stop worrying and love specification) - - PowerPoint PPT Presentation

Testing (or how I learned to stop worrying and love specification)

David Nutter

Biomathematics and Statistics Scotland http://www.bioss.ac.uk

May 28th 2020

Why test

So far, we’ve been testing to confirm things: Functionality (the code does what I expect) Regressions (the change I made has not broken anything) Bugfixes (the code no longer exhibits a bug) There is another reason

Specification

A specification defines What your software should do How it should do it (if you want to get really thorough) . . . Usually writing this is BORING! Even more so than writing unit tests!

Specification

A specification defines What your software should do How it should do it (if you want to get really thorough) . . . Usually writing this is BORING! Even more so than writing unit tests! And the tools are HORRIBLE! Let me show you just how bad it can be. . .

Flashback

To my undergrad years - behold:

Figure 1: (Spivey, J.M. and Abrial, J.R., 1992. The Z notation. Hemel Hempstead: Prentice Hall.)

Wossat then?

That’s Z notation. Clear, precise, provable next to no help when writing a program, (or talking to a collaborator about your program). Two main problems:

1

It’s as hard, or harder, to write a good spec as a good program (validation)

2

Checking your program conforms to the spec is hard (mapping, automated verification) Also, “Fork my Z specification on github”. . .

Wossat then?

That’s Z notation. Clear, precise, provable next to no help when writing a program, (or talking to a collaborator about your program). Two main problems:

1

It’s as hard, or harder, to write a good spec as a good program (validation)

2

Checking your program conforms to the spec is hard (mapping, automated verification) Also, “Fork my Z specification on github”. . . . . . said no-one ever!

A closer look

A closer look

What this spec describes is slightly tricksy to do in R due to mutable state.

• But. . . I code by HACKING AWAY

..not writing a load of rubbish first! And you can. You can also get a starter set of unit tests “for free” And how?

• But. . . I code by HACKING AWAY

..not writing a load of rubbish first! And you can. You can also get a starter set of unit tests “for free” And how? The answer is to write your tests first Madness!?!!!

• But. . . I code by HACKING AWAY

..not writing a load of rubbish first! And you can. You can also get a starter set of unit tests “for free” And how? The answer is to write your tests first Madness!?!!! No! This is Test Driven Development (TDD) and will force you to think about two things:

1

What your code should be doing

2

Making your code easy to use

(rather than easy to write. . . )

Back to the birthday book

What should it do? Let’s write a test: BirthDayBook = ... ? ... add_birthday = ... ? ... test_that("addbirthday", { expect_equal(length(BirthDayBook), 0) add_birthday(BirthDayBook, "David", "1980-01-04") expect_equal(length(BirthDayBook), 1) expect_true( "David" %in% names(BirthDayBook) ) add_birthday(BirthDayBook, "David", "1980-06-04") expect_equal(length(BirthDayBook), 1) }) The test specifies some of the properties that any add_birthday function must possess, just like the formal specification. You can read it, if you know R And we can run it, directly

But does it pass?

Nope! We have implementation decisions to make. But we already know: some of the what (arguments expected by add_birthday) some (not all) success/fail conditions So a good start. Further decisions:

1

What is BirthDayBook itself?

2

How should add_birthday do its thing

But does it pass?

Nope! We have implementation decisions to make. But we already know: some of the what (arguments expected by add_birthday) some (not all) success/fail conditions So a good start. Further decisions:

1

What is BirthDayBook itself?

2

How should add_birthday do its thing For 1, an environment is mutable. That’ll do: BirthDayBook=new.env()

Implementing

And for 2, knowing what BirthDayBook is we can now implement the add_birthday function: add_birthday = function(book, name, date) { if (! name %in% names(book)) { book[[name]]=as.Date(date) } }

Implementing

And for 2, knowing what BirthDayBook is we can now implement the add_birthday function: add_birthday = function(book, name, date) { if (! name %in% names(book)) { book[[name]]=as.Date(date) } } Later we could add more sanity checks and so on. Knock yourself out, but a key idea in TDD is to write the simplest code possible to satisfy the test

Implementing

And for 2, knowing what BirthDayBook is we can now implement the add_birthday function: add_birthday = function(book, name, date) { if (! name %in% names(book)) { book[[name]]=as.Date(date) } } Later we could add more sanity checks and so on. Knock yourself out, but a key idea in TDD is to write the simplest code possible to satisfy the test Adding more functionality requires more tests

Finally, verify

Then we can run the test again: test_that("addbirthday", { expect_equal(length(BirthDayBook), 0) add_birthday(BirthDayBook, "David", "1980-01-04") expect_equal(length(BirthDayBook), 1) expect_true( "David" %in% names(BirthDayBook) ) add_birthday(BirthDayBook, "David", "1980-06-04") expect_equal(length(BirthDayBook), 1) expect_equal(BirthDayBook[["David"]], as.Date("1980-01-04")) }) And it passes! (We could also do this with R.oo, closure-as-object, or in a more R-like way in the first place (state-free))

Things to think about when writing spec tests

Similar to when writing tests generally. How lucky!

1

“Typical” use cases (functionality check). We started with this, and you should too

2

Invalid input.

Example: My routine expects a positive int, what should happen when I put in a negative float? Or NA, NULL, the contents of your sock drawer . . . A stop() error, warn() or some special value returned? Decide before you write code with subtle bugs!

3

Zero, one, many.

Basically what should happen when you give certain routines no input, a single input item or lots of input R has some “features” which make this important

4

Boundary conditions

If there’s an expected change in behaviour for certain input values, test each side of the change point Guards against off-by-one errors and suchlike

Finally, some other considerations concentrate your spec/testing effort on the “core” parts of your code that will be used heavily Your data input code would likely benefit from thorough testing But it’s probably not worth writing loads of spec for one-off plotting routines and so forth. Though a little might help clarify your thoughts

And when to start implementing?

No hard and fast rules: When you feel you have enough spec to build a more or less correct implementation (of part of your software) When you get bored of specifying. Iterative development is vastly better than building a vast software castle on shaky foundations

Learnings

Hopefully I’ve shown you how test-driven development can help Clarify your thoughts at an early stage in development Avoid tedious/useless specification tasks Give you a head-start on the process of software testing Catch a number of annoying programming errors Obviously, I’ve only scratched the surface. Some resources The “bible” for this technique is “Test Driven Development by Example” (Beck, 2003). Not R-based. Online Introduction to Test Driven Development (TDD) Online Test Driven Development in R A small amount of supplementary material follows this slide

Zero, one, many tests

R sometimes tries to be helpful. Helpful like a toddler. I once wrote a function that returned different column(s) from a data.frame according to the input. The caller always expected a data.frame though. Contrived example, starting with the zero case: foo=mtcars[,c()] expect_true(is(foo,"data.frame")) Good?

Zero, one, many tests

R sometimes tries to be helpful. Helpful like a toddler. I once wrote a function that returned different column(s) from a data.frame according to the input. The caller always expected a data.frame though. Contrived example, starting with the zero case: foo=mtcars[,c()] expect_true(is(foo,"data.frame")) Good? Looks good. The “one” case: bar=mtcars[,c("cyl")] expect_true(is(bar,"data.frame")) Problem ????

Zero, one, many tests

R sometimes tries to be helpful. Helpful like a toddler. I once wrote a function that returned different column(s) from a data.frame according to the input. The caller always expected a data.frame though. Contrived example, starting with the zero case: foo=mtcars[,c()] expect_true(is(foo,"data.frame")) Good? Looks good. The “one” case: bar=mtcars[,c("cyl")] expect_true(is(bar,"data.frame")) Problem ???? Oh dear, the type is numeric!

The “many” case is OK: baz=mtcars[,c("cyl","disp","hp")] expect_true(is(baz,"data.frame"))

The “many” case is OK: baz=mtcars[,c("cyl","disp","hp")] expect_true(is(baz,"data.frame")) So let’s fix our “one” case: bar=mtcars[,c("cyl"), drop=FALSE] expect_true(is(bar,"data.frame")) That’s better!

Boundary conditions

Say we have some data: dat = sample(1:10,20, TRUE) We want to replace values ‘1-5 with “Bad”, 6-8 with “OK” and “9-10” with “Good”. We want a function called boundaries to do this: dat2 = sapply(dat,boundaries) Let’s test the boundaries: expect_equal(boundaries(1),"Bad") expect_equal(boundaries(5),"Bad") expect_equal(boundaries(6),"OK") expect_equal(boundaries(8),"OK") expect_equal(boundaries(9),"Good") expect_equal(boundaries(10),"Good") Done?

Not quite. We also need: expect_error(boundaries(0)) expect_error(boundaries(11)) And a function: boundaries=function(x) { if (x<5) return("Bad") if (x>5 && x<9) return("OK") if (x>=9) return("Good") stop(paste("Value ",x,"out of range")) } Will it pass?

Not quite. We also need: expect_error(boundaries(0)) expect_error(boundaries(11)) And a function: boundaries=function(x) { if (x<5) return("Bad") if (x>5 && x<9) return("OK") if (x>=9) return("Good") stop(paste("Value ",x,"out of range")) } Will it pass?

• Nope. Not one, but THREE bugs that will be caught by our tests.

And at least one that won’t.

Not quite. We also need: expect_error(boundaries(0)) expect_error(boundaries(11)) And a function: boundaries=function(x) { if (x<5) return("Bad") if (x>5 && x<9) return("OK") if (x>=9) return("Good") stop(paste("Value ",x,"out of range")) } Will it pass?

• Nope. Not one, but THREE bugs that will be caught by our tests.

And at least one that won’t. boundaries(NULL) Deary me. One bug per line!

Closure-as-object implementation of BirthDayBook

Like this: BirthDayBook2<-function() { birthday<-list() list(add_birthday=function(name,date) { if (! name %in% names(birthday)) { birthday[[name]]<<-as.Date(date) } }, known=function(name) { name %in% names(birthday) }, get_birthday=function(name) { birthday[[name]] }) }

We use it like this: book = BirthDayBook2() book$add_birthday("David","1980-10-02") expect_equal(book$get_birthday("David"), as.Date("1980-10-02"))