Testability Reid Holmes Testability The degree to which a system - - PowerPoint PPT Presentation

▶
testability
SMART_READER_LITE
LIVE PREVIEW

Testability Reid Holmes Testability The degree to which a system - - PowerPoint PPT Presentation

Mar 31, 2023 203 likes •411 views

http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf https://www.youtube.com/watch?v=1keyEiJHqPw https://www.youtube.com/watch?v=XcT4yYu_TTs Testability Reid Holmes Testability The degree to which a system

slide-1
SLIDE 1

Testability

Reid Holmes

http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf https://www.youtube.com/watch?v=1keyEiJHqPw https://www.youtube.com/watch?v=XcT4yYu_TTs

slide-2
SLIDE 2

REID HOLMES - CPSC 410: Advanced Software Engineering

Testability

The degree to which a system

  • r component facilitates the

establishment of test objectives and the execution of tests to determine whether those

  • bjectives have been achieved.
slide-3
SLIDE 3

REID HOLMES - CPSC 410: Advanced Software Engineering

Given a finite amount of time and resources, how can we validate that the system has an acceptable risk of costly or dangerous defects.

—Bob Binder

slide-4
SLIDE 4

REID HOLMES - CPSC 410: Advanced Software Engineering

Develop Test Check- In

slide-5
SLIDE 5

REID HOLMES - CPSC 410: Advanced Software Engineering

Why not test?

  • Good reasons:
  • I don’t know how!
  • Legacy code
  • Bad reasons:
  • Bad design
  • Doesn’t catch bugs (now)
  • Slow
  • Boring
  • Hard to change
  • That’s QA’s job
slide-6
SLIDE 6

REID HOLMES - CPSC 410: Advanced Software Engineering

Common assumption

  • “The cost of fixing faults rises exponentially with

how late [e.g., requirements, design, implementation, deployment] they are detected.”

  • This is commonly stated but is based on evidence

from 20+ years ago.

  • This assumption does not seem to hold for

modern processes, tools, and languages.

  • That said, it is still necessary to validate that the

system works.

  • Also important that it continues to work as the

system evolves.

slide-7
SLIDE 7

REID HOLMES - CPSC 410: Advanced Software Engineering

Terminology

  • Efficiency: number of tests per unit of effort.
  • Effectiveness: the probability of detecting a bug per

unit of effort.

  • Higher testability: more effective tests, same cost.
  • Lower testability: fewer weaker tests, same cost.
  • Repeatability: the likelihood that running the same

test twice will yield the same result.

  • SUT/CUT: System/Code Under Test
  • White-box: tests consider internals of CUT.
  • Black-box: tests are oblivious of internals of CUT.

http://robertvbinder.com/wp-content/uploads/rvb-pdf/talks/GTAC-2010-Binder-Testability.pdf

slide-8
SLIDE 8

REID HOLMES - CPSC 410: Advanced Software Engineering

Anatomy of a test

  • To reveal a fault, a test must:
  • Reach some code
  • Trigger a defect
  • Propagate an incorrect result
  • The result must be observed
  • The result must be interpreted as incorrect
  • Test threats:
  • Non-deterministic dependencies
  • Threading/Race Conditions/Deadlock
  • Shared data
slide-9
SLIDE 9

REID HOLMES - CPSC 410: Advanced Software Engineering

Properties of Testability

  • Controllability
  • The extent to which the SUT can be made to

perform specific actions of interest.

  • Observability
  • The extent to which the response of the SUT to a

test can be verified.

  • Isolateability
  • The degree to which the element under test can

be validated on its own.

  • Automatability
  • The ability to execute the test programmatically.
slide-10
SLIDE 10

REID HOLMES - CPSC 410: Advanced Software Engineering

Controllability

  • What do we have to do to run a test?
  • How expensive is it?
  • Does the SUT make running a test impractical?
  • Give a test goal, do we have enough information to

create an adequate suite?

  • How much tooling can we afford?
slide-11
SLIDE 11

REID HOLMES - CPSC 410: Advanced Software Engineering

Observability

  • What do we have to do to identify pass/fail?
  • How expensive is it to do this?
  • Can we extract the result from the SUT?
  • Do we know enough to identify pass/fail?
slide-12
SLIDE 12

REID HOLMES - CPSC 410: Advanced Software Engineering

Isolateability

  • Can the element being tested be isolated?
  • What is the cost to do this?
  • If an element cannot be naturally isolated, can we

simulate it (e.g., with mocks / stubs)?

  • Why bother?
  • Isolated components are:
  • Simpler to reason about (e.g., root cause analysis)
  • Less prone to non-determinism
  • Faster
  • Simulated dependencies can also more enable

validating unusual states.

slide-13
SLIDE 13

REID HOLMES - CPSC 410: Advanced Software Engineering

Automatability

  • Can tests be executed without human intervention?
  • Huge economic advantages:
  • Setting up automation: 5 hours
  • Running manual test: 30 minutes.
  • Automation pays for itself after just 10 iterations.
  • What is the cost of automated infrastructure?
  • What is the benefits of using a test infrastructure?
  • Executions can be batched.
  • Run on same configuration / hardware.
  • Global visibility of results.
  • Enables regression testing.
slide-14
SLIDE 14

REID HOLMES - CPSC 410: Advanced Software Engineering

Challenges

  • Tests are code too.
  • Also subject to their own faults.
  • Not all test failures uncover faults:
  • Defect in test itself
  • Flaky test (due to some form of non-determinism)
  • Requirements shortcoming (undefined behaviour)
  • Implicit assumptions often surfaced by tests
  • How to retrospectively recognize a ‘true’ failure?
  • Developer changed the source (not test), test

passed on the next iteration.

slide-15
SLIDE 15

REID HOLMES - CPSC 410: Advanced Software Engineering

Develop Test Check- In

Must: Be Fast Be Reliable Isolate Failures

slide-16
SLIDE 16

REID HOLMES - CPSC 410: Advanced Software Engineering

Kinds of tests

slide-17
SLIDE 17

REID HOLMES - CPSC 410: Advanced Software Engineering

Test value

Unit tests isolate failures. Even if a product contains millions of lines of code, if a unit test fails, you only need to search that small unit under test to find the bug. Writing effective unit tests requires skills in areas such as dependency management, mocking, and hermetic testing. I won't cover these skills here, but as a start, the typical example offered to new Googlers (or Nooglers) is how Google builds and tests a stopwatch. With end­to­end tests, you have to wait: first for the entire product to be built, then for it to be deployed, and finally for all end­to­end tests to run. When the tests do run, flaky tests tend to be a fact of life. And even if a test finds a bug, that bug could be anywhere in the product. Although end­to­end tests do a better job of simulating real user scenarios, this advantage quickly becomes

  • utweighed by all the disadvantages of the end­to­end feedback loop:

Unit End­toEnd Fast Reliable Isolates Failures Simulates a Real User Unit tests do have one major disadvantage: even if the units work well in isolation, you do not know if they work well

  • together. But even then, you do not necessarily need end­to­end tests. For that, you can use an integration test. An

integration test takes a small group of units, often two units, and tests their behavior as a whole, verifying that they coherently work together. If two units do not integrate properly, why write an end­to­end test when you can write a much smaller, more focused integration test that will detect the same bug? While you do need to think larger, you only need to think a little larger to verify that units work together. Even with both unit tests and integration tests, you probably still will want a small number of end­to­end tests to verify the system as a whole. To find the right balance between all three test types, the best visual aid to use is the testing

  • pyramid. Here is a simplified version of the testing pyramid from the opening keynote of the 2014 Google Test

Automation Conference:

Unit Tests vs. End­to­End Tests Integration Tests

Testing Pyramid

Kirkland (1) Kurt Alfred Kluever (1) Mark Ivey (1) Mark Striebeck (1) Marko Ivanković (1) Michael Bachman (1) Mike Wacker (1) Mona El Mahdy (1) Noel Yap (1) Patricia Legaspi (1) Peter Arrenbrecht (1) Phil Rollet (1) Pooja Gupta (1) Radoslav Vasilev (1) Rajat Dewan (1) Rajat Jain (1) Rich Martin (1) Richard Bustamante (1) Roshan Sembacuttiaratchy (1) Ruslan Khamitov (1) Sean Jordan (1) Sharon Zhou (1) Stephen Ng (1) Tejas Shah (1) Test Analytics (1) Vojta Jína (1) October 2015 (1) August 2015 (1) June 2015 (1) May 2015 (2) April 2015 (2) March 2015 (1) February 2015 (1) January 2015 (2) December 2014 (2) November 2014 (1) October 2014 (2) September 2014 (2) August 2014 (2) July 2014 (3) June 2014 (3) May 2014 (2) April 2014 (2)

Archives

This site uses cookies from Google to deliver its services, to personalise ads and to analyse traffic. Information about your use of this site is shared with Google. By using this site, you agree to its use of cookies.

LEARN MORE GOT IT

slide-18
SLIDE 18

REID HOLMES - CPSC 410: Advanced Software Engineering

Test pyramid

http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html

slide-19
SLIDE 19

REID HOLMES - CPSC 410: Advanced Software Engineering

Continuous Integration

  • Every project will build and execute automated tests.
  • TravisCI (https://travis-ci.com/ (NOT ORG)) has

provided the class with large numbers of commits

  • You can request access online, we will grant it
slide-20
SLIDE 20

REID HOLMES - CPSC 410: Advanced Software Engineering

Activity

  • In your groups:
  • Choose one of your use cases
  • Describe one end to end test for this use case
  • Describe 2 integration tests for this use case
  • Describe 3 unit tests for this use case