"I Wouldn't Have Seen It If I Hadn't Believed It: Confirmation - - PDF document

T2

Track 4/29/2010 9:45 AM

"I Wouldn't Have Seen It If I Hadn't Believed It: Confirmation Bias in Testing"

Presented by: Michael Bolton DevelopSense

Brought to you by:

330 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Michael Bolton

Michael Bolton has been teaching and providing consulting on software testing on five continents for nine years. He is the co-author (with senior author James Bach) of Rapid Software Testing, a course that presents a methodology and mindset for testing software expertly in uncertain conditions and under extreme time pressure. He has been Program Chair for TASSQ and is a co-founder of the Toronto Workshops on Software Testing. Michael wrote the Test Connection columns in Better Software magazine for three years and sporadically produces his own newsletter. Michael lives in Toronto, Canada, with his wife and two children and can be reached at mb@developsense.com or at developsense.com.

1

“I Wouldn't Have Seen It I Wouldn t Have Seen It If I Hadn't Believed It: Confirmation Bias in Testing”

Mi h l B lt Michael Bolton

michael@developsense.com

DevelopSense

http://www.developsense.com

STAREAST April 2010

Testing is About Making Sure That People Aren’t Being Fooled

…and we have to start with ourselves

2

The Big Theme of This Workshop

• Jerry Weinberg

Feedback Loop: Perception & Conception

What we b li I wouldn’t have seen it if I believe What we can see hadn’t believed it. What we’ve been trained to do

3

Confirmation Bias The tendency to

see believe test evaluate behave investigate

in ways that fit with your

hypotheses goals

y

interpret filter choose expectations hypotheses predictions preconceptions goals

rather than challenging them.

beliefs biases

Confirmation Bias: Variations

anchoring bias congr ence he ristic availability bias bandwagon effect endowment effect recency bias belief perseverance selection bias valence effect congruence heuristic Pollyanna principle survivorship bias endowment effect recency bias assimilation bias choice-supportive bias congruence bias concurrency bias experimenter bias and, for today’s talk…

4

Positive Test Strategy

• “A tendency to test cases that are expected (or

known) to have the property of interest rather than known) to have the property of interest rather than those expected (or know) to lack that property.”

• “…can be a very good heuristic for determining the

truth or falsehood of a hypothesis under realistic conditions.” It can however lead to systematic errors or

• It can, however, lead to systematic errors or

inefficiencies.

• Klayman and Ha, 1987

Positive Test Strategy

• “When concrete, task-specific information is

lacking or cognitive demands are high people rely lacking, or cognitive demands are high, people rely

• n the positive test strategy as a general default

heuristic.” BUT…

• “emphasis on the sufficiency of one’s actions is

enhanced when one is rewarded for each individual success rather than only for the final rule individual success rather than only for the final rule discovery.”

• Klayman and Ha, 1987

5

The Great Traps of Test Cases If you want to know how the system works,

Escaping Confirmation Bias

• Practice the skill of factoring

Ch f t i d i ti

• Change a factor in your description
• Change one of the nouns to something else
• Be more specific
• Add adjectives to your description
• Consider the opposite

M th f f tt ti

• Manage the focus of your attention
• Alternate between focusing and defocusing
• Reverse figure and ground

6

Factoring: Dimensions of Interest

• Factoring is the process of analyzing an object,

event or model to determine the elements that event, or model to determine the elements that comprise it.

• When we factor for the purposes of test design, we

identify elements that we may need to observe or control during a test. We call those factors. Factors may be

• Factors may be
• intrinsic or relational
• variable or static

Managing Attention

• To test for anticipated problems…
• To test a simple product or part of a complex product very
• To test a simple product, or part of a complex product very

thoroughly…

• To pinpoint an observed problem…
• To confirm that a fix has been made…
• To maximize test integrity…
• To stay in grooves…

y g

7

Focusing Heuristics

• Start the test from a known, clean state
• reset the system; observe established pre-conditions

reset the system; observe established pre conditions

• Prefer simple, deterministic actions
• focus on a relatively small number of tests
• Vary One Factor At a Time (OFAT)
• Develop and test to a specified model
• trace test steps to that model

F ll t bli h d d i t t l b d

• Follow established and consistent lab procedures
• Make specific predictions, observations and records
• test to observe that the system fits the patterns
• Make the test easy to reproduce
• automated input may help

Managing Attention

• To find unexpected problems…
• To find elusive problems in sustained field use
• To find elusive problems in sustained field use…
• To test whether a fix has broken something else…
• To discover new dimensions of the product or the testing

mission…

• To get out of ruts…

8

Defocusing Heuristics

• Start from a variety of different states
• not necessarily clean; don’t necessarily reset the system
• not necessarily clean; don t necessarily reset the system
• Prefer complex, challenging actions
• considering perform a huge number of tests
• Vary Many Factors At a Time (MFAT)
• Generate tests from a variety of models
• or without reference to a conscious model
• Question your lab procedures and tools
• Try to see things with open expectations
• let the patterns come to you
• Make the test hard to pass, instead of easy to reproduce
• automatic logging and screen recording may help

Reverse Figure and Ground

9

Heuristic: A vs. THE

• Example: “A problem…” instead of “THE problem…”
• Using “A” instead of “THE” helps us to avoid several

kinds of critical thinking errors

• single path of causation
• confusing correlation and causation
• single level of explanation

Heuristic: Unless…

• To test your description, try adding “unless…” to it,

and performing a test based on that

10

Heuristic: The Rule of Three

• Special case of the Rule Of At Least Three:

Readings

• Klayman, Joshua; Young-Won Ha (1987). "Confirmation,

Disconfirmation and Information in Hypothesis Testing". yp g

http://www.stats.org.uk/statistical-inference/KlaymanHa1987.pdf

• Tools of Critical Thinking (Levy)
• Exploring Requirements (Weinberg)
• Perfect Software and Other Illusions About Testing

(Weinberg)

• Lessons Learned in Software Testing (Kaner Bach and
• Lessons Learned in Software Testing (Kaner, Bach, and

Pettichord)

• Quality Software Management, Vol. 1: Systems Thinking

(Weinberg)

• Quality Software Management, Vol. : First-Order

Measurement (Weinberg)

11

Readings

• How To Lie With Statistics (Huff)

Th Bl k S (T l b)

• The Black Swan (Taleb)
• An Introduction To General Systems Thinking

(Weinberg)

• Measuring and Managing Performance in

Organizations (Austin)

• Software Testing as a Social Science (Kaner)
• http://www.kaner.com/pdfs/KanerSocialScienceSTEP.pdf
• How To Solve It (Polya)
• Politics and the English Language (Orwell)