How to improve your manual testing without getting bored! Alex - - PowerPoint PPT Presentation

▶
how to improve your
SMART_READER_LITE
LIVE PREVIEW

How to improve your manual testing without getting bored! Alex - - PowerPoint PPT Presentation

Feb 06, 2024 467 likes •764 views

How to improve your manual testing without getting bored! Alex Schladebeck (exploratory testing and session-based test management) 23.10.2014 1 Short introduction BREDEX This talk Individual software Why exploratory testing?

slide-1
SLIDE 1

1

How to improve your manual testing

… ¡without getting bored! Alex Schladebeck

23.10.2014

(exploratory testing and session-based test management)

slide-2
SLIDE 2

2

Short introduction

BREDEX

  • Individual software

development

  • Enterprise applications
  • Quality assurance
  • Based on Jubula
  • Tool-independent services

This talk

  • Why exploratory testing?
  • How to use it
  • How to plan and organise it

23.10.2014 Eclipse Con Europe 2014

slide-3
SLIDE 3

3

Wait, what?!

If you‘re such test automation champions, why are we talking about manual testing?

23.10.2014 Eclipse Con Europe 2014

slide-4
SLIDE 4

4

Testing as detective work

23.10.2014 Eclipse Con Europe 2014

Testing is gathering information with the intention of informing a decision – Jerry Weinberg

slide-5
SLIDE 5

5

Gathering information

It is impossible to plan tests in advance to cover every possible condition If you try to plan / write all possible tests, you will

  • nly ever write them, never execute them

You can only plan based on what you know

  • If the map and the territory differ, believe the territory

23.10.2014 Eclipse Con Europe 2014

Elisabeth Hendrickson, Explore It!

slide-6
SLIDE 6

6

Testing vs. Checking

23.10.2014 Eclipse Con Europe 2014

Scripted test (manual or automated) Unscripted test (manual only)

Checking Testing (Exploring)

Are there any

  • ther

risks? Intended behaviour?

slide-7
SLIDE 7

7

A couple of videos

23.10.2014 Eclipse Con Europe 2014

slide-8
SLIDE 8

8

  • 1. Observational blindness

Focus on one aspect makes us miss other things

  • The narrower the view, the wider

the ignorance

Errors avoid well-tested paths

23.10.2014 Eclipse Con Europe 2014

slide-9
SLIDE 9

9

  • 2. Orders of ignorance

23.10.2014 Eclipse Con Europe 2014

because as we know, there are known knowns—there are things we know ¡we ¡know… ¡ We also know there are known unknowns; that is to say we know there ¡are ¡some ¡things ¡we ¡do ¡not ¡know… ¡ But there are also unknown unknowns—the ones we don't know we don't know.

Donald Rumsfeld, US Secretary of Defense. Department of Defense News Briefing. Feb 12 2002

slide-10
SLIDE 10

10

  • 2. Orders of ignorance

0th order ignorance

  • We provably know something
  • a scripted test that shows

us the same questions being answered again and again

1st order ignorance

  • We ¡know ¡what ¡we ¡don’t ¡know
  • scripted and exploratory

testing can help here: ask the questions that will give us the knowledge

2nd order ignorance

  • We ¡don’t ¡even ¡know ¡what ¡we ¡

don’t ¡know

  • exploratory testing to find out

what ¡we ¡don’t ¡know

  • Repeat thinking, not steps
  • Find the holes in the net

23.10.2014 Eclipse Con Europe 2014

slide-11
SLIDE 11

11

Not scripted? Clicking around?

23.10.2014 Eclipse Con Europe 2014

slide-12
SLIDE 12

12

Exploratory testing

Exploratory testing is not ad hoc Test design, execution and analysis all happen at the same time

  • No scripting – but normal test design practices can be used (personas,

decision tables, ¡…)

  • Pre-defined test results not necessary
  • Thoughts and process not documented in advance

Situational and emergent practice – context is important

  • The puzzle itself influences the strategies for solving it
  • Steps and test cases are designed based on context and on observation

… ¡This sounds kind of difficult, but it‘s actually our normal human practice of reasoning.

23.10.2014 Eclipse Con Europe 2014

slide-13
SLIDE 13

13

Exploratory testing

23.10.2014 Eclipse Con Europe 2014

slide-14
SLIDE 14

14

When should I use Exploratory Testing?

To find out unknown unknowns To find out how the software works

  • E.g. when the specifications aren‘t detailed

When you need to find important information quickly To keep test script writing to a minimum To find new information and new questions To avoid observational blindness

23.10.2014 Eclipse Con Europe 2014

slide-15
SLIDE 15

15

How do I use Exploratory Testing?

How do I know how the program should behave? What should I test? How do I know what is a problem?

23.10.2014 Eclipse Con Europe 2014

slide-16
SLIDE 16

16

Sources and oracles

Sources

  • Places you can get

information

  • Specification
  • Requirements
  • Documentation
  • Experience
  • Sales promises
  • Discussions
  • Customer
  • Users
  • Product menus, toolbars,

context menus

Oracles

  • How to know if there is a

problem

  • Mental models
  • Own gut feeling
  • Consistency
  • Is there a problem here?

Heuristics

  • Rules of thumb to guide

testing and to identify problems

23.10.2014 Eclipse Con Europe 2014

slide-17
SLIDE 17

17

Heuristics – what to look at / for

CRUD: Create, read, update, delete Goldilocks: Too big, too small, just right Users and scenarios: personas, extreme personalities, soap operas, tours Follow the data: Enter – Search – Report – Export – Import – Update – View HICCUPSF: History, Image, Comparable Products, Claims, User expectations, Product, Purpose, Statutes, Familiar Problems Areas of potential instability: interoperation, multiple files, network, ¡…

23.10.2014 Eclipse Con Europe 2014

slide-18
SLIDE 18

18

Advantages and disadvantages

Advantages

  • Less preparation than

scripting

  • Complete specification not

required

  • Adaptable to new

information

  • Important errors found

quickly

  • More interesting

Disadvantages

  • Not reviewable in advance
  • Not reproducible
  • Not good in low-availability

environments

  • Hard to organise and

control

  • Where to start?
  • Session-based test

management

23.10.2014 Eclipse Con Europe 2014

slide-19
SLIDE 19

19

Session-based test management

A way of organising and reporting on Exploratory Testing

23.10.2014 Eclipse Con Europe 2014

Session-based test management recipe Charters What areas to test Sessions Time to do the testing Notes Observations, questions, feelings, tests performed Debriefing Information transfer

slide-20
SLIDE 20

20 23.10.2014 Eclipse Con Europe 2014

Define charters Prioritise and assign Session – down the rabbithole! Observe and note Debrief

slide-21
SLIDE 21

21

Charters

Explore [area] with [resources] to discover [information] Charters come from:

  • Stakeholders, team, risks, quality attributes, places where we

require information, the nightmare headline game

Defined in advance but can be changed (Re)prioritised before each new session

23.10.2014 Eclipse Con Europe 2014

slide-22
SLIDE 22

22

Session – down the rabbit hole

Uninterrupted timebox

  • 30-120 minutes

One charter

  • Down the rabbithole
  • If you find another hole, write it down new charter for next

session

Use heuristics to create and execute tests

23.10.2014 Eclipse Con Europe 2014

slide-23
SLIDE 23

23

Notes

What did I do? What did I observe?

  • Videos, screenshots

How do I feel about the quality of this charter? Ratio test:setup Ratio charter:opportunity

  • Opportunity = new areas that occur while testing

23.10.2014 Eclipse Con Europe 2014

slide-24
SLIDE 24

24

Debrief

PROOF

  • Past: what happened
  • Results: what was achieved?
  • Obstacles: what blocked the testing?
  • Outlook: What needs to be done?
  • Feeling: how do I feel about this?

23.10.2014 Eclipse Con Europe 2014

slide-25
SLIDE 25

25

Test day

8.00 Define, prio, choose charters 8.30 Session 10.00 Debrief 10.15 Break 10.30 Define, prio, choose charters 10.45 Session 12.00 Debrief 12.15 Lunch 1.15 Define, prio, choose charters 1.30 Session 3.00 Debrief 3.15 Bug entry

23.10.2014 Eclipse Con Europe 2014

slide-26
SLIDE 26

26

When to reproduce / enter bugs?

23.10.2014 Eclipse Con Europe 2014

slide-27
SLIDE 27

27

A good mix

Daily / in each sprint Automated tests

  • Regression tests
  • As part of acceptance

testing for new features tests that will become regression tests

Exploratory testing

  • Manual testing of new

features

  • Can use SBTM

Before a release Manual test checklists

  • Semi-scripted

Exploratory testing days with session based test management

23.10.2014 Eclipse Con Europe 2014

slide-28
SLIDE 28

28

Thanks!

23.10.2014 Eclipse Con Europe 2014

alexandra.schladebeck@bredex.de @alex_schl www.bredexsw.com testing.bredex.de www.eclipse.org/jubula

slide-29
SLIDE 29

29

References

Jerry Weinberg, Perfect Software and other Illusions about Testing Orders of ignorance http://www.corvusintl.com/CACM002-5OI.htm Exploratory testing

  • Explore it! Elisabeth Hendrickson (Pragmatic Bookshelf)
  • http://www.developsense.com/blog/2012/04/all-oracles-are-heuristic/
  • http://www.satisfice.com/articles/what_is_et.shtml
  • http://www.satisfice.com/tools/procedure.pdf
  • http://www.flickr.com/photos/softwaretestingclub/7173828227/sizes/o/in/photostream/

Cheat sheets and mnemonics

  • http://www.qualityperspectives.ca/resources_mnemonics.html
  • http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf
  • Session-based test management
  • http://www.ministryoftesting.com/resources/session-based-test-management/
  • http://www.satisfice.com/articles/sbtm.pdf

23.10.2014 Eclipse Con Europe 2014