FG1 Friday, May 5, 2000 9:00AM S OAP O PERA T ESTI NG Hans Buwalda - - PDF document

P R E S E N T A T I O N

Friday, May 5, 2000 9:00AM

SOAP OPERA TESTI NG

Hans Buwalda

CMG TestFrame Research Center

FG1

Presentation Bio

S o ap O p era T e stin g

H a n s Bu w a l d a C M G - T e st Fr a m e Re se a r c h Ce n t e r h an s. b u w al d a@c m g . n l w w w . t est f r am e. co m H a n s Bu w a l d a C M G - T e st Fr a m e Re se a r c h Ce n t e r h an s. b u w al d a@c m g . n l w w w . t est f r am e. co m

A genda

• int roduct ion
• underlying archit ecture: th e T estF r am e

M o d el

• S oap O pera’s
• u sag e

Th e C hallenges f or a Test P r o cess

• te st ing should be fun
• te st ing should be effect ive
• te st ing should be ef f icient
• te st ing should be under cont rol

Th e “m ech an ical appr oach ” for t est developm ent ( exam ple)

• s tart w it h ( preferably long) list of requirement s
• m a k e a te st ca se for ever y r equ ir em en t
• u se a standar dized t est t echnique t o t r anslat e t he

r equir em ent s int o t he t est cases

• hir e ( m any) people t o pef or m t he t est s by hand
• … .

S om e pit f alls w it h a t oo m echanical approach

• no f un at all
• shut t ing dow n creat ivit y
• cover age is only at single r equir em ent level
• an y def ect s sh ou ld pr obably h ave been f ou n d in

an ear lier t est

• su g g ests false sen se of con t r ol
• te stse t hard t o m aint ain
• doesn’t cat ch m ist akes in t he r equir em ent s
• . . .

Qu e stio n s t o answ er w ith a te st collect ion

• 1. d o es t he syst em com ply t o t he

requirement s

• 2. a re ther e any pr oblem s ( def ect s and/or

fa ilur es) w e should know about

• 3. w ill t he syst em w ork in pract ice

Th e Test Fr am e p r o d u cts

• high level business orient ed t est s
• product ion accept ance t est s
• f unct ional t est s
• t echnical t est s
• low level f unct ional t est s
• t echnical t est s

specif icat ions design programming

high level act ions int er mediat e level act ions low level act ions

Test Execut ion

Development and navigation in TestFrame

test development test execution

test cluster navigation scheme

…

check balance enter customer

…

A B C D . . . transfer Houston Klein 210 check balance Klein 210 transfer Savy Klein 150 check balance Klein 360 . . .

test execution navigation scheme target system(s) separation reporting test design test clusters test plan

• actual results
• comparison with

expectations

• management information
• input data
• expected outcomes
• documentation

management system development QA/Auditors end users

TestFrame Product overview

S oap O per as

Ashley hears about Jack's deposit when he thought he had to go. Victoria lectures her father about what's wrong with him and Nikki but Victor advises her that it's none of her business Olivia learns Dru has no regrets about leaving and takes great satisfaction in having Lily as her companion. Dru then asks Olivia why she is raking Malcolm over the coals. Stopping by Gina's, Nikki spots Brad and sits with him, admitting she doesn't want to be alone tonight. Victor stops by Mack's party at the Crimson

• Lights. Ashley takes a home pregnancy test. Worried about Billy,

Raul makes call and J.T. claims he doesn't know where Billy is. Raul rushes over and finds Billy out cold in the snow Raul worries when he can't find a pulse. . . .

P ropert ies of S o ap Op er as

• ab ou t “ real life”
• bu t con den sed
• and m or e ext r em e

S oap O peras for t est ing

• def in e a scope of th e te st t o develop
• ident if y w it h t he business environm ent
• w h ich elem en ts w ou ld m ake things difficult
• draf t scenario’s ( t ypical som e dozen lines)
• w rit e t hem dow n in clust ers

E xam ples of st or y lines w hen used f or te sting

Pe n si o n Fu n d Wor ld Wid e Tr an sact ion Sy st em f or an in t er n at ion al Ban k

William starts as a metal worker for Industrial Entropy Incorporated in 1955. During his career he becomes ill, works part time, marries, divorces, marries again, gets 3 children, one

• f which dies, then his wife dies and he marries again and gets 2

more children…. A fish trade company in Japan makes a payment to a vendor on

• Iceland. It should have been a payment in Icelandic Kronur, but

it was done in Yen instead. The error is discovered after 9 days and the payment is revised and corrected, however, the interest calculation (value dating)…

E xam ple of t est lines

from to amount valuta trans nr

enter payment

123421344 4124244123 120000 yen &keep tx1

check value dating

&tx1 $0.47

wait days

9

• rder to reverse

&tx1

from to amount valuta trans nr

enter payment

123421344 4124244123 1200000000 IKr &keep tx2

check value dating

&tx2 $7,701.56

. . . .

S oap O peras ( in t est ing) are not necessar ily:

• “extrem e”
• fa r fe tch ed
• long and elaborat e
• pieces of ar t and cr eativity

“K iller So a p s”

• m ore specif ically aim ed at f inding hidden

pr oblem s

• r u n w h en ever yt h in g else h as passed
• on e op t ion: put a killer soap at t he end of a

norm al clust er

• ask t he “ specialist s” for input

Re a so n s f or scenar ios like soaps

• te st collection can b e m ad e m or e com p act
• it is m or e f un t o m ake
• specialists used m ore ef f ect ively
• te st ing m ore of t he applicatio n
• less direct ly dependent on f unct ional specs,

so catching more pit falls

W h a t to use it f or pr im ar y use

• high level f unct ional accept ance t est ing

but also:

• m odu le te sting
• sy stem te sting
• int egrat ion t est ing
• ...

W h a t is n ot int erest ing for soaps

• screen stuff
• routine tests
• any ot her st raight f orw ard com pliance

te sting

source: Cem Kaner, James Bach, Star West 1999

A L ist of “P ar adigm s”

• D o m a in dr iven
• S t r ess dr iven
• Sp e ci fic a t ion driven
• R isk dr iven
• Ra n d o m / st at ist ical
• Fu n ct ion
• Re g r ession
• Scen ar io
• U s e r t est ing
• Explor at or y
• Securit y

Re la tion t o u se cases

• friendly cousins
• soap oper a’s ar e m or e dir ect ly aim ed at

te st ing, f or exam ple by exaggerat ing and using ( non local) com binat ions

• less an alytical “ top down” , but “ outside in” :

translat ed f rom end user pract ice

Wh o cou ld m ak e S oap s

• near ly ever ybody:

en d u ser s specialized t est er s developer s audit ors ...

W a y s to g et them

• coach in g en d u ser s or bu sin ess specialists
• int erviews
• ow n f ant asy
• w or kbooks
• using joint developm ent sessions

Joint Test w are Dev elop m en t (JT D ) ™

• JT D = A technique f or struc tur ed te st

developm en t w it hout ( complet e) d ocu m en ta tio n

• JT D < > Ju st ta lk t o t he end-u ser s an d write

dow n w h at th ey say

• JT D < > A subsit ut e f or developing

d ocu m en ta tion (requirement s , design s, user-guides)

W h a t can joint sessions give you

• T e st S tra teg y
• Acce p tan ce Cr ite ria
• C lust er G rouping
• T e st conditions
• E valuat ion of Re su lts
• S ta rt ing up developm ent of scenarios

Joint sessions

• m oderat or / chair m an
• users
• bu sin ess specialists
• developers
• te sters

S et up of a j oint session f or a t elecom provider

• 1 st sessi o n

In t r od u ct ion b y m od er at or an d p r oj ect m an ag er ex p lan at ion ab ou t t h e JTD p r oced u r e ex p lan at ion of t h e f u n ct ion al ar ea b y a sp ecialized u ser

• 2 n d sessi o n

s tar t of pr oduct ion of t est condit ions

• 3 r d session

s tar t of pr oduct ion of t est scenar ios

• 4 th sessi o n

ev al u at ion t est scen ar ios

S truc tur ed t est developm ent and S oap s

• soap s ar e n ot the n at ural w ay t o get

“cover age”

• additional t echniques can help, exam ples in

T e stF ra m e :

te st condit ions te st design t em plat es

• reco m m en d ation: do “m atching” aft erw ards

T e st analysis and t est creat ion

Test Analysis:

• what do we want
• what do we need

Test Creation:

• confrontation with reality
• put it to the test

separate relate and specifications analytical techniques business environment

E x am p le Test C ondit ion

nr description

... 3.51 it is checked that the exit date is after the entry date ...

test condition 3.51 name entry date exit date enter employment Bill Goodfellow 1999-10-02 1999-10-01 check error message The exit date must be after the entry date.

coupling with the actual test lines in the cluster

M atching C ondit ions

condition description severity tested in scenario: MB01 Ent ering cust omers using manual numbering MB02 Aut omat ic account numbering MBT-C02 high MB01 Ent ering cust omers using manual numbering MB02 Aut omat ic account numbering MBT-C03 a cust omer wit h a negat ive balance cannot t ransf er money t o anot her cust omer 2 MBT-C04 high MB01 Ent ering cust omers using manual numbering MB02 Aut omat ic account numbering MBT-C05 high MB01 Ent ering cust omers using manual numbering MB02 Aut omat ic account numbering MBT-C06 account numbers can be ent ered manually by the user medium MB01 Ent ering cust omers using manual numbering MBT-C07 account numbers can be generat ed aut omat ically by t he syst em medium MB02 Aut omat ic account numbering MBT-C08 ... ... high t he balance of t he receiving cust omer is increased wit h t he sum payed t he balance of t he paying cust omer is decreased wit h t he sum payed MBT-C01 a cust omer ent ered in t he ent er client screen is present in t he dat abase a cust omer wit h a posit ive balance can t ransf er money t o anot her cust omer

T e st D e sign Tem plat es

• sp r ead sh eet b ased technique f or designing t est s
• in t r od u ced in 1998 as an ext en sion t o Test Fr am e

b y Ed w ar d Kit

• f riendly t echnique and yet very st rong, f or exam ple

in ident if ying needed sit uat ions and com binat ions

• for a f ur t her descr ipt ion please have a look at Ed ’s

art icle in S of t w ar e Developm ent M ag azi n e:

ht t p: / / w w w . sdm agazine. com / br eakr m / f eat ur es/ s992f 2. sht m l

Ex a m p l e o f T est Desi g n T em p l at es

( applied t o SD T ’ s Rev i ew Pr o p ack ag e f or suppor t ing in sp ect ion s an d r ev i ew s)

Feature Hierarchy: Default Entry Type Standard Matrix Summary: Check that the default (non-customized) Entry Types can be selected/used Test Design: Technique Feature Combination Risk Analysis: Impact Likelihood Test Case ID: DETS01 DETS02 DETS03 DETS04 Test Case Validity: Priority: Test Condition Log Entry Input Fields * * * * Document Reviewed = Location = Type = defect question external issue praise Severity = Status = Disposition Code = Summary = Attachments = Select Default Entry * Select Entry #2 * Select Entry #3 * Select Entry #4 * Expected Results Output= Go to Reviewer's Log View, Log is in the Reviewer's Log Forum, Correct Entry Type is selected in the Log Entry Output= Go to Reviewer's Log View, Log is in the Reviewer's Log Forum, Correct Entry Type is selected in the Log Entry Output= Go to Reviewer's Log View, Log is in the Reviewer's Log Forum, Correct Entry Type is selected in the Log Entry Output= Go to Reviewer's Log View, Log is in the Reviewer's Log Forum, Correct Entry Type is selected in the Log Entry

Usa g e o f T e st D e sign Tem plates as p or tal to S oap O per as

Template ID: MB des 1 MB des 2 MB des 3 MB des 4 …. customer * * * * last name first name balance positive too low positive positive number confirmation letter yes yes automatic numbering yes …. tested in scenario: MB01 MB01, MB02 MB02 matching

A life cycle f or t est developm ent

clustering test conditions test design templates test scenario’s

review execution maintenance ...

cluster level

• verall

Qu e stio n s t o answ er w ith a te st collect ion

• 1. d o es t he syst em com ply t o t he

requirement s

• 2. a re their any problem s ( def ect s and/or

fa ilur es) w e should know about

• 3. w ill t he syst em w ork in pract ice

Qu e stio n s t o answ er w ith a te st collect ion

mechanical soaps soaps + techniques

1 does the system comply to the requirements

*** * **

2 are their any problems (defects and/or failures) we should know about

* *** ***

3 will the system work in practice

** ** ***

Th e t hree “ holy grails” of Test Developm ent e ffect ive clust ering of t he t est s t he proper level of t he act ion choosing t he right t echnique per clust er

E xper iences

• t he appr oach can w or k w ell once people ar e

u sed to it

• it can be har d t o get aw ay from t radit ional

f unctional t est ing

• not m agic, use it w here applicable ( grail 2)
• good clust ering is essent ial ( grail 1)
• sta rt w it h a coach

Ack n ow led g em en ts

• E dw ar d K it
• C em Kan er
• Ja m e s Ba ch
• H a n s Sch a e fe r
• Den n i s Jan ssen

Hans Buwalda

Hans Buwalda is project director at CMG, a leading European information technology services group. He has over 15 years of experience in IT. Working for numerous

• rganizations

in sectors like finance, government, manufacturing, trade, and telecommunications, he has focussed on issues of IT management and strategy. Today he is responsible for new developments around the TestFrame approach for testing and test automation, of which he is the main architect. He started the approach in 1994. In 1996 he presented the main ideas behind it for the first time to an international audience in a speech called "Testing with Action Words, Abandoning Record and Playback". Since then the approach is being used in an increasing number of countries and Hans has become a frequent speaker at industry conferences, tutorials, and workshops.