"I Didn't Know I Knew That: A Story of Self-Learning" - - PDF document
WK2 Keynote 10/5/2011 10:00 AM "I Didn't Know I Knew That: A Story of Self-Learning" Presented by: David Hayman Qual IT Softw are Testing Solutions Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888 268
Keynote 10/5/2011 10:00 AM
Presented by: David Hayman Qual IT Softw are Testing Solutions
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
David Hayman has been in IT for twenty-seven years with twenty-three of those focused on software testing. Originally from the UK, David has lived in New Zealand for eight years. He is currently the test practice manager for Qual IT Software Testing Solutions in Auckland, delivering testing solutions, and mentoring and training IT teams across a gamut of industries and applications. A contributing author to The Testing Practitioner, David is a thought-provoking and entertaining speaker at events in the UK, Europe, and Australasia. Actively involved in the ISEB/ISTQB certification program since 1998 as a trainer and exam writer, David is currently the Chair of the Australia and New Zealand Testing Board.
1
1
David Hayman
StarWest, Disneyland Hotel, Anaheim, California October 2011 david.hayman@qualit.co.nz + 64 21 293 6604
Identify what can be done by people using a process and a positive attitude p p Differentiating the ‘what’ and the ‘how’ What were the objectives of the testing? Look at the approaches taken How test case design techniques were used
2
How test case design techniques were used Consider what is really in a name Share a few good and bad things I learnt
2
A voice activated internal Directory system We’ve got a couple of possible tools It’ll need performance testing in conjunction
with a DTMF system in production
Any business experience? Any testers with experience?
3
Any testers with experience? Any help at all? Why are we doing this? Why me?
What do I know?
Wh d I k ?
Who do I know? Where can I look for help? What have I done that is similar? How hard can it be? Why did I say yes?
“If k h t i th h I f il?” ☺
4
“If no one knows what success is then how can I fail?” ☺ On the other hand… … “If no one knows what success is then how can I succeed?”
3
5
Health Banking Insurance Surveys Shopping
Ed ti
Education Communication [ Anti] Social Media
6
4
We may not know exactly how but
y y we know why
Sometimes it’s just another name for
the same thing
Artificial boundaries and constraints If I need to be told to think I’m in the If I need to be told to think I m in the
wrong job!
7
I’ve learned more since I prepared
p p this presentation
We don’t all have the opportunities
afforded to some
Do we need to know everything
about the ‘how’ before we can even consider starting – of course not
8
5
The Internet [ Anti] Social media [ Anti] Social media ISTQB BS7925-2; test case design techniques Users Customers BA’s
9
BA s Testing Gurus Other companies with voice systems Usability groups Me – I like a challenge
Risks to the project and product Was the testing we planned: Was the testing we planned: Valid Valuable Complete As they say – “we didn’t know what we didn’t
know” and it wasn’t just us! j
Equally no one knew what they were signing
10
6
Good IVR more ‘personal’ and easier
p for people with some disabilities e.g. the blind [ or visually impaired] , dyslexic or paralysed than DTMF.
Bad IVR can give you blood pressure
g y p and make you throw things!
11
5 seconds to answer the call Go round three times [ became twice] Bad language Unrecognised name Silence Hang Ups Hand off to operator Department vs. individual All lines busy – engaged tone Required line busy – go to voicemail
12
7
Turning requirements and call flows into
d l models
Establishing 100% coverage Identify test conditions, input data and
expected results
Used for impact analysis on change
requests Used to identify key areas for regression
13
Used to identify key areas for regression
14
8
Follow the yellow brick road What is it?
We tested usability in at the start rather than
the usual at the end approach
Gap analysis Test driven design A review with a fancy name?
15
A review with a fancy name?
How does it work?
Pseudo Scrum included Marketing
Did it work?
Yup
Whakatane
k = wak a tane
In the Maori alphabet there are only 10
consonants and 5 vowels and no ‘F’
It’s all about pronunciation What we say the way we say it and what What we say, the way we say it and what
people hear
Whakatane
= fuk a tar nay
16
9
Types of name
Individuals Individuals Departments
Fun with test data:
Mark Eting The Reverend Ndabaningi Sithole John Smith/ Joe Smith/ Rob Smith/ Bob Smith
17
Phonetics
35 Accents [ all within IT department] –
wide involvement in the project
Phonetic vs. pronunciation
Home Mobile Mobile Skype Speaker Hands Free Public call boxes – 2 common issues Car Hands free
18
Where is the caller, what is going on in the
background
Background noise and how the IVR copes –
again relate to ‘noise’ in a performance test but this can be done on an individual call
10
Wav files First in NZ to use this tool UI and reporting issues Tool recognition thresholds – levels of
correctness - Nuance
Codec changes introduced ‘clicks’
Jitter was an issue [ Ji
i l i
Jitter was an issue [ Jitter, an irregular time
variation of period signal properties, such as small, unpredictable delays in scheduling]
19
Step/Actor Response Step 1 IVR Text to Speech "Hello and welcome to the StarWest Interactive Voice Response system. To help us direct your call please choose one of the following options. Say "Speakers", “Tickets“ , "Free Gifts” or "Operator" Step 2 Caller Silence Step 3 IVR Text to Speech I'm sorry I didn't catch that. To help us direct your call please choose one of the following options. Say "Speakers", “Tickets“ , "Free Gifts” or "Operator" Step 4 C ll "Ti k t "
20
Caller "Tickets" Step 5 IVR Text to Speech To confirm did you say "Tickets"? Step 6 Yes Caller Step 7 IVR Text to Speech Thank you. Putting you through to tickets
11
21
22
12
23
Roger Roff Learning could be fooled Bad language Silence Background noise Stepping over the threshold Establishing “confidence levels” Example - Peter Gouws
24
13
Tools
IBM/ Ci R t t k it i
IBM/ Cisco Router network monitoring Cyara ‘Cruncher’ Tool configured as a
service not in-house – more realistic
Operational Profile
Limited to 120 lines and handled a mixture
Test’
25
Test’
Manage scripts to ensure silence
added so that call was fully answered before the message started
Careful use of Production switches and
network and phone lines
Test environments – had to use external
hardware, therefore control over quality was impossible.
DTMF ‘tromboning’ issue caused problems
with IVR throughput and line availability
Other than that all good ☺
26
14
Social engineering
g g
How to stop people phishing Stutters, lisps and other speech
impediments
27
Good functional and performance testing
with the product supported by the business BUT
The quality of the product was overridden
by the feeling in the market
Voice systems at the time were getting a
bad rap
28
bad rap
Therefore a good system that could have
improved the image of the company was lost to the general public because of the business risk assessment.
15
Voice Talent – new career Prevention rather than cure – always good Gap analysis – we missed how much? Fully Documented – OK I know! Enthused the business and others to get
involved in testing Very formal test process
Very formal test process Configured the system rather than tested it
29
Test driven design or test driven
configuration?
Replaced ‘Good Enough’ with
BEWECODOGITHELIOUS or BEWE Testing
A Review by any other name Improved the UI on the Cruncher tool
Made some friends
Made some friends Learnt a lot about the tool, myself and my
team
Not all systems that pass acceptance
testing and are essentially bug free will go into production!!!
30
16
Alok Kulkarni, Bonny Malik and Thomas Fejes @
Cyara Solutions www cyarasolutions com Cyara Solutions www.cyarasolutions.com
Nick Brown and Piers Langridge are to testing
what Batman and Robin are to crime fighting
31
Akash Jattan @ IBM New Zealand now in Oz Anna Creery – IVR voice – available for work
now ☺
32
17
David Hayman
lit
33
www.qualit.co.nz david.hayman@qualit.co.nz + 64 21 293 6604