"I Didn't Know I Knew That: A Story of Self-Learning" - PDF document

WK2 Keynote 10/5/2011 10:00 AM "I Didn't Know I Knew That: A Story of Self-Learning" Presented by: David Hayman Qual IT Softw are Testing Solutions Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888 ‐ 268 ‐ 8770 ∙ 904 ‐ 278 ‐ 0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

David Hayman Qual IT Softw are Testing Solutions David Hayman has been in IT for twenty-seven years with twenty-three of those focused on software testing. Originally from the UK, David has lived in New Zealand for eight years. He is currently the test practice manager for Qual IT Software Testing Solutions in Auckland, delivering testing solutions, and mentoring and training IT teams across a gamut of industries and applications. A contributing author to The Testing Practitioner , David is a thought-provoking and entertaining speaker at events in the UK, Europe, and Australasia. Actively involved in the ISEB/ISTQB certification program since 1998 as a trainer and exam writer, David is currently the Chair of the Australia and New Zealand Testing Board.

Welcome I didn’t know I knew that; until I asked myself David Hayman StarWest, Disneyland Hotel, Anaheim, California October 2011 david.hayman@qualit.co.nz + 64 21 293 6604 1 Today’s Objectives Identify what can be done by people using a process and a positive attitude p p Differentiating the ‘what’ and the ‘how’ What were the objectives of the testing? Look at the approaches taken How test case design techniques were used How test case design techniques were used Consider what is really in a name Share a few good and bad things I learnt 2 1

Can you test this please? � A voice activated internal Directory system � We’ve got a couple of possible tools � It’ll need performance testing in conjunction with a DTMF system in production � Any business experience? � Any testers with experience? � Any testers with experience? � Any help at all? � Why are we doing this? � Why me? 3 Sure, now what was the question again? � What do I know? � Who do I know? Wh d I k ? � Where can I look for help? � What have I done that is similar? � How hard can it be? � Why did I say yes? “If no one knows what success is then how can I fail?” ☺ I f il?” ☺ “If k h t i th h On the other hand… … “If no one knows what success is then how can I succeed?” � 4 2

It’s not the ‘what’ but the ‘how’ 5 We know the ‘what’ � Health � Banking � Insurance � Surveys � Shopping � Education Ed ti � Communication � [ Anti] Social Media 6 3

The ‘how’ is different � We may not know exactly how but y y we know why � Sometimes it’s just another name for the same thing � Artificial boundaries and constraints � If I need to be told to think I’m in the � If I need to be told to think I m in the wrong job! 7 Constraints � I’ve learned more since I prepared p p this presentation � We don’t all have the opportunities afforded to some � Do we need to know everything about the ‘how’ before we can even consider starting – of course not 8 4

Sources of inspiration � The Internet � [ Anti] Social media � [ Anti] Social media � ISTQB � BS7925-2; test case design techniques � Users � Customers � BA’s � BA s � Testing Gurus � Other companies with voice systems � Usability groups � Me – I like a challenge 9 What didn’t we know? � Risks to the project and product � Was the testing we planned: � Was the testing we planned: � Valid � Valuable � Complete � As they say – “we didn’t know what we didn’t know” and it wasn’t just us! j � Equally no one knew what they were signing off, but they did know what they wanted. 10 5

Key Considerations � Good IVR more ‘personal’ and easier p for people with some disabilities e.g. the blind [ or visually impaired] , dyslexic or paralysed than DTMF. � Bad IVR can give you blood pressure g y p and make you throw things! 11 Business Requirements � 5 seconds to answer the call � Go round three times [ became twice] � Bad language � Unrecognised name � Silence � Hang Ups � Hand off to operator � Department vs. individual � All lines busy – engaged tone � Required line busy – go to voicemail 12 6

State Transition Models � Turning requirements and call flows into models d l � Establishing 100% coverage � Identify test conditions, input data and expected results � Used for impact analysis on change requests � Used to identify key areas for regression Used to identify key areas for regression 13 14 7

Wizard of Oz Testing � Follow the yellow brick road � What is it? � We tested usability in at the start rather than the usual at the end approach � Gap analysis � Test driven design � A review with a fancy name? � A review with a fancy name? � How does it work? � Pseudo Scrum included Marketing � Did it work? � Yup 15 Say Something � Whakatane = wak a tane k � In the Maori alphabet there are only 10 consonants and 5 vowels and no ‘F’ � It’s all about pronunciation � What we say the way we say it and what � What we say, the way we say it and what people hear � Whakatane = fuk a tar nay 16 8

Put the ‘Fun’ in Functional � Types of name � Individuals � Individuals � Departments � Fun with test data: � Mark Eting � The Reverend Ndabaningi Sithole � John Smith/ Joe Smith/ Rob Smith/ Bob Smith � Phonetics � 35 Accents [ all within IT department] – wide involvement in the project � Phonetic vs. pronunciation 17 Platforms � Home � Mobile � Mobile � Skype � Speaker � Hands Free � Public call boxes – 2 common issues � Car Hands free � Where is the caller, what is going on in the background � Background noise and how the IVR copes – again relate to ‘noise’ in a performance test but this can be done on an individual call 18 9

Actual Recordings � Wav files � First in NZ to use this tool � UI and reporting issues � Tool recognition thresholds – levels of correctness - Nuance � Codec changes introduced ‘clicks’ � Jitter was an issue [ Jitter, an irregular time Jitter was an issue [ Ji i l i variation of period signal properties, such as small, unpredictable delays in scheduling] 19 Script #1 Step/Actor Response Step 1 IVR Text to Speech "Hello and welcome to the StarWest Interactive Voice Response system. To help us direct your call please choose one of the following options. Say "Speakers", “Tickets“ , "Free Gifts” or "Operator" Step 2 Caller Silence Step 3 IVR Text to Speech I'm sorry I didn't catch that. To help us direct your call please choose one of the following options. Say "Speakers", “Tickets“ , "Free Gifts” or "Operator" Step 4 Caller C ll "Tickets" "Ti k t " Step 5 IVR Text to Speech To confirm did you say "Tickets"? Step 6 Yes Caller Step 7 20 IVR Text to Speech Thank you. Putting you through to tickets 10

Call #1 – Well that’s a good start 21 Call #2 - Did you hear that? 22 11

Call #3 – Are you sure? 23 Sample problems � Roger Roff � Learning could be fooled � Bad language � Silence � Background noise � Stepping over the threshold � Establishing “confidence levels” � Example - Peter Gouws 24 12

Performance Testing � Tools � IBM/ Cisco Router network monitoring IBM/ Ci R t t k it i � Cyara ‘Cruncher’ Tool configured as a service not in-house – more realistic � Operational Profile � Limited to 120 lines and handled a mixture of DTMF and IVR – included a 4 hour ‘Soak Test’ Test’ � Manage scripts to ensure silence added so that call was fully answered before the message started 25 Test Environment � Careful use of Production switches and network and phone lines � Test environments – had to use external hardware, therefore control over quality was impossible. � DTMF ‘tromboning’ issue caused problems with IVR throughput and line availability � Other than that all good ☺ 26 13

Other issues � Social engineering g g � How to stop people phishing � Stutters, lisps and other speech impediments 27 The end result � Good functional and performance testing with the product supported by the business BUT � The quality of the product was overridden by the feeling in the market � Voice systems at the time were getting a bad rap bad rap � Therefore a good system that could have improved the image of the company was lost to the general public because of the business risk assessment. 28 14

Good things that happened Part 1 � Voice Talent – new career � Prevention rather than cure – always good � Gap analysis – we missed how much? � Fully Documented – OK I know! � Enthused the business and others to get involved in testing � Very formal test process Very formal test process � Configured the system rather than tested it 29 Good things that happened Part 2 � Test driven design or test driven configuration? � Replaced ‘Good Enough’ with BEWECODOGITHELIOUS or BEWE Testing � A Review by any other name � Improved the UI on the Cruncher tool � Made some friends Made some friends � Learnt a lot about the tool, myself and my team � Not all systems that pass acceptance testing and are essentially bug free will go into production!!! 30 15