IMGD 1001 - The Game Development Process: QA by by Aric Rubin , - PDF document

10/8/2007 IMGD 1001 - The Game Development Process: QA by by Aric Rubin , Linden Lab Robert W . Lindem an (gogo@wpi.edu) Kent Quirk (kent_quirk@cognitoy.com) Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science QA • Quality Assurance ✦ Also known as • QE (Quality Engineering) • SQA (Software QA) • Testing • Over The Wall ✦ (more on this later) Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 2 1

10/8/2007 What is Quality? • Ideas? Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 3 What is Quality • It’s NOT: ✦ An abstract measure of goodness g ✦ An indication of zero bugs • It IS: ✦ Meeting expectations ✦ A measure of state ✦ A risk management tool Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 4 2

10/8/2007 Timex vs Rolex • Timex: ✦ Costs $20 $ ✦ Made of plastic ✦ Tells time accurately • Rolex: ✦ Costs $2000 ✦ Made of gold ✦ Tells time accurately • Did customers get what they expected? Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 5 Quality is related to Value • A “Good deal” ✦ exceeds quality expectations for the price q y p p • usability • works as expected • doesn’t crash • A ripoff ✦ paid too much Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 6 3

10/8/2007 How to test games • Mini-outline ✦ Capture user requirements p q ✦ Set quality goals ✦ Measure a baseline ✦ Test against the baseline ✦ Verify fixes Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 7 Capture user requirements • Expectations ✦ Although many are implied: g y p • Fun • Functional • Matches genre expectations • Get as much detail as you have time for Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 8 4

10/8/2007 Set quality goals • Based on concrete measurements ✦ You need quantitative results q ✦ Example measurements: • Bug priority • Bug severity ✦ Goals: • Zero known crash bugs • < 10 critical bugs < 10 critical bugs • etc Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 9 Measure baseline • If possible ✦ New titles may not be in shape to get baseline y p g quality measurements • Create test plans against requirements • Methodical repeatability is key • But human interaction too ✦ You can’t automate all game testing You can’t automate all game testing ✦ Have to test for: • Fun • Unanticipated bugs Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 10 5

10/8/2007 Start measuring (testing) • When a feature is delivered ✦ Execute plans for that feature p ✦ Start baseline testing if you didn’t do it earlier Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 11 Verification • When developers claim a bug is fixed ✦ Verify it y ✦ Shouldn’t be adversarial ✦ Avoid it with: • Good product requirements • Clear milestones • Good quality criteria defined FIRST Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 12 6

10/8/2007 The Wall • Devs sometimes want to “throw it over the wall” ✦ They think of QA as a separate universe • THIS IS VERY BAD ✦ Fight the urge • Relationship should be Relationship should be cooperative Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 13 Interesting measurements • Total number of bugs • Bug find rate • Bug find rate • Bug fix rate ✦ When fix > find, project has turned corner • Test case execution percentage • Use for predictions of: ✦ Stability ✦ Release dates Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 14 7

10/8/2007 Skills needed for QA • Technical knowledge • Good communications skills • Good communications skills • Curiosity and persistence -- puzzle skills • Strong personality • Attention to detail • Tolerance for repetition Tolerance for repetition Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 15 Special considerations • MMO environments • Open source • Open source Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 16 8