IMGD 1001 - The Game Development Process: QA by by Aric Rubin , - - PDF document

10/8/2007 1

IMGD 1001 - The Game Development Process: QA

by

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science

by Aric Rubin, Linden Lab Robert W . Lindem an (gogo@wpi.edu) Kent Quirk (kent_quirk@cognitoy.com)

QA

• Quality Assurance

✦ Also known as

• QE (Quality Engineering)
• SQA (Software QA)
• Testing
• Over The Wall

✦ (more on this later) Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 2

10/8/2007 2

What is Quality?

• Ideas?

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 3

What is Quality

• It’s NOT:

✦ An abstract measure of goodness

g

✦ An indication of zero bugs

• It IS:

✦ Meeting expectations ✦ A measure of state ✦ A risk management tool

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 4

10/8/2007 3

Timex vs Rolex

• Timex:

✦ Costs $20

$

✦ Made of plastic ✦ Tells time accurately

• Rolex:

✦ Costs $2000 ✦ Made of gold

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 5

✦ Tells time accurately

• Did customers get what they expected?

Quality is related to Value

• A “Good deal”

✦ exceeds quality expectations for the price

q y p p

• usability
• works as expected
• doesn’t crash
• A ripoff

✦ paid too much

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 6

10/8/2007 4

How to test games

• Mini-outline

✦ Capture user requirements

p q

✦ Set quality goals ✦ Measure a baseline ✦ Test against the baseline ✦ Verify fixes

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 7

Capture user requirements

• Expectations

✦ Although many are implied:

g y p

• Fun
• Functional
• Matches genre expectations
• Get as much detail as you have time for

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 8

10/8/2007 5

Set quality goals

• Based on concrete measurements

✦ You need quantitative results

q

✦ Example measurements:

• Bug priority
• Bug severity

✦ Goals:

• Zero known crash bugs
• < 10 critical bugs

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 9

< 10 critical bugs

• etc

Measure baseline

• If possible

✦ New titles may not be in shape to get baseline

y p g quality measurements

• Create test plans against requirements
• Methodical repeatability is key
• But human interaction too

You can’t automate all game testing

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 10

✦ You can’t automate all game testing ✦ Have to test for:

• Fun
• Unanticipated bugs

10/8/2007 6

Start measuring (testing)

• When a feature is delivered

✦ Execute plans for that feature

p

✦ Start baseline testing if you didn’t do it earlier

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 11

Verification

• When developers claim a bug is fixed

✦ Verify it

y

✦ Shouldn’t be adversarial ✦ Avoid it with:

• Good product requirements
• Clear milestones
• Good quality criteria defined FIRST

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 12

10/8/2007 7

The Wall

• Devs sometimes want to

“throw it over the wall”

✦ They think of QA as a

separate universe

• THIS IS VERY BAD

✦ Fight the urge

• Relationship should be

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 13

Relationship should be cooperative

Interesting measurements

• Total number of bugs
• Bug find rate
• Bug find rate
• Bug fix rate

✦ When fix > find, project has turned corner

• Test case execution percentage
• Use for predictions of:

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 14

✦ Stability ✦ Release dates

10/8/2007 8

Skills needed for QA

• Technical knowledge
• Good communications skills
• Good communications skills
• Curiosity and persistence -- puzzle skills
• Strong personality
• Attention to detail
• Tolerance for repetition

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 15

Tolerance for repetition

Special considerations

• MMO environments
• Open source
• Open source

Lindeman & Quirk (& Claypool) - WPI Dept. of Computer Science 16