"What Would MacGyver Do?" Presented by: Harry Robinson - - PDF document

W2

Track 10/7/2009 11:30 AM

"What Would MacGyver Do?"

Presented by: Harry Robinson Microsoft

Brought to you by:

330 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Harry Robinson

Harry Robinson is Principal Software Development Engineer in Test for Microsoft's Bing team. He has twenty years of software development and testing experience at AT&T Bell Labs, Hewlett-Packard, Microsoft, and Google as well as time spent in the startup trenches. While at Bell Labs, Harry created a model-based testing system that won the 1995 AT&T Award for Outstanding Achievement in the Area of Quality. At Microsoft, he pioneered the model-based test generation technology, which won the Microsoft Best Practice Award in 2001. Harry holds two patents for software test automation methods, maintains the site www.model-based- testing.org, and speaks and writes frequently on software testing and automation issues.

9/9/2009 1

What Would MacGyver Do?

Harry Robinson Microsoft Principal SDET, Bing

What Would Ed Smylie Do?

Harry Robinson Microsoft Principal SDET, Bing

9/9/2009 2

April 14, 1970

“The contraption wasn't very handsome, but it worked.” ‐ James Lovell, Apollo 13

June 1, 2009

bing!

9/9/2009 3

Smart thumbs

yipes!

Popular now

Deals on underwater cameras Plan your next vacation today Obama education speech · <tacky phrase here> · Laura Bush · Tropica…

9/9/2009 4

Autosuggest

Deals on underwater cameras Plan your next vacation today Obama education speech · Congressional hearing · Laura Bush · Trop…

Spell correction

“porn viedo” “porn video” “porn viedo”

9/9/2009 5

Images/Shopping We considered the usual options

• Testers
• Automation scripts
• A huge test lab
• Relevance judges
• Bug bashes
• Fuzzers

C d i

… as well as some not‐so‐usual options

• Crowdsourcing
• Logged user queries

Desperation tends to make one sort of flexible – Angus MacGyver

9/9/2009 6

Welcome to the monkey trap

“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k h i d i dd l t d Well, he should … see if things he thought were important really were important and, well, stop yanking and just stare at the coconut for a while. … what general advice would you give the poor monkey? monkey reaches in and is suddenly trapped ... Before long he should get a nibble from a little fact wondering if he is interested in it.” ‐Robert Pirsig Zen and the Art of Motorcycle Maintenance

Wh hi ' b k h i hi i h i When something's broken, the easiest thing is to throw it away, forget about it. But if you just step back and take a look at what you've got, you find a totally different way for it to work. ‐ Angus MacGyver

9/9/2009 7

Components of creativity

Expertise Creative‐thinking skills Motivation Creativity

“Creative thinking … refers to how people approach problems and solutions – their capacity to put existing ideas together in new combinations.” ‐ Teresa Amabile Motivating creativity in organizations

Motivation

9/9/2009 8

Expertise

• Lawyers
• Project managers
• Domain experts
• Feature testers
• Senior testers
• Automation specialists

Conceptual blocks

“… mental walls that block the problem‐solver from correctly perceiving a problem or conceiving its solution.” ‐ James L. Adams Conceptual Blockbusting

Perceptual

• Stereotyping
• Difficulty isolating the problem
• Delimiting the problem too closely
• Saturation

Emotional

• Fear of risk
• No appetite for chaos
• Judging rather than generating ideas
• Inability to incubate
• Complacency

Cultural Environmental

p g

Cultural

• Taboos
• Humor (lack of)
• Reason vs intuition
• Tradition vs change

Environmental

• Supportive team
• Supportive management
• Incorporating feedback
• Hidden agendas

9/9/2009 9

Voices in our heads: automation

“Automated tests represent the automation of a manual process.” Harold F Tipton Micki Krause – Harold F. Tipton, Micki Krause “The most important benefit of automated testing over conventional manual testing is the minimization of costs over repeated tests.” – Markus Helfen, Michael Lauer “We have found that it is essential for tests to execute very quickly; our target is We have found that it is essential for tests to execute very quickly; our target is under 30 seconds for the typical test run.” – Shaun Smith, Gerard Mezaros

The less you know sometimes, the better. – Angus MacGyver

Voices in our heads: new bugs

“I have never been convinced that finding ‘new’ bugs is a realistic expectation for test automation.” – I.M. Testy “After you have run your automated tests for the first time, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe “Running automated test scripts can not be used to find new bugs in the software …” – Cordell Vail

9/9/2009 10

Voices in our heads: test oracles

“Oracle: A mechanism to produce the predicted outcomes to compare with p p p the actual outcomes of the software under test.” – BS 7925‐1 “A test oracle is a mechanism for predicting the expected results.” ‐ Andreas Spillner; Tilo Linz; Hans Schaefer “In real testing the outcome is predicted and documented before the test is run.” i i – Boris Beizer

Challenge assumptions

• Is it necessary to predict the outcome of a test?
• Must an automated test behave like a human?
• Must an automated test behave like a human?
• Can random inputs be used well?
• Does a test program really need to run quickly?
• How do humans judge if a result is correct?
• How efficient must a test program be?
• Why can’t a test program be used for exploratory testing?

Why can t a test program be used for exploratory testing?

Never laugh at what you don't know. – Angus MacGyver

9/9/2009 11

Blockbusting

• Challenge assumptions
• Investigate the problem
• Question all constraints
• Look at object attributes
• Brainstorm ideas
• Who else has done something like this?
• Imagine you have unlimited resources …

Imagination is the most important thing the human mind has. – Angus MacGyver

Part 1: Break the problem up

9/9/2009 12

Part 2: Smart thumbs/Popular now

• Smart thumbs:

– Create explicit domain

• Popular now:

– Validate upon data entry – Validate upon data entry

Part 3: Autosuggest

• Bug bash?

G t d h k?

• Generate and check?

– Input from

• Log data
• Exhaustive combinations

– Check with

• Homemade utility
• Homemade utility
• PoliCheck
• Simple solution: Check source file

9/9/2009 13

Part 4: Spell correction

• Generate and check

– Input from

• Log data
• Mutated combinations

– Check with

• Homemade utility
• PoliCheck

Part 5: Images/Shopping

• Generate and check

– Input from

• Log data
• Exhaustive combinations

– Check with

• PoliCheck

If you don't have the right equipment for the job, you just have to make it yourself. – Angus MacGyver

• Homemade utilities
• Autonomation (people + machines)

9/9/2009 14

What should you do?

• Approach test as an engineering problem
• Study up on

– Test approaches – Your product – Available tools – Creative problem‐solving

• Testing books: Beizer, Binder, Jorgensen

h l b

• Papers: Test generation, heuristic oracles, combinatorics

Always carry around a knapsack to pick up things you find along the way. – Angus MacGyver

April 19, 2005

Ed Smylie accepts the first Great Moments in Engineering award

• n behalf of NASA’s Crew Systems Division who improvised an air filter

that kept astronauts alive on the ill‐fated mission 35 years ago.

Any problem can be solved with a little ingenuity. – Angus MacGyver

9/9/2009 15 The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you. – Angus MacGyver

10/7/2009 1

What Would MacGyver Do?

Harry Robinson Microsoft Principal SDET, Bing

What Would Ed Smylie Do?

Harry Robinson Microsoft Principal SDET, Bing

10/7/2009 2

April 14, 1970

“The contraption wasn't very handsome, but it worked.” ‐ James Lovell, Apollo 13

June 1, 2009

bing!

10/7/2009 3

Smart thumbs

yipes!

Popular now

Deals on underwater cameras Plan your next vacation today Obama education speech · tacky phrase goes here · Laura Bush · Trop…

10/7/2009 4

Autosuggest

Deals on underwater cameras Plan your next vacation today Obama education speech · Congressional hearing · Laura Bush · Trop…

Spell correction

“porn viedo” “porn video” “porn viedo”

10/7/2009 5

Images/Shopping We considered the usual options

• Testers
• Automation scripts
• A huge test lab
• Relevance judges
• Bug bashes
• Fuzzers

C d i

… as well as some not‐so‐usual options

• Crowdsourcing
• Logged user queries

Desperation tends to make one sort of flexible – Angus MacGyver

10/7/2009 6

Welcome to the monkey trap

“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k h i d i dd l t d monkey reaches in and is suddenly trapped ...

Welcome to the monkey trap

“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k h i d i dd l t d … what general advice would you give the poor monkey? monkey reaches in and is suddenly trapped ...

10/7/2009 7

Welcome to the monkey trap

“The trap consists of a hollowed‐out coconut chained to a stake. The coconut has some rice inside which can be grabbed through a small hole. The hole is big enough so that the monkey's hand can go in, but too small for his fist with rice in it to come out. The k h i d i dd l t d Well, he should … see if things he thought were important really were important and, well, stop yanking and just stare at the coconut for a while. … what general advice would you give the poor monkey? monkey reaches in and is suddenly trapped ... Before long he should get a nibble from a little fact wondering if he is interested in it.” ‐Robert Pirsig Zen and the Art of Motorcycle Maintenance

Wh hi ' b k h i hi i h i When something's broken, the easiest thing is to throw it away, forget about it. But if you just step back and take a look at what you've got, you find a totally different way for it to work. ‐ Angus MacGyver

10/7/2009 8

Components of creativity

Expertise Creative‐thinking skills Motivation Creativity

“Creative thinking … refers to how people approach problems and solutions – their capacity to put existing ideas together in new combinations.” ‐ Teresa Amabile Motivating creativity in organizations

Motivation

10/7/2009 9

Expertise

• Lawyers
• Project managers
• Domain experts
• Feature testers
• Senior testers
• Automation specialists

Creative‐thinking skills

“Conceptual blocks are mental walls that block the problem‐solver from correctly perceiving a problem or conceiving its solution.” ‐ James L. Adams

Perceptual

• Stereotyping
• Difficulty isolating the problem
• Delimiting the problem too closely
• Saturation

Emotional

• Fear of risk
• No appetite for chaos
• Judging rather than generating ideas
• Inability to incubate
• Complacency

Conceptual Blockbusting

Cultural

• Taboos
• Humor (lack of)
• Reason vs intuition
• Tradition vs change

Environmental

• Supportive team
• Supportive management
• Incorporating feedback
• Hidden agendas

10/7/2009 10

Ping‐Pong Pipe Problem

A steel pipe is embedded in the concrete floor of a bare room… The inside diameter is 0.06” larger than the diameter of the ping‐pong ball (1.50”) that is resting at the bottom of the pipe. You are one of a group of six people in the room, along with the following objects:

• 100 feet of clothesline
• A box of granola
• A light bulb
• A file

from Conceptual Blockbusting by James L. Adams List as many ways as you can think of to get the ball out of the pipe without damaging the ball, the pipe or the floor.

• A carpenter’s hammer
• A wire coat hanger
• A chisel
• A monkey wrench

Ping‐Pong Pipe Problem

A steel pipe is embedded in the concrete floor of a bare room… The inside diameter is 0.06” larger than the diameter of the ping‐pong ball (1.50”) that is resting at the bottom of the pipe. You are one of a group of six people in the room, along with the following objects:

• 100 feet of clothesline
• A box of granola
• A light bulb
• A file

from Conceptual Blockbusting by James L. Adams List as many ways as you can think of to get the ball out of the pipe without damaging the ball, the pipe or the floor.

• A carpenter’s hammer
• A wire coat hanger
• A chisel
• A monkey wrench

Notice that having tools can distract you from a good solution!

10/7/2009 11

So, Are Tools Bad?

MacGyver Gets Lazy

http://xkcd.com/444/

You can do more with a gun that just fire it. – Angus MacGyver

Voices in our heads: automation

“Automated tests represent the automation of a manual process.” Harold F Tipton Micki Krause – Harold F. Tipton, Micki Krause “The most important benefit of automated testing over conventional manual testing is the minimization of costs over repeated tests.” – Markus Helfen, Michael Lauer “We have found that it is essential for tests to execute very quickly; our target is We have found that it is essential for tests to execute very quickly; our target is under 30 seconds for the typical test run.” – Shaun Smith, Gerard Mezaros

The less you know sometimes, the better. – Angus MacGyver

10/7/2009 12

Voices in our heads: new bugs

“I have never been convinced that finding ‘new’ bugs is a realistic expectation for test automation.” – I.M. Testy “After you have run your automated tests for the first time, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe “Running automated test scripts can not be used to find new bugs in the software …” – Cordell Vail

Voices in our heads: test oracles

“Oracle: A mechanism to produce the predicted outcomes to compare with p p p the actual outcomes of the software under test.” – BS 7925‐1 “A test oracle is a mechanism for predicting the expected results.” ‐ Andreas Spillner; Tilo Linz; Hans Schaefer “In real testing the outcome is predicted and documented before the test is run.” i i – Boris Beizer

10/7/2009 13

Challenge assumptions

• Is it necessary to predict the outcome of a test?
• Must an automated test behave like a human?
• Must an automated test behave like a human?
• Can random inputs be used well?
• Does a test program really need to run quickly?
• How do humans judge if a result is correct?
• How efficient must a test program be?
• Why can’t test automation be used for exploratory testing?

Why can t test automation be used for exploratory testing?

Never laugh at what you don't know. – Angus MacGyver

Blockbusting

• Challenge assumptions
• Investigate the problem
• Question all constraints
• Look at object attributes
• Brainstorm ideas
• Who else has done something like this?
• Imagine you have unlimited resources …

I i j i d f M

• Imagine you just arrived from Mars …

Imagination is the most important thing the human mind has. – Angus MacGyver

10/7/2009 14

Part 1: Break the problem up Part 2: Smart thumbs/Popular now

• Smart thumbs:

– Create explicit domain

• Popular now:

– Validate upon data entry – Validate upon data entry

10/7/2009 15

Part 3: Autosuggest

• Bug bash?

G t d h k?

• Generate and check?

– Input from

• Log data
• Exhaustive combinations

– Check with

• Homemade utility
• Homemade utility
• PoliCheck
• Simple solution: Check source file

Part 4: Spell correction

• Generate and check

– Input from

• Log data
• Mutated combinations

– Check with

• Homemade utility
• PoliCheck

10/7/2009 16

Part 5: Images/Shopping

• Generate and check

– Input from

• Log data
• Exhaustive combinations

– Check with

• PoliCheck

If you don't have the right equipment for the job, you just have to make it yourself. – Angus MacGyver

• Homemade utilities
• Autonomation (people + machines)

Part 6: Bing overall

• Use Best‐of‐Breed as a test oracle

Commercial filter

Do these results differ?

Results web Results

10/7/2009 17

Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to

• zero. There had been no airport nearby; he couldn’t land. Unavoidable.”
• zero. There had been no airport nearby; he couldn t land. Unavoidable.

Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to

• zero. There had been no airport nearby; he couldn’t land. Unavoidable.”
• zero. There had been no airport nearby; he couldn t land. Unavoidable.

Outlaw: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places – smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing …

10/7/2009 18

Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to

• zero. There had been no airport nearby; he couldn’t land. Unavoidable.”
• zero. There had been no airport nearby; he couldn t land. Unavoidable.

“Outlaw”: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places – smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing … “It’s up to us to keep flight alive in a world of airplane drivers – the people who come out of your school, concerned only with moving straight and level from airport to airport. We’re trying to keep a few real pilots left in the air. “ Narrator: “It had been an unavoidable accident; one of those rare times when fog forms right out of mid‐air and in five minutes the visibility goes from ten miles down to

• zero. There had been no airport nearby; he couldn’t land. Unavoidable.”
• zero. There had been no airport nearby; he couldn t land. Unavoidable.

“Outlaw”: “[Your] rule number eight told him never to land away from an airport. In his last five minutes of visibility, he flew over eight hundred thirty‐seven landing places – smooth fields and level pastures – but they were not ‘designated airports’ … so he didn’t even think about landing … “It’s up to us to keep flight alive in a world of airplane drivers – the people who come out of your school, concerned only with moving straight and level from airport to airport. We’re trying to keep a few real pilots left in the air. “ ‐ Richard Bach A Gift of Wings

10/7/2009 19

What can you do?

• Approach test as an engineering problem
• Study up on

– Test approaches – Your product – Available tools – Creative problem‐solving

• Testing books: Beizer, Binder, Jorgensen

h l b

• Papers: Test generation, heuristic oracles, combinatorics

Always carry around a knapsack to pick up things you find along the way. – Angus MacGyver

April 19, 2005

Ed Smylie accepts the first Great Moments in Engineering award

• n behalf of NASA’s Crew Systems Division who improvised an air filter

that kept astronauts alive on the ill‐fated mission 35 years ago.

Any problem can be solved with a little ingenuity. – Angus MacGyver

10/7/2009 20 The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you. – Angus MacGyver The most important tool for survival isn't duct tape or a Swiss Army Knife, it's your wits. Always keep them about you. – Angus MacGyver

Thanks for attending!