[PDF] - Evaluation C.W. Johnson, Univ ersit y of Glasgo w, Glasgo PDF Document

SLIDE 1 Evaluation C.W. Johnson, Univ ersit y

f

Glasgo w, Glasgo w, G12 8QQ. Scotland. johnson@dcs.gla.ac.uk, h ttp://www.dcs.gla .a c.uk/ johnso n Octob er 2001 CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001 1

SLIDE 2 Intro duction

F
rmative

evaluation:

heuristic

evaluation, co

p

erative evaluation.

Summative

evaluation:

lab-based

techniques, dia ries, ethnometho dology .

Current

p roblems:

mobile

systems; fun and games. CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 3 F

rmative

Evaluation

Helps

to fo rm design decisions.

Should

w e use a menu here

r

icons? { build p encil and pap er p rotot yp es

f

b

th;

{ do some user testing, thro w

ne

a w a y .

Will

results from p rotot yp es accurately p redict

p

erfo rmance with the nal computer-based system? CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 4 Heuristic Evaluation

Check

to see design meets guidelines. 1. Strive for consistency 2. Enable frequent users to use shortcuts 3. Offer informative feedback 4. Design dialogues to yield closure 5. Offer error prevention and simple error handling 6. Permit easy reversal

f

actions 7. Support locus

f

control 8. Reduce short-term memory load Shneiderman's Designing the User Interfac e, Chapter 2, P age 74-75.

Can

b e dicult to agree in sp ecic cases. CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 5 Co-op erative Evaluation

Involves

real users...

Relatively

simple p ro cedure:

ask

users to p erfo rm a sp ecied task;

nly

intervene to help them if they get stuck;

if

they get stuck this indicated need fo r redesign;

get

them to `think aloud' as they use the system. CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 6 Co-op erative Evaluation

Involves

real users... Ac kno wledgemen t: BBC

Problems:
`thinking

aloud' can b e unnatural;

`thinking

aloud' can interrupt thought p ro cesses;

users

ma y p erfo rm w ell b y guessing and hitting lucky . CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 7 Summative Evaluation

T

ak es place at the end

f

the design p ro cess.

Check

to see if interface meets requirements.

F

rom lecture 3:

p

rovide automated

rdering

facilities fo r all sta;

sta

should complete rst

rder

with 1 da y training.

Need

some resources left if p roblems a re found? CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 8 Lab-Based Exp erimentation

Exp

erimental metho d:

clea

rly dened hyp

thesis;
app

rop riate metho d to supp

rt

hyp

thesis;
results

describ ed accurated;

conclusions

connexct results to hyp

thesis.

Ac kno wledgemen t: IBM

Must

constrain the environment:

counter-balancing
f

tasks fo r lea rning eects;

app

rop riate sample

f

p

tential

users;

exact

replication

f

conditions b et w een tests...

Is

this rep resentative

f

the real w

rld?

CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 9 Usage Dia ries and Logs

When

system is almost ready to deliver:

p

rovide few users with advance cop y;

ask

them to k eep usage dia ries

f

any p roblems;

collect

results p rio r to nal debugging etc.

Are

users robust enough to cop e with ea rly release?

if

so, a re they rep resentative

f

the eventual users?

Less

fo rmal app roach via b eta-releases. CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 10 Problems

Observation

aects the

bserved.

Ac kno wledgemen t: BBC

Hawthorne

Ee ct:

if

y

u

kno w y

u're

b eing w atched y

u

act dierently;

p

eople a re mo re ca reful under exp erimental conditions? CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 11 Ethnometho dology

No

p redetermined hyp

theses.
Observe

patterns

f

use in w

rking

environment.

Ma

y later try to interp ret meaning

f

those patterns.

Highly

skilled, little understo

d,

not widely used.

Ethnography
r

ethnometho dology? CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 12 Problems

Don't

kno w eect

f

system until it's used. Ac kno wledgemen t: Nokia

Hermeneutics:
new

systems designed to supp

rt

existing tasks but

new

systems change existing tasks and create new

nes;
eg
n-line

shopping changes nature

f

shopping. CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 13 Current Problems: Mobile Evaluation

Lab-based

techniques a ren't very mobile.

Observational

techniques involve chasing p eople.

T

endency to eld trial rst:

high

costs and p

tentially

big losses;

will

users buy the service

r

p ro duct? CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 14 Current Problems: F un and Games

Can

p eople have fun in a lab setting?

Can

y

u

tak e the time to evaluate?

Highly

subjective issue:

extreme

resp

nses

dep ending

n

user;

some

will love a game that

thers

hate.

Mo

re general p roblems with W eb:

can

y

u

get p eople to `b ro wse' under

bservation?
is

it ethical to log p erfo rmance routinely? CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001

SLIDE 15 Conclusion

F
rmative

evaluation:

heuristic

evaluation, co

p

erative evaluation.

Summative

evaluation:

lab-based

techniques, dia ries, ethnometho dology .

Current

p roblems:

mobile

systems; fun and games. CS-1Q: HCI (Lecture 3) c

C.W.

Johnson, 2001

SLIDE 16 F urther Reading

Same

as fo r p revious lecture!

Shneiderman
n:
design

p ro cess

pp.

95-117;

evaluation
pp.

124-150.

He

combines elicitation and evaluation. CS-1Q: HCI (Lecture 4) c

C.W.

Johnson, 2001