Evaluation C.W. Johnson, Univ ersit y of Glasgo w, Glasgo - PDF document

Evaluation C.W. Johnson, Univ ersit y of Glasgo w, Glasgo w, G12 8QQ. Scotland. johnson@dcs.gla.ac.uk, h ttp://www.dcs.gla .a c.uk/ � johnso n Octob er 2001 c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001 1

Intro duction F o rmative evaluation: � - heuristic evaluation, co op erative evaluation. Summative evaluation: � - lab-based techniques, dia ries, ethnometho dology . Current p roblems: � - mobile systems; fun and games. c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

F o rmative Evaluation Helps to fo rm design decisions. � Should w e use a menu here o r icons? � { build p encil and pap er p rotot yp es of b oth; { do some user testing, thro w one a w a y . Will results from p rotot yp es accurately p redict � - p erfo rmance with the �nal computer-based system? c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Heuristic Evaluation Check to see design meets guidelines. � 1. Strive for consistency 2. Enable frequent users to use shortcuts 3. Offer informative feedback 4. Design dialogues to yield closure 5. Offer error prevention and simple error handling 6. Permit easy reversal of actions 7. Support locus of control 8. Reduce short-term memory load Shneiderman's e , Chapter 2, P age 74-75. Designing the User Interfac Can b e di�cult to agree in sp eci�c cases. � c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Co-op erative Evaluation Involves real users... � Relatively simple p ro cedure: � - ask users to p erfo rm a sp eci�ed task; - only intervene to help them if they get stuck; - if they get stuck this indicated need fo r redesign; - get them to `think aloud' as they use the system. c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Co-op erative Evaluation Involves real users... � Ac kno wledgemen t: BBC Problems: � - `thinking aloud' can b e unnatural; - `thinking aloud' can interrupt thought p ro cesses; - users ma y p erfo rm w ell b y guessing and hitting lucky . c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Summative Evaluation T ak es place at the end of the design p ro cess. � Check to see if interface meets requirements. � F rom lecture 3: � - p rovide automated o rdering facilities fo r all sta�; - sta� should complete �rst o rder with 1 da y training. Need some resources left if p roblems a re found? � c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Lab-Based Exp erimentation Exp erimental metho d: � - clea rly de�ned hyp othesis; - app rop riate metho d to supp o rt hyp othesis; - results describ ed accurated; - conclusions connexct results to hyp othesis. Ac kno wledgemen t: IBM Must constrain the environment: � - counter-balancing of tasks fo r lea rning e�ects; - app rop riate sample of p otential users; - exact replication of conditions b et w een tests... Is this rep resentative of the real w o rld? � c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Usage Dia ries and Logs When system is almost ready to deliver: � - p rovide few users with advance cop y; - ask them to k eep usage dia ries of any p roblems; - collect results p rio r to �nal debugging etc. Are users robust enough to cop e with ea rly release? � - if so, a re they rep resentative of the eventual users? Less fo rmal app roach via b eta-releases. � c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Problems Observation a�ects the observed. � Ac kno wledgemen t: BBC Hawthorne E�e ct : � - if y ou kno w y ou're b eing w atched y ou act di�erently; - p eople a re mo re ca reful under exp erimental conditions? c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Ethnometho dology No p redetermined hyp otheses. � Observe patterns of use in w o rking environment. � Ma y later try to interp ret meaning of those patterns. � Highly skilled, little understo o d, not widely used. � Ethnography o r ethnometho dology? � c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Problems Don't kno w e�ect of system until it's used. � Ac kno wledgemen t: Nokia Hermeneutics : � - new systems designed to supp o rt existing tasks but - new systems change existing tasks and create new ones; - eg on-line shopping changes nature of shopping. c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Current Problems: Mobile Evaluation Lab-based techniques a ren't very mobile. � Observational techniques involve chasing p eople. � T endency to �eld trial �rst: � - high costs and p otentially big losses; - will users buy the service o r p ro duct? c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Current Problems: F un and Games Can p eople have fun in a lab setting? � Can y ou tak e the time to evaluate? � Highly subjective issue: � - extreme resp onses dep ending on user; - some will love a game that others hate. Mo re general p roblems with W eb: � - can y ou get p eople to `b ro wse' under observation? - is it ethical to log p erfo rmance routinely? c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001

Conclusion F o rmative evaluation: � - heuristic evaluation, co op erative evaluation. Summative evaluation: � - lab-based techniques, dia ries, ethnometho dology . Current p roblems: � - mobile systems; fun and games. c CS-1Q: HCI (Lecture 3) � C.W. Johnson, 2001

F urther Reading Same as fo r p revious lecture! � Shneiderman on: � - design p ro cess - pp. 95-117; - evaluation - pp. 124-150. He combines elicitation and evaluation. � c CS-1Q: HCI (Lecture 4) � C.W. Johnson, 2001