ISD Development and Evaluation Semester 2, 2009 Notes This topic - - PDF document

ISD Development and Evaluation

Semester 2, 2009

Notes This topic is about the development and evaluation of interactive systems, looking first mainly at the evaluation side.

1 Usability Analysis

Analysing Peformance

• GOMS analysis methods

– Goals, Operators, Methods, Selection rules – including KLM, Keystroke-Level Modelling

• Cognitive walkthrough
• Heuristic evaluation aka usability inspection

Notes These are three ways in which you can analyse and evaluate the usability of an interactive system. The techniques described here are typically applied by the designers to designs, prototypes, or actual working systems. Unlike user studies (which we’ll look at later), they don’t directly involve the users. They all involve the designers imagining what the users might do: with KLM in a fairly mechanical way; with the others more open ended. Cognitive Walkthrough

• Particularly appropriate for analysing walk-up-and-use interfaces for

ease of learning by first-time users.

• Also appropriate for analysing changes to a system.

• Similar in spirit to code walkthrough.
• Evaluators step through the sequence of interface actions required to

complete some task.

• At each step, determine why it is or isn’t appropriate for a new user.

Cognitive Walkthrough

• Based on a user model of exploratory learning:
• 1. The user starts with a rough plan of what is to be achieved—a

task to be performed.

• 2. The user explores the system, via the user interface, looking for

actions that might contribute to performing the task.

• 3. The user selects the action whose description or appearance

most closely matches the goal.

• 4. The user interprets the system’s response and assesses whether

progress has been made towards completing the task.

• Exposes design flaws that may interfere with exploratory learning.

Notes Clearly, there are parallels between Exploratory Learning and Norman’s Model of Human Task Perform. You might almost say they were the same idea, just expressed in different words. There are several differences in em- phasis, however. For one, Exploratory Learning emphasizes how users learn, while Norman’s model emphasizes how users do. Cognitive Walkthrough Requirements To do a cognitive walkthrough, as an evaluator you need

• sufficiently detailed description of the prototype,
• description of the task the user is to perform,
• list of actions needed to perform the task with the prototype,
• indication of the user’s experience and knowledge,

2

Cognitive Walkthrough Questions As you step through, ask:

• 1. Will the correct action be made sufficiently evident to the users?
• 2. Will the users connect the correct action’s description with what they

are trying to do?

• 3. Will the users interpret the system’s response to the chosen action

correctly? That is, will the users know if they have made a right or a wrong choice?

• 4. Will the user’s mental model be affected? Will new concepts be added,
• r existing concepts lost?

Notes So the process (ritual?) of a cognitive walkthrough is for you to go step by step through the steps your user has to go through, and at each step ask these four “magic” questions, and try to answer them, given what you know about your users. The first three questions are the crucial ones. If the answer to any of them is no, then that’s a sign that that part of your interface needs fixing. The fourth question is a reminder to be aware how interaction with your interface will affect your user’s mental model, for good or bad. Keep in mind that the user’s mental model also lurks behind the the first three questions, in that how the user interacts with the system will depend not only on what the user can see, but also on what the user thinks about how the system works. There will always be something of a subjective element in this process, since you the designer are making judgements about what you think your users will see and do and think. Heuristic Evaluation

• also called usability inspection
• more appropriate for designs where method of operation is less pre-

dictable

• typically done by small team of evaluators

3

Heuristic Evaluation

• apply usability heuristics (general-purpose guidelines)

– simple and natural dialogue – speak the user’s language – minimize memory load – be consistent – provide feedback – provide clearly marked exits – provide shortcuts – provide good error messages – prevent errors – . . .

• informal walkthroughs as needed

Notes The guidelines given here are just examples, of guidelines that might be

• used. We’ll look at guidelines in more detail later in the semester.

Heuristic Evaluation, Comments

• rather loose and flexible
• low cost, compared with other methods
• little or no advance planning required
• can be used early in development process
• problem oriented
• design inertia
• more varied outcome, less repeatable

4

Notes Heuristic evaluation is a less formal, more open-ended kind of analysis than cognitive walkthrough. You need to have ahead of time a list of guidelines

• r usability heuristics that you think your interface should conform to. You

then look at the interface and evaluate whether indeed it does follow those

• guidelines. Because one team member may see what other team members

don’t see, heuristic evaluation works better with a small team of evaluators than with a solo evaluator. Heuristic evaluation has a number of sources of variability: what guide- lines are chosen, how the evaluators perceive the interface and interpret the

• guidelines. So the results may be variable, but still useful.

Heuristic evaluation can be done at any stage of the development pro- cess, but can be done in quite early stages of design, even before there are prototypes. One problem that afflicts heuristic evaluation (and also prototyping) is design inertia: Designers can get stuck in evaluating the interface in terms

• f the current design.

2 User Studies

User Studies Can be with respect to:

• existing system
• prototypes of new system
• completed new system

User Studies Can be for:

• understanding users, their work and environment
• identifying problems
• eliciting, clarifying, and validating requirements
• system evaluation: “acceptance testing”

5

User Studies

• May need to conduct initial studies to define problems then follow-up

studies to address particular issues

• Tension between thoroughness and availability of resources

Some Methods of Data Collection

• Interviews
• Questionnaires
• Observation
• Other methods, e.g. focus groups, server logs for websites (see later)

Notes These are ways in which you can gather information about your users. As with all user studies, they can be to find out about how users work with an existing system (which you’re planning to replace or improve); with a new system under development (say through prototypes); or with a newly completed system to confirm that it does indeed do the job. Interviews

• Who to interview?

– users: workers, supervisors? – number of people and range depends on allocated resources

• How long should interview be?

– half an hour to an hour

• Logistics of scheduling interviews
• Need to be structured, but should also allow for open-ended exploration

6

Interviews Some issues to address:

• explaining purpose of interview
• enumerating user’s activities/responsibilities
• work methods: how is the job done?
• tracing interconnections with other people
• performance (may be sensitive, seeming to reflect on user’s competence)
• following up on exceptions: non-routine aspects of work
• knowledge of the domain (interviewer’s homework)

Interviews

• May be set questions to ask, but need also allow for interviewee to

volunteer information

• Approach needs to be conducive to finding out, i.e., non-threatening to

users

• Lack of anonymity, users may feel intimidated
• Record may be interviewer’s notes, or audio recording (with permis-

sion)

• Labor-intensive and time-consuming, so relatively costly and only few

can be interviewed Questionnaires Questionnaires need to be devised carefully—a whole area of study in itself

• only one chance—can’t seek clarification or elaboration as in an inter-

view

• questions need to be unambiguous
• users have only limited time to answer

– make things easy for the subject 7

• avoid leading questions
• beware of ordering effects, biases
• . . .

Questionnaires Kinds of questions, e.g.:

• multiple choice
• tick boxes
• rating on a scale: numeric, or converted to numbers
• short text or numeric answers
• open-ended free-text comments

Questionnaires Can be anonymous

• may get more honest answers, but less “responsible” answers
• still may need “demographic” data

Questionnaires

• More advance planning needed than for interviews
• Less costly per person
• Can potentially obtain data from a large number of people

– more representative, particularly for statistical analysis

• May need to offer incentives to get people to take time to answer
• Can be paper, postal, email, or web form. . . —different characteristics
• Usually advisable to test questionnaire on small group beforehand

8

Observation

• Observing user in action
• May be in normal workplace, or under controlled conditions (e.g., us-

ability lab)

• May be on normal work tasks, or on specially devised set tasks

Observation May be passive or active observation

• “Passive” means observer does not influence performance of activities;

it does not preclude asking questions

• An extreme version of active observation is where observer participates

in the work: “action research” Observation Recording:

• note-taking by observer
• video recording

– need to see and synchronize both user’s actions and expression, and screen display – problems with resolution and flicker if screen is videoed: direct feed better

• audio recording

– get user to speak aloud thought processes: “concurrent verbal protocol”

• instrumented software
• transcripts, protocols: time consuming
• may end up with huge volume of data to analyse

9

Observation Beware of the Hawthorne Effect

• That is, people may work better (or differently) just because they know

they’re being observed.

• In extreme cases, observer may work “under cover” Any such study

would have to be done with due regard for ethical concerns about ob- serving people without their consent. Notes The Hawthorne Effect refers to studies done at the Western Electric Hawthorne Works in Chicago, from 1927 to 1932, by Elton Mayo and others. They found that almost any change in conditions in their studies led to increased pro- ductivity by the workers.

3 Evaluation by Controlled Experiments

Evaluation by Controlled Experiments

• Why conduct controlled experiments?
• Some issues in designing and running experiments
• Steps in evaluation:
• 1. learning from prototype construction
• 2. informal user tests
• 3. field tests (informal or more formal)
• 4. laboratory tests
• Controlled experiments usually during 3 and 4.
• In 2 good and bad features of design will probably be obvious immedi-

ately; may also conduct simple performance measurements. Field Tests

• identifying the issues to be addressed
• planning
• coordination with client organization—commitment/approval needed

10

• prototype

– engineered to adequate level of reliability and performance

• selecting users and tasks
• experimental design (see later)
• running tests

– may require training, creation/conversion of databases, running

• ld system in parallel. . .
• data collection and analysis
• drawing conclusions

Controlled Experiments

• purpose: to make accurate measurements and inferences about usabil-

ity

• limitation: situations where usage and circumstances are predictable
• types of questions that can be answered:

– Is design A preferable to design B in terms of usability? – Does this design meet a certain requirement?

• experimental design

– choose situation (users and tasks) and measurements so that re- sults are most meaningful

• statistical analysis

– Are results statistically significant or did they just occur by chance? 11

Example: Drawing Package Menu

Plain Double Dashed Arrow Dbl Arrow Custom LineType

• r

Dashed Plain Double Arrow Custom Dbl Arrow Line Type

• What is the effect of switching to pie menus?
• What kind of testing?
• What to measure? Speed? Error rate?

Experimental Design—Issues

• focus on specific aspects of user behavior and specific components of

interface

• consider two (or more) experimental conditions that differ only in one

controlled aspect – changes in performance can be attributed to the single changed characteristic Experimental Design—Issues

• select subjects

– representative – sufficiently big sample 12

Experimental Design—Issues

• experimental variables

– independent variables: what’s controlled to produce different con- ditions, e.g., interface style; order, size of menu items,. . . – dependent variables: measured in the experiment, e.g., time taken, number of errors,. . . – “nuisance variables” Experimental Design—Issues

• hypothesis to be tested

– prediction of outcome framed in terms of independent and depen- dent variables, e.g., “Time taken with pie menu is at least 25% faster than with pull-down menu.” Experimental Design—Issues

• experimental design, e.g.

– between groups: have each subject perform under only one con- dition – within groups (“matched pairs”): collect data on each subject under each condition – issues: statistical power, learning/ordering effects, randomization, etc. Experimental Design—Issues

• data analysis

– Is there a difference, and is it statistically significant? – various kinds of statistical tests, e.g. Student’s t-test, chi-squared. . . — beyond scope of 371 13

4 Interactive Systems Development

Interactive Systems Development

• Standard software-development model:

– requirements – specification – design – coding – testing – installation – maintenance

• Interactive systems more-or-less follow same model, but flavor and em-

phasis is different – actual usability is pervasive issue ISD Design Processes

• user study
• modelling user activity
• design specification
• design analysis
• empirical evaluation (can be on prototypes)

General Design Activities Not just for software

• analysis of alternative solutions
• conducting investigations to fill gaps in knowledge
• drawing up draft specifications
• making performance and cost estimates
• building and testing prototypes
• modifying the design and revising prototype if necessary
• a systematic process, but not a recipe

14

General Design Issues

• Human problems and errors when dealing with technology are often a

result of design failure

• Good design makes allowance for human capabilities and frailties
• Principles of good design
• Designers help things work well by providing a good conceptual model
• Good designs are targeted to (classes of) users

– Know your user User Interface Design Activities

• Adopting a style of interaction appropriate to the task performed

and the person performing it, and achievable with available tech- nology

• Creating a conceptual design, ensuring that the users have an ade-

quate mental model of the system to support them in deciding what steps to perform and in understanding the system’s responses User Interface Design Activities

• Documenting the user interface, with the aid of notations
• Making decisions about specific aspects of the design, based on design

guidelines

• Prototyping the design
• Performing usability analysis

Information Gathering for Design

• interviews, focus groups
• observation, protocol taking
• questionnaires
• prototypes

15

• usability evaluation

– analytical – empirical Prototyping in ISD

• Prototyping used across all areas of design, including software.
• Particularly in interactive systems, since so much depends on actual

usability of system.

• Very difficult to elicit requirements properly without a prototype.

Prototyping in ISD

• Prototypes can span range:

– pencil-and-paper sketched prototype (e.g., for layout) – partially functioning system – full functioning system (evolutionary)

• Prototyping appropriate for:

– requirements elicitation – investigating design trade-offs Prototyping in ISD

• Less costly than full implementation, but still allows adequate assess-

ment of usability

• Sacrifices, e.g.:

– functionality – limits (say limit to small hard-wired database) – efficiency – error checking and recovery – maintainability – . . . 16

Prototyping Issues

• a prototype provides some but not all features of intended system

– How much realism to provide? More realism means: ∗ better simulation, more accurate evaluation ∗ longer, more expensive development – How can performance requirements be evaluated without full func- tionality?

• a prototype can be a useful tool for communication with the user to

elicit requirements, feedback

• can help uncover misunderstandings
• beware of design inertia

Design Evaluation Summative evaluation Evaluation when design is complete Formative evaluation Evaluation interleaved with ongoing (re)design

• Designs are evaluated against requirements
• Evaluation performed by testing under “live” conditions
• Results of evaluation are fed into next stage of design enhancement

Formative Evaluation

• design and build prototypes, discovering design problems that weren’t

apparent earlier

• informal user tests of initial prototype, by designers or willing users
• field tests of progressive enhancements
• more formal testing under controlled conditions (usability lab)

Three Approaches to Prototyping Throwaway

• construct a prototype explicitly for the purpose of elicit-

ing, understanding and validating requirements

• design knowledge gained is used to inform building of the produc-

tion version

• prototype is discarded

17

Three Approaches to Prototyping Evolutionary

• gather requirements, build prototype, assess
• iterate until acceptable production version is reached
• prototype not discarded—used as basis for each iteration

Three Approaches to Prototyping Incremental

• combines evolutionary approach with traditional SE life-

cycle

• one overall design for final system, but partitioned into smaller,

independent components

• components delivered incrementally in successive releases of prod-

uct Throwaway Prototyping

• prototyping can consume all available resources

– unless carefully managed

• clients may not want to pay for production version after seeing the

features they want in the prototype – but prototype not maintainable

• proper client and project management needed

Evolutionary Prototyping

• evolution can be so rapid that no design rationale is kept
• project planning difficult—no schedule of deliverables
• difficult to provide good documentation—short time-lines
• iterative improvement can be never ending—when to stop?
• again, proper client and project management needed

18

Incremental Prototyping Advantages

• maintains project discipline for each increment—planning

and documentation

• successive deliverables ensure regular feedback

Disadvantages

• hard to specify requirements appropriately
• integration of successive increments may be difficult (and diffi-

cult to anticipate) Interface Prototyping Tools

• Ease of developing and modifying layouts
• Support for a range of types of user interfaces and input devices
• Means of calling external programs and accessing external text, graph-

ics and other media

• e.g., Tcl/Tk, Python, NetBeans, HyperCard, Interface Builder, Glade,

. . .

• Note: some of these features may not be good for a production system:

maintainability, design documentation, version control. . . Maintaining Design Rationale

• Reasons behind design decisions
• Ideally, documented throughout the lifecycle

– provides a communication mechanism among design-team mem- bers – documents why one alternative was chosen over others—reasoning which might be reused in other contexts – assists good design by enforcing a discipline of deliberating carefully about design decisions – facilitates recovering from mistakes, bad design decisions 19

Maintaining Design Rationale

• Especially important in interface design—generally no best solution

– encourages documentation of the criteria used to arrive at a particular solution – encourages documentation of alternatives so proper compar- isons can be made

5 Summary

Summary

• Experiments: uses and issues
• ISD development: design/evaluation/prototyping

20