Feedback from Usability Evaluation to User I nterface Design: Are - PowerPoint PPT Presentation

Feedback from Usability Evaluation to User I nterface Design: Are Usability Reports Any Good? Christian M. Nielsen 1 Presented by: Michael Overgaard 2 Christian M. Nielsen 1 Michael B. Pedersen 3 Jan Stage 2 Jan Stage 2 1 Novo Nordisk I T 2 Aalborg University 3 ETI

Overview Overview � Motivation � Experim ent � Results � Conclusion Feedback 2

Motivation (1) Motivation (1) � The interplay betw een user Design interface design and usability products evaluation � User interface design User Usability Interface provides design products: Evaluation Design • The system Feedback • An operational prototype • A user interface specification • A paper prototype • … � Usability evaluation provides feedback to user interface design � This feedback form s the basis for redesign and further developm ent Feedback 3

Motivation (2) Motivation (2) 1 . Executive sum m ary � Typical feedback: 1 . Executive sum m ary 2 . Method A usability report 2 . Method a) Purpose a) Purpose � Characteristics: b) Procedure b) Procedure • c) Test participants 40-80 pages c) Test participants d) Test procedure d) Test procedure • 30-80 usability problems: e) Location & equipm ent e) Location & equipm ent A list and a detailed description f) I dentification & categorization of f) I dentification & categorization of of each problem s problem s • Problems are identified through 3 . Results 3 . Results user-based tests a) W orkload ( NASA-TLX) a) W orkload ( NASA-TLX) b) Tim e used • Problems are categorized b) Tim e used c) Problem overview (cosmetic, serious, critical) c) Problem overview d) Detailed description of problem s d) Detailed description of problem s • Additional: TLX and task 4 . Conclusion 4 . Conclusion completion times 5 . Appendix 5 . Appendix • Log files transcribed from the a) Tasks a) Tasks video (15-20 pages) b) I nterview guide b) I nterview guide c) Questionnaires c) Questionnaires d) Video log-files d) Video log-files e) System log-files e) System log-files f) Task solutions f) Task solutions Feedback 4

Motivation (3) Motivation (3) � There are very different opinions about the usefulness of a usability report Usability reports are boring and nobody ever reads them � Basic question: To w hat extent can a usability report influence developers' opinion about strengths and w eaknesses of a system Feedback 5

Experiment: Overview Experiment: Overview � A system for registering use � Tw o usability evaluations of of tim e and m aterials the system ( by tw o team s) � Used by tradesm en Field Laboratory Critical 15 14 � Running on a m obile Severe 16 14 telephone w ith a barcode Cosmetic 17 6 scanner Total 48 34 � Codes for tasks and m aterials Feedback 6

Experiment: Steps Experiment: Steps Step Developer A Developer B #1 Outline the process for the developers, without revealing details. Understanding of usability discussed. Semi-structured interview on initial opinions on Semi-structured interview on initial opinions on #2 advantages and disadvantages advantages and disadvantages Recieve and read the laboratory usability report. Semi-structured interview based on step #2. Recieve and read the field usability report. Semi- Interview is conducted by one of the writers of the structured interview based on step #2. Interview is laboratory report. A field report writer takes conducted by one of the writers of the field report. #3 notes. A laboratory report writer takes notes. Recieve and read the field report. Semi-structured Recieve and read the laboratory report. Semi- interview based on step #3. The developer is structured interview based on step #3. The asked to comment on the usefulness of the reports developer is asked to comment on the usefulness and the individual parts. Interview is conducted by of the reports and the individual parts. Interview is one of the writers of the field report. A laboratory conducted by one of the writers of the laboratory #4 report writer takes notes. report. A field report writer takes notes. Group discussion where the developers are presented with each other's list of advantages and #5 disadvantages. The two developers are asked to agree on a joint list. Feedback 7

Results: Views on Usability Results: Views on Usability � Before reading the usability reports both developers w ere asked to express how they understood the term usability � Developer A: • ‘intuitive’ is the word that described it best • also mentioned ‘easy’ and ‘straightforward’ to use, without having to read several manuals � Developer B: • the specific screens in the system, where the design of the screens should target the user and the information presented should be relevant • The user interface should be easily understood and nice to look at Feedback 8

Results: Developer A Results: Developer A Advantages (before reading reports) Disadvantages (before reading reports) 1. Online: The system can provide relevant 1. GPRS: Limited coverage. real time information. 2. Barcodes are used to interact with the system instead of 2. Barcode scanners: All interaction begins the mobile phone. with the user scanning. 3. Online: problem when no connection is available. 3. No software on the mobile phone. Disadvantages (after reading report 1) Advantages (after reading report 1) 1. No manual or documentation. 1. Online / No software on mobile phone. 2. Error messages. 2. The use of barcode technology. 3. Handling of logical errors. 3. Customizable. 4. Input of data through the mobile phone is problematic in relation to target user group. 5. Human resistance towards the system. Advantages (after reading report 2) Disadvantages (after reading report 2) 1. Online / No software on mobile phone. 1. Human resistance towards the system. Employees feel that they are under surveillance. 2. Customizable. 2. No manual or documentation. 3. The use of barcode technology. 3. Many barcodes needed to navigate the system. 4. Hardware: mobile phone. Everybody knows it. 4. Browser technology/phone restrictions: Input of data through the mobile phone is problematic in relation to target user group. 5. Error messages and handling of logical errors. Feedback 9

Results: Developer B Results: Developer B Advantages (before reading reports) Disadvantages (before reading reports) 1. Hardware: mobile phone. 1. Screen size. 2. Few scans necessary. 2. Problems with GPRS. Often slow. 3. Customizable. 3. No manual or documentation. 4. Online – real time. 4. Only works on some types of mobile phones. 5. Simple solution with limited interaction. Disadvantages (after reading report 1) Advantages (after reading report 1) 1. The text describing each of the barcodes. 1. Hardware: mobile phone. Everybody is familiar with the technology. 2. More user education in needed. 2. Displays only necessary information. 3. System reply time. 3. Customizable. 4. Screen size. Difficult to maintain an overview. 4. Online all the time. 5. System is interpreted differently on different 5. The system is simple and uniform. phones. Advantages (after reading report 2) Disadvantages (after reading report 2) 1. Hardware: mobile phone. Everybody knows it. 1. The text describing each of the barcodes. 2. The system is simple and uniform. 2. More user education in needed. 3. Customizable. 3. System reply time. 4. Displays only necessary information. 4. Screen size. Difficult to maintain an overview. 5. Online all the time. 5. System is interpreted differently on different phones. Feedback 10

Results: Joint List Results: Joint List Advantages (Developer A – originally) Disadvantages (Developer A – originally) 1. Online: The system can provide relevant real 1. GPRS: Limited coverage. time information. 2. Barcodes are used to interact with the system 2. Barcode scanners: All interaction begins with instead of the mobile phone. the user scanning. 3. Online: problem when no connection is 3. No software on the mobile phone. available. Advantages (Developer B – originally) Disadvantages (Developer B – originally) 1. Hardware: mobile phone. 1. Screen size. 2. Few scans necessary. 2. Problems with GPRS. Often slow. 3. Customizable. 3. No manual or documentation. 4. Online – real time. 4. Only works on some types of mobile phones. 5. Simple solution with limited interaction. Advantages (Joint) Disadvantages (Joint) 1. Online – real time. 1. Online: Problems with GPRS. 2. Customizable. 2. No manual or documentation. 3. Rely on commonly known technology: 3. Human resistance towards the system. mobile phone. Employees feel that they are under surveillance. 4. Simple and small barcode-scanner. 4. More user education in needed. 5. The system is simple and uniform. 5. Error messages and handling of logical errors. Feedback 11