Evaluations, Studies, and Research 707.031: Evaluation Methodology - PowerPoint PPT Presentation

Evaluations, Studies, and Research 707.031: Evaluation Methodology Winter 2014/15 Eduardo Veas

Research Projects @ KTI • Connected world • build connected coffee machine • build sensing and intelligence into appliances • Augmented Data • how can we augment the real world with data? • investigate different display devices • investigate different visual techniques • Augmented Knowledge Spaces • Use space to organize and interact with technology • Use natural mobility to interact with augmentations 2

3

4

5

Why do we evaluate? Motivation 6

What are evaluations? Why do we need them? 7

Why do we evaluate? • to make a product more efficient • to know whether we are going in the right path • find out if people can do what they wanted to do with the tool • to obtain new ideas • choose between options in the design • for comparing interfaces 8

Continuous Evaluation Methods for D & D 9

Waterfall Model of Software Engineering Initiation Application Analysis Description Requirement Design specification System Implementation Design Product 10

Design Build Test Design Build Test Fab. errors Design errors Alice Agogino. NASA Jet Propulsion Lab 11

UCD: ISO9241-210 Plan the Human Centered Design process Understand and specify the context of use Iterate where appropriate Evaluate the designs Specify the user against requirements requiremets Produce design solutions to meet Designed user requirements solution meets requirements 12

THEOC, the scientific method Theory Hypothesis Experiment Observation Conclusion 13

Creative Problem Solving [Korberg and Bagnall ’71] 14

Creative Problem Solving [Korberg and Bagnall ’71] Ideate Define Select Analyze Implement Accept Situation Evaluate 15

Design Thinking 16

Design Thinking Principles • Heterogeneous teams • Cooperative work • Fail often and soon 17

A Process of Iterative Design Design Prototype Evaluate 18

A Process of Iterative Design Design Prototype Evaluate 19

Continuous Evaluation • Iterative methods expose several stages • We evaluate at every stage • Different evaluation methods for different purposes 20

Why do we evaluate? • to make a product more efficient • to know whether we are going in the right path • find out if people can do what they wanted to do with the tool • to obtain new ideas • choose between options in the design • for comparing interfaces 21

We evaluate to understand a process and design solutions. We evaluate to validate our designs. Use evaluation to create and critique 22

Evaluation Goals Never stop exploring 23

How do we evaluate? • stage defines goals and methods for evaluation • evaluation informs iteration or continuation to next stage 24

Goals • Find out about your users: • what do they do? • in which context? • how do they think about their task? • Evaluation goals: • users and persona definition • task environment • scenarios 25

Goals • Select initial designs • use sketches, brainstorming exercises, paper mockups • is the representation appropriate? • Evaluation goals: • elicit reaction to design • validate/invalidate ideas • conceptual problems/ new ideas 26

Goals • Iterative refinement • evolve from low-> high fidelity prototypes • look for usability bugs • Evaluation goals • elicit reaction to design • find missing features • find bugs • validate idea 27

Goals • Acceptance • did the product match the requirements • revisions: what needs to be changed • effects: changes in user workflow • Evaluation goals • usability metrics • end user reactions • validation and bug list 28

Where do we use this knowledge? • Visualization • Social Computing • Human Computer Interaction • Big Data analytics • Virtual / Augmented Reality 29

707.031: Evaluation Methodology a research methodology 30

707.031: Evaluation Methodology This course is about learning from mistakes, knowing when to move to the next stage and when to go back to the drawing board. 31

707.031: Evaluation Methodology • Scheduled annually since this year. Depending on students. • First time as block lecture (2-week course) • This may be your only chance to take it • If you find this course valuable, you have to score it, so other students will have the opportunity in the future. (Lehrveranstaltungsevaluierung) 32

707.031: Evaluation Methodology • is not an intro to HCI, InfoVis, Visual Analytics, Augmented Reality. • is not an Advanced Statistics, (Web) Usability, Interface Design. • is appropriate for students (PhD. and Msc.) and researchers investigating: • novel metaphors to interact with machines • user behaviour and how it is influenced by technology 33

707.031: Evaluation Methodology WYG What you get: • organize your research problem • collect data about the problem and solutions • compare different evaluation methods • understand when which evaluation is appropriate • properly report methodology and results 34

§ • D1: Model Human Processor • D2: Visual Processing • D3: Visual Processing 2 • D4: Haptics ? • D5: Crowdsourced studies ? • D6: Descriptive and Correlational Research Methods • D7: Two-Sample Experimental Designs: • D8: Multi-Sample Experimental Designs • D9: Putting it all together • D10: Evaluation 35

707.031: Evaluation Methodology Grading • 30% participation (in class) • 40% evaluator • 30% participant • (bonus 15% for each study you take part in) 36

Project Topics • Glove Study • AR Study • Collection Study • Visualization Study 37

Source of Variability ensuring the vitality of species 38

The Human Homunculus 39

The Human Homunculus 40

The Human Homunculus 41

Measuring performance 42

Comparing Human Responses • Humans can rarely repeat an action exactly even when trying hard • People can differ a great deal from one another • How can we compare responses from different adaptive systems? 43

Model Human Processor • Is there a way to approximate responses of people? • Can we predict usability of interface designs? • …without user involvement? 44

Model Human Processor Source: Card et al 1983 45

Model Human Processor(2): Processors • Processing typical value and window. • Window [a,b] defined by extremes • Typical value is not average. It conforms to studied behavior 46

Model Human Processor (4): Memory • Decay: how long memory lasts • Size: number of things • Encoding: type of things 47

Model Human Processor (4): Memory • WM: percepts and active products of thinking in (7+/-2) chunks. • WM Decay ~ 7s / 3chunks. Competition / discrimination • LTM: Infinite mass of knowledge in connected chunks. 48

BCSBMICRA Read aloud 49

CBSIBMRCA Read aloud 50

Model Human Processor: Read Aloud • Tool • Pen • Window • Coat • Cow • Paper 51

Model Human Processor: Read Aloud • Orange • Black • Pink • Red • Green • Blue 52

Model Human Processor (3): Perception • encodes input in a physical representation • stored in temp. visual / auditory memory • new frames in PM activate frames in WM and possibly in LTM • Unit percept: input faster than Tp combines 53

Model Human Processor (3): Cognition • Recognize-act cycle • Uncertainty increases cycle time • Load decreases cycle time 54

Model Human Processor (3): Motor • controls movement of body, • combining discrete micromovements (70ms) • activates action patterns from thought. • head-neck, arm-hand- finger 55

Model Human Processor: cycle time • A user sitting at the computer must press a button when a symbol appears. What is the time between stimulus and response? 56

Model Human Processor: cycle time • Red pill / blue pill. A user sitting at the computer must press a button when a blue symbol appears. What is the time between stimulus and response? 57

Hicks Law: Decision Time • Models cognitive capacity in choice-reaction experiements • Time to make decision increases with uncertainty • H = log2(n + 1), for n equiprobable ∑ • H = p log ( 1 / p 1 ) + i 2 i i 1 = 58

Model Human Processor: Motor action • At stimulus onset, 9 participant has 5 S to move the mouse to target and click. How long does it take? D 59

Fitts Law • Motion as a sequence of motion-correction. • Each cycle covers remaining distance • Time T for arm-hand system to reach target of size S at distance D: T = a + b * log 2 ( D / S + 0.5 ) • where a: y-intercept, b: slope S D 60

Model Human Processor: Summary • Top down analysis of response • Reasonable approximation of response and boundaries (Fastman, Middleman, Slowman) • For each expected goal • analyze motor actions • analyze perceptual actions • analyze cognitive steps transferring from perception to action • BUT • missing parts: motor- memory, other senses (haptic / olfactory), interference model, reasoning model 61