Trade Studies A Hypothetical Example Based On a Real Study 1 Human - - PowerPoint PPT Presentation

▶
trade studies a hypothetical example based on a real study
SMART_READER_LITE
LIVE PREVIEW

Trade Studies A Hypothetical Example Based On a Real Study 1 Human - - PowerPoint PPT Presentation

Apr 18, 2023 485 likes •771 views

Trade Studies A Hypothetical Example Based On a Real Study 1 Human Factors Research to Inform the Human-Machine Systems Engineering Process Needs, Problems, Opportunities Generalizable Research Question(s) Hypothesis Formulation

slide-1
SLIDE 1

1

Trade Studies A Hypothetical Example Based On a Real Study

slide-2
SLIDE 2

2

Users, Operators, Subject Matter Experts

Human Factors Research to Inform the

Human-Machine Systems Engineering Process

Needs, Problems, Opportunities Operation,Test & Evaluation Analysis Design Implementation Design Specifications Requirements HMS: Humans, Machines, Processes

(Model, Mockup, Prototype, Product) Data Collection Data Analysis & Hypothesis Testing Interpretation & Application of Results Generalizable Research Question(s) HFE Principles & Guidelines Research Design Hypothesis Formulation

slide-3
SLIDE 3

3

Users, Operators, Subject Matter Experts

Human Factors Trade Studies to Inform the

Human-Machine Systems Engineering Process

Needs, Problems, Opportunities Operation,Test & Evaluation Analysis Design Implementation Design Specifications Requirements HMS: Humans, Machines, Processes

(Model, Mockup, Prototype, Product) HFE Principles & Guidelines Data Collection Data Analysis Interpretation & Application of Results Design Options Study Design Prototype Construction

slide-4
SLIDE 4

4

Conducting a Trade Study: An Example

  • NB:

1)This example is hypothetical, but based on the instructor's experience with real trade studies. 2)It is probably more elaborate than most trade studies, but it illustrates what can be done when right answers to the questions are important.

slide-5
SLIDE 5

5

Design Question

For a digital camera UI, which menu geometry configuration yields the best speed, accuracy, and user satisfaction?

  • Linear Horizontal?
  • Linear Vertical?
  • Circular?
slide-6
SLIDE 6

6

Menu Geometry Design Options

A: Linear Horizontal B: Linear Vertical C: Circular

Camera Settings Flash Timer Close Playback Camera Settings Flash Timer Close Playback Camera Settings Flash Timer Close Playback

slide-7
SLIDE 7

7

Menu Geometry Design Options

A: Linear Horizontal B: Linear Vertical C: Circular

Camera Settings Flash Timer Close Playback Camera Settings Flash Timer Close Playback Camera Settings Flash Timer Close Playback Settings Settings Quality Red Eye Beep No Card Close Rev Time Quality Red Eye Beep No Card Close Rev Time Settings Beep On Close Off Quality Red Eye Beep No Card Close Rev Time Quality Red Eye Beep No Card Close Rev Time Settings Beep On Off Close Settings On Off Close Settings Beep

slide-8
SLIDE 8

8

Study Design

  • Participants
  • Apparatus
  • Data Collection
  • Data Analysis
  • Interpretation of Results
slide-9
SLIDE 9

9

Study Design

  • Participants: representative sample of 12 users

–

6 female, 6 male

–

various ages

–

novices & experts

slide-10
SLIDE 10

10

Study Design

  • Participants: representative sample of 12 users

–

6 female, 6 male

–

various ages

–

novices & experts

  • Apparatus:
slide-11
SLIDE 11

11

Apparatus Options

  • 1. Fully functional prototype camera

– most expensive – longest time to build – highest fidelity

  • 2. Part-functional prototype (e.g., just display with menu and a few

controls)

– less expensive – less time consuming – moderate fidelity (but interactive)

  • 3. PowerPoint slides with links

– minimal cost – quick – low fidelity (limited interaction)

  • 4. Paper

– minimal cost – quick? – lowest fidelity (not interactive)

slide-12
SLIDE 12

12

Study Design

  • Participants: representative sample of 12 users

–

6 female, 6 male

–

various ages

–

novices & experts

  • Apparatus: part-functional prototype camera

–

display

–

menu/photos

–

D-pad control

–

fire (shutter-release) button

  • Data Collection:

–

each participant uses 3 menus (random order) and for each

  • hears explanation, instructions
  • performs series of tasks (speed & accuracy recorded)
  • answers SUS questionnaire
  • answers comparison questionnaire: Hardest/Easiest?, etc.
  • answers interview questions: 3 best things? 3 worst things?

Comments? Suggestions?

–

ranks 3 menus, best to worst

slide-13
SLIDE 13

13

Data Collection

Participant Tasks

1.Set Date and Time 2.Turn on Date Display 3.Turn on Flash 4.Select Red-Eye Reduction 5.Quickly “take” 3 photos 6.Review photos on display 7.Delete 2nd photo

(Prototype records time & action in log file for later analysis.)

Participant Menu Order

  • 1. CHV
  • 2. VCH
  • 3. HCV
  • 4. VCH
  • 5. CVH
  • 6. CHV
  • 7. CVH
  • 8. VHC
  • 9. HVC
  • 10. HCV
  • 11. HVC
  • 12. VHC
slide-14
SLIDE 14

14

Study Design

  • Participants: representative sample of 12 users

–

6 female, 6 male

–

various ages

–

novices & experts

  • Apparatus: part-functional prototype camera

–

display

–

menu/photos

–

D-pad control

–

fire (shutter-release) button

  • Data Collection:

–

each participant uses 3 menus (random order) and for each

  • hears explanation, instructions
  • performs series of tasks (speed & accuracy recorded)
  • answers SUS questionnaire
  • answers comparison questionnaire: Hardest/Easiest?, etc.
  • answers interview questions: 3 best things? 3 worst things?

Comments? Suggestions?

–

ranks 3 menus, best to worst

  • Data Analysis → Results:

–

Descriptive statistics

–

Box-and-whisker charts

–

Paired t-tests

slide-15
SLIDE 15

15

Data Analysis: Data

Completon Time (sec) Data

Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Horiz Completon Time (sec) 14.5 14.4 14.0 22.6 21.4 19.4 21.7 25.8 25.3 20.4 12.1 20.9 Vert Completon Time (sec) 14.5 14.8 14.5 14.2 16.0 16.7 18.0 17.7 16.9 12.5 16.7 17.0 Circ Completon Time (sec) 13.3 26.0 19.6 17.4 15.5 14.6 24.7 19.0 21.3 27.8 25.2 17.7

Errors Data

Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Horiz Errors 4 6 2 3 3 3 5 1 3 3 1 2 Vert Errors 2 2 2 2 3 2 2 4 Circ Errors 7 2 2 10 3 8 6 2 4 8

SUS Overall Score Data

Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Horiz SUS Overall Score 60 64 67 62 83 87 78 74 71 64 66 63 Vert SUS Overall Score 87 98 100 90 79 92 72 82 91 86 88 85 Circ SUS Overall Score 96 53 69 76 79 78 80 43 72 58 77 77

Best/Worst Comparisons Data

Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Comparison: Easiest to use Vert Circ Horiz Horiz Vert Vert Horiz Circ Vert Vert Horiz Vert Comparison: Hardest to use Horiz Horiz Circ Circ Horiz Circ Circ Horiz Circ Horiz Circ Circ Comparison: Most efcient / Fastest Vert Vert Vert Circ Circ Horiz Horiz Vert Vert Vert Vert Vert Comparison: Least efcient / Slowest Circ Horiz Horiz Horiz Horiz Circ Circ Circ Horiz Circ Horiz Horiz Comparison: Easiest to learn Horiz Vert Horiz Circ Vert Horiz Circ Horiz Circ Vert Vert Vert Comparison: Hardest to learn Circ Circ Circ Horiz Horiz Circ Horiz Circ Horiz Circ Circ Horiz Comparison: Most intuitve Vert Vert Horiz Vert Vert Horiz Circ Horiz Circ Vert Vert Vert Comparison: Least intuitve Circ Circ Circ Circ Horiz Circ Horiz Vert Horiz Circ Circ Circ

Rankings Data

Data p01 p02 p03 p04 p05 p06 p07 p08 p09 p10 p11 p12 Vert Circ Vert Horiz Vert Vert Circ Vert Vert Vert Horiz Vert Horiz Vert Horiz Vert Circ Horiz Vert Horiz Horiz Circ Vert Horiz Circ Horiz Circ Circ Horiz Circ Horiz Circ Circ Horiz Circ Circ Ranking: 1st Ranking: 2nd Ranking: 3rd

slide-16
SLIDE 16

16

Data Analysis & Results: Task Completion Time

Completon Time (sec) Statstcs

Data n min mean median max Horiz Completon Time (sec) Horiz 12 12.1 14.5 19.4 20.7 21.9 25.8 Vert Completon Time (sec) Vert 12 12.5 14.5 15.8 16.3 17.0 18.0 Circ Completon Time (sec) Circ 12 13.3 16.9 20.2 19.3 24.8 27.8

Completon Time (sec)

Paired t-Tests Diferences & Paired t-Tests n mean

  • std. dev.

t statstc p(2-tailed) Horiz – Vert 12 3.6 4.3 2.900 2.201 True 0.0144 Horiz – Circ 12

  • 0.8

7.1

  • 0.391

2.201 False 0.7033 Vert – Circ 12

  • 4.4

5.3

  • 2.852

2.201 True 0.0157 1st qtl 3rd qtl t(α=0.05, ν=n-1) |t stat| > t(α=0.05, ν=n-1)

Horiz Vert Circ 5 10 15 20 25 30

Completion Time

Time (sec): min, 1st qtl, median, 3rd qtl, max

slide-17
SLIDE 17

17

Data Analysis & Results: Errors

Errors Statstcs

Data n min mean median max Horiz Errors Horiz 12 1.0 2.0 3.0 3.0 3.3 6.0 Vert Errors Vert 12 0.0 0.0 1.6 2.0 2.0 4.0 Circ Errors Circ 12 0.0 2.0 4.3 3.5 7.3 10.0

Errors

Paired t-Tests Diferences & Paired t-Tests n mean

  • std. dev.

t statstc p(2-tailed) Horiz – Vert 12 1.4 1.9 2.609 2.201 True 0.0243 Horiz – Circ 12

  • 1.3

4.2

  • 1.109

2.201 False 0.2909 Vert – Circ 12

  • 2.8

4.2

  • 2.279

2.201 True 0.0436 1st qtl 3rd qtl t(α=0.05, ν=n-1) |t stat| > t(α=0.05, ν=n-1)

Horiz Vert Circ 2 4 6 8 10 12

Errors

Errors: min, 1st qtl, median, 3rd qtl, max

slide-18
SLIDE 18

18

Horiz Vert Circ 20 40 60 80 100 120

SUS Overall Score

SUS Score: min, 1st qtl, median, 3rd qtl, max

Data Analysis & Results: Errors

SUS Overall Score Statstcs

Data n min mean median max Horiz SUS Overall Score Horiz 12 60.0 63.8 69.9 66.5 75.0 87.0 Vert SUS Overall Score Vert 12 72.0 84.3 87.5 87.5 91.3 100.0 Circ SUS Overall Score Circ 12 43.0 66.3 71.5 76.5 78.3 96.0

SUS Overall Score

Paired t-Tests Diferences & Paired t-Tests n mean

  • std. dev.

t statstc p(2-tailed) Horiz – Vert 12

  • 17.6

13.6

  • 4.472

2.201 True 0.0009 Horiz – Circ 12

  • 1.6

16.5

  • 0.332

2.201 False 0.7462 Vert – Circ 12 16.0 17.3 3.199 2.201 True 0.0085 1st qtl 3rd qtl t(α=0.05, ν=n-1) |t stat| > t(α=0.05, ν=n-1)

slide-19
SLIDE 19

19

Data Analysis & Results: Comparisons & Rankings

Best/Worst Comparisons

Counts Data n Horiz Vert Circ Comparison: Easiest to use 12 4 6 2 Comparison: Hardest to use 12 5 7 Comparison: Most efcient / Fastest 12 2 8 2 Comparison: Least efcient / Slowest 12 7 5 Comparison: Easiest to learn 12 4 5 3 Comparison: Hardest to learn 12 5 7 Comparison: Most intuitve 12 3 7 2 Comparison: Least intuitve 12 3 1 8

Horiz Vert Circ 5 10 15 20 25 30 35

Menu Ranking Score Rankings

Counts Data n Horiz Vert Circ rank weight 1st 12 2 8 2 3 2nd 12 6 4 2 2 3rd 12 4 8 1 22 32 18 menu ranking score Ranking: 1st Ranking: 2nd Ranking: 3rd

slide-20
SLIDE 20

20

Data Analysis & Results: Some Interview Question Responses

  • Horizontal

–

Similar to other camera menus I've used

–

Good menu organization & layout

–

Menus were large & easy to read

–

Knew how to use controls / controls worked as expected

–

Easy to navigate

–

etc.

  • Vertical

–

The menu layout was simple & straightforward

–

Controls worked as expected (greater number than horizontal)

–

Easy to navigate menu

–

Easy to find menu & sub-menu options

–

Similar to a computer menu

–

etc.

  • Circular

–

Controls did not perform as expected

–

Did not like the dispersed layout of menu options

–

Menu appeared to be unorganized

–

Appeared cluttered

–

Did not like scrolling to get to the menu options

–

Difficulty searching for menu options

–

etc.

slide-21
SLIDE 21

21

Interpretation of Results

  • Discussion
  • Decision
slide-22
SLIDE 22

22

What would make this trade study a human factors experiment?

  • A more general question to answer, a Research Question, like

–

For the UI for any hand-held, special purpose device, which menu geometry configuration yields the best speed, accuracy, and user satisfaction?

  • A more abstract (more generalized) apparatus.
  • A wider range of participants, more representative of the general

population

  • More participants – to go for statistically significant results.
slide-23
SLIDE 23

23

Appendix

Paired t-Tests

slide-24
SLIDE 24

24

Using Paired t-tests To Compare Means In a Within-Subjects Study

H 0: μD=μ1−μ2=0 (means are the same) H 1: μD=μ1−μ2≠0 (means are different) t= ¯ d sd/√n where: ¯ d=sample meanof differences sd=sample standard deviationof differences Critical region(two−tailed test):t<−t(α/2,n−1) and t>t(α/2,n−1)

slide-25
SLIDE 25

25

Example: Compare speed on task with two user interface designs, UI 1 & UI 2

slide-26
SLIDE 26

26

t-Table

slide-27
SLIDE 27

27

Example: Compare speed on task with two user interface designs, UI 1 & UI 2