Foundations of experimental research 707.031: Evaluation Methodology - - PowerPoint PPT Presentation

Foundations of experimental research

707.031: Evaluation Methodology Winter 2014/15

Eduardo Veas

THEOC, the scientific method

2

Theory Hypothesis Experiment Observation Conclusion

Source of variability

3

Source: Card et al 1983

Curiosity Human behaviour

foundation of experimental research

4

Ideas, Theories and Hypothesis

5

Curiosity in motion

Experimental research

• Establish relationships between circumstances

and behaviors

• Fit these relationships into an orderly body of

knowledge

6

FEAR OF IDEAS

7

Fear your ideas

Anyone doing research is a genius, I don’t come even close GENIEPHOBIA

8

Fear your ideas

I am having a hard time coming with original ideas IMITATOPHOBIA

9

Fear your ideas

Having to use complex hardware… I got a headache PARAPHERNALIO-PHOBIA If there is complex equipment involved, it must be good research MANUPHOBIA

10

Fear your ideas

If it is simple it can’t be science FEAR OF SIMPLICITY

11

Fear your ideas

numbers! numbers! FEAR OF STATISTICS

12

Fear your ideas

Something is missing here, I just know it IMPERFECTAPHOBIA

13

Fear your ideas

Lorem ipsum… FEAR OF NOT SOUNDING SCIENTIFIC

14

Fear your ideas

FEAR OF WORK

15

Fear your own ideas

• Geniephobia
• Imitatophobia
• Paraphernalio-phobia /

Manuphobia

• Fear of simplicity
• Fear of math
• Imperfectaphobia
• Fear of not sounding

scientific

• Fear of work

16

Generating ideas

17

Systematic reduction of idea-phobia

Experimental research

• Establish relationships between circumstances

and behaviors

• Fit these relationships into an orderly body of

knowledge

18

Observation

• Sit in your computer and stare at your keyboard

until your eyes start to bleed

19

Observation

• Sit in your computer and stare at your keyboard

until your eyes start to bleed

• We are interested in human, rather than

keyboard behavior

20

Public observation

write up ideas that come up as you stroll through campus.

21

Public observation

write up ideas that come up as you stroll through campus. you got 7 minutes. statements in the form circumstance => behavior

22

ROT test

Experimental ideas must be:

• R epeatable
• O bservable
• T estable

23

Correlational or observational?

• Label your ideas now

24

Theories

25

Title Text

Theory

• choose one idea and convert it into a theory
• use that theory to make predictions
• each prediction forms a hypothesis

26

induction deduction

Relationship Theory-Hypothesis-Experiment

27

Theory EXPERIMENT Observation induction deduction Predicted Observation Confirmed Observation Disconfirmed Observation induction deduction Theory supported Theory false

Expected results of experiment

• proving a prediction: does not prove but

supports a hypothesis, thus the theory.

• disproving a prediction: not enough evidence

was found to prove the hypotheses/theory

28

Does theory precede data?

29

OBSERVATION THEORY HYPOTHESES EXPERIMENT

Experimental Methodology

30

Formal Curiosity

Experimental research

• Establish relationships between circumstances

and behaviors

• Fit these relationships into an orderly body of

knowledge

31

Experimental research

32

Circumstances

• Behavior

Experimental research

33

Circumstances Behavior heavy breakfast birthday party last night rainy day hot office bright office bad coffee smelly

• ffice

press button read email text girlfriend text boyfriend grab more coffee increase light intensity sunny day loud pitch sound

Experimental research

34

Circumstances Behavior heavy breakfast birthday party last night rainy day hot office bright office bad coffee smelly

• ffice

press button read email text girlfriend text boyfriend grab more coffee increase light intensity sunny day loud pitch sound

Experimental research: causal statements

35

Increase in light intensity press button WHEN DONE CORRECTLY change in measured behavior is due to manipulation of circumstance

Variables in experimental research

36

Title Text

Experimental research: variables

37

Circumstances Behavior press button increase light intensity INDEPENDENT

Experimental research: variables

38

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT

Experimental research: hypothesis

39

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT

the hypothesis is a statement about the expected outcome

Experimental research: hypothesis

40

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT

H1: Participants will be significantly faster in pressing a button in the 100 Lux condition.

Experimental research: variables

41

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT heavy breakfast birthday party last night rainy day hot office bright office bad coffee smelly

• ffice

sunny day loud pitch sound

• ?-

Experimental research: variables

42

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT heavy breakfast birthday party last night rainy day hot office bright office bad coffee smelly

• ffice

sunny day loud pitch sound

• control variables-

Experimental research: external validity

• validity of experimental method: is drawing

conclusions about cause justifiable?

• the more highly controlled the experiment, the

less generalizable its results.

43

Experimental research: variables

44

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT heavy breakfast birthday party last night rainy day hot office bright office bad coffee smelly

• ffice

sunny day loud pitch sound

• control variables-

learning

Experimental research: variables

45

Circumstances Behavior press button increase light intensity INDEPENDENT DEPENDENT heavy breakfast birthday party last night rainy day hot office bright office bad coffee smelly

• ffice

sunny day loud pitch sound

• control variables-

learning

• random variables-

Experimental research: random variables

• random selection of participants.
• random assignment of circumstances to levels of

the independent variable

46

Experimental research: variables

47

CONTROL RANDOM RANDOM WITHIN CONSTRAINTS

generalizable

Experimental research: variables

circumstance that changes systematically as the experimenter manipulates the independent variable CONFOUNDING VARIABLES

48

VALIDITY

49

Where we see it all fail

Experimental Method: validity

• External: is it justifiable to generalize causation

from the results

• Internal: are there confounding variables which

have not been taken into account? are there unconsidered threats?

50

Threats to internal validity

• History
• Maturation
• Selection
• Differential mortality
• Testing
• Statistical regression
• Interactions with selection

51

Experimental Design

52

Blueprint of a proof

Experimental design questions

• HOW MANY INDEPENDENT VARIABLES?
• HOW MANY DIFFERENT VALUES DOES

EACH VARIABLE HAVE?

53

Experimental design: Decision

54

Design study Basic design Factorial design Between groups Within groups Between groups Within groups Split plot Number of independent variables >1? YES NO Number of values per independent variable

Between participant design

• Each participant is exposed to one level only
• Divide participants in groups (one per

condition)

• Compare measurements between groups

55

Between participants design: advantages

• Rules out learning effects.
• Cannot contaminate behavior in other levels
• Can collect more data per level / more

participant time per level.

56

Between participant design: disadvantages

• differences between groups of participants =

differences between conditions

57

Within participants design

• Each participant is exposed to every condition

58

Within participant design: advantages

• requires fewer participants
• minimizes individual differences between levels
• f the independent variable

59

Within participant design: disadvantages

• Needs to account for learning effects
• Needs to account for ordering effects
• Combinatorial explosion limits number of

conditions

60

Number of participants

• Depends on
• effect size
• study design
• Calculated through power analysis (statistical

power)

61

Experimental design: overview

• Trial: independent unit of measurement
• Measurement:
• quantitative: measurable indicators (task

completing time, error rates, mouse movement)

• qualitative: subjective feedback (satisfaction,

preference)

• observations

62

Experimental design: am I ready?

• Ethical constraints?
• Number of participants?
• How long will it take?
• Should I set participant restrictions?
• Should I set criteria for eliminating participants?
• Can I operationally define all my variables?
• Have I organized equipment?
• Do I know how I will analyze my data?
• How will I interpret results?

63

Experimental design: Pilot test

• rule out planning errors
• get acquainted with methodology
• test analysis tools (if pilot big enough)

64

Experiment execution

• Repeat:
• all participants in a group are expected to

receive the same stimuli

• for each participant
• repeat instructions exactly
• respect execution
• plan breaks
• DO NOT CHANGE THE RULES!!!

65

Reporting 101

66

if you live to tell the story

Assignment: Experimental protocol

• Introduction:
• observation
• statement of purpose
• hypotheses
• Method
• procedure:
• design (within-between)
• combinations of conditions
• one experiment session

67

Outline of report

• Introduction:
• statement of purpose
• hypotheses
• Method
• participants
• apparatus/materials
• procedure
• Results
• descriptive statistics
• statistical tests used
• results in standard way
• Discussion

68

R

69

Title Text

set work directory

• setwd("/new/work/directory")
• getwd()

70

packages

• #installing packages
• install.packages("package.name")
• # loading a package
• library(package.name)
• # disambiguating functions
• package::function()

71

72

Readings

• Doing Psychological Experiments (Martin): Ch2-

Ch3-Ch8