Last Class COGS 105 Research Methods for Cognitive Scientists In - - PowerPoint PPT Presentation

COGS 105

Research Methods for Cognitive Scientists

Week 3, Class 2: Behavioral Methods I: Reliability and Validity

Last Class

• In any behavioral research we need to design

measures, develop tasks, and recruit people to participate in them.

• Lots of sampling methods; usually we are stuck

with nonprobability “haphazard” sampling, and we often assume that our recruitment (e.g., SONA) is “effectively random.”

Our LDT Task

• We started with a simple

Lexical Decision Task: Are you faster at processing uncommon or common words?

• General thrust of the result:

common words (“higher frequency”) are faster to process than uncommon words (“lower frequency”)

Between vs. Within

common word stimuli uncommon word stimuli common word stimuli uncommon word stimuli

RT(uncommon) vs. RT(common) RT(uncommon) vs. RT(common)

RT(uncommon)

Between vs. Within

vs. RT(common)

common word stimuli uncommon word stimuli common word stimuli uncommon word stimuli

RT(uncommon) vs. RT(common)

within-subject experiment between-subject experiment

Pervasive Sampling Issues

• We sample subjects, we sample words as stimuli, and for

each participant in our task we have to sample the stimuli we chose for presentation in a given order.

• All of these can involve biases.
• Participant biases: e.g., WEIRD
• Stimulus biases: e.g., you choose words that are not

perfectly comparable only in the variable of interest (commonality, aka frequency)

• Presentation biases: you order the words in a way that

influences responses.

E.g., Stimulus Biases

• If we want to compare common vs. uncommon words, we need

to isolate this one difference, and our target stimuli (common

• vs. uncommon) should be:
• Overall matched for length
• Overall matched for pronounceability
• Overall matched for concreteness in meaning
• Etc.
• Such extensive controls are difficult to achieve but possible with

some available tools.

Example Tool

• English Lexicon Project!
• Large-scale project helping you select stimuli for your

word experiments (used often for LDT).

• Can help you avoid certain “stimulus biases,” to

make sure words are differ only on one dimension.

• Completely free to use; you can use it next week for

your lab!

• http://elexicon.wustl.edu/

E.g., Control for Length

sort the data that is emailed to you

Another Bias: Order

Respond with your dominant hand if you see a real word.

• rder bias!

fludl made pragrl walking suggest fort “no-go trial”

uncommon, but faster because they

• ccur in order?

Construct Validity

• Now that you have the task in mind… consider…

construct validity.

• We wish to make an inference about how people

process words.

• Thus LDT is a method (an operationalization) of mental

processing that is supposed to tell us something about a construct: word processing.

• You typically cannot directly observe the construct; your
• perationalization (your measures) help you make

inferences about it.

Validity of What?

LDT language processing

method construct theoretical concept or proposal under study the method is a proposed

• perationalization
• f construct

Kinds of Validity

• “In face validity, you look at the operationalization

and see whether "on its face" it seems like a good translation of the construct.”

also: ecological validity

LDT carefully choose a bunch of words show ‘em one at a time separated by carefully controlled time intervals in a quiet room in front of a computer and you’re asked to “just recognize them”

Kinds of Validity

• In predictive validity, “we assess the
• perationalization's ability to predict something it

should theoretically be able to predict.”

• E.g., can LDT be used to measure other aspects of

language processing? For example, can it demonstrate that positive vs. negative words are processed differently? Can it show that longer words and processed more slowly than shorter words? Etc.

Kinds of Validity

• “In convergent validity, we examine the degree to

which the operationalization is similar to (converges

• n) other operationalizations that it theoretically should

be similar to.”

• Eye movements while reading?
• Naming times? Rather than responding to word/

nonword, respond by speaking the sequence of letters (common words also faster!).

• LDT should “converge” with these tasks.

Word-Naming Task

• WNT is a variant of LDT that is often used for similar
• purposes. Let’s give it a try. Just speak these words

as you see them as quickly, but naturally, as you can.

We expect WNT to have “convergent validity” with results in LDT. symbol plenty

• ther

also

reliability = “consistency” validity = “accuracy”

LDT valid?

LDT predicts word recognition, vocabulary size, and fluency to some degree

Why RT / LDT?

• These kinds of measures are very simple, and

seemingly artificial, however they have massive and broad applicability!

• Two case studies:
• 1) Lumosity
• 2) The IAT (as in lab)

Lumosity

Online brain-training system that uses basic cognitive task

• perationalization

IAT

• “Implicit Association Task”: Uses basic RT to tap

into potential biases or stereotypes you might have.

Demo...

Left hand: Good Right hand: Bad smelly stupid delicious friendly evil pleasant Affordable Care Act

Example use of IAT in business / marketing

“First, explicit measures and IAT measures of attitudes and other marketing constructs converge when consumers are willing and able to report their feelings and beliefs.” Brunel, F. F., Tietje, B. C., & Greenwald, A. G. (2004). Is the implicit association test a valid and valuable measure of implicit consumer social cognition?. Marketing, 4.

Construct?

• Construct: Political affiliation, or race?
• Operationalization: reaction time (RT) to responses

that are mapped onto the same hand.

• Construct validity:
• Face validity?
• Predictive validity?
• Convergent validity?

1

what’s your question? frame it as a hypothesis develop your task sample from people conduct analysis

sample from the world for your stimuli / task

• perationalize

with a method measure behavior statistical tools (PSYC 10!) don’t fool

• urselves

critique, replicate integrate

1

• “These subtle distinctions,

about sampling, validity, reliability, and so on… really it is becoming clear that the

• nly way to really

understand these things is to get in there and do studies…”

Next class…

• Let’s move into some methodological specifics:

Details of using reaction time.

• Lab: You will build your own reaction-time

experiment.

• You can build your own creative experimental

idea using the overall process just described.