2/9/2015 ! - - PDF document

SLIDE 1

2/9/2015 1

• !

!""

##

"

$

%

"

#" #" ""#

#$% &''%(%)

&" '$"

!*

( )$**+

,$"$

"-#$$)**+ .$

/"$"

$)#0*0+ "0

)-"+

%

$#1 #

"**- .$"

"

"""# $#")"*

**+

("##"#

)"$0 + "$)$*23#$"

#45+

SLIDE 2

2/9/2015 2

+'$!

"""

#"".$"

/#)

"0$-+

$*-

$"1 "#"$*6

$$"*-*

)$*"*#+

"7$*$$$ $7%8%" $$7 8%" 89"

#*$

&$

2:(05

;$

" $#

• ")205+

',$-*' "

3")+$4

$(

/""4 ;$#$

4

"4 3$4

*.'$

7"

$*#"*

• 7
• -"

:$-

/$'012!"

7:(#$$"#$

2(5"###:( "$#$$ "$($"#*" ($#($ ####:($"# "#*($"# ($$$$#""#$ #"(" ##3#*#( #$$$#"#<#$# #*#$#"#" '(:($$*#"#" #(($=#"(# "#### (##"(#"*"""#

/$'012!"

2;$0*>(

(#?"0>#( :#<#* ("## *"#$($ 5

SLIDE 3

2/9/2015 3

*.'$

7*$*

"$

#7#$

@

• #

*.'$

$7

" "< "0$#

3(7

/""$ " A#$

12!"

7B"#"#( %"4 7$#"##2(5(

"#$**##>"$"# #*$(0*> #"# "" #1 $($*: $$$*>"##$> >("$"#( 1 "$"#>*$ #>**>(#> #">"#$** $>""$#*"#>( #

'!2!3

:#$)"#+ /#

:"" /2($5 /25

#

/"#$"")**+ ($

%-4'"

$7 %"*

$(#)&=7% #+*"#"#" (0*.#*# "#4 **0

• &#$"$

"4

/#"$

#)$*#+

''$5

7"

$*#"*

• 7
• -"

:$-

SLIDE 4

2/9/2015 4

/$'012!"

7:(#$$

$$$#($#:*" $$(##*(#" #"#= #"(#"# ###(##"(#"* """#

C>7>((#(

#""#)3:A 3*:BD+7"#($# 3#($"* #"44

• .'"

&"73"

25"#4

$7&#.(

#4

&.$"7B##

$"4

7B#""#

" %"4

#7/#("

""$ 4/#(" #4

73##("

"4

E7B#43

#

''$

7*$*

"$)"(0 +

&$ $

#7

''$

$7

##

3(7

# 0"#

#)+

• -3

/

• /#)"+

(#"

• ""

$(0

12!"

:($#(#*""# ##4 FFFFFFBA, FFFFFF,: A;7B#4 A;73"2"5"#4

SLIDE 5

2/9/2015 5

6*$*--4'"

/#$" &#$"$# G""$ "" /#""$ &$"" /# $$ #"$

#

#"# ("4

7*$

$$

#"

$

/>

• "

3(

A#"-#)##<+ #$

Types of Behavior Coding

• Real-time coding
• Coding from tapes

Sample Size Issues

• For diagnostic analyses, van der Zouwen & Smit (2004)

recommend ~100 cases

• Zukerberg et al. (1995) claim that:
• “a small behavior coding pretest [n=15 cases] can be as efficient at identifying

questionnaire flaws as a larger one [30-50 cases].”

Analytic Approaches

• Classical approach
• Analysis of code patterns (i.e., frequencies) of respondent and/or

interviewer behaviors

• 15% rule (Morton-Williams, 1979)
• Diagnostic approach
• Analysis of problematic interaction sequences
• Argument that many behaviors that initially indicate problems are resolved

successfully by the interviewer

• Percentage paradigmatic sequences
• Van der Zouwen & colleagues (2002; 2004)
• These two approaches are strongly correlated

Advantages of Classical Approach

• 1. A powerful strategy for identifying

questionnaire problems

• 2. Easy to implement
• 3. Objective
• 4. Reliable
• 5. Replicable

SLIDE 6

2/9/2015 6

Advantages of Diagnostic Approach

• 1. Is not restricted to information about the first

phases of the question-answer sequence only.

• 2. Makes use of content of what is said.
• 3. Permits analysis of the interaction between

respondent & interviewer

Diagnostic Approach Sequence Typology

• 1. Paradigmatic sequences
• 2. Problematic sequences
• 3. Inadequate sequences

Problematic Sequences

• 1. Tuning – premature responses
• 2. Corrections – interviewer mistakes that are

adequately corrected

• 3. Irrelevancies –superfluous conversation

Inadequate Sequences

• 1. Changing question meaning
• 2. Changing response categories
• 3. Risky or wrong interpretations of responses
• 4. Suggesting an answer
• 5. Insufficient probing

Comparisons of Behavior Coding with Other Pretest Methods

• Presser & Blair (1994)
• BC the most reliable method (compared to cognitive

interviewing, traditional pretesting and key informants.

• Van der Zouwen & Dijkstra (2002)
• Compared interaction analysis with jury ratings & cognitive

task analysis.

• Interaction analysis identified problems with questions not

detected by other methods.

• Van der Zouwen & Smit (2004)
• Compared 6 pretest methods
• Correlations between ex-ante evaluations and DA/BC

methods is close to zero

Some Limitations of BC

• Provides no information about the cause of any problems

identified.

• But can serve as the basis for interviewer debriefing
• Will not detect problems when respondents give

acceptable answers to questions that they have misinterpreted.

• Depends on outward manifestations of mental processes

and on the matching of behaviors with their underlying cognitive processes.

SLIDE 7

2/9/2015 7

Coding Schemes

• Vary in complexity and emphasis

Presser & Blair (1994)

1.

R have difficult coming to an understanding of question Meaning?

2.

R have any difficulty remembering the question?

3.

R have any difficulty understanding meaning of specific words or concepts?

4.

Different R’s have different understandings of what question refers to?

5.

R have any difficulty recalling, formulating, or reporting an answer?

6.

Interviewer have any problem reading question or recording answer?

7.

Analyst have any problem using data?

Original Cannell (1968) Coding Scheme

1.

Task-related behaviors

1.

Interviewer (n=5 codes)

2.

Respondent (n=6 codes)

2.

Unrelated conversation initiations

1.

Interviewer (n=4)

2.

Respondent (n=4)

3.

Total codes (n=19)

Original Cannell Coding Scheme: Task- Related Behaviors

1.

Task-related behaviors

1.

Interviewer

1.

Repeats question as worded

2.

Other nondirective probes

3.

Directive probes

4.

Clarifies meaning of question

5.

Suggests other sources of information be consulted

Original Cannell Coding Scheme: Task- Related Behaviors

1.

Task-related behaviors

1.

Respondent

1.

Acceptable answer

2.

Inadequate answers

3.

Elaborated answers

4.

Requests clarification

5.

Consults other information sources

6.

Questions adequacy of answer

Original Cannell Coding Scheme: Unrelated Conversational Initiations

1.

Interviewer or respondent:

1.

Respondent talks about interviewer/interviewer talks about respondent

2.

Talks about self, family, friends

3.

Talks about interview

4.

Laughs, jokes

SLIDE 8

2/9/2015 8

Cahalan et al. (1994)

1.

Interviewer Behaviors

1.

Question asked appropriately?

2.

Question asked as written?

3.

Any probing or clarification by interviewer?

4.

Interviewer demonstrate affect (laughter, sympathy)? 2.

Respondent Behaviors

1.

Was a “correct” response obtained?

2.

Was a repeat of the question requested?

3.

Was a clarification requested?

4.

Was there any indication by the respondent that the question was sensitive? 3.

Total codes = 8

Bates and Good (1996) Coding Scheme (also used

by Zukerberg et al.)

1.

Question Asking Codes

1.

Exact wording or slight change

2.

Major change in question wording

3.

Verification

4.

Omission 2.

Response Codes

1.

Adequate answer

2.

Inadequate answer

3.

Interrupt

4.

Clarification

5.

Other respondent behavior 3.

Total codes = 9

Hunter & Landreth (2005) Coding Scheme

1.

Interviewer behaviors (n=6)

2.

Respondent behaviors (n=9)

3.

Total codes 15

Sykes and Collins (1992) Coding Scheme

Question delivery

Introductions 4 subcodes Questions 6 subcodes Explanations associated with question delivery 3 subcodes

Question answering

Respondent 19 subcodes Interviewer 22 subcodes

Total codes = 54

Johnson, Holbrook and Cho Coding Scheme

Goal of assessing difficulties with different aspects of question answering, rather than identifying problem questions per se. A total of 24 codes that attempt to decompose behaviors by stage of cognitive processing:

1.

Question interpretation

2.

Memory retrieval

3.

Judgment formation

4.

Response mapping

5.

Social desirability concerns

12!'

$"#"$"

$*

$

H H$

"

$*

SLIDE 9

2/9/2015 9

/!!12!'

"$ #"-" :"$-"):+

• $2!

;. 3. E7#)#+

/!"

$$#( "$&)"%I#+ $"

• "<"#

8!*$

/"##" ;) 99!+ &"*$*$

*-

• """

$$$

""

• $$#"
• $"

"

$"" $

"*$"-*

• #

"##)+

"

9!*' $"

$< $""#*

$"$<

6

25"$#

;>$#

"" ($#2(5(2

$5

$1 $

#$"4

&"#$

)#*"*+

SLIDE 10

2/9/2015 10

$!$"

00# "#

B(0*-"*"4

.*!"

• "#

)$*"+

%

,"" B"## )$+4

J

&"* #)+$"# #4

!

3#*#"#"" )"+

K

E""." 7*#$*( *

L

C*# .$"-$$# ""# $"$

*0:''

" E#0 ($$)

$4+

H-4

E)

+

""$$)$*

**+

:(#M H4