Contact Strategies in F2F and Telephone Surveys Steve Schwarzer, - - PowerPoint PPT Presentation

▶
contact strategies in f2f and telephone
SMART_READER_LITE
LIVE PREVIEW

Contact Strategies in F2F and Telephone Surveys Steve Schwarzer, - - PowerPoint PPT Presentation

Apr 04, 2024 384 likes •632 views

Worth the effort? An Examination of Re- Contact Strategies in F2F and Telephone Surveys Steve Schwarzer, Patrick Moynihan, Martha McRoy CSDI Workshop, Limerick, Ireland | March 27, 2018 Research Area Declining respons onse e rates

slide-1
SLIDE 1

Worth the effort? An Examination of Re- Contact Strategies in F2F and Telephone Surveys

Steve Schwarzer, Patrick Moynihan, Martha McRoy CSDI Workshop, Limerick, Ireland | March 27, 2018

slide-2
SLIDE 2

April 18, 2018

2 www.pewresearch.org

Research Area

  • Declining respons
  • nse

e rates general ally ly seen as an indicat ation for less qu quali lity, ,

  • A lot of effort goes into maintaining response rates
  • Non-re

respon sponse e is seen as one cause of survey y error and bias

  • Non-re

respon sponse e bias as the difference nce bet etween estimates s for survey y respon pondent ents s and overal all l popu pulat latio ion

  • Increases of there is a relationship between study subject and likelihood of responding
  • Inverse of response rate

Ho How do survey estima mates es change ge with differen erent t level els s of fieldw dwor

  • rk

k effor

  • rt

t (re relati tive and absol solut ute e non-response esponse bias)? )?

slide-3
SLIDE 3

April 18, 2018

3 www.pewresearch.org

Literature

  • Keet

eter er at all ll 2000, 2006 – reduci ucing ng non-resp respons

  • nse

e in telep elepho hone ne surveys ys

  • Groves

es and Peytche tcheva 2008 2008 – meta eta analysis alysis of studies es of absol

  • lut

ute e NR bias as

  • Very low correlation between response rate and non response bias
  • Great variation within studies then across studies
  • Sturg

urgis s et et all l 2017 – meta eta analysi alysis focus cusing ng on relat ative e NR bias as and d FW effor

  • rt
  • On average, questions about beliefs and attitudes tended to respond most to FW effort,

behavioral questions not so much

  • Small number of variables with high relative bias
  • 4 call design has the most appeal
  • Sztabi

abins nski ki 2017, Fuchs hs et et all l 2013 13 – effor

  • rt and data

a qu quali ality ty for ESS S data a

slide-4
SLIDE 4

April 18, 2018

4 www.pewresearch.org

Data Used

  • 2017 Global Attitudes Spring surveys: Nationally representative telephone and face-

to-face interviews with adults 18 years and older

  • Scope would be 40 countries, but
  • 10 telephone countries are included (all)
  • 5 f2f countries included

BUT

  • 15 f2f countries excluded due to response rates higher than 70% on first visit
  • 10 countries excluded because contact data unusable
slide-5
SLIDE 5

Fieldwork Efforts all F2F Countries

April 18, 2018

www.pewresearch.org 5 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 4 3 2 1 Source: Global Attitudes survey, 2017.

slide-6
SLIDE 6

Response Categories F2F Countries

April 18, 2018

www.pewresearch.org 6 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Not eligibile Non-Contact Refusal Interview Source: Global Attitudes survey, 2017.

slide-7
SLIDE 7

CATI Countries

April 18, 2018

www.pewresearch.org 7 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 7.00 6.00 5.00 4.00 3.00 2.00 1.00 Source: Global Attitudes survey, 2017.

slide-8
SLIDE 8

CATI Countries

April 18, 2018

www.pewresearch.org 8 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Not eligibile Non-Contact Refusal Interview Source: Global Attitudes survey, 2017.

slide-9
SLIDE 9

Percentage of Interviews Completed by Fieldwork Effort

Telephone (10 countries)

Callb llbac acks Number of complet leted intervie iews ws Number of comple leted intervie iews ws 1 34% 34% 2 22% 36% 3 14% 4 10% 17% 5 7% 6 4% 7% 7 3%

In-person (5 countries)

Callb llbac acks Number of comple leted intervie iews ws Number of comple leted intervie iews ws 1 56% 56% 2 29% 29% 3 14% 16% 4 2%

April 18, 2018

www.pewresearch.org 9 Source: Global Attitudes survey, 2017.

slide-10
SLIDE 10

April 18, 2018

10 www.pewresearch.org

Cumulative Interview Rates – Telephone Countries

0.00 100.00 200.00 300.00 400.00 500.00 600.00 700.00 800.00 900.00 1000.00 1100.00 1200.00 1300.00 1400.00 1500.00 1 2 3 4 5 6 7 Australia Canada France Germany Great Britain Japan Netherlands Korea Spain USA Sweden Source: Global Attitudes survey, 2017.

slide-11
SLIDE 11

April 18, 2018

11 www.pewresearch.org

Cumulative Interview Rates – F2F Countries

Source: Global Attitudes survey, 2017. .00 200.00 400.00 600.00 800.00 1000.00 1200.00 1 2 3 4 Israel Italy Poland Hungary Greece

slide-12
SLIDE 12

April 18, 2018

12 www.pewresearch.org

Response Rates by Attempt – CATI Countries

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 1 2 3 4 5 6 7 Australia Canada France Germany Great Britain Japan Netherlands Korea Spain USA Sweden Source: Global Attitudes survey, 2017.

slide-13
SLIDE 13

April 18, 2018

13 www.pewresearch.org

Response Rates by Attempt – F2F Countries

.00 .10 .20 .30 .40 .50 .60 .70 .80 1 2 3 4 Israel Italy Poland Hungary Greece Source: Global Attitudes survey, 2017.

slide-14
SLIDE 14

April 18, 2018

14 www.pewresearch.org

Evaluating Variable Bias

Absolute bias (distance from true value – population estimate) measured after different FW effort levels measured for the following variables

  • Sex of respondent
  • Binary age variable (up to 49 years; 50 and more years of age)
  • Binary education variable (higher secondary education or less; above secondary education)

Relative bias (distance from final estimate) measured after different FW effort levels

  • 4 attitudinal questions - perception and respect of US, favorability Trump & Obama
  • 4 technology related questions - use of internet, own cell, own smartphone, social networking

Source: Global Attitudes survey, 2017.

slide-15
SLIDE 15

Differences by Attempt Structure – Phone (10 countries)

Demographics

Sex Sex Age (50+) Edu (4+)

Easy to get (1) 0 (-6;8) 1 (-4;6)

  • 2 (-7;2)

Medium effort (2-3) 0 (-3;4) 2 (-5;8) 1 (-3;9) More effort (4-7) 0 (-11;7)

  • 4 (-14;5)

2 (-5;8) Easy to get (1) vs. Medi dium effort (2;3) 0 (-9;6) 1 (-7;8) 4 (-5;13) Easy to get (1) vs. More effort (4 or more)

  • 1 (-19;12)
  • 5 (-19;6)

4 (0;12)

Substantive questions

A1 A1 A2 A2 A3 A3 A4 A4 T1 T1 T2 T2 T3 T3 T4 T4

  • 1 (-7;3)

0 (-1;4)

  • 1 (-3;1)

1 (-2;5)

  • 1 (-4;3)

0 (-4;4)

  • 2 (-7;1)

0 (-5;14) 1 (-4;7)

  • 1 (-12;5)

1 (-1;3) 0 (-4;3)

  • 1 (-9;3)
  • 1 (-5;2)

1 (-4;7)

  • 1 (-8;5)

0 (-6;4) 0 (-6;4) 0 (-2;5)

  • 1 (-7;5)

2 (-2;7) 2 (-2;7) 1 (-4;8) 0 (-5;6) 1 (-5;14) 2 (-14;6) 2 (-2;5)

  • 2 (-5;5)

0 (-10;7)

  • 1 (-6;5)

2 (-4;10)

  • 1 (-22;10)

1 (-8;7) 0 (-8;6) 1 (-4;7)

  • 2 (-11;5)

2 (-3;7) 2 (-5;9) 3 (-1;14) 0 (-20;10)

April 18, 2018

www.pewresearch.org 17 Source: Global Attitudes survey, 2017.

slide-16
SLIDE 16

Differences by Attempt Structure – F2F (5 countries)

Demographics

Sex Sex Age (50+) Edu (4+)

Easy to get (1) 1 (-2;4)

  • 1 (-6;3)

0 (-1;1) Medium effort (2)

  • 3 (-6-;-1)

2 (-4;4) 1 (-4;4) More effort (3-4)

  • 5 (-51;17)
  • 3 (-32;22)
  • 1 (-15;14)

Easy to get (1) vs. Medium effort (2)

  • 4 (-8;1)

4 (-7;14) 1 (-5;4) Easy to get (1) vs. More effort (3 or more)

  • 4 (-51;19)
  • 9 (-32;11)
  • 2 (-11;4)

Substantive questions

A1 A1 A2 A2 A3 A3 A4 A4 T1 T1 T2 T2 T3 T3 T4 T4

  • 3 (-6;-1)
  • 1 (-3;1)

0 (-2;2)

  • 1 (-4;4)
  • 1 (-1;3)
  • 1 (-4;2)

0 (-3;4)

  • 1 (-4;1)

3 (0;6) 1 (-4;9) 0 (-6;4) 4 (0;11) 0 (-6;4) 0 (-4;3) 1 (-6;4) 0 (-3;3)

  • 4 (-23;11)

6 (-7;22)

  • 4 (-58;21)
  • 3 (-9;1)
  • 3 (-32;12)
  • 3 (-26;5)
  • 9 (-16;-4)

11 (6;16) 6 (4;12) 1 (-4;12) 1 (-8;8) 6 (-1;14) 0 (9;7)

  • 1 (-5;5)

0 (-11;8) 5 (-2;15) 2 (-18;13) 4 (-7;13)

  • 9 (-57;11)
  • 3 (-15;11)
  • 3 (-32;11)
  • 3 (-26;7)
  • 4 (-17;9)

9 (-2;20)

April 18, 2018

www.pewresearch.org 18 Source: Global Attitudes survey, 2017.

slide-17
SLIDE 17

Demographics by Fieldwork Attempt (CATI)

Low effort Medium effort

April 18, 2018

www.pewresearch.org 20

High effort

  • 0.2
  • 0.15
  • 0.1
  • 0.05

0.05 0.1 0.15 sex_par age_par Q001edu

  • .15
  • .10
  • .05

.00 .05 .10 .15 sex_par age_par Q001edu

  • .15
  • .10
  • .05

.00 .05 .10 .15 sex_par age_par Q001edu

slide-18
SLIDE 18

Substantive Differences by Fieldwork Effort (CATI)

Low effort Medium effort

April 18, 2018

www.pewresearch.org 21

  • .15
  • .10
  • .05

.00 .05 .10 .15

A1 A2 A3 A4 T1 T2 T3 T4

  • .15
  • .10
  • .05

.00 .05 .10 .15

A1 A2 A3 A4 T1 T2 T3 T4

Source: Global Attitudes survey, 2017.

slide-19
SLIDE 19

Demographics by Fieldwork Effort (F2F)

Low effort Medium effort

April 18, 2018

www.pewresearch.org 23

  • 0.15
  • 0.1
  • 0.05

0.05 0.1 Greece Hungary Israel Italy Poland sex_par age_par Q001edu

  • 0.15
  • 0.1
  • 0.05

0.05 0.1 Greece Hungary Israel Italy Poland sex_par age_par Q001edu

  • 0.15
  • 0.1
  • 0.05

0.05 0.1 Greece Hungary Israel Italy Poland sex_par age_par Q001edu

High effort

slide-20
SLIDE 20

Substantive Differences by Fieldwork Effort (F2F)

Low effort Medium effort

April 18, 2018

www.pewresearch.org 24 Source: Global Attitudes survey, 2017.

  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04 0.06 0.08 0.1 0.12 Greece Hungary Israel Italy Poland A1 A2 A3 A4 T1 T2 T3 T4

  • 0.08
  • 0.06
  • 0.04
  • 0.02

0.02 0.04 0.06 0.08 0.1 0.12 Greece Hungary Israel Italy Poland A1 A2 A3 A4 T1 T2 T3 T4

slide-21
SLIDE 21

April 18, 2018

www.pewresearch.org 25 0.2 0.4 0.6 0.8 1 1.2 France Korea Canada Spain Japan Hungary Sweden Netherlands Greece Australia Great Britain USA Poland Israel Germany

Mediu ium effor

  • rt

0.2 0.4 0.6 0.8 1 1.2 Germany Greece Japan Australia Great Britain USA Spain Hungary Netherlands France Israel Sweden Canada Korea Poland

Most st effor

  • rt

0.0 0.2 0.4 0.6 0.8 1.0 1.2 Germany Israel Poland Sweden USA Korea Netherlands Hungary Great Britain Greece Australia Japan Canada Spain France

Low effor

  • rt

Response Patterns Differ Across Countries (CATI)

Source: Global Attitudes survey, 2017.

slide-22
SLIDE 22

April 18, 2018

26 www.pewresearch.org

Response Patterns Differ Across Countries (F2F)

0.2 0.4 0.6 0.8 1 1.2 Israel Poland Hungary Greece Italy

Low effor

  • rt

0.0 0.2 0.4 0.6 0.8 1.0 1.2 Italy Hungary Greece Poland Israel

Mediu ium effor

  • rt

0.0 0.2 0.4 0.6 0.8 1.0 1.2 Greece Italy Hungary Israel Poland

Most st effor

  • rt

Source: Global Attitudes survey, 2017.

slide-23
SLIDE 23

April 18, 2018

28 www.pewresearch.org

Conclusion and Limitations

  • Fiel

eldwor

  • rk

k effor

  • rt

t matt tter ers, which ch is a refl flect ection

  • n of the desi

sign gn

  • But

t some me effec ects ts seem em to cancel ncel each ch ot

  • ther

her out

in CATI countries age and education it is a country specific story for F2F

  • Substantiv

bstantive e resul sults ts shift t when n the sampl ple e compos position tion by fiel eldwork k effor

  • rt

t changes nges

  • Limitat

mitations

  • ns

F2F designs with a 1+2 design provide limited insights, but one can still find differences Surveys in developing countries still show very high response rates generally, and 1st contact success rates that make the effort assessment difficult, if not impossible