Reporting Standards for Social Science Experiments Kevin Esterling - - PowerPoint PPT Presentation

Reporting Standards for Social Science Experiments

Kevin Esterling University of California - Riverside

Summer Institute June 2014

Introduction

• Reporting and disclosure are essential for contributing to

the accumulation of knowledge

– Minimize researcher degrees of freedom – Enables others to assess the plausibility of your identifying assumptions, validity, generalizability, etc.

• Disclosure is most important when you generate your
• wn data

– Today’s focus will be on social science RCTs – For surveys, see AAPOR standards – For archival data, do your best!

BITSS Summer Institute 2 June 2014

Costs and Benefits of Disclosure

• High costs

– RCTs are “inconsistent with the human spirit”

• Data collection is often messy
• Studies do not go as planned

– Time consuming to keep track of this in real time

• Benefits of disclosure:

– Ethical norms; your own identity as an ethical person – Accumulation of knowledge; you are a member of society – Disclosure provides checklist of crucial design and analysis elements that you need to think about anyway – (*) Costly signal about the quality and merits of your research; your reputation and the venues for publication

BITSS Summer Institute 3 June 2014

Illustration

BITSS Summer Institute 4 June 2014

Reporting Standards

• Existing standards

– CONSORT for biomedical research – CONSORT-SPI in development – Many others… (including AAPOR for certain survey experiments)

• Social science: Organized Section on Experimental Research of

APSA (XPS)

– Minimum reporting standards – Developed by the Experimental Standards Committee – Adopted by the Journal of Experimental Political Science

• XPS standards are a checklist with six sections

– Hypotheses – Subjects and Context – Allocation methods – Treatments – Results – Other information

BITSS Summer Institute 5 June 2014

Hypotheses

• Specific objectives or hypotheses

– State the questions the experiment designed to address – What are the specific hypotheses to be tested?

• Be sure to delineate which hypotheses and subgroup analyses were

developed in advance of the data analysis

• Also, note primary and secondary outcome measures

BITSS Summer Institute 6 June 2014

Subjects

• Eligibility and exclusion criteria for participants

– Why was this subject pool selected? – Who was eligible to participate in the study?

• Procedures used to recruit and select participants

– Recruitment dates defining the periods of recruitment – What would result in the exclusion of a participant? – Were any aspects of recruitment changed (such as the exclusion criteria) after recruitment began?

• Sample size

– Intended number of participants per cell or plan to stop recruitment – Best practice to conduct power analysis

BITSS Summer Institute 7 June 2014

Allocation Method

• Procedures used to generate the assignment sequence

(e.g., randomization)

– Details of procedure (e.g., any restrictions, blocking) – Unit of randomization (individuals, groups, households, etc). – Provide evidence that assignment was successfully implemented, such as balance scores on pretreatment variables

• Blinding

– Were participants, those administering the interventions, and those assessing the outcomes unaware of condition assignments? – If blinding took place, include a statement regarding how it was accomplished and how the success of blinding was evaluated

BITSS Summer Institute 8 June 2014

Context

• Settings and locations where the data were collected

– Field – Lab – Classroom – Online

• Other relevant specifics of the population

– Large public university vs. small private university – Geographic location – Social networks or proximity of subjects

• Timeframes

– When the experiments were conducted – Dates of any repeated measurements as part of a follow-up

BITSS Summer Institute 9 June 2014

Treatments

• Description of the interventions and their timing in each

treatment condition

– Method of delivery (paper, computer, face-to-face, telephone) – Was deception used? Were incentives given? – Manipulation checks; other evidence on whether the treatment was delivered as intended – Report any instructional anomalies or problems in administration

• For lab experiments (and other experiments, when relevant):
• Report the number of repetitions, group rotation, ordering of treatments,

piggybacking of other protocols

• How long did each experiment last? How many sessions were subjects expected to

attend? Amount of time between sessions

• Were subjects given quizzes on the experimental instructions?
• Were there practice rounds? If so, how many and what were the results?
• Did subjects complete a post-experiment debriefing, interview, or questionnaire?

If so, is there evidence that subjects understood the instructions and treatments?

• Descriptions should be sufficient to allow replication: verbatim

treatment materials in appendix

BITSS Summer Institute 10 June 2014

Results: Non-Compliance and Attrition

• Complete CONSORT Participant Flow Diagram (if non-

trivial non-compliance or attrition)…

BITSS Summer Institute 11 June 2014

Consort 2010 Flow Diagram

BITSS Summer Institute 12 June 2014

Dropped from Sample Random Assignment Provide Background Information Deliberative Session Information Only Screen: Willing to Participate in Session? Constituent+Member Session Post Treatment Survey Post Midterm Election Survey Post Treatment Survey Post Treatment Survey Post Midterm Election Survey Post Midterm Election Survey Provide Background Information Initial Survey Constituent Only Session Yes No True Control Random Assignment .33 .25 .33 .33 .25 .5 Figure 1. Project Overview

Here was our design going into the field:

Here was the actual compliance among subjects:

Here was the actual compliance among subjects:

Results: Treatment Effects

• Summarize Outcome Measures and Covariates

– For indices, provide exact description of how they are formed – Clearly state which of the outcomes and subgroup analyses were specified prior to the experiment and which were the result of exploratory analysis

• Statistical Analysis

– Report ITT and local effect estimates (reporting or weighting by blocks if appropriate) + identification strategy – Discuss reasons for noncompliance and attrition and examine if related to pretreatment variables – Report missing data by group and methods for addressing missing data – Note if level of analysis differs from level of randomization and estimate appropriate standard errors

BITSS Summer Institute 16 June 2014

Other Information

• Was the experiment reviewed and approved by an IRB?
• If the experimental protocol was registered, where and

how can the filing be accessed?

• What was the source of funding? What was the role of the

funders in the analysis of the experiment?

– Were there any restrictions or arrangements regarding what findings could be published? – Any funding sources where conflict of interest might reasonably be an issue?

• If a replication data set is available, provide the URL

BITSS Summer Institute 17 June 2014

From the Dictionary of Useful Research Phrases

• “Three of the samples were chosen for detailed study….”

– The results of the others didn’t make sense

• “Typical results are shown….”

– The best results are shown

• “A careful analysis of obtainable data….”

– Three pages of notes were obliterated when I knocked over a glass of beer

• “While it has not been possible to provide definite

answers to these questions….”

– An unsuccessful experiment, but I still hope to get it published

BITSS Summer Institute 18 June 2014