Transparency and Replication in Cross-National Survey Research: - - PowerPoint PPT Presentation

Transparency and Replication in Cross-National Survey Research: Quality of Reporting

Elena Damian, Bart Meuleman & Wim van Oorschot Centre for Sociological Research KU Leuven

Centre for Sociological Research

Transparency and replication in cross-national survey research

Centre for Sociological Research 2

Two main functions of research transparency:

• Evaluate the validity and reliability of study’s findings (Evaluation

transparency)

• Enable direct replications by independent researchers (Replicability

transparency)

Transparency and replication in cross-national survey research

Centre for Sociological Research 3

Particular transparency issues

• 1. Data collection is complex and comes with extensive metadata
• 2. Secondary analysis is often a complex process
• 3. High risk of data dependent analysis (Gelman & Loken, 2014)

Transparency and replication in cross-national survey research

Centre for Sociological Research 4

General aim

Open the debate about transparency in cross-national survey research

Contributions

• 1. Model of responsibility for achieving research transparency
• 2. Empirical evidence regarding the current reporting practices in cross-

national survey research

• 3. Checklist for reporting cross-national survey research

Current reporting practices in cross-national social studies

Centre for Sociological Research 5

Data

• Random sample of 305 comparative studies (1986 – 2016)
• 29 peer-reviewed journals (Sociology, Political Science, Cross-Cultural

Psychology)

• Article selection criteria: (1) main data from Afrobarometer, Eurobarometer,

European Social Survey, European Values Study, International Social Survey Program, World Values Survey, and/or Latinobarometer; (2) five or more countries; (3) any type of comparative analysis ; (4) do not have a purely methodological aim

Current reporting practices in cross-national social studies

Centre for Sociological Research 6

Descriptive statistics (1) survey data or contextual data form external sources (2) data preparation procedures (3) replication materials

Table 1: Percentage of articles disclosing information about data and sampling (N=305)

Centre for Sociological Research 7

Results

Table 2: Percentage of articles disclosing information about variables

Centre for Sociological Research 8

Results

Graph 1 & 2. Percentage of academic articles by evaluation and replicability transparency index (N=305)

Centre for Sociological Research 9

Results

Note: The values of the two indices range from 0 to 1, with a high value representing higher transparency

Centre for Sociological Research 10

Table 3. Checklist for achieving higher transparency in cross-national social research

Centre for Sociological Research 11

Concluding remarks

Centre for Sociological Research 12

Additional slides

Figure 2. Identification process for article selection

Centre for Sociological Research 13

Article selection criteria:

• uses as main data Afrobarometer,

Eurobarometer, European Social Survey, European Values Study, International Social Survey Program; World Values Survey, and/or Latinobarometer

• analyses five or more countries
• any type of comparative analysis
• does not have a purely

methodological aim

Centre for Sociological Research 14 List of journals

• Acta Politica
• Acta Sociologica
• American Journal of Political Science
• American Journal of Sociology
• American Sociological Review
• Comparative European Politics
• Comparative Political Studies
• Cross-Cultural Research
• European Journal of Political Research
• European Political Science, European

Societies

• European Sociological Review
• International Journal of Comparative

Sociology

• International Journal of Public Opinion

Research

• International Journal of Social Welfare
• International Political Science Review
• Journal of Cross-Cultural Psychology
• Journal of European Public Policy
• Journal of European Social Policy
• PLoS ONE
• Political Studies
• Politics and Policy
• Public Opinion Quarterly
• Social Forces
• Social Indicators Research
• Social Problems
• Social Science Research
• The Journal of Politics
• West European Politics