Transparency and Replication in Cross-National Survey Research: - PowerPoint PPT Presentation

Transparency and Replication in Cross-National Survey Research: Quality of Reporting Elena Damian, Bart Meuleman & Wim van Oorschot Centre for Sociological Research KU Leuven Centre for Sociological Research

Transparency and replication in cross-national survey research Two main functions of research transparency:  Evaluate the validity and reliability of study’s findings ( Evaluation transparency )  Enable direct replications by independent researchers ( Replicability transparency ) 2 Centre for Sociological Research

Transparency and replication in cross-national survey research Particular transparency issues 1. Data collection is complex and comes with extensive metadata 2. Secondary analysis is often a complex process 3. High risk of data dependent analysis (Gelman & Loken, 2014) 3 Centre for Sociological Research

Transparency and replication in cross-national survey research General aim Open the debate about transparency in cross-national survey research Contributions 1. Model of responsibility for achieving research transparency 2. Empirical evidence regarding the current reporting practices in cross- national survey research 3. Checklist for reporting cross-national survey research 4 Centre for Sociological Research

Current reporting practices in cross-national social studies Data Random sample of 305 comparative studies (1986 – 2016) • • 29 peer-reviewed journals (Sociology, Political Science, Cross-Cultural Psychology) Article selection criteria: (1) main data from Afrobarometer, Eurobarometer, • European Social Survey, European Values Study, International Social Survey Program, World Values Survey, and/or Latinobarometer; (2) five or more countries; (3) any type of comparative analysis ; (4) do not have a purely methodological aim 5 Centre for Sociological Research

Current reporting practices in cross-national social studies Descriptive statistics (1) survey data or contextual data form external sources (2) data preparation procedures (3) replication materials 6 Centre for Sociological Research

Results Table 1: Percentage of articles disclosing information about data and sampling (N=305) 7 Centre for Sociological Research

Results Table 2: Percentage of articles disclosing information about variables 8 Centre for Sociological Research

Results Graph 1 & 2. Percentage of academic articles by evaluation and replicability transparency index (N=305) Note: The values of the two indices range from 0 to 1, with a high value representing higher transparency 9 Centre for Sociological Research

Table 3. Checklist for achieving higher transparency in cross-national social research 10 Centre for Sociological Research

Concluding remarks 11 Centre for Sociological Research

Additional slides 12 Centre for Sociological Research

Figure 2. Identification process for article selection Article selection criteria: • uses as main data Afrobarometer, Eurobarometer, European Social Survey, European Values Study, International Social Survey Program; World Values Survey, and/or Latinobarometer • analyses five or more countries • any type of comparative analysis • does not have a purely methodological aim 13 Centre for Sociological Research

List of journals • Acta Politica Journal of Cross-Cultural Psychology • • Acta Sociologica Journal of European Public Policy • American Journal of Political Science • • Journal of European Social Policy American Journal of Sociology • • PLoS ONE American Sociological Review • Political Studies • • Comparative European Politics Politics and Policy • • Comparative Political Studies Public Opinion Quarterly • Cross-Cultural Research • • Social Forces European Journal of Political Research • Social Indicators Research • European Political Science, European • Social Problems • Societies Social Science Research • European Sociological Review • The Journal of Politics • International Journal of Comparative • • West European Politics Sociology International Journal of Public Opinion • Research International Journal of Social Welfare • International Political Science Review • 14 Centre for Sociological Research