Publication bias in QCA Publication bias in QCA Publication bias in - - PowerPoint PPT Presentation

▶
publication bias in qca publication bias in qca
SMART_READER_LITE
LIVE PREVIEW

Publication bias in QCA Publication bias in QCA Publication bias in - - PowerPoint PPT Presentation

Mar 04, 2023 150 likes •542 views

24.1.2019 Publication bias in QCA Publication bias in QCA Publication bias in QCA Publication bias in QCA Meaning, diagnosis and remedies Meaning, diagnosis and remedies Meaning, diagnosis and remedies Ingo Rohlng Ingo Rohlng Ingo

slide-1
SLIDE 1

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 1/39

Publication bias in QCA Publication bias in QCA Publication bias in QCA

Meaning, diagnosis and remedies Meaning, diagnosis and remedies Meaning, diagnosis and remedies

Ingo Rohlng Ingo Rohlng Ingo Rohlng 2019/01/25 2019/01/25 2019/01/25

1 / 18 1 / 18 1 / 18

slide-2
SLIDE 2

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 2/39

A short intro to QCA

2 / 18

slide-3
SLIDE 3

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 3/39

Terminology and concepts

Independent variable = condition (set) 3 / 18

slide-4
SLIDE 4

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 4/39

Terminology and concepts

Independent variable = condition (set) Dependent variable = outcome (set) 3 / 18

slide-5
SLIDE 5

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 5/39

Terminology and concepts

Independent variable = condition (set) Dependent variable = outcome (set) Interaction = configuration/conjunction 3 / 18

slide-6
SLIDE 6

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 6/39

Terminology and concepts

Independent variable = condition (set) Dependent variable = outcome (set) Interaction = configuration/conjunction Search for set relations Sufficiency: If A, then Y 3 / 18

slide-7
SLIDE 7

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 7/39

Terminology and concepts

Independent variable = condition (set) Dependent variable = outcome (set) Interaction = configuration/conjunction Search for set relations Sufficiency: If A, then Y Necessity: Only if A, then Y 3 / 18

slide-8
SLIDE 8

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 8/39

Terminology and concepts

Independent variable = condition (set) Dependent variable = outcome (set) Interaction = configuration/conjunction Search for set relations Sufficiency: If A, then Y Necessity: Only if A, then Y Types of sets Crisp sets: 0 or 1 Fuzzy sets: Between 0 and 1 (including) 3 / 18

slide-9
SLIDE 9

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 9/39

From variables to sets: Calibration

Uncalibrated variable 4 / 18

slide-10
SLIDE 10

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 10/39

From variables to sets: Calibration

Calibrated variable 5 / 18

slide-11
SLIDE 11

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 11/39

Does the data support a set-relational claim?

Consistency: Is data consistent with an inference of sufficiency? 6 / 18

slide-12
SLIDE 12

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 12/39

Does the data support a set-relational claim?

Consistency: Is data consistent with an inference of sufficiency? Consistency of 0.8 A 0 1 Y 4 2 1 6 8 6 / 18

slide-13
SLIDE 13

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 13/39

Does the data support a set-relational claim?

Consistency: Is data consistent with an inference of sufficiency? Consistency of 0.8 A 0 1 Y 4 2 1 6 8 Consistency of 0.5 A 0 1 Y 4 5 1 6 5 6 / 18

slide-14
SLIDE 14

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 14/39

Does the data support a set-relational claim?

7 / 18

slide-15
SLIDE 15

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 15/39

Distribution of solution consistency

Results for 359 solutions from 170 articles (2009-2017) 8 / 18

slide-16
SLIDE 16

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 16/39

Distribution of solution consistency

Results for 359 solutions from 170 articles (2009-2017) 8 / 18

slide-17
SLIDE 17

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 17/39

Caliper test for solution consistency

9 / 18

slide-18
SLIDE 18

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 18/39

Caliper test for solution consistency

Results for 359 solutions from 170 articles (2009-2017) cutoff range lower upper above below pvalue 0.75 0.05 0.725 0.775 13 3 1.063538e-02 0.75 0.10 0.700 0.800 40 5 3.939192e-08 0.75 0.20 0.650 0.850 93 10 2.599305e-18 9 / 18

slide-19
SLIDE 19

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 19/39

Caliper test for solution consistency

Results for 359 solutions from 170 articles (2009-2017) cutoff range lower upper above below pvalue 0.75 0.05 0.725 0.775 13 3 1.063538e-02 0.75 0.10 0.700 0.800 40 5 3.939192e-08 0.75 0.20 0.650 0.850 93 10 2.599305e-18 Results for 300 solutions from 50 dissertations (2009-2017) cutoff range lower upper above below pvalue 0.75 0.05 0.725 0.775 15 8 0.1050198078 0.75 0.10 0.700 0.800 33 20 0.0491853242 0.75 0.20 0.650 0.850 67 31 0.0001772669 9 / 18

slide-20
SLIDE 20

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 20/39

Should we bother about positive results?

10 / 18

slide-21
SLIDE 21

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 21/39

Should we bother about positive results?

Exploration/induction vs confirmation/deduction in 170 QCA articles 10 / 18

slide-22
SLIDE 22

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 22/39

Should we bother about positive results?

Exploration/induction vs confirmation/deduction in 170 QCA articles 10 / 18

slide-23
SLIDE 23

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 23/39

Researchers' degrees of freedom in QCA

11 / 18

slide-24
SLIDE 24

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 24/39

Researchers' degrees of freedom in QCA

  • 1. Hypotheses
  • 2. Choice of conditions and outcome

11 / 18

slide-25
SLIDE 25

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 25/39

Researchers' degrees of freedom in QCA

  • 1. Hypotheses
  • 2. Choice of conditions and outcome
  • 3. Delineation of sample/population

11 / 18

slide-26
SLIDE 26

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 26/39

Researchers' degrees of freedom in QCA

  • 1. Hypotheses
  • 2. Choice of conditions and outcome
  • 3. Delineation of sample/population
  • 4. Calibration (anchors, link function)

11 / 18

slide-27
SLIDE 27

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 27/39

Researchers' degrees of freedom in QCA

  • 1. Hypotheses
  • 2. Choice of conditions and outcome
  • 3. Delineation of sample/population
  • 4. Calibration (anchors, link function)
  • 5. Truth table parameters (cutoffs, remainder handling)
  • 6. Solution parameters (type, remainder handling)

11 / 18

slide-28
SLIDE 28

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 28/39

Degrees of freedom: Sample and optional stopping

12 / 18

slide-29
SLIDE 29

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 29/39

Degrees of freedom: Sample and optional stopping

12 / 18

slide-30
SLIDE 30

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 30/39

Degrees of freedom: Samples and populations in articles

13 / 18

slide-31
SLIDE 31

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 31/39

Degrees of freedom: Samples and populations in articles

Flexibility in delineating population (space, time, substance) 13 / 18

slide-32
SLIDE 32

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 32/39

Degrees of freedom: Samples and populations in articles

Flexibility in delineating population (space, time, substance) Sizeable number of QCA articles use apparent populations or samples 13 / 18

slide-33
SLIDE 33

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 33/39

Degrees of freedom: Calibration

14 / 18

slide-34
SLIDE 34

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 34/39

Degrees of freedom: Calibration

15 / 18

slide-35
SLIDE 35

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 35/39

Opportunity for preregistration

16 / 18

slide-36
SLIDE 36

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 36/39

Opportunity for results-blind review

17 / 18

slide-37
SLIDE 37

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 37/39

Opportunity for results-blind review

Information for judging quality of theory and design in results section? 17 / 18

slide-38
SLIDE 38

24.1.2019 Publication bias in QCA file:///C:/data/sync/qca/pubbias/text/pubbias_data/ossc_slides.html#1 38/39

Opportunity for results-blind review

Information for judging quality of theory and design in results section? Blinding in QCA No results section Blinded truth table and raw data (if N small) Mock truth table parameters 17 / 18