Measuring disagreement in science Dakota Murray, Wout Lamers, Kevin - - PowerPoint PPT Presentation

▶
measuring disagreement in science
SMART_READER_LITE
LIVE PREVIEW

Measuring disagreement in science Dakota Murray, Wout Lamers, Kevin - - PowerPoint PPT Presentation

Jan 09, 2023 110 likes •1.34k views

Slides at: https://... @dakotasmurray Measuring disagreement in science Dakota Murray, Wout Lamers, Kevin Boyack, Vincent Larivire, Cassidy R. Sugimoto, Nees Jan van Eck, Ludo Waltman SciTech Strategies Disagreement in science

slide-1
SLIDE 1

Measuring disagreement in science

Dakota Murray, Wout Lamers, Kevin Boyack, Vincent Larivière, Cassidy R. Sugimoto, Nees Jan van Eck, Ludo Waltman

SciTech Strategies

Slides at: https://... @dakotasmurray

slide-2
SLIDE 2

Disagreement in science

slide-3
SLIDE 3

Disagreement in science

  • Dialectic debate in early history
slide-4
SLIDE 4

Disagreement in science

  • Dialectic debate in early history
  • Consensus in Comte's hierarchy of

sciences

slide-5
SLIDE 5

Disagreement in science

  • Dialectic debate in early history
  • Consensus in Comte's hierarchy of

sciences

  • Popper’s falsification,
  • Kuhn’s anomalies
  • Latour’s controversies
  • more
slide-6
SLIDE 6

Disagreement is integral to science.

slide-7
SLIDE 7

Disagreement is integral to science...but where and when does it happen?

slide-8
SLIDE 8

Past work?

Disagreement

slide-9
SLIDE 9

Past work?

“uncertainty”

Chen, C., Song, M., & Heo, G. E. (2018). A scalable and adaptive method for finding semantically equivalent cue words

  • f uncertainty. JOI

Disagreement

slide-10
SLIDE 10

Past work?

“uncertainty”

Chen, C., Song, M., & Heo, G. E. (2018). A scalable and adaptive method for finding semantically equivalent cue words

  • f uncertainty. JOI

“Negative citations”

Catalini, C., Lacetera, N., & Oettl, A. (2015). The incidence and role of negative citations in science. PNAS

Disagreement

slide-11
SLIDE 11

Past work?

“uncertainty”

Chen, C., Song, M., & Heo, G. E. (2018). A scalable and adaptive method for finding semantically equivalent cue words

  • f uncertainty. JOI

“Negative citations”

Catalini, C., Lacetera, N., & Oettl, A. (2015). The incidence and role of negative citations in science. PNAS

But no holistic analysis of disagreement...

Disagreement

slide-12
SLIDE 12

Goal:

Develop a methodological framework to quantify the degree of disagreement (controversy, disconsensus, and dissonance...) in scientific literature

slide-13
SLIDE 13

Operationalizing disagreement

slide-14
SLIDE 14

Our focus is explicit, interpersonal disagreement

Author of cited paper Some cited paper Cites and explicitly disagrees with

slide-15
SLIDE 15

General disagreement: direct contradiction

“In contrast to past results [1], we find that coffee causes cancer”

Author of citing paper

slide-16
SLIDE 16

General disagreement: different model assumptions

“The model used by Kim et al [2] predicts coffee drinker’s life expectancy to be 80 years, whereas our model predicts 85 years

Author of citing paper

slide-17
SLIDE 17

Disagreement in the literature

X = y X != y X ~ y X = 2y

“There is disagreement in past studies over whether coffee does or does not causes cancer [3-10]”

Author of citing paper

slide-18
SLIDE 18

Approach

slide-19
SLIDE 19

Data

  • Our interest is interpersonal

disagreement, and so we focus on in-text citations

  • Over 3 million full text articles from

english articles published in ScienceDirect between 1998 to 2015

  • Citances extracted

Lorem ipsum dolor sit amet, consectetur adipiscing elit [1] Donec vel fermentum ligula [2-4] Nam sit amet lorem sit amet libero pharetra tristique quis non orc [5-9]

... ... ... ... ... ...

slide-20
SLIDE 20

How can we precisely and reliably identify a citation sentence as disagreement?

slide-21
SLIDE 21

A long, iterative process of identifying, validating, and refining queries

slide-22
SLIDE 22
  • --- Signal terms ----

Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

slide-23
SLIDE 23
  • --- Signal terms ----

“...recruiting participants was challenging...”

query retrieve sentences

Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

slide-24
SLIDE 24
  • --- Signal terms ----

“...recruiting participants was challenging...” X Invalid

query retrieve sentences

Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

slide-25
SLIDE 25

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----
slide-26
SLIDE 26

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----

studies; previous works; earlier works; literature, analyses; reports

slide-27
SLIDE 27

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----

“...recruiting participants was challenging...”

slide-28
SLIDE 28

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----

“...recruiting participants was challenging...” “However, recent studies have disagreed with this approach”

slide-29
SLIDE 29

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----

“However, recent studies have disagreed with this approach” “This theory have since been disproven” “...recruiting participants was challenging...”

slide-30
SLIDE 30

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----

“However, recent studies have disagreed with this approach” “This theory have since been disproven” “...recruiting participants was challenging...” “These ideas were influenced by Popper’s Conjectures and Refutations”

slide-31
SLIDE 31

_standalone_ +studies +ideas +methods +results Challenge* Conflict* Contradict* Contrary Contrast* Contravers* Debat* Differ* Disagree* Disprov* No consensus Questionable* Refut*

  • --- Signal terms ----
  • --- Filter terms ----

“However, recent studies have disagreed with this approach”

✓ Valid

“This theory have since been disproven” ✓ Valid “These ideas were influenced by Popper’s Conjectures and Refutations” X Invalid “...recruiting participants was challenging...” X Invalid

slide-32
SLIDE 32

_standalone_ +studies +ideas +methods +results Challenge*

? ? ? ? ?

Conflict*

? ? ? ? ?

Contradict*

? ? ? ? ?

Contrary

? ? ? ? ?

Contrast*

? ? ? ? ?

Contravers*

? ? ? ? ?

Debat*

? ? ? ? ?

Differ*

? ? ? ? ?

Disagree*

? ? ? ? ?

Disprov*

? ? ? ? ?

No consensus

? ? ? ? ?

Questionable*

? ? ? ? ?

Refut*

? ? ? ? ?

  • --- Signal terms ----
  • --- Filter terms ----

Which queries are the most valid?

slide-33
SLIDE 33

_standalone_ +studies +ideas +methods +results Challenge*

50 citances 50 citances 50 citances 50 citances 50 citances

Conflict*

50 citances 50 citances 50 citances 50 citances 50 citances

Contradict*

50 citances 50 citances 50 citances 50 citances 50 citances

Contrary

50 citances 50 citances 50 citances 50 citances 50 citances

Contrast*

50 citances 50 citances 50 citances 50 citances 50 citances

Contravers*

50 citances 50 citances 50 citances 50 citances 50 citances

Debat*

50 citances 50 citances 50 citances 50 citances 50 citances

Differ*

50 citances 50 citances 50 citances 50 citances 50 citances

Disagree*

50 citances 50 citances 50 citances 50 citances 50 citances

Disprov*

50 citances 50 citances 50 citances 50 citances 50 citances

No consensus

50 citances 50 citances 50 citances 50 citances 50 citances

Questionable*

50 citances 50 citances 50 citances 50 citances 50 citances

Refut*

50 citances 50 citances 50 citances 50 citances 50 citances

  • --- Signal terms ----
  • --- Filter terms ----

Which queries are the most valid? For each combination, query a random sample

  • f 50 citances

Two coders manually identify each citance as

✓ Valid or X Invalid

Calculate % agreement and % validity

slide-34
SLIDE 34

% agreement

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

slide-35
SLIDE 35

% agreement

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

slide-36
SLIDE 36

% agreement

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

Overall, 85% agreement This demonstrates that our coding system was robust

slide-37
SLIDE 37

% agreement

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

Overall, 85% agreement This demonstrates that our coding system was robust

slide-38
SLIDE 38

% Validity

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

slide-39
SLIDE 39

% Validity

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

slide-40
SLIDE 40

% Validity

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

Lots of heterogeneity

slide-41
SLIDE 41

% Validity

Coder 1: ✓ Valid Coder 2: ✓ Valid Coder 1: ✓ Valid Coder 2: X Invalid Coder 1: X Invalid Coder 2: ✓ Valid Coder 1: X Invalid Coder 2: X Invalid

Lots of heterogeneity

slide-42
SLIDE 42

Both

Ordered by Validity

slide-43
SLIDE 43

Both

  • % Agreement stable, %valid varied widely

Ordered by Validity

slide-44
SLIDE 44

Both

  • % Agreement stable, %valid varied widely
  • We don’t want obviously bad queries

Ordered by Validity

slide-45
SLIDE 45

Both

  • % Agreement stable, %valid varied widely
  • We don’t want obviously bad queries
  • Set a validity threshold—80%

Ordered by Validity

slide-46
SLIDE 46

Both

  • % Agreement stable, %valid varied widely
  • We don’t want obviously bad queries
  • Set a validity threshold—80%
  • 23 queries accounting for ~450,000 citation

sentences.

slide-47
SLIDE 47

Both

Disagreement

  • % Agreement stable, %valid varied widely
  • We don’t want obviously bad queries
  • Set a validity threshold—80%
  • 23 queries accounting for ~450,000 citation
  • sentences. Non-exhaustive, but precise!
slide-48
SLIDE 48

Both

  • % Agreement stable, %valid varied widely
  • We don’t want obviously bad queries
  • Set a validity threshold—80%
  • 23 queries accounting for ~450,000 citation
  • sentences. Non-exhaustive, but precise!

Disagreement

slide-49
SLIDE 49

Assess disagreement based on...

  • Discipline
  • Over time
  • Age of cited paper
  • Self Citation

Disagreement

slide-50
SLIDE 50

Measuring disagreement by discipline

slide-51
SLIDE 51

Disagreement by field

Auguste Comte’s Hierarchy of sciences

slide-52
SLIDE 52

Disagreement by field

Auguste Comte’s Hierarchy of sciences

More complex, Less consensus Less complex More consensus

slide-53
SLIDE 53

Disagreement by field

Auguste Comte’s Hierarchy of sciences

More complex, Less consensus Less complex More consensus

slide-54
SLIDE 54

Disagreement by field

Auguste Comte’s Hierarchy of sciences

More complex, Less consensus Less complex More consensus

Too coarse-grained?

slide-55
SLIDE 55

Digging deeper

Meso-level fields (clusters of papers)

slide-56
SLIDE 56

Digging deeper

Meso-level fields (clusters of papers) Area maps to total citation sentences Distance reflects relatedness

slide-57
SLIDE 57

Digging deeper

Meso-level fields (clusters of papers) Area maps to total citation sentences Distance reflects relatedness Color indicates ratio of % disagreement to the average

slide-58
SLIDE 58

Digging deeper

Meso-level fields (clusters of papers) Area maps to total citation sentences Distance reflects relatedness Color indicates ratio of % disagreement to the average “Horseshoe” shape moves from social sciences to Mathematics and computer science Soc & Hum Bio & Health Life & Earth Phys & Engr Math & Comp

slide-59
SLIDE 59

Hierarchy repeated

Soc & Hum Bio & Health Life & Earth Phys & Engr Math & Comp

slide-60
SLIDE 60

Hierarchy repeated...

Soc & Hum Bio & Health Life & Earth Phys & Engr Math & Comp

slide-61
SLIDE 61

Hierarchy repeated...with heterogeneity

Soc & Hum Bio & Health Life & Earth Phys & Engr Math & Comp

slide-62
SLIDE 62

Biomedical and Health Sciences

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

slide-63
SLIDE 63

Biomedical and Health Sciences

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

“Health Affairs” “Health Policy” “Pharmacoeconomics”

slide-64
SLIDE 64

Similar in Mathematics and Computer Science

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

slide-65
SLIDE 65

Similar in Mathematics and Computer Science

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

“Transportation Research Records” “Accident Analysis and Prevention”

slide-66
SLIDE 66

Similar in Life & Earth Science

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

slide-67
SLIDE 67

Similar in Life & Earth Science

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

“Ecological Economics” “Land Use Policy” “Ecological Indicators”

slide-68
SLIDE 68

Similar in Life & Earth Science

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

“Ecological Economics” “Land Use Policy” “Ecological Indicators”

More “social” fields often have more disagreement

slide-69
SLIDE 69

Similar in Life & Earth Science...but also new patterns

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp More “social” fields often have more disagreement

“Journal of Volcanology” “Earth and Planetary Science Letters” “Journal of vertebrate paleontology” “Cretaceous Research” “Sedimentary Research”

slide-70
SLIDE 70

Similar in Life & Earth Science...but also new patterns

Soc & Hum Bio & Health

Life & Earth

Phys & Engr Math & Comp

“Journal of vertebrate paleontology” “Cretaceous Research” “Sedimentary Research” “Journal of Volcanology” “Earth and Planetary Science Letters”

Fields that can’t experiment have more disagreement More “social” fields often have more disagreement

slide-71
SLIDE 71

Takeaways

  • Re-confirm the hierarchy of sciences using disagreement

○ Additional evidence that our approach is robust ○ Supports bast bibliometric studies of the hierarchy (Fanelli & Glänzel, 2013)

  • Identify heterogeneity

○ Based on epistemic characteristics of the fields

Fanelli, D., & Glänzel, W. (2013). Bibliometric Evidence for a Hierarchy of the Sciences. PLoS ONE, 8(6), e66938. https://doi.org/10.1371/journal.pone.0066938

slide-72
SLIDE 72

Measuring disagreement

  • ver time
slide-73
SLIDE 73

Disagreement over time

slide-74
SLIDE 74

Disagreement over time

  • Mostly stable, slight

decrease

slide-75
SLIDE 75

Disagreement over time

  • Mostly stable, slight

decrease

  • Mostly driven by a few fields
slide-76
SLIDE 76

Disagreement over time

  • Mostly stable, slight

decrease

  • Mostly driven by a few fields
  • Takeaways

○ Is this real? ○ Are these changes significant?

slide-77
SLIDE 77

Measuring disagreement by age of cited paper

slide-78
SLIDE 78

Disagreement and age

Relative age of cited paper (years)

slide-79
SLIDE 79

Disagreement and age

  • After a bump, older papers

are disagreed with less

Relative age of cited paper (years)

slide-80
SLIDE 80

Disagreement and age

  • After a bump, older papers

are disagreed with less

  • Field differences

Relative age of cited paper (years)

slide-81
SLIDE 81

Disagreement and age

  • After a bump, older papers

are disagreed with less

  • Field differences
  • Takeaways

○ Disagreement relates to age ○ Selection effect? Relative age of cited paper (years)

slide-82
SLIDE 82

Measuring disagreement… by self citation

slide-83
SLIDE 83

Disagreement and Self-citation

  • People should disagree with

themselves less

slide-84
SLIDE 84

Disagreement and Self-citation

  • People should disagree with

themselves less

slide-85
SLIDE 85

Disagreement and Self-citation

  • People should disagree with

themselves less

  • Similar across fields

Presence of author overlap

slide-86
SLIDE 86

Disagreement and Self-citation

  • People should disagree with

themselves less

  • Similar across fields
  • Takeaways

○ Confirms expectations ○ Perfunctory citations? Presence of author overlap

slide-87
SLIDE 87

So far, our measure has revealed insights based on...

  • Discipline
  • Time
  • Age of cited paper
  • Self citation

Disagreement

slide-88
SLIDE 88

So far, our measure has revealed insights based on...

  • Discipline
  • Time
  • Age of cited paper
  • Self citation
  • And the process of keyword-based

studies itself

Disagreement

slide-89
SLIDE 89

Our approach is transparent

  • We can open the black box and see

exactly where issues occur ○ In contrast to common Machine Learning techniques

  • Contextualizes our analysis
  • Informs future keyword-based studies
slide-90
SLIDE 90

When a signal is a proper noun

“...the conflict monitoring theory [6] provides a slightly different way to account for conflict effects.”

slide-91
SLIDE 91

When a signal is an object of study

“Caselli and Coleman study conflicts based on ethnicity....”

slide-92
SLIDE 92

When a signal is just in reference to the wrong thing

“...consumers are usually self-motivated and enthusiastic in exchanging, sharing and debating ideas... (Anderson & Weitz, 1992).”

slide-93
SLIDE 93

When it’s somebody’s name

“Measurement items for perceived risks were also developed from Debatin et al.’s (2009) study.”

slide-94
SLIDE 94

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

slide-95
SLIDE 95

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

slide-96
SLIDE 96

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

Each line corresponds to a meso-field Color matches ratio to average across all fields

slide-97
SLIDE 97

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

  • Debat* overrepresented in Soc & Hum
slide-98
SLIDE 98

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

  • Debat* overrepresented in Soc & Hum
  • No consensus in Bio & Health
slide-99
SLIDE 99

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

  • Debat* overrepresented in Soc & Hum
  • No consensus in Bio & Health
  • Disagree* in Phys & Engr, Life & Earth
slide-100
SLIDE 100

Isolated, or systematic?

  • Overall, we capture disagreement,

○ Heterogeneity by field

  • Debat* overrepresented in Soc & Hum
  • No consensus in Bio & Health
  • Disagree* in Phys & Engr, Life & Earth
  • Are these driven by artefacts?
slide-101
SLIDE 101

Our approach has limitations, but...

  • They can be identified
  • Can be easily adjusted

○ Different validity threshold ○ Refine existing queries ○ Add new signal and filter terms

  • Future analyses of scientific text

will need to consider these artefacts

slide-102
SLIDE 102

In Summary

slide-103
SLIDE 103

In Summary

  • Developed a framework for identifying

instances of disagreement

○

Robust

○

Transparent

○

Adjustable

slide-104
SLIDE 104

In Summary

  • Developed a framework for identifying

instances of disagreement

○

Robust

○

Transparent

○

Adjustable

  • Confirmed a hierarchy of sciences

○

But with greater heterogeneity at the field level

slide-105
SLIDE 105

In Summary

  • Developed a framework for identifying

instances of disagreement

○

Robust

○

Transparent

○

Adjustable

  • Confirmed a hierarchy of sciences

○

But with greater heterogeneity at the field level

  • Observed minor temporal changes
slide-106
SLIDE 106

In Summary

  • Developed a framework for identifying

instances of disagreement

○

Robust

○

Transparent

○

Adjustable

  • Confirmed a hierarchy of sciences

○

But with greater heterogeneity at the field level

  • Observed minor temporal changes
  • Relative age matters
slide-107
SLIDE 107

In Summary

  • Developed a framework for identifying

instances of disagreement

○

Robust

○

Transparent

○

Adjustable

  • Confirmed a hierarchy of sciences

○

But with greater heterogeneity at the field level

  • Observed minor temporal changes
  • Relative age matters
  • Differences by self-citation
slide-108
SLIDE 108

In Summary

  • Developed a framework for identifying

instances of disagreement

○

Robust

○

Transparent

○

Adjustable

  • Confirmed a hierarchy of sciences

○

But with greater heterogeneity at the field level

  • Observed minor temporal changes
  • Relative age matters
  • Differences by self-citation
  • Disciplinary heterogeneity in word use requires

consideration in scientific text analysis

slide-109
SLIDE 109

Now that we have an indicator of disagreement

  • Answer questions about how disagreement

differs based on author characteristics

○ Gender ○ Race ○ National affiliation ○ Career stage

  • Incorporate into traditional scientometric

analysis

○ How does disagreement relate to citation impact? ○ Are novel or interdisciplinary papers more likely to be disagreed with? ○ Does team size relate to disagreement

Disagreement

slide-110
SLIDE 110

More soon!

slide-111
SLIDE 111

Thank you!

Dakota Murray Indiana University Bloomington dakmurra@iu.edu

Vincent Larivière Université de Montréal vincent.lariviere@umontreal.ca

Cassidy R. Sugimoto Indiana University Bloomington sugimoto@indiana.edu Nees Jan van Eck CWTS Leiden University ecknjpvan@cwts.leidenuniv.nl Ludo Waltman CWTS Leiden University waltmanlr@cwts.leidenuniv.nl Wout Lamers CWTS Leiden University w.s.lamers@cwts.leidenuniv.nl Kevin Boyack SciTech Strategies kboyack@mapofscience.com

@dakotasmurray dakotamurray.me Slides at:

slide-112
SLIDE 112

Supporting slides

slide-113
SLIDE 113

Signal term Variants Specific negations Results challenge* 405,613 contradict* 115,375 contrast* 1,257,866 contrary 171,711 conflict* 212,246 disagree* “not agree*”, “no agreement” “range”, “scale”, “kappa”, “likert”, “agree*” and/or “disagree” within a ten-word range of each other. 52,615 differ* “different*” 2,003,677 controvers* 154,608 disprov* “prove*” and “disprove*” within a ten-word range 2,938 refut* “refutab*” 10,322 debat* “parliament* debat*”, “congress* debat*”, “senate* debat*”, “polic* debat*”, “politic* debat*”, “public* debat*”, “societ* debat*” 150,617 no consensus “lack of consensus” “consensus sequence”, “consensus site” 16,632 questionable 24,244

Signal terms

slide-114
SLIDE 114

Filter terms

studies Studies; study; previous work; earlier work; literature; analysis; analyses; report; reports ideas idea*; theory; theories; assumption*; hypothesis; hypotheses methods model*, method*, approach*; technique* results result*; finding*; outcome*; evidence; data; conclusion*; observation*

slide-115
SLIDE 115

Representation

The expected (red line) vs. actual (dot) number of citances conflict* more common in Social Sciences disprov* more common in Maths disagree* more common in Physics & Engineering

slide-116
SLIDE 116

Representation

Same for filter terms +ideas over-represented in Social Sciences and Humanities +methods over-represented in Math & Comp Sci, also Physics and Engineering

slide-117
SLIDE 117

Representation

But there is huge heterogeneity controvers* over-represented in Bio & Health sciences, except with +ideas contradict* over-represented in Math & Comp Sci, except with +results and +studies

slide-118
SLIDE 118

Different temporal dynamics by query

slide-119
SLIDE 119

Text progression

slide-120
SLIDE 120

Reference meso-field map

slide-121
SLIDE 121

Self-citation and filter term

slide-122
SLIDE 122

Relative age and filter term

slide-123
SLIDE 123

Disagreement and gender

Based on gender of the author of the citing paper Not really any convincing evidence of an effect