Consistent bibliometric rankings of Scientists and Departments - - PowerPoint PPT Presentation

Consistent bibliometric rankings

• f Scientists and Departments

Denis Bouyssou Thierry Marchant

CNRS Paris, France Ghent University Ghent, Belgium

EBPM Workshop, Paris December 

If you do not know Thierry. . .

Introduction

Introduction

Evaluation of research: the old way papers (informally but seriously) evaluated by peers scientists (informally but seriously) evaluated by peers departments (informally but seriously) evaluated by peers

2

Introduction

Introduction

Evaluation of research: the old way papers (informally but seriously) evaluated by peers scientists (informally but seriously) evaluated by peers departments (informally but seriously) evaluated by peers Problems cost time (some) documented biases

2

Introduction

Introduction

Evaluation of research: the old way papers (informally but seriously) evaluated by peers scientists (informally but seriously) evaluated by peers departments (informally but seriously) evaluated by peers Problems cost time (some) documented biases Evaluation of research: the new way bibliometric indices

supposedly objective supposedly at low cost supposedly giving the right incentives

2

Introduction

Bibliometric indices

Many possible indices counting of papers counting of citations sum of Impact Factors Markovian indices h-index and its variants

3

Introduction

Potential problems

Indices field normalization (Mathematics vs Molecular Biology) coauthors multiple affiliations books & other publications publications not in English . . .

4

Introduction

Potential problems

Indices field normalization (Mathematics vs Molecular Biology) coauthors multiple affiliations books & other publications publications not in English . . . Database adequate coverage of publications? adequate time window for collecting citations? quality of data?

names with diacritical signs and/or T EX ligatures (Fran¸ cois-´ Eric ´ Effla¨ ır) complicated names (Bou y Sou)

4

Introduction

Properties of Bibliometric indices

Bibliometric Indices what properties? how to compare them? how to combine them?

5

Introduction

Properties of Bibliometric indices

Bibliometric Indices what properties? how to compare them? how to combine them? Motivation choosing adequate bibliometric indices should be a subject of scientific investigation this choice is tremendously important this choice should not be in the hands of evaluation bureaucrats

5

Introduction

Potential problems 1/2

Evaluation of authors h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) author f: 4 papers with 4 citations each author g: 3 papers with 6 citations each

ih(f) = 4 > ih(g) = 3

6

Introduction

Potential problems 1/2

Evaluation of authors h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) author f: 4 papers with 4 citations each author g: 3 papers with 6 citations each

ih(f) = 4 > ih(g) = 3 both authors publish a new paper with 6 citations ih(f ∗) = 4 = ih(g∗) = 4

6

Introduction

Potential problems 1/2

Evaluation of authors h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each) author f: 4 papers with 4 citations each author g: 3 papers with 6 citations each

ih(f) = 4 > ih(g) = 3 both authors publish a new paper with 6 citations ih(f ∗) = 4 = ih(g∗) = 4 both authors publish a new paper with 6 citations ih(f ∗∗) = 4 < ih(g∗∗) = 5

6

Introduction

Potential problems 2/2

Evaluation of authors and departments h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each)

7

Introduction

Potential problems 2/2

Evaluation of authors and departments h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each)

Department a author a1: 4 papers each one cited 4 times author a2: 4 papers each one cited 4 times

h-index of both authors is 4 h-index of the department is 4

7

Introduction

Potential problems 2/2

Evaluation of authors and departments h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each)

Department a author a1: 4 papers each one cited 4 times author a2: 4 papers each one cited 4 times

h-index of both authors is 4 h-index of the department is 4

Department b author b1: 3 papers each one cited 6 times author b2: 3 papers each one cited 6 times

h-index of both authors is 3 h-index of the department is 6

7

Introduction

Potential problems 2/2

Evaluation of authors and departments h-index

the h-index of an author is x if this author x papers having at least x citations each (and her other papers have at most x citations each)

Department a author a1: 4 papers each one cited 4 times author a2: 4 papers each one cited 4 times

h-index of both authors is 4 h-index of the department is 4

Department b author b1: 3 papers each one cited 6 times author b2: 3 papers each one cited 6 times

h-index of both authors is 3 h-index of the department is 6

the “best” department contains the “worst” authors!

7

Outline

Outline

1 Setting 2 Axioms 3 Scoring rules 4 Limitation and future research

8

Setting

Outline

1 Setting

9

Setting

Model of Authors

Authors an author is a function f from N to N f(x) is the number of papers by this author having received x citations

10

Setting

Model of Authors

Authors an author is a function f from N to N f(x) is the number of papers by this author having received x citations Set of all Authors X is the set of all functions f from N to N such that

• x∈N

f(x) is finite

10

Setting

Model of Authors

Authors an author is a function f from N to N f(x) is the number of papers by this author having received x citations Set of all Authors X is the set of all functions f from N to N such that

• x∈N

f(x) is finite Objective build a binary relation on X f g is “given their publication/citation record, scientists f is at least as good as scientist g”

10

Setting

Model of Authors

Authors an author is a function f from N to N f(x) is the number of papers by this author having received x citations Set of all Authors X is the set of all functions f from N to N such that

• x∈N

f(x) is finite Objective build a binary relation on X f g is “given their publication/citation record, scientists f is at least as good as scientist g” Limitations coauthors are ignored in this talk

10

Setting

Notation and remarks

Notation 0 is an author without any paper 1x is an author with 1 paper having received x citations

11

Setting

Notation and remarks

Notation 0 is an author without any paper 1x is an author with 1 paper having received x citations Remarks it makes sense to add two authors f and g: f + g it makes sense to multiply an author f by an integer n: n · f

11

Setting

Model of Departments

Departments a department of size k is an element of X k: (f1, f2, . . . , fk)

12

Setting

Model of Departments

Departments a department of size k is an element of X k: (f1, f2, . . . , fk) Set of all Departments D =

• k∈N

X k

12

Setting

Model of Departments

Departments a department of size k is an element of X k: (f1, f2, . . . , fk) Set of all Departments D =

• k∈N

X k Objective build a binary relation on D A B is “given their publication/citation record of the scientists in departments A and B, department A is at least as good as department B”

12

Setting

Model of Departments

Departments a department of size k is an element of X k: (f1, f2, . . . , fk) Set of all Departments D =

• k∈N

X k Objective build a binary relation on D A B is “given their publication/citation record of the scientists in departments A and B, department A is at least as good as department B” Limitations multiple affiliations are ignored field normalization is ignored

12

Axioms

Outline

2 Axioms

13

Axioms

Axioms

Consistency Let A = (a1, a2, . . . , ak) and B = (b1, b2, . . . , bk) be two departments of size k. If ai bi, for all i ∈ {1, 2, . . . , k} then A B Furthermore if ai ≻ bi, for some i ∈ {1, 2, . . . , k} then A ⊲ B

14

Axioms

Axioms

Consistency Let A = (a1, a2, . . . , ak) and B = (b1, b2, . . . , bk) be two departments of size k. If ai bi, for all i ∈ {1, 2, . . . , k} then A B Furthermore if ai ≻ bi, for some i ∈ {1, 2, . . . , k} then A ⊲ B Independence For all f, g ∈ X and all x ∈ N f g ⇔ f + 1x g + 1x

14

Axioms

Axioms

Consistency Let A = (a1, a2, . . . , ak) and B = (b1, b2, . . . , bk) be two departments of size k. If ai bi, for all i ∈ {1, 2, . . . , k} then A B Furthermore if ai ≻ bi, for some i ∈ {1, 2, . . . , k} then A ⊲ B Independence For all f, g ∈ X and all x ∈ N f g ⇔ f + 1x g + 1x Transfer For all A = (a1, a2, . . . , ak) ∈ D, all i, j ∈ {1, 2, . . . , k} and all x ∈ N (a1, . . . , ai + 1x, . . . , ak) (a1, . . . , aj + 1x, . . . , ak)

14

Axioms

Interpretation and Results

Interpretation Consistency appears uncontroversial Independence appears uncontroversial Transfer is strong (but used quite often)

“Inequalities” within departments are ignored

15

Axioms

Interpretation and Results

Interpretation Consistency appears uncontroversial Independence appears uncontroversial Transfer is strong (but used quite often)

“Inequalities” within departments are ignored

Proposition 1 If and are linked by Consistency and if satisfies Transfer then satisfies Independence

15

Axioms

Interpretation and Results

Interpretation Consistency appears uncontroversial Independence appears uncontroversial Transfer is strong (but used quite often)

“Inequalities” within departments are ignored

Proposition 1 If and are linked by Consistency and if satisfies Transfer then satisfies Independence Corollary If is the ranking of authors based on the h-index then there is no such that Transfer and Consistency hold

15

Scoring rules

Outline

3 Scoring rules

16

Scoring rules

Scoring rules for scientists

Definition 1 is a scoring rule for scientists (s-scoring rule) if there is a real valued function u on N such that f g ⇔

• x∈N

f(x)u(x) ≥

• x∈N

g(x)u(x) u(x) gives the worth of one publication with x citations many bibliometric indices are scoring rules (but not the h-index) all scoring rules satisfy independence

17

Scoring rules

Rules for departments

Definition 2 is a scoring rule for departments (d-scoring rule) if there is a real valued function v on N such that (a1, a2, . . . , ak) (b1, b2, . . . , bℓ) ⇔

k

• i=1
• x∈N

ai(x)v(x) ≥

ℓ

• i=1
• x∈N

bi(x)v(x)

18

Scoring rules

Rules for departments

Definition 2 is a scoring rule for departments (d-scoring rule) if there is a real valued function v on N such that (a1, a2, . . . , ak) (b1, b2, . . . , bℓ) ⇔

k

• i=1
• x∈N

ai(x)v(x) ≥

ℓ

• i=1
• x∈N

bi(x)v(x) Definition 3 is an averaging rule for departments (d-averaging rule) if there is a real valued function v on N such that (a1, a2, . . . , ak) (b1, b2, . . . , bℓ) ⇔ 1 k

k

• i=1
• x∈N

ai(x)v(x) ≥ 1 ℓ

ℓ

• i=1
• x∈N

bi(x)v(x)

18

Scoring rules

Remarks

Proposition 2 If is an s-scoring rule and is a d-scoring rule or a d-averaging rule with u = v then they are linked by Consistency

19

Scoring rules

Axioms

Archimedeanness For all f, g, f ′, g′ ∈ X such that f ≻ g there is n ∈ N such that f ′ + n · f g′ + n · g

20

Scoring rules

Axioms

Archimedeanness For all f, g, f ′, g′ ∈ X such that f ≻ g there is n ∈ N such that f ′ + n · f g′ + n · g Dummy Scientist For all k ∈ N and all (a1, a2, . . . , ak) ∈ D (a1, a2, . . . , ak) (a1, a2, . . . , ak, 0)

20

Scoring rules

Axioms

Archimedeanness For all f, g, f ′, g′ ∈ X such that f ≻ g there is n ∈ N such that f ′ + n · f g′ + n · g Dummy Scientist For all k ∈ N and all (a1, a2, . . . , ak) ∈ D (a1, a2, . . . , ak) (a1, a2, . . . , ak, 0) Homogeneity For all k, n ∈ N and all (a1, a2, . . . , ak) ∈ D (a1, a2, . . . , ak) (a1, a1, . . . , a1

• n

, a2, a2, . . . , a2

• n

, . . . , ak, ak, . . . , ak

• n

)

20

Scoring rules

Remarks

all s-scoring rules satisfy Archimedeanness Dummy Scientist is satisfied by d-scoring rules but not by d-averaging rules Homogeneity is satisfied by d-averaging rules but not by d-scoring rules

21

Scoring rules

Main results

Theorem 1 The relations and are linked by Consistency, satisfies Transfer and Dummy Scientist, satisfies Archimedeanness if and only if is an s-scoring rule and is a d-scoring rule with u = v The function u is unique up to the multiplication by a positive constant

22

Scoring rules

Main results

Theorem 1 The relations and are linked by Consistency, satisfies Transfer and Dummy Scientist, satisfies Archimedeanness if and only if is an s-scoring rule and is a d-scoring rule with u = v The function u is unique up to the multiplication by a positive constant Theorem 2 The relations and are linked by Consistency, satisfies Transfer and Homogeneity, satisfies Archimedeanness if and only if is an s-scoring rule and is a d-averaging rule with u = v The function u is unique up to the multiplication by a positive constant

22

Scoring rules

Extensions and remarks

Extension add additional conditions to restrict the shape of u

u is nondecreasing u is constant u is linear

23

Scoring rules

Extensions and remarks

Extension add additional conditions to restrict the shape of u

u is nondecreasing u is constant u is linear

Remark the conditions used in Theorems 1 & 2 are independent

23

Limitation and future research

Outline

4 Limitation and future research

24

Limitation and future research

Discussion

Axioms Consistency is highly desirable Independence is highly desirable (but violated by the h-index) Archimedeanness is technical Transfer is more debatable

25

Limitation and future research

Discussion

Axioms Consistency is highly desirable Independence is highly desirable (but violated by the h-index) Archimedeanness is technical Transfer is more debatable Transfer strong condition mixing two aspects

anonymity is imposed inequality is ignored

accepted by most bibliometric researchers

25

Limitation and future research

Future research

coauthors multiple affiliations single profile vs multi profile approaches what about Consistency without Transfer?

26

References

Bouyssou, D., Marchant, T. (2010) Ranking scientists and departments in a consistent manner Working Paper, 19 pages.