Best practice in assessing Research Excellence by Public Research - - PowerPoint PPT Presentation

▶
best practice in assessing research excellence by public
SMART_READER_LITE
LIVE PREVIEW

Best practice in assessing Research Excellence by Public Research - - PowerPoint PPT Presentation

May 20, 2023 11 likes •214 views

Best practice in assessing Research Excellence by Public Research Funders Quality over quantity! The German Research Foundation (DFG) Who we are 1. The DFG is Germanys largest independent Basic Research Funding Agency 2. The DFG is an

slide-1
SLIDE 1

Best practice in assessing Research Excellence by Public Research Funders

Quality over quantity!

slide-2
SLIDE 2

The German Research Foundation (DFG)

  • 1. The DFG is Germany‘s largest independent Basic

Research Funding Agency

  • 2. The DFG is an association of the German Universities

and the large Research Organisations

  • 3. The DFG receives its budget from the German Federal

Government and the 16 German Länder (States)

  • 4. The DFG is spending ~3 bn € per year on Bottom Up

Research

  • 5. The DFG is serving all fields of Science and the

Humanities

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 2

Who we are

slide-3
SLIDE 3

Assessing Research Quality

Function and Purpose of Scientific Publications:

  • Disseminate and safeguard Scientific Findings and Knowledge
  • Attribute Creatorship/ Authorship and subsequently Reputation to the Scientists

Funding facts:

  • 1. Publicly funded Research usually results in Publications
  • 2. Publications play an important role in assessing Public Research

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 3

The role of publications

slide-4
SLIDE 4

Assessing Research Quality

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 4

The role of publications

Money Research Findings / Knowledge Publications

slide-5
SLIDE 5

DFG Review Boards: Shares of Publication Types

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 5

Humanities and Social Sciences

M

  • n
  • g

r a p h s A n t h

  • l
  • g

i e s J

  • u

r n a l a r t i c l e s P r e

  • p

r i n t s N

  • n

p u b l i s h e r p u b l i c a t i

  • n

s D a t a p a c k a g e s S

  • f

t w a r e / c

  • d

e P a t e n t s A u d i

  • v

i s u a l m e d i a I m a g e s / g r a p h s A l t e r n a t i v e p u b l i c a t i

  • n

t y p e s O t h e r

DFG Review Board

4 4 4 1 2 1 1 1 1 5 1

  • Ancient Cultures

5 5 5 1 2 1 1 1 1 2 2

  • History

5 2 1 1 1 1 1 1 1 1 1

  • Fine Arts, Music, Theatre and Media Studies

3 3 4 2 4 2 2 1 1 1 1

  • Linguistics

3 3 3 1 1 1 1 1 1 1 1

  • Literary Studies

4 3 3 2 2 1 1 1 2 2 2

  • Social and Cultural Anthropology, Non-

European Cultures, Jewish Studies and Religious Studies

5 4 3 1 2 1 1 1 1 2 2

  • Theology

3 3 3 2 2 1 1 1

  • 1

2

  • Philosophy

3 4 3 1 2 2 1 1 1 1 1

  • Educational Research

2 1 5 1 1 1 2 1 1 2 2

  • Psychology

3 3 3 2 2 2 2 1 1 1 2

  • Social Sciences

1 1 5 4 1 2 2 1 1 1 1

  • Economics

5 3 3 1 2 1 1 1 1 2 2

  • Jurisprudence

Humanities and Social Sciences

slide-6
SLIDE 6

DFG Review Boards: Shares of Publication Types

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 6

Humanities and Social Sciences

M

  • n
  • g

r a p h s A n t h

  • l
  • g

i e s J

  • u

r n a l a r t i c l e s P r e

  • p

r i n t s N

  • n

p u b l i s h e r p u b l i c a t i

  • n

s D a t a p a c k a g e s S

  • f

t w a r e / c

  • d

e P a t e n t s A u d i

  • v

i s u a l m e d i a I m a g e s / g r a p h s A l t e r n a t i v e p u b l i c a t i

  • n

t y p e s O t h e r

DFG Review Board

4 4 4 1 2 1 1 1 1 5 1

  • Ancient Cultures

5 5 5 1 2 1 1 1 1 2 2

  • History

5 2 1 1 1 1 1 1 1 1 1

  • Fine Arts, Music, Theatre and Media Studies

3 3 4 2 4 2 2 1 1 1 1

  • Linguistics

3 3 3 1 1 1 1 1 1 1 1

  • Literary Studies

4 3 3 2 2 1 1 1 2 2 2

  • Social and Cultural Anthropology, Non-

European Cultures, Jewish Studies and Religious Studies

5 4 3 1 2 1 1 1 1 2 2

  • Theology

3 3 3 2 2 1 1 1

  • 1

2

  • Philosophy

3 4 3 1 2 2 1 1 1 1 1

  • Educational Research

2 1 5 1 1 1 2 1 1 2 2

  • Psychology

3 3 3 2 2 2 2 1 1 1 2

  • Social Sciences

1 1 5 4 1 2 2 1 1 1 1

  • Economics

5 3 3 1 2 1 1 1 1 2 2

  • Jurisprudence

Humanities and Social Sciences

slide-7
SLIDE 7

DFG Review Boards: Shares of Publication Types

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 7

Life Sciences

M

  • n
  • g

r a p h s A n t h

  • l
  • g

i e s J

  • u

r n a l a r t i c l e s P r e

  • p

r i n t s N

  • n

p u b l i s h e r p u b l i c a t i

  • n

s D a t a p a c k a g e s S

  • f

t w a r e / c

  • d

e P a t e n t s A u d i

  • v

i s u a l m e d i a I m a g e s / g r a p h s A l t e r n a t i v e p u b l i c a t i

  • n

t y p e s O t h e r

DFG Review Board

1 1 5 2 3 2 1 1 1 1 1

  • Basic Research in Biology and Medicine

1 2 5 3 4 3 2 2 2 1 2

  • Plant Sciences

1 1 5 2 2 1 1 1 1 1 1

  • Zoology

1 2 5 3 4 4 2 2 1 1 1

  • Microbiology, Virology and Immunology

1 1 4 1 3 1 1 1 1 1 2

  • Medicine

1 1 5 2 2 2 1 1 1 1 1

  • Neurosciences

1 2 5 2 2 3 1 1 1 1 2

  • Agriculture, Forestry and Veterinary

Medicine Life Sciences

slide-8
SLIDE 8

DFG Review Boards: Shares of Publication Types

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 8

Natural Sciences

Monographs Anthologies Journal articles Pre-prints Non publisher publications Data packages Software/code Patents Audio-visual media Images/ graphs Alternative publication types Other

DFG Review Board

2 2 5 2 2 2 1 2 1 1 1

  • Molecular Chemistry

2 2 5 2 2 2 1 2 1 1 1

  • Chemical Solid State and Surface Research

1 2 5 2 1 2 1 1 1 1 1

  • Physical and Theoretical Chemistry

1 1 5 1 2 1 1 2 1 1 1

  • Analytical Chemistry, Method Development

(Chemistry)

1 1 5 2 2 1 1 2 1 1 1

  • Biological Chemistry and Food Chemistry

2 2 5 2 2 2 1 3 1 2 1

  • Polymer Research

1 2 5 3 3 2 2 2 1 1 1

  • Condensed Matter Physics

1 2 5 4 2 2 2 2 1 1 2 1

Optics, Quantum Optics and Physics of Atoms, Molecules and Plasmas

1 1 5 5 3 3 3 2 1 1 2

  • Particles, Nuclei and Fields

1 2 5 3 3 2 2 2 1 1 1

  • Statistical Physics, Soft Matter, Biological

Physics, Nonlinear Dynamics

1 1 5 4 2 3 2 1 1 1 1

  • Astrophysics and Astronomy

4 2 5 5 2 2 2 1 1 2 1 1

Mathematics

1 2 4 1 1 2 2 1 1 1 1 1

Atmospheric Science, Oceanography and Climate Research

2 2 5 2 1 1 1 1 1 1 1

  • Geology and Palaeontology

1 2 4 2 3 2 2 1 1 1 1

  • Geophysics and Geodesy

1 2 5 1 3 1 1 1 1 1 1

  • Geochemistry, Mineralogy and

Crystallography

1 1 5 1 3 2 2 1 1 1 1

  • Geography

1 1 5 1 2 1 1 1 1 1 1

  • Water Research

Natural Sciences

slide-9
SLIDE 9

DFG Review Boards: Shares of Publication Types

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 9

Engineering Sciences

Monographs Anthologies Journal articles Pre-prints Non publisher publications Data packages Software/code Patents Audio-visual media Images/ graphs Alternative publication types Other

DFG Review Board

1 2 4 1 2 1 1 2 1 1 2

  • Production Technology

2 2 3 1 1 2 2 2 1 1 1 3

Mechanics and Constructive Mechanical Engineering

1 1 5 2 4 4 4 5 2 2 1

  • Process Engineering, Technical Chemistry

1 1 5 2 4 4 4 5 2 2 1

  • Heat Energy Technology, Thermal Machines,

Fluid Mechanics

1 2 5 2 2 2 2 3 1 1 1

  • Materials Engineering

1 2 5 3 1 4 2 2 1 1 1 4

Materials Science

1 1 3 1 2 2 2 2 2 2 1

  • Systems Engineering

2 3 4 3 4 3 3 4 2 2 1

  • Electrical Engineering and Information

Technology

1 1 2 4 5 2 5 1 4 4 1

  • Computer Science

1 1 2 4 5 2 5 1 4 4 1

  • Construction Engineering and Architecture

Engineering Sciences

slide-10
SLIDE 10

Assessing Research Quality

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 10

The role of publications

slide-11
SLIDE 11

The incredible usefulness of Journal Publications

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 11

You can count and do calculations on them

Honjo, Tasuko Kyoto Allison, James P. Houston Documents 632 350 Cited by (No. of) documents 36.322 29.901 Co-authors 1.739 1.506 h-index 124 111 Total citations 63.187 49.622 Link

https://www.scopus.com/authid/det ail.uri?authorId=36013822600 https://www.scopus.com/authid/detail.u ri?authorId=7401475344

Scopus-Search, 2019/05/19. Figure out the better scientist:

slide-12
SLIDE 12

The incredible usefulness of Journal Publications

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 12

What is good Science?

https://www.nobelprize.org/prizes/medicine/2018/summary/

The Nobel Prize in Physiology or Medicine 2018:

James P. Allison

Prize share: ½

Tasuko Honjo

Prize share: ½

slide-13
SLIDE 13

Journal and Publication metrics

►Observed use of Journal /Publication Metrics for…

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 13

What to use it for?

useful not useful

  • Research on Research
  • Monitoring purposes
  • Mapping Research
  • Identifying Funding Gaps
  • Measuring institutional Productivity
  • Measuring Researchers‘ Productivity
  • Calculating Investment in Institutions
  • Calculating Investment in Researchers
  • Tenuring and hiring of Researchers
  • Setting up Outcome Targets for Research
  • Optimizing individual Researchers‘ Reputation
slide-14
SLIDE 14

Non-useful application of journal/publication metrics

  • results in strong Incentives for Researchers to optimize their individual

reputation by optimizing their publication metrics

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 14

Setting bad incentives

Using publication metrics for… setting up outcome targets for research… funding, tenuring and hiring of researchers…

  • results in strong Incentives for Journals to optimize their individual

reputation by optimizing their journal metrics

slide-15
SLIDE 15

Use of journal/publication metrics as research targets

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 15

Why it won‘t work

Charles Albert Eric Goodhart (1975): "When a measure becomes a target, it ceases to be a good measure”1 (Goodhart’s Law)

1 see: Marilyn Strathern (1997). ‘Improving ratings’: audit in the British University system.

European Review, 5, pp 305-321 doi:10.1002/(SICI)1234-981X(199707)5:33.0.CO;2-4

slide-16
SLIDE 16

Use of journal/publication metrics as research targets

Incentives to optimize an individual researcher’s publication metrics result in further incentives for a spectrum of either undesirable behaviour or misconduct:

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 16

Side effects imply on research integrity

►Slicing of Publications ►Rushed Research, rushed Submissions

  • Incl. insufficient quality control, p-hacking etc.

►Non-adequate Choice of Publication Type/ Journal

  • Incl. delaying the publication process (Cascading)

►Non-justified Authorships ►Choice of Research Topic based on its Publishing-Potential ►…

slide-17
SLIDE 17

Quality over Quantity

►What DFG (and other funders) should respond:

  • „Dear researcher, if you are trying to impress us, don‘t ‚metric‘ us!
  • Keep doing science, keep trying to find answers to your big questions!
  • Keep publishing using the publication types and channels required to reach your

target groups

  • We will figure out the quality of your project by assessing…
  • its Originality and the anticipated contribution to knowledge,
  • the Objectives and Work Programme by their working hypotheses and distinct topic,
  • the soundness of the Preliminary Work, the Quality of Publications and your

Qualifications as the applicant,

  • your Work and Research Environment.
  • Don‘t h-index, slice, cascade or p-hack!
  • It‘s the research story!”

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 17

It‘s the Content not the Package that counts

slide-18
SLIDE 18

Assessment of Research Quality

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 18

Old View

Money Research Findings / Knowledge Publications,

  • pref. many

and „top“- Journal Articles

slide-19
SLIDE 19

Assessment of Research Quality

Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG Hongkong, June 5th 2019 19

Suggestion

Money Research Findings / Knowledge All kinds of Publications

slide-20
SLIDE 20

Further information

► DFG: https://www.dfg.de/en/index.jsp ► Funding atlas: https://www.dfg.de/en/dfg_profile/facts_figures/funding_atlas/index.html ► GEPRIS – German Project Information System:

http://gepris.dfg.de/gepris/OCTOPUS/?language=en&module=gepris

► GERiT – German Research Institutions: https://gerit.org/en/

Hongkong, June 5th 2019 Best practice in assessing Research Excellence by Public Research Funders / Quality over quantity! – Dr. Tobias Grimm, DFG 20