Is our knowledge of ecology becoming more precise and accurate over - - PowerPoint PPT Presentation

Is our knowledge of ecology becoming more precise and accurate over time? A meta-analysis of meta-analyses

Jeremy Fox University of Calgary

Lab: foxlabcalgary.wordpress.com Blog: dynamicecology.wordpress.com

Is progress in ecology more like this? The big vague motivating question Or this?

The “decline effect”: effect size estimates decrease in magnitude over time

Implicit bias (Charlesworth & Banaji 2019): Autistic vs. non-autistic neurology (Rødgaard et al. 2019): Ego depletion (Vidallo 2019):

The decline effect in ecology and evolution

Status signaling in male house sparrows (Sánchez-Tojár et al. 2018) EEB effect sizes estimates typically are negatively correlated with publication year (Jenions & Moller 2001)

Why might we expect decline effects?

ZSL

Because unusual things get noticed.

The much more specific questions that I’ll actually answer

• 1. Do effect size estimates tend to decline in magnitude as more

studies are published?

• 2. Does the precision of the estimated mean effect size typically

improve as more studies are published?

• 3. Is sampling error or effect size estimates declining?
• 4. How much heterogeneity is there among ecological effect size

estimates, and how does it change as more studies are conducted?

Methods overview: meta-analysis of meta-analyses

• Meta-analysis: statistically summarizes different estimates
• f the “same” effect
• Various measures of effect size
• Hedge’s d, log(response ratio), etc.
• Cumulative random effects meta-analysis
• Random effects: variation in effect size among studies

(papers), within studies, and due to sampling error

• Cumulative: add in one study at a time, in order of

publication, recalculate the meta-analysis

• Look for patterns across the meta-analyses

• 1. Are decline effects widespread?

No widespread decline effects

1995 2000 2005 2010 2015 500 1000 1500 2000 Study year Absolute value of weighted effect size

r=0.07

• Same results whether or not you weight by the inverse of the sampling variance
• Same results when using untransformed effect sizes instead of absolute values
• The extreme correlations are mostly from meta-analyses with very few studies
• See the backup slides for additional analyses

Corr(abs(weighted effect size), study year) Frequency

• 1.0
• 0.5

0.0 0.5 1.0 5 10 15 20 25

• 2. Do our estimates of mean effect

size get more precise as more studies are published?

Precision of the cumulative mean effect size usually (not always) increases over time

Frequency

• 1.0
• 0.5

0.0 0.5 1.0 5 10 15 20 25 30

Cor(95% c.i. width, final study year)

• In 13% of meta-analyses, precision

decreases on avg. as more studies are performed!

• Often due to increasing among-

study heterogeneity

• 3. Does sampling error tend to

decline over time?

No general trend for sampling error to decrease (or increase) over time

1995 2000 2005 2010 2015 2 4 6 8

r=0.07 Study year Sampling variance of effect size

Corr(sampling variance, study year) Frequency

• 1.0
• 0.5

0.0 0.5 1.0 10 20 30 40 50

• 4. How large are among- and

within-study heterogeneity? And how do they change as more studies are conducted?

• Typically ~85% of the variance in effect size
• confirms Senior et al. 2016
• Heterogeneity becomes an increasingly important source of

variation as more studies are conducted

Summary

• Don’t worry about the decline effect
• Instead, worry about heterogeneity:
• Within and among studies
• Often keeps precision low, can prevent it from improving

much as more studies are conducted

• Anecdotally, we usually can’t explain much of it with

moderator variables

Ecologists often study small effects (and if they don’t look small, it’s often because they’re estimated imprecisely)

Mean effect size Frequency

• 2

2 4 6 5 10 15 20 25 30

• 2

2 4 6 1 2 3 4 Mean effect size se(mean effect size)

• 59% of mean effect sizes are significantly different from 0

A slight excess of both the decline effect, and its opposite?

Real data Permuted data

Corr(abs(weighted effect size), study year) Frequency

• 1.0
• 0.5

0.0 0.5 1.0 5 10 15 20 25 Corr(abs(weighted effect size), permuted study year) Frequency

• 1.0
• 0.5

0.0 0.5 1.0 10 20 30 40

10 20 30 40

• 0.5

0.0 0.5 1.0 1.5

Early studies often are not representative of later studies

Mean effect size (95% c.i.) Years since first study Final mean effect size estimate is 1.07 s.e. below the estimate from the first two studies (z=-1.07) Estimated mean effect size from first two studies, ± 1.96 s.e.

Frequency

• 10
• 5

5 10 10 20 30 40

z score of final mean effect size Only 71% of final means fall within the 95% c.i. from the first two studies

• Probably because no two studies are representative of the others!

Slight excess of both increases and decreases in sampling error over time?

Corr(sampling variance, study year) Frequency

• 1.0
• 0.5

0.0 0.5 1.0 10 20 30 40 50 Corr(sampling variance, permuted study year) Frequency

• 1.0
• 0.5

0.0 0.5 1.0 10 20 30 40 50

How does precision of the estimated mean effect size change over time as more studies are conducted?

10 15 20 25 30 35

• 1.0
• 0.5

0.0 0.5 1.0 4 6 8 10 12 14 0.0 0.2 0.4 0.6 0.8 1.0 2006 2008 2010 2012 2014 0.10 0.15 0.20 0.25 0.30 0.35 0.40

r=0.71

Cumulative mean effect size (95% c.i.) Cumulative mean ean effect size (95% c.i.) 95% c.i. width

1985 1990 1995 2000 2005 2010 2015 0.6 0.8 1.0 1.2 1.4 1.6 1.8

r=-0.95

95% c.i. width Final year Final year Years since first study Years since first study

Variation in effect size estimates is mostly among- and within-study heterogeneity, not sampling error

Total heterogeneity (%) Frequency 20 40 60 80 100 10 20 30 40 50

• Confirms Senior et al. 2016
• Heterogeneity increases with # of studies in the meta-analysis

Heterogeneity becomes a more important source of variation in effect size as more studies are conducted

• In 80% of meta-analyses, % of variation due to heterogeneity

increases over time as more studies are conducted

100 200 300 20 40 60 80 100 Studies Total heterogeneity (%)

How can we address heterogeneity?

• Lower our expectations?
• Maybe we should be happy to explain any heterogeneity?
• Ask narrower questions?
• Probably not
• Narrower questionfewer studiesreduced precision
• If you don’t know the sources of heterogeneity, you don’t know

which narrow question to ask

• Narrow questions often of narrow interest
• Distributed experiments?
• Learn to love heterogeneity?
• Increasing heterogeneity = diversifying our study subjects?
• Seek other sorts of generalizations besides those provided

by meta-analyses

• Fox (in press), Philosophical Topics