SLIDE 1 Publication bias – teaching materials
- The attached slides can be used to teach people
about publication bias
- There are notes beneath the slides with
suggestions of how they might be used
- Prepared by Professor Amanda Burls and available
from http://openaccess.city.ac.uk/13488/
- Amanda.Burls.1@city.ac.uk
- Publication bias by Dr Amanda Burls is licensed
under a Creative Commons Attribution 4.0 International License.
SLIDE 2
Who’s heard of friendly bacteria?
SLIDE 3
You are seeing a patient and she asks…
“Should I buy Activia™ to help with my tummy pain and improve my constipation?” You have 1 min inute to dis iscuss wit ith your neighbour what you would tell her?
1:00 0:59 0:58 0:57 0:56 0:55 0:54 0:53 0:52 0:51 0:50 0:49 0:48 0:47 0:46 0:45 0:44 0:43 0:42 0:41 0:40 0:39 0:38 0:37 0:36 0:35 0:34 0:33 0:32 0:31 0:30 0:29 0:28 0:27 0:26 0:25 0:24 0:23 0:22 0:21 0:20 0:19 0:18 0:17 0:16 0:15 0:14 0:13 0:12 0:11 0:10 0:09 0:08 0:07 0:06 0:05 0:04 0:03 0:02 0:01 End
SLIDE 4 Which of the following is closest to your answer?
1. 2. 3. 4. 5. 6. 7. 3% 21% 0% 39% 18% 5% 13%
- 1. Try it - it can help some people
2. You need to ask a doctor a question like that
- 3. Yes, but any live yoghurt will do
- 4. No, there’s no evidence it helps
5. I don’t know 6. I don’t know, but it can’t do any harm 7. I don’t know, but I’ll look up the evidence and tell you next week
SLIDE 5 What is the evidence?
- What sort of study design would be best for
answering this question?
SLIDE 6 Randomised Control Trial (RCT) of live yoghurt for Inflammatory Bowel Syndrome (IBS)
- P: Adults with IBS
- I: Activia (2 pots per day)
- C: Identical yoghurt with no live
bacteria (2 pots per day)
- O: Adequate symptom relief
- T: 12 weeks
- Funded by Danone
- Undertaken by independent
researchers at the University of Birmingham
SLIDE 7 RCT comparing live yoghurt with
- rdinary yoghurt IBS
- The trial ended in 2005
- What do you think the
results showed?
SLIDE 8
Results for primary outcome
2:00 1:59 1:58 1:57 1:56 1:55 1:54 1:53 1:52 1:51 1:50 1:49 1:48 1:47 1:46 1:45 1:44 1:43 1:42 1:41 1:40 1:39 1:38 1:37 1:36 1:35 1:34 1:33 1:32 1:31 1:30 1:29 1:28 1:27 1:26 1:25 1:24 1:23 1:22 1:21 1:20 1:19 1:18 1:17 1:16 1:15 1:14 1:13 1:12 1:11 1:10 1:09 1:08 1:07 1:06 1:05 1:04 1:03 1:02 1:01 1:00 0:59 0:58 0:57 0:56 0:55 0:54 0:53 0:52 0:51 0:50 0:49 0:48 0:47 0:46 0:45 0:44 0:43 0:42 0:41 0:40 0:39 0:38 0:37 0:36 0:35 0:34 0:33 0:32 0:31 0:30 0:29 0:28 0:27 0:26 0:25 0:24 0:23 0:22 0:21 0:20 0:19 0:18 0:17 0:16 0:15 0:14 0:13 0:12 0:11 0:10 0:09 0:08 0:07 0:06 0:05 0:04 0:03 0:02 0:01 End
SLIDE 9
Results for primary outcome
SLIDE 10
One participant in trial finally wrote to the investigator when these results had not been published five years later. This was the reply…
SLIDE 11 Sent: 19 January 2010 15:45 To: Dr Amanda Burls Subject: RE: Yoghurt trial Dear Amanda, The trial is not yet in press - this is in part due to the much longer than anticipated further analysis of the data at the funders request. In summary this was a negative trial - although both groups demonstrated benefit, those in the active product group did not show greater benefit and at times the difference actually favoured the control product….
SLIDE 12
Publication bias – the tendency not to publish negative findings
SLIDE 13
The trial was finally published eight years later (after pressure to do so)...
SLIDE 14 However, publication bias is
- Not only about not publishing
- Can also be about reporting outcomes that were
positive but not mentioning those that were not (even when they were the primary outcome)
- Putting more positive results in the Abstract of the
paper
- Distorting the Conclusions
SLIDE 15
Let’s compare your conclusions to those in the paper
SLIDE 16
Let’s compare our implications for practice with those of the authors
SLIDE 17 A biased conclusion - great for Danone!
- Activia was worse than the control yoghurt
- But, let’s recommend fermented dairy products
anyway!
- To back up such a recommendation randomized
control trials are needed showing that fermented dairy products help patients with IBS compared to not having them.
- Otherwise we can’t exclude bias or regression to
the mean (that people get better anyway, without treatment!)
SLIDE 18
Further slides for later classes on publication bias
SLIDE 19
Looking for bias in systematic reviews
SLIDE 20 A funnel plot
Size of study Estimate of truth
SLIDE 21 Funnel plots…
- …are scatter plots of treatment effect
estimated from individual studies (x axis) against a measure of each study’s sample size (y axis).
- The precision of the estimates of the
treatment effect increases as sample size increases.
- Effect estimates from small studies
scatter more widely at the bottom of the graph, with the spread narrowing among larger studies.
- In the absence of bias the plot should
resemble a symmetrical inverted funnel.
Size of study Treatment effect
SLIDE 22 A funnel plot
Size of study
SLIDE 23 A funnel plot
Size of study
SLIDE 24 A funnel plot
Size of study
SLIDE 25
Publication bias distorts results
SLIDE 26 A funnel plot
Size of study Favours treatment Favours control
SLIDE 27 A funnel plot
Size of study Favours treatment Favours control
SLIDE 28 Sources of asymmetry
- Publication bias
- Poor methodological quality of smaller studies
- True heterogeneity i.e. size of effect differs
according to study size
- for example, due to differences in the intensity of
interventions or differences in underlying risk between studies of different sizes
