Efforts to Improve Data Transparency at the JBC Roger J. Colbran - - PowerPoint PPT Presentation

2

Efforts to Improve Data Transparency at the JBC

Roger J. Colbran Associate Editor

Credit: Amanda Fosang, Associate Editor

3

Why Improve Transparency?

• Irreproducibility of pre-clinical studies
• Negative impact on political and public

will to fund research…...

4

Reason for Irreproducibility

• Poor methodological descriptions
• Reagents not described adequately

and/or validated – antibodies, siRNAs, small molecules

• Inappropriate data interpretation

5

NIH Principles and Guidelines

http://www.nih.gov/research-training/rigor-reproducibility

• Rigorous statistical analysis
• Transparency in reporting
• Establish guidelines for:
• Image based data
• Western blots
• Descriptions of biological materials
• Antibodies: source, dilutions, validation
• Cell lines: source, authentication, mycoplasma
• Animals: species, source, strain, sex, age

6

Efforts Toward a Solution…...

Improve transparency of JBC papers:

1. Improve description of methods, including reagents 2. Clearly define reproducibility (technical and biological replicates)

7

JBC Policies and Guidelines

• 1. Experimental uncertainty, statistics
• 2. Graphical presentation of data
• 3. Western blots – presentation/quantitation

8

• 1. Experimental uncertainty, statistics

NIH:

• Require full reporting of statistics, including tests used,

exact value of N, dispersion/precision (e.g., mean, median, SD, SEM, confidence intervals).

• Inclusion and exclusion criteria: State criteria used for

exclusion of any data or subjects.

• Policies for statistical analysis should be included in

Information for Authors.

• Journals should have mechanisms to check for statistical

accuracy.

9

• 1. Experimental uncertainty, statistics

"There are three kinds of lies: lies, damned lies, and statistics."

• Attr. (by Mark Twain)

Benjamin Disraeli, British Prime Minister

Transparency is the key…....

10

• 1. Experimental uncertainty, statistics

JBC Instructions to Authors:

• Authors must include information on uncertainty and

reproducibility of data in figure legends.

• Authors must state numbers of independent samples

(biological replicates) and replicate samples (technical replicates) analyzed and report how many times each experiment was repeated.

• Experimental variability/precision should be reported by

standard deviation (SD) (strongly preferable), confidence intervals (CI). Standard error of the mean (SEM) may be acceptable when combined with visual scatter plots.

11

• 1. Experimental uncertainty, statistics

Data can be plotted to be misleading……

Motulsky, 2014. J. Pharmacol. Exp. Ther. 351:200

12

• 1. Experimental uncertainty, statistics

Instructions to Authors: Animal studies

• State whether or not animals and/or samples

analyzed were randomized, and indicate how.

• State whether studies were blinded or not. If so,

indicate the method of blinding.

• State whether any data were excluded. If so,

indicate the reason and/or criteria for exclusion.

13

• 1. Experimental uncertainty, statistics

http://xkcd.com/882 CC BY-NC 2.5 license.

14

• 2. Graphical presentation of data

Mean ± SEM bar graphs can be misleading……..

Weissgerber et al., 2015. PLOS Biology 13: e1002128

15

• 2. Graphical presentation of data

Mean ± SEM bar graphs can be misleading

Weissgerber et al., 2015. PLOS Biology 13: e1002128

16

• 2. Graphical presentation of data

Instructions to authors:

• Scatter plots strongly recommended for small data

sets; use box and whisker plots for large data sets, if necessary.

17

• 3. Western blots - antibodies

Instructions to authors:

• Define species of origin and source of all antibodies used,

including catalog/lot numbers.

• Describe how novel antibodies were generated, including

preparation/purification of epitope/antigen.

• Describe data supporting antibody specificity, including post-

translational modifications or neoepitopes.

• If possible, demonstrate loss of immunoreactivity following

genetic or other molecular modification to the antigen.

18

• 3. Western blots - presentation

Instructions to authors:

• Immunoblots should be cropped in a way that retains

information about antigen size, antibody specificity.

• Cropped images should ideally include positions of

molecular weight markers above AND below the band(s) of interest.

• Avoid assembling figures of blots by splicing lanes from

different sections of a gel. If blots must be spliced, borders must be clearly marked and explained in the figure legend.

19

• 3. Western blots - presentation

Keep original data and unprocessed scans!

20

• 3. Western blots – presentation

21

• 3. Western blots - quantitation

Instructions to authors:

• Explain how data were obtained, linearity of signal intensity with

antigen loading, and how protein loading was normalized. Some detection methods (e.g., ECL) have a very limited linear range.

• Strongly prefer normalizing signal intensity to total protein

loading (assessed by staining membranes for total protein). “House-keeping” proteins should not be used for normalization without evidence that manipulations do not affect expression.

• Phospho-specific antibody signals should be normalized to total

levels of the target protein.

Supported by a large literature making the same point.

22

• 3. Western blots

Instructions to authors:

During the editorial review process, authors may be asked to provide high resolution images of original immunoblots, quantification details, and antibody validation data.

Keep your original data and the unprocessed scans!

23

Implementation

• 1. Clearly define expectations/policies
• 2. Author education
• 3. Consistency in reviews (by EBMs!)

24

Implementation – an author’s response

“The revision process provided us an opportunity to learn more about using total protein as a loading control, and we are glad to learn that the total protein staining of the Western blots is a superior and simpler method. We will use this method in our

• ngoing and future experiments/projects.”