Characterizing the long-term PM 2.5 - mortality response function: - - PowerPoint PPT Presentation

Characterizing the long-term PM2.5 - mortality response function: Comparing the strengths and weaknesses of research synthesis approaches

Neal Fann*, Elisabeth Gilmore & Katherine Walker*

* Usual institutional disclaimers 1

Why study PM2.5? Large estimates of benefits and costs of regulations

Source: http://www.whitehouse.gov/sites/default/files/omb/inforeg/2013_cb/draft_2013_cost_benefit_report.pdf.

total ann. (2002-2012) in billions $ 2

Measuring Clean Air Act Progress:

Costs vs. Benefits 1990 – 2020

(Section 812 Prospective Analysis)

Benefits are primarily from reduced mortality

• Mostly from

reductions in particulate matter, PM2.5

3

Hot issue: What is the PM2.5-mortality concentration response relationship?

• Magnitude of relationship between

exposure and response (e.g. relative risk)

• Shape of the function (e.g. linear, non-

linear, threshold effects, etc…)

• Level of confidence or uncertainty
• Likelihood that relationship is causal….

4

Background: Risk analyses have relied heavily

• n a few key epidemiologic cohort studies

100 200 300 400 500 600

Number of papers in Pub Med

Year Published All other Epidemiologic studies

Harvard 6-City Study (Dockery 1993) ACS study ( Pope et al. 2002) ACS study ( Pope et al. 1995)

5

Background: Publications on air pollution and particulate matter have exploded over time…

100 200 300 400 500 600

Number of Published papers in Pub Med

Year Published All other Epidemiologic studies

HEI reanalysis 6C &ACS study (Krewski, 2009) Harvard 6-City Study (Dockery 1993) ACS study ( Pope et al. 2002) ACS study ( Pope et al. 1995) Harvard 6-City Study (LePeule, 2012)

6

Where are we on the continuum from ignorance to perfect information?

No information Perfect information More Judgment More Knowledge 7

Compare 4 research synthesis approaches Approach

Examples Systematic reviews EPA’s Integrated Science Assessments (ISA) for Particulate Matter Formal elicitation of expert judgments

• EPA (2004) pilot PM2.5

elicitation of 5 experts

• EPA (2006) extended PM2.5

elicitation of 12 experts Meta-analysis Illustrative analyses using, pre- and post-2006 cohort studies Integrated exposure response assessment Shin et al. (in this series)

8

Systematic Reviews

• Application: EPA lntegrated Science

Assessments (ISA) for Particulate Matter

• 5-year reviews under the Clean Air Act
• Used for qualitative judgments about likelihood of a

causal relationship

• Informs CASAC decisions on National Ambient Air Quality

Standards (NAAQS)

• Basis for identification of studies most appropriate for

quantitative risk analyses

• Primary analysis - American Cancer Society (ACS) study
• Sensitivity analysis – Harvard Six City (H6C) study

9

Expert Judgment Elicitation

• Application: EPA pilot (2004) and extended

(2006) PM2.5 elicitation

• Multi-disciplinary expert selection through

stratified peer nomination process

• Briefing materials and detailed interview protocol
• Full-day interviews to elicit quantitative

assessments:

• Likelihood of a causal relationship, threshold
• % change in all-cause mortality per 1 µg/m3 decrease in

PM2.5 under a specific scenario

• “Credible intervals” (5th, 25th, 50th, 75th, 95th)

10

Illustrative meta-analyses

Studies available pre- 2006 (available to expert elicitation) Updates to pre- 2006 studies and new studies available up to mid-2013 Pooled hazard ratios % change in risk per 10 ug/m3 11

Evaluation approach #1 -Criteria

Broad Questions Attributes How valid are the methods and results?

• Appropriate disciplines involved
• Completeness of data and analysis
• Analytical methods appropriate to data
• Verification/Validation of methods
• Rigorous peer review

How transparent are the methods and results?

• How analysts are selected and represented
• How cognitive biases are dealt with
• Clarity of hypotheses, models, assumptions
• Exploration of variability and uncertainty in

inputs and results

How suited is are the data, methods, and results to the policy problem?

• Relevance
• External validity
• Transferability/Transportability

How suited is the method to the user needs and resource constraints?

• Ability to use existing data
• Value of output commensurate with costs
• Repeatability/Updateability
• Communicability

12

#2 - Illustrative Policy Application:

EPA Mercury and Air Toxics Rule (2011)

• Applied the C-R functions from the different methods to

the benefits analysis of the avoided PM2.5-related premature deaths due to air quality improvements

• Used air quality model results from the Regulatory

Impact Assessment (RIA) for the Mercury and Air Toxics Rules

• Combined with the population and incidence values from

the environmental Benefits Mapping and Analysis Program—Community Edition (BenMAP-CE) v0.63 (USEPA, 2013)

13

Observations: PM ISA

Strengths

• Essential starting point for any research synthesis method
• Systematic selection of studies – virtually all published studies
• Over time, has improved structured consideration of evidence
• Multi-disciplinary teams

Challenges

• Qualitative assessment
• Hard to know how disparate evidence is ultimately weighed and

integrated

• Methods for structured evaluation of evidence evolving (e.g.

Cochrane reviews, PRISMA, National Tox. Program - Office of Health Assessment and Translation (OHAT), EPA Next Gen Chemical Risk Assessment)

14

Observations: Expert Judgment Elicitation

Strengths

• Structured, explicit characterization of experts’ judgments about

what they know and don’t know

• Extent of and basis for inter-expert agreement or disagreement
• Quantitative estimates of the quantity of interest (e.g. causal

likelihood; existence of thresholds, and more complete assessment of uncertainty)

• Uses available evidence, even limited
• Multidisciplinary
• Independent, structured process for selection of experts

Challenges

• Assessing the quality of judgments: How well do experts estimate

the ‘truth’ and how well they know it?

• Dealing with ‘strategic’ judgments in highly political debates
• Capturing/communicating transparently the basis for judgments
• Whether and how best to combine experts
• Harder to update easily

15

Observations: Meta-Analyses

Strengths

• Structured, explicit approach for selection and combination of

evidence

• Draws strength from multiple studies
• Easy to test sensitivity to existing studies and updatable with new

information

• Simpler to communicate
• Transparent

Challenges

• Pooling estimates from studies can be problematic
• Can only reflect the existing studies, their biases and uncertainties

(e.g. standard errors)

16

Observations: Integrated Exposure Response

Strengths

• Structured, explicit approach for selecting and combining evidence
• Draws from multiple studies and types of exposures
• Quantitative estimates of the shape of the CR function over broader

concentration range and more complete assessment of uncertainty

• Has been tested against real-world results (e.g. China)
• Easier to conduct sensitivity analyses and to update with new

information

Challenges

• Requires a lot of compatible evidence from different exposures
• It is still a model
• Evidence for low level population exposures still limited
• Would benefit from replication/repetition/validation by others

with alternative assumptions and choices of data

17

Comparison of Avoided Premature Deaths Using Different C-R functions for the Mercury and Air Toxics Rule*

* Results for IER analysis not yet available 18

Summary

• Systematic, structured, critical review is essential to all these

research synthesis methods

• Growing number of guidelines in many disciplines
• Quantitative exploration and analyses needs to be suited to

the data and question of interest or decision

• The PM2.5 example shows that more data supports more

comprehensive and sophisticated analysis

• The wealth of data we have for PM2.5 is rare
• All of these methods require the considerable judgment of

scientists, individually and collectively

• Challenge is to recognize when analyses may be sensitive to

differences in judgment and might benefit from more structured examination

19