Characterizing the long-term PM 2.5 - mortality response function: Comparing the strengths and weaknesses of research synthesis approaches Neal Fann*, Elisabeth Gilmore & Katherine Walker* 1 * Usual institutional disclaimers
Why study PM 2.5 ? Large estimates of benefits and costs of regulations total ann. (2002-2012) in billions $ 2 Source: http://www.whitehouse.gov/sites/default/files/omb/inforeg/2013_cb/draft_2013_cost_benefit_report.pdf .
Benefits are primarily from reduced mortality Mostly from • reductions in particulate matter, PM 2.5 Measuring Clean Air Act Progress: Costs vs. Benefits 1990 – 2020 (Section 812 Prospective Analysis) 3
Hot issue: What is the PM 2.5 -mortality concentration response relationship? • Magnitude of relationship between exposure and response (e.g. relative risk) • Shape of the function (e.g. linear, non- linear, threshold effects, etc…) • Level of confidence or uncertainty • Likelihood that relationship is causal…. 4
Background: Risk analyses have relied heavily on a few key epidemiologic cohort studies 600 All other 500 Epidemiologic Number of papers in Pub Med studies ACS study ( Pope et al. 400 ACS study 2002) ( Pope et al. 1995) 300 Harvard 6-City Study (Dockery 200 1993) 100 0 5 Year Published
Background: Publications on air pollution and particulate matter have exploded over time … Harvard 6-City Study (LePeule, 2012) 600 HEI reanalysis All other 6C &ACS study Number of Published papers in Pub Med (Krewski, 2009) 500 Epidemiologic studies 400 ACS study ACS study ( Pope et al. ( Pope et al. 2002) 1995) 300 Harvard 6-City Study 200 (Dockery 1993) 100 6 0 Year Published
Where are we on the continuum from ignorance to perfect information? Perfect information No information More Judgment More Knowledge 7
Compare 4 research synthesis approaches Approach Examples Systematic reviews EPA’s Integrated Science Assessments (ISA) for Particulate Matter Formal elicitation of • EPA (2004) pilot PM 2.5 expert judgments elicitation of 5 experts • EPA (2006) extended PM 2.5 elicitation of 12 experts Meta-analysis Illustrative analyses using, pre- and post-2006 cohort studies Integrated exposure Shin et al. (in this series) 8 response assessment
Systematic Reviews • Application: EPA lntegrated Science Assessments (ISA) for Particulate Matter • 5-year reviews under the Clean Air Act • Used for qualitative judgments about likelihood of a causal relationship • Informs CASAC decisions on National Ambient Air Quality Standards (NAAQS) • Basis for identification of studies most appropriate for quantitative risk analyses • Primary analysis - American Cancer Society (ACS) study • Sensitivity analysis – Harvard Six City (H6C) study 9
Expert Judgment Elicitation • Application: EPA pilot (2004) and extended (2006) PM 2.5 elicitation • Multi-disciplinary expert selection through stratified peer nomination process • Briefing materials and detailed interview protocol • Full-day interviews to elicit quantitative assessments: Likelihood of a causal relationship, threshold • % change in all-cause mortality per 1 µg/m 3 decrease in • PM 2.5 under a specific scenario 10 “Credible intervals” (5 th , 25 th , 50 th , 75 th , 95 th ) •
Illustrative meta-analyses Pooled hazard ratios % change in risk per 10 ug/m 3 Studies available pre- 2006 (available to expert elicitation) Updates to pre- 2006 studies and new studies available up to mid-2013 11
Evaluation approach #1 -Criteria Broad Questions Attributes Appropriate disciplines involved How valid are the methods and • Completeness of data and analysis results? • Analytical methods appropriate to data • Verification/Validation of methods • Rigorous peer review • How transparent are the methods How analysts are selected and represented • How cognitive biases are dealt with and results? • Clarity of hypotheses, models, assumptions • Exploration of variability and uncertainty in • inputs and results How suited is are the data, Relevance • External validity methods, and results to the policy • Transferability/Transportability problem? • Ability to use existing data How suited is the method to the • Value of output commensurate with costs user needs and resource • 12 Repeatability/Updateability • constraints? Communicability •
#2 - Illustrative Policy Application: EPA Mercury and Air Toxics Rule (2011) • Applied the C-R functions from the different methods to the benefits analysis of the avoided PM 2.5 -related premature deaths due to air quality improvements • Used air quality model results from the Regulatory Impact Assessment (RIA) for the Mercury and Air Toxics Rules • Combined with the population and incidence values from the environmental Benefits Mapping and Analysis Program—Community Edition (BenMAP-CE) v0.63 (USEPA, 2013) 13
Observations: PM ISA Strengths • Essential starting point for any research synthesis method • Systematic selection of studies – virtually all published studies • Over time, has improved structured consideration of evidence • Multi-disciplinary teams Challenges • Qualitative assessment • Hard to know how disparate evidence is ultimately weighed and integrated • Methods for structured evaluation of evidence evolving (e.g. Cochrane reviews, PRISMA, National Tox. Program - Office of 14 Health Assessment and Translation (OHAT), EPA Next Gen Chemical Risk Assessment)
Observations: Expert Judgment Elicitation Strengths • Structured, explicit characterization of experts’ judgments about what they know and don’t know • Extent of and basis for inter-expert agreement or disagreement • Quantitative estimates of the quantity of interest (e.g. causal likelihood; existence of thresholds, and more complete assessment of uncertainty) • Uses available evidence, even limited • Multidisciplinary • Independent, structured process for selection of experts Challenges • Assessing the quality of judgments: How well do experts estimate the ‘truth’ and how well they know it? • Dealing with ‘strategic’ judgments in highly political debates • Capturing/communicating transparently the basis for judgments 15 • Whether and how best to combine experts • Harder to update easily
Observations: Meta-Analyses Strengths • Structured, explicit approach for selection and combination of evidence • Draws strength from multiple studies • Easy to test sensitivity to existing studies and updatable with new information • Simpler to communicate • Transparent Challenges • Pooling estimates from studies can be problematic • Can only reflect the existing studies, their biases and uncertainties (e.g. standard errors) 16
Observations: Integrated Exposure Response Strengths • Structured, explicit approach for selecting and combining evidence • Draws from multiple studies and types of exposures • Quantitative estimates of the shape of the CR function over broader concentration range and more complete assessment of uncertainty • Has been tested against real-world results (e.g. China) • Easier to conduct sensitivity analyses and to update with new information Challenges • Requires a lot of compatible evidence from different exposures • It is still a model • Evidence for low level population exposures still limited 17 • Would benefit from replication/repetition/validation by others with alternative assumptions and choices of data
Comparison of Avoided Premature Deaths Using Different C-R functions for the Mercury and Air Toxics Rule * 18 * Results for IER analysis not yet available
Summary • Systematic, structured, critical review is essential to all these research synthesis methods • Growing number of guidelines in many disciplines • Quantitative exploration and analyses needs to be suited to the data and question of interest or decision • The PM 2.5 example shows that more data supports more comprehensive and sophisticated analysis • The wealth of data we have for PM 2.5 is rare • All of these methods require the considerable judgment of scientists, individually and collectively • Challenge is to recognize when analyses may be sensitive to 19 differences in judgment and might benefit from more structured examination
Recommend
More recommend
