NMEC Working Group Wednesday, June 12, 2019 at 1:00-2:00pm Agenda - - PowerPoint PPT Presentation

NMEC Working Group

Wednesday, June 12, 2019 at 1:00-2:00pm

Agenda

• 1. Introduction/Welcome
• 2. Recap of Working Group Meetings
• 3. Draft Report Overview
• 4. Review Recommendations on Priority Buckets
• 5. Working Group Next Steps

Welcome

Coby Rudolph, CPUC

• Thank you!

– To Common Spark Consulting, PG&E and all working group participants

• Next from CPUC:

– Need your eyes to identify corrections. – Common Spark to finalize their report. – Report will be made public & posted on NMEC Working Group webpage – CPUC to issue draft population-level NMEC rules – Opportunity for formal comment from parties to proceeding. – CPUC to issue final population-level NMEC rules.

Recap of Activities

• Meeting 1: May 6
• Over 70 participants across government, PA, third-party, contractor, consultant,

NGO

• Definition and delineation of “population-level NMEC”
• 65+ Responses to the Survey/Work Group Sign-up
• Meeting 2: May 15
• Additional work on definition of Population-Level NMEC
• Determined three areas for developing guidance:
• Definition and Population Eligibility
• Modeling Guidance
• Process and Roles for Determining Savings
• Small Group work
• Developed straw proposals
• Small group calls held May 29-30th

Recap of Activities

• Meeting 3: June 4
• Update on straw proposals, guidance for population-level NMEC
• Break out groups to confirm, refine, and test recommendations; propose any

additional high-consensus items

• Report backs from small groups / full group feedback and discussion

Recap: Priority Buckets

Defining Population NMEC; Aggregate population eligibility Process, Roles, Review, and Evaluation

• Population-level vs. Site-level vs. Aggregate vs.

RCT/experimental

• Aggregating sites in the population using the same approach vs.

pooled approaches or another approach?

• Other Factors
• Permissible project types or site types, qualifying measures

(do they need to be the same?)

• Expected savings impact
• What is sufficient/needed to form a "population" – Quantity?

Level of statistical power?

• Significance of factors:
• Savings claim (program level)
• Cohort size
• Building type/use
• Building size/scale of energy use or savings
• Model fit
• Other factors?
• How to handle different approaches for pooled methodologies?
• The goal is to provide standard requirements for M&V plans,

that once approved by the PA, can be paid upon once the M&V implementation has been verified to match the pre- approved M&V.

• Need to balance the risk of the implementers and the

participants; what are factors that PAs and implementers should consider in balancing that risk?

• In the interim: “Payable” savings – may be (for a number of

factors) different than savings PAs claim

Recap: Priority Buckets

Modeling: Baseline, Normalization, Comparison Groups, Exogenous Factors, NREs, and Outlier Sites Baseline and Normalization

• What is normalized, in the context of population NMEC (Weather normalization? Other factors?)
• Identify factors up-front that need to be adjusted to achieve common operating conditions across time periods

Comparison Groups

• When and/or why is a comparison group appropriate/necessary?
• What risks/uncertainty does a comparison group mitigate? (e.g., exogenous factors?)
• What are appropriate criteria for evaluating a matching approach?

NREs, Exogenous, Outlier Sites

• When do NREs effectively cancel out in populations? What factors matter?
• What approaches for identifying and addressing NREs are appropriate within a population-level program? -
• NREs vs. Outlier sites?
• Outlier sites: What impact do outliers have at a population level? How are outlier sites identified and treated?
• What should happen when a site becomes disqualified (e.g., due to an EV or solar)?

Overview of Draft Report

• Review of meeting activities
• Summaries of May 6, 15, and June 4 meetings
• Prioritization of Issues
• Identification of Priority, Next Round, and Other Buckets
• Recommendations developed during June 4 meeting
• Documents changes in text discussed during meeting
• Next steps
• Rulebook updates and formal comment process
• Next round of topics for working group

Recommendations: Pop-Level NMEC Definition & Eligibility

Topic #1: Defining Population-level NMEC

• Population NMEC is an energy savings calculation approach in which results are based on energy usage data
• bserved at the meter, and aggregated across a portfolio/program/population rather than a modeled

engineering forecast or deemed value.

• Characteristics of Population NMEC Programs:
• For a Population NMEC program, savings may aggregate site-level estimates where all sites use the same

modeling methods; or savings may be pooled, modeling savings across a population.

• Population NMEC uses a consistent approach to measure savings across all sites within a specific program

means that the same data collection, processing, and analytical methods should be applied to all participating sites to obtain the aggregate result for a specific program.

• Data from all sites are collected and prepared for analysis the same way; same data collected from all sites,

and data are treated consistently (i.e., same rules to determine outliers). Values may differ across sites.

• Population NMEC programs are those in which savings are claimed for an aggregate or portfolio of sites

with similar characteristics.

Topic #2: Aggregate Population Eligibility

• 1. To use a population NMEC approach, the forecasted number of sites, projected savings, and projected baseline model

error for individual sites should be sufficient that fractional savings uncertainty (FSU) for the group of sites as a whole is not predicted to exceed 25% at a 90% confidence level, otherwise the implementer and program administrator should seek an exception from the CPUC. Commission staff, in collaboration with stakeholders, should re-evaluate the 25/90 precision/confidence threshold after NMEC programs have been implemented and operational for one year.

• 2. The Rulebook should specify that this eligibility threshold applies only to population-level NMEC at the whole-group level,

and not site-level NMEC or individual sites within a population-level NMEC group.

• 3. Implementers, program administrators, and evaluators will use only daily or monthly, and not hourly, data when

estimating the FSU.

• 4. Commission staff should conduct, or delegate to program administrators to conduct, a study on the reasonableness of

this threshold and alternate eligibility options.

• 5. Commission staff should allow implementers and/or program administrators to propose programs with a higher FSU or

lower confidence level, which would be subject to additional review by Commission staff. If such a program is proposed, implementers and/or program administrators would need to demonstrate how the threshold addresses risks to realizing savings.

Recommendations: Pop-Level NMEC Definition & Eligibility

Recommendations: Modeling and Methodology

• Topic #1: Model/Methodology Transparency and Access
• Any methodology including calculations used should be available for verification, replication, and
• evaluation. Methodology may be public or open-source, and at least available for the aforementioned

verification activities. Ideally, the methodology would have demonstrated performance based on a generally accepted testing methodology.

• Topic #2: M&V Plan Template
• The working group did not recommend an M&V template at this time. However, in the M&V plan,

program administrators and/or implementers should describe how raw data will be processed into a

• result. The M&V plan should also discuss why the program is appropriate for population-level NMEC,

and why the calculation methodology and variables used for normalization are appropriate for the program and type of sites treated.

Recommendations: Modeling and Methodology

Topic #3: M&V Thresholds for Population-level NMEC Programs

Criteria to consider in an M&V plan include:

• Normalizing for Weather and other Factors: How does the program normalize for weather? Does the program also

normalize for other factors? If so, how?

• Comparison Groups and Baseline: Does the program utilize a comparison group? If so, for what purposes? How is the

comparison group composed?

• Outlier Site Identification and Treatment: Under what condition(s) will a site be excluded from a population-based

program after enrollment, and who should get to decide?

• Non-Routine Events Identification and Treatment: Does the program allow for NREs and adjustments, and if so of what

magnitude, and what types of change(s) to building use or other factors will qualify for allowing a non-routine adjustment to be made? What type of documentation and verification will be required for a non-routine adjustment, what criteria will be used to determine whether the adjustment or treatment it is sufficient?

• Program Risks and Risk Management: How do the M&V approach, modeling, and other activities address risks of not

realizing savings or overpaying for savings? Do payable savings differ from claimable savings? If so, why is this appropriate and how do program activities or program design (e.g., site exclusion protocol established up-front, rules

• n eligibility, use of a comparison group or other basis for adjustments) address risk?

Recommendations: Process and Roles

Topic #1: Forecasting, Reporting, and Claiming Savings

Forecasting Savings: Prior to program launch, program administrators must forecast program savings for planning and cost-effectiveness

• purposes. In order to forecast Expected Useful Life (EUL) and demonstrate how the program would achieve projected impacts, forecasted

savings may include measure-level savings values and installation rates. Forecasted savings should be a best estimate that could be based

• n sources including DEER values, engineering estimates, information from prior program years, and/or other transparent sources as they

become available. Reporting and Claiming Savings: After program launch, program administrators report savings to the CPUC prior to formal evaluation, measurement, and verification (EM&V). Program-level savings should be reported in program administrators’ Quarterly and Annual Reports. NMEC savings claims are expected to be based on at least 12 months of post-installation usage data. However, in the year in which installation is completed, but before one year of post-installation data are available, options for CPUC to consider for reported savings include:

• Using the forecasted values identified for planning purposes, possibly discounted based on the predicted FSU;
• Updating forecasted values by extrapolating interim NMEC results (e.g., based on number of installations to date) and expenditures;
• Holding NMEC savings (and potentially costs) until one full year of post-installation data are available - note that current reporting

requirements prescribe that program administrator costs are reported in the year they are incurred. If a savings claim is held while program money is spent and reported, program administrators’ cost-effectiveness would be inaccurate. The CPUC may wish to set interim guidelines for claiming NMEC savings until a long-term process is identified.

Recommendations: Process and Roles

Topic #2: Data

Stakeholders should move towards the use of common data sets for program management, savings claims, and CPUC-led ex post evaluation (and where possible, forecasting). Program administrators should maintain and manage the data on each of their programs (including usage and other data). Data should flow from the implementer forward through the program chain to the evaluator (as opposed to relying on the evaluator’s data).

Topic #3: Pay-for-Performance

The CPUC should encourage pay for performance, an arrangement in which program administrators compensate implementers based on NMEC energy savings. However, there is no recommendation at this time

• n whether the CPUC should prescribe that a minimum proportion of implementer compensation must be

pay-for-performance. The broader issue related to this recommendation is to minimize program risk and that increasing pay-for-performance program designs would decrease risk to ratepayers. Overall, implementers and program administrators should be considering which parties are subject to risks of overspending for savings or underperformance of realized savings and how those risks could be minimized.

What next?

➔Working group participants review the draft report to ensure it

accurately represents working group outcomes

➔Corrections deadline: COB Tuesday 6/18 ➔And, the earlier the better! ➔Facilitation team will finalize the draft report and submit it to PG&E

and CPUC

➔Commission staff will update the Rulebook based on the draft report ➔Updated Rulebook will be formally issued for public comment

Thank you & Contact Info

• Coby Rudolph, CPUC
• Caroline Massad Francis, PG&E
• Katie Wu (report drafting)

Please send corrections to: wu.katie.c@gmail.com