Part 1 Part 1 I ntroduction Review of I ntroduction Review of I - - PowerPoint PPT Presentation

Part 1 I ntroduction Review of Part 1 I ntroduction Review of I ntroduction, Review of Agenda, Agenda Revisions I ntroduction, Review of Agenda, Agenda Revisions

2

Part 2 Review of ACT 129 and Other Part 2 Review of ACT 129 and Other Review of ACT 129 and Other Historical Documents Review of ACT 129 and Other Historical Documents

3

Part 3 Part 3 Part 3 Review of Objectives for the Part 3 Review of Objectives for the Statewide Evaluation and Contents of the Audit Plan Statewide Evaluation and Contents of the Audit Plan Contents of the Audit Plan Contents of the Audit Plan

4

Act 129 Mandate Act 129 Mandate Act 129 Mandate Act 129 Mandate

The Energy Efficiency and Conservation Program shall include: The Energy Efficiency and Conservation Program shall include:

(2) An evaluation process, including a process to monitor and verify data collection quality assurance and (2) An evaluation process, including a process to monitor and verify data collection quality assurance and and verify data collection, quality assurance and results of each plan and the program. and verify data collection, quality assurance and results of each plan and the program. (3) An analysis of the cost and benefit of each plan submitted under subsection (B) in accordance with a total resource cost test approved by the Commission (3) An analysis of the cost and benefit of each plan submitted under subsection (B) in accordance with a total resource cost test approved by the Commission total resource cost test approved by the Commission. total resource cost test approved by the Commission.

5

Source: House Bill No. 2200, page 51

Objectives of Statewide l i Objectives of Statewide l i Evaluation Evaluation

• Identify the cost-effectiveness of each plan in accordance with
• Identify the cost-effectiveness of each plan in accordance with

the Commission adopted Total Resource Cost Test Manual.

• Provide reasonable assurance that the claimed measures are

being properly installed and utilized. the Commission adopted Total Resource Cost Test Manual.

• Provide reasonable assurance that the claimed measures are

being properly installed and utilized.

• Provide reasonable assurance that the installed measures are
• btaining the claimed energy savings or reductions in accordance

with the Commission adopted Technical Reference Manual or

• Provide reasonable assurance that the installed measures are
• btaining the claimed energy savings or reductions in accordance

with the Commission adopted Technical Reference Manual or metered savings.

• Identify areas for improvement.
• Identify best practices that may be implemented on a

metered savings.

• Identify areas for improvement.
• Identify best practices that may be implemented on a

y p y p going-forward basis. y p y p going-forward basis.

6

Requirements for Contents of the A dit Pl Requirements for Contents of the A dit Pl Audit Plan Audit Plan

1. High-level summary of audit plan. 2 EE&C measures to be audited and the plan measure logic/theory 1. High-level summary of audit plan. 2 EE&C measures to be audited and the plan measure logic/theory 2. EE&C measures to be audited and the plan measure logic/theory. 3. The type and format of documentation identifying EDC plan expenditures. 4. A description of the audit objectives, audit questions and audit rigor level for each EDC plan. 5 A description of the methodologies procedures and data tracking systems to be used by 2. EE&C measures to be audited and the plan measure logic/theory. 3. The type and format of documentation identifying EDC plan expenditures. 4. A description of the audit objectives, audit questions and audit rigor level for each EDC plan. 5 A description of the methodologies procedures and data tracking systems to be used by 5. A description of the methodologies, procedures and data tracking systems to be used by the Contractor to verify the impact evaluations and project verifications for each plan, including data gathering, sampling and analysis methods. 6. Audit approaches to be used for each plan in accordance with the TRM or other approved EM&V plans 5. A description of the methodologies, procedures and data tracking systems to be used by the Contractor to verify the impact evaluations and project verifications for each plan, including data gathering, sampling and analysis methods. 6. Audit approaches to be used for each plan in accordance with the TRM or other approved EM&V plans EM&V plans. 7. Audit approaches to be used for each plan in accordance with approved custom measures. 8. Strategies to conduct audit studies coordinated on a statewide basis, such that results can be extrapolated back to individual EDC EE programs. 9 Verification and due diligence procedures for site inspections and validating information EM&V plans. 7. Audit approaches to be used for each plan in accordance with approved custom measures. 8. Strategies to conduct audit studies coordinated on a statewide basis, such that results can be extrapolated back to individual EDC EE programs. 9 Verification and due diligence procedures for site inspections and validating information 9. Verification and due diligence procedures for site inspections and validating information provided by EDCs.

• 10. Data and information needed from EDCs and other sources.
• 11. Budget, schedule and manpower plan for Contractor.

9. Verification and due diligence procedures for site inspections and validating information provided by EDCs.

• 10. Data and information needed from EDCs and other sources.
• 11. Budget, schedule and manpower plan for Contractor.

7

Source: PA PUC RFP for Statewide Evaluator , page 31

Value of I nformation (VOI ) Value of I nformation (VOI )

• What is the value of

information and data?

• What is the value of

information and data?

• What is the cost? What is it

worth? What is the return?

• VOI balance cost and rigor
• What is the cost? What is it

worth? What is the return?

• VOI balance cost and rigor
• Deterministic sampling
• Reduction of uncertainty
• Adjust sample size according to

the cost of changes compared to

• Deterministic sampling
• Reduction of uncertainty
• Adjust sample size according to

the cost of changes compared to the cost of changes compared to changes in value

• Sampling plans guided by

expectations

the cost of changes compared to changes in value

• Sampling plans guided by

expectations e pectat o s

• Audits structured to test the

validity of reported results – reasonable scientific checks e pectat o s

• Audits structured to test the

validity of reported results – reasonable scientific checks

8

Value of I nformation (VOI ) Value of I nformation (VOI ) ( ) ( )

• Actual reported impacts may
• Actual reported impacts may

vary by program and EDC – VOI considerations employed when evaluation efforts may b l d i k vary by program and EDC – VOI considerations employed when evaluation efforts may b l d i k be large compared to risk, funding and impacts

• Frequent stratification

be large compared to risk, funding and impacts

• Frequent stratification

Frequent stratification scheme – weight the sample towards significant attributes Frequent stratification scheme – weight the sample towards significant attributes

• Greater focus on impacts

with greater uncertainty – sensitivity analysis

• Greater focus on impacts

with greater uncertainty – sensitivity analysis

9

Timetable for Development of A dit Pl Timetable for Development of A dit Pl Audit Plan Audit Plan

• Draft – November 1, 2009
• Draft – November 1, 2009

,

• Final – December 2, 2009

,

• Final – December 2, 2009

10

Table of Contents of Audit Plan Table of Contents of Audit Plan Table of Contents of Audit Plan Table of Contents of Audit Plan

Section 1 Introduction and Purpose of the Audit Plan Section 1 Introduction and Purpose of the Audit Plan Section 2 Requirements for EDC Evaluation Section 3 Audit Plan Section 2 Requirements for EDC Evaluation Section 3 Audit Plan Section 4 EDC Reporting Guidelines Section 5 EDC Data Requirements Section 4 EDC Reporting Guidelines Section 5 EDC Data Requirements Section 6 Statewide Evaluator Reporting Activities Section 7 Expected Deadlines Section 6 Statewide Evaluator Reporting Activities Section 7 Expected Deadlines p

11

Part 4 Development of Overall Part 4 Development of Overall Development of Overall Statewide Evaluation Plan Development of Overall Statewide Evaluation Plan

12

Topics to be Discussed Topics to be Discussed Topics to be Discussed Topics to be Discussed

• Procedures for measurement and verification
• Procedures for measurement and verification
• Procedures for measurement and verification
• Procedures for impact evaluation
• Procedures for measurement and verification
• Procedures for impact evaluation
• Procedures for process evaluation
• Procedures for process evaluation
• Data reporting requirements
• Data reporting requirements

13

Overview of Evaluation Plans S b itt d b EDC Overview of Evaluation Plans S b itt d b EDC Submitted by EDCs Submitted by EDCs

• The EDC submitted plans on July 15, 2009
• The EDC submitted plans on July 15, 2009

The EDC submitted plans on July 15, 2009

• The PUC is in process of reviewing the plans
• The EDC plans provide highlights of the EDC

The EDC submitted plans on July 15, 2009

• The PUC is in process of reviewing the plans
• The EDC plans provide highlights of the EDC
• The EDC plans provide highlights of the EDC

evaluation approach

• The EDC plans provide highlights of the EDC

evaluation approach

14

I mportant Aspects of l i l d i i i I mportant Aspects of l i l d i i i Evaluation Plans and Activities Evaluation Plans and Activities

• Verification
• Verification
• M&V Option Selection
• Source of Stipulated Data
• M&V Option Selection
• Source of Stipulated Data
• Baseline Definition
• Monitoring Duration

d h

• Baseline Definition
• Monitoring Duration

d h

• Adjustments – Weather, Economic, etc.
• Calibration Criteria
• Additional Provisions
• Adjustments – Weather, Economic, etc.
• Calibration Criteria
• Additional Provisions
• Additional Provisions
• Additional Provisions

15

Review of Best Practices Review of Best Practices Review of Best Practices Review of Best Practices

• National Action Plan for Energy Efficiency
• National Action Plan for Energy Efficiency

National Action Plan for Energy Efficiency

• Statewide Impact Evaluations

– New York

National Action Plan for Energy Efficiency

• Statewide Impact Evaluations

– New York New York – Wisconsin – California New York – Wisconsin – California California

• Other Program Evaluations Metrics

– Cost-effectiveness: Connecticut California

• Other Program Evaluations Metrics

– Cost-effectiveness: Connecticut Cost effectiveness: Connecticut Cost effectiveness: Connecticut

16

National Action Plan for Energy Effi i G l E l ti P National Action Plan for Energy Effi i G l E l ti P Efficiency – General Evaluation Process Efficiency – General Evaluation Process

• Set evaluation objectives in the context of the program
• Set evaluation objectives in the context of the program

j p g policy objectives

• Select an approach, define baseline scenarios and prepare

a plan that takes into account the critical issues j p g policy objectives

• Select an approach, define baseline scenarios and prepare

a plan that takes into account the critical issues a plan that takes into account the critical issues

• Compare energy use and demand before and after the

program is implemented to determine energy savings and calc lated a oided emissions a plan that takes into account the critical issues

• Compare energy use and demand before and after the

program is implemented to determine energy savings and calc lated a oided emissions calculated avoided emissions

• Report the evaluation results and, as appropriate, work

with program administrators to implement calculated avoided emissions

• Report the evaluation results and, as appropriate, work

with program administrators to implement recommendations for current or future program improvements recommendations for current or future program improvements

17

Source: Model Energy Efficiency Program Impact Evaluation Guideline, a resource of the National Action Plan for Energy Efficiency, November 2007

National Action Plan for Energy Effi i T i l I t R t d National Action Plan for Energy Effi i T i l I t R t d Efficiency – Typical I mpact Reported Efficiency – Typical I mpact Reported

• Estimates of Gross Savings
• Estimates of Gross Savings

– M&V methods used to determine savings from sample of projects to be applied to all projects in the program – Deemed Savings based on historical and verified data, applied to conventional measures – M&V methods used to determine savings from sample of projects to be applied to all projects in the program – Deemed Savings based on historical and verified data, applied to conventional measures – Statistical analyses of large volumes of metered energy usage data

• Estimates of Net Savings

– Self-reporting surveys – Statistical analyses of large volumes of metered energy usage data

• Estimates of Net Savings

– Self-reporting surveys – Statistical models (i.e., compare usage and knowledge of participants and non-participants) – Stipulated net-to-gross ratios

• Estimate of Co-Benefits (e.g., avoided air emissions)

– Statistical models (i.e., compare usage and knowledge of participants and non-participants) – Stipulated net-to-gross ratios

• Estimate of Co-Benefits (e.g., avoided air emissions)

Estimate of Co Benefits (e.g., avoided air emissions) – Emissions factors – Emissions scenario analyses (e.g., computer models) Estimate of Co Benefits (e.g., avoided air emissions) – Emissions factors – Emissions scenario analyses (e.g., computer models)

18

Source: Model Energy Efficiency Program Impact Evaluation Guideline, a resource of the National Action Plan for Energy Efficiency, November 2007

New York: E l ti P d t New York: E l ti P d t Evaluation Products Evaluation Products

• Impact evaluation plans specific to each
• Impact evaluation plans specific to each

Impact evaluation plans specific to each program type

• Annual evaluation reports provide gross and

Impact evaluation plans specific to each program type

• Annual evaluation reports provide gross and

Annual evaluation reports provide gross and net energy savings

– At the portfolio level,

Annual evaluation reports provide gross and net energy savings

– At the portfolio level, – For the largest energy savors, – By sector, and – For the largest energy savors, – By sector, and – By specific program. – By specific program.

19

Source: New York Energy $mart Program Evaluation and Status Report: Year Ending December 31, 2008. Report to the Public Service Commission; Final Report – May 2009.

New York: I t E l ti P t l New York: I t E l ti P t l I mpact Evaluation Protocols I mpact Evaluation Protocols

Objectives & Activities Objectives & Activities Savings Estimates Savings Estimates Objectives & Activities Objectives & Activities

• Research Objectives – Determine:

– Realization rates for gross savings

• Research Objectives – Determine:

– Realization rates for gross savings

• Billing Analysis
• On-site measurements

d

• Billing Analysis
• On-site measurements

d

Savings Estimates Savings Estimates

savings – Attribution – Precision and bias Activities savings – Attribution – Precision and bias Activities

• Deemed savings
• Engineering models
• Energy simulations
• Deemed savings
• Engineering models
• Energy simulations
• Activities

– Gross impacts – Demand savings N i t

• Activities

– Gross impacts – Demand savings N i t – Non-energy impacts – Attribution – Non-energy impacts – Attribution

20

Source: New York Energy $mart Program Evaluation and Status Report: Year Ending December 31, 2008. Report to the Public Service Commission; Final Report – May 2009.

New York: Examples of Data Requirements New York: Examples of Data Requirements Examples of Data Requirements Examples of Data Requirements

• Project Level Information:

– Address & contact information for site-owner and/or contractor Measure Level Information:

• Project Level Information:

– Address & contact information for site-owner and/or contractor Measure Level Information:

• Measure Level Information:

– Description, quantity installed, energy savings, demand savings, measure life, installation costs

• Participant Level Information:
• Measure Level Information:

– Description, quantity installed, energy savings, demand savings, measure life, installation costs

• Participant Level Information:
• Participant Level Information:

– Base measure characteristics and/or statistics

• Other Information:

– Utility consumption data for participant

• Participant Level Information:

– Base measure characteristics and/or statistics

• Other Information:

– Utility consumption data for participant – Utility consumption data for participant – Utility consumption data for comparable non-participant – Outside influences – weather, economic conditions, etc.

• Sample Size:

– Utility consumption data for participant – Utility consumption data for comparable non-participant – Outside influences – weather, economic conditions, etc.

• Sample Size:
• Sample Size:

– Statistically significant samples as appropriate

• Sample Size:

– Statistically significant samples as appropriate

21

Source: New York Energy $mart Program Evaluation and Status Report: Year Ending December 31, 2008. Report to the Public Service Commission; Final Report – May 2009.

New York: Summary of Evaluation Results New York: Summary of Evaluation Results y o

• y o
• Benefits

Through Year- End 2004 Through Year- End 2005 Through Year- End 2006 Through Year- End 2007 Through Year- End 2008

Electricity Savings from Energy Efficiency and On-Site Generation (Annual GWh) 1,400 1,950 2,350 3,070 3,220 P k D d R d ti (MW) 860 1 040 1 113 1 200 1 275 Peak Demand Reduction (MW) 860 1,040 1,113 1,200 1,275 Permanent Measures (MW 325 445 495 650 700b Curtailable 1 535 595 618 550 575 Net Fuel Savings (Annual MMBTu) 2 600 000 4 000 000 4 049 000 4 460 000 5 400 000 Net Fuel Savings (Annual MMBTu) 2,600,000 4,000,000 4,049,000 4,460,000 5,400,000 Annual Energy Bill Savings to Participating Customers ($ Million) $195 $275 $330 $570 $590 Renewable Energy Generation (Annual GWh) 102 103 105 106 106 Net Additional Jobs2 2,500 3,100 3,600 4,500 4,900 Nox Emissions Reductions (Annual Tons)3 1,280 1,750 2,060 2,570 2,800 SO2 Emissions Reductions (Annual Tons)3 2,320 3,170 3,800 4,720 5,120 CO2 Emissions Reductions (Annual Tons)3 1,000,000 1,400,000 1,600,000 2,000,000 2,200,000 Equivalent number of cars removed from NY roadways 200,000 275,000 320,000 400,000 435,000

22

Source: New York Energy $mart Program Evaluation and Status Report: Year Ending December 31, 2008. Report to the Public Service Commission; Final Report – May 2009.

Wisconsin: I mpact Evaluation Guidelines Wisconsin: I mpact Evaluation Guidelines I mpact Evaluation Guidelines I mpact Evaluation Guidelines

Business Programs Business Programs Residential Programs Residential Programs

I t kW d kWh i I t kW d kWh i

• Impacts – kW and kWh savings
• Impacts – kW and kWh savings
• Impact – kW and kWh savings
• Determine verified gross and net

impacts for measures implemented

• Gross savings – tracked by measure
• Impact – kW and kWh savings
• Determine verified gross and net

impacts for measures implemented

• Gross savings – tracked by measure
• Impacts

kW and kWh savings

• Direct and indirect impact analysis
• Gross savings – determined by

deemed savings and engineering i

• Impacts

kW and kWh savings

• Direct and indirect impact analysis
• Gross savings – determined by

deemed savings and engineering i g y

• Gross savings adjustment factor &

attribution factor determined at sector level

• Estimate first year net savings (Y1NS)

g y

• Gross savings adjustment factor &

attribution factor determined at sector level

• Estimate first year net savings (Y1NS)

review

• Program specific impacts

review

• Program specific impacts

Energy I mpacts Reported By: Energy I mpacts Reported By:

• Estimate first year net savings (Y1NS)

and life cycle net savings (LCNS)

• Estimate first year net savings (Y1NS)

and life cycle net savings (LCNS)

Reported By: Reported By:

• Program area,
• Specific Program,

E d t h l

• Program area,
• Specific Program,

E d t h l

• End-use technology,
• Geographic Region – County,

Service Territory, Assembly, Senate District

• End-use technology,
• Geographic Region – County,

Service Territory, Assembly, Senate District

23

Source: State of Wisconsin, Public Service Commission of Wisconsin – Focus on Energy Evaluation, Evaluation Calendar Year 2009, Detailed Evaluation Plans. Final: April 16, 2009. Prepared by: The Focus on Energy Evaluation Team.

Wisconsin: Sampling Steps i Wisconsin: Sampling Steps i – Business Programs – – Business Programs –

• Sample stratified by:

Sector and end use;

• Sample stratified by:

Sector and end use; – Sector and end-use; – State facility; – High-level technology type: Custom incentive, CFL, non-CFL deemed, prescriptive incentive non-deemed; – Sector and end-use; – State facility; – High-level technology type: Custom incentive, CFL, non-CFL deemed, prescriptive incentive non-deemed; – Analysis cost category (all measures/projects assigned to cost category based

• n size, type and measure complexity); and

– Fuel source

• Each sample cell assigned to:

– Analysis cost category (all measures/projects assigned to cost category based

• n size, type and measure complexity); and

– Fuel source

• Each sample cell assigned to:
• Each sample cell assigned to:

– Engineering Review: gross savings verification and net-to-gross analysis – Automated Survey: net-to-gross analysis and installation confirmation

• Field resources allocated across all sectors
• Each sample cell assigned to:

– Engineering Review: gross savings verification and net-to-gross analysis – Automated Survey: net-to-gross analysis and installation confirmation

• Field resources allocated across all sectors

d

• u

a o a d a

• a
• Sample size by sector based on contribution to Business Programs gross

savings and likely variance of the gross savings adjustment factor

d

• u

a o a d a

• a
• Sample size by sector based on contribution to Business Programs gross

savings and likely variance of the gross savings adjustment factor

24

Source: State of Wisconsin, Public Service Commission of Wisconsin – Focus on Energy Evaluation, Evaluation Calendar Year 2009, Detailed Evaluation Plans. Final: April 16, 2009. Prepared by: The Focus on Energy Evaluation Team.

Wisconsin: Data Requirements i Wisconsin: Data Requirements i – Business Programs – – Business Programs –

• For deemed measures:
• For deemed measures:

– Application documentation – no. units installed, deemed algorithm parameters, measure qualification info. – Invoices that verify no. of units installed and measure qualification i f – Application documentation – no. units installed, deemed algorithm parameters, measure qualification info. – Invoices that verify no. of units installed and measure qualification i f info. – Energy Advisor impact statements

• For non-deemed measures:

A li ti d t ti d i i id tif d d ib info. – Energy Advisor impact statements

• For non-deemed measures:

A li ti d t ti d i i id tif d d ib – Application documentation and invoices – identify and describe project and equipment installed – Documentation of savings calculations – supporting docs. For assumptions and/or calculation methodology – Application documentation and invoices – identify and describe project and equipment installed – Documentation of savings calculations – supporting docs. For assumptions and/or calculation methodology assumptions and/or calculation methodology – Energy Advisor impact statements assumptions and/or calculation methodology – Energy Advisor impact statements

25

Source: State of Wisconsin, Public Service Commission of Wisconsin – Focus on Energy Evaluation, Evaluation Calendar Year 2009, Detailed Evaluation Plans. Final: April 16, 2009. Prepared by: The Focus on Energy Evaluation Team.

Wisconsin: Savings Estimates – Residential Programs – Wisconsin: Savings Estimates – Residential Programs – Residential Programs Residential Programs

• Energy Star Lighting

– Deemed savings, no. of program CFLs sold, installation rates

A li d Pl L d

• Energy Star Lighting

– Deemed savings, no. of program CFLs sold, installation rates

A li d Pl L d

• Appliance and Plug Loads

– Deemed savings from WI water heater research , DOE, CA, etc.

• Apartment and Condo Efficiency Services

– M&V – measure installation and use characteristics (e g quantities equipment

• Appliance and Plug Loads

– Deemed savings from WI water heater research , DOE, CA, etc.

• Apartment and Condo Efficiency Services

– M&V – measure installation and use characteristics (e g quantities equipment M&V measure installation and use characteristics (e.g., quantities, equipment efficiencies, operating hours) – Sample Sizes

• Existing buildings in-unit direct install – randomly selected ensuring projects w/ greatest savings

are sampled with certainty

M&V measure installation and use characteristics (e.g., quantities, equipment efficiencies, operating hours) – Sample Sizes

• Existing buildings in-unit direct install – randomly selected ensuring projects w/ greatest savings

are sampled with certainty

• Existing buildings contracted energy advisors – dependent on no. of projects completed
• New constructions – likely to be a census (all measures are sampled, although counts for

individual measures are low)

• Neighborhood Efficiency Pilot Program

S i t b d t i d th h ti i t l

• Existing buildings contracted energy advisors – dependent on no. of projects completed
• New constructions – likely to be a census (all measures are sampled, although counts for

individual measures are low)

• Neighborhood Efficiency Pilot Program

S i t b d t i d th h ti i t l – Savings to be determined through participant sample

• No energy impact estimates proposed for current evaluation period

– Energy Star Homes, Home Performance w/ Energy Star, Efficient Heating and Cooling. Targeting Home Performance w/ Energy Star – Savings to be determined through participant sample

• No energy impact estimates proposed for current evaluation period

– Energy Star Homes, Home Performance w/ Energy Star, Efficient Heating and Cooling. Targeting Home Performance w/ Energy Star

26

Source: State of Wisconsin, Public Service Commission of Wisconsin – Focus on Energy Evaluation, Evaluation Calendar Year 2009, Detailed Evaluation Plans. Final: April 16, 2009. Prepared by: The Focus on Energy Evaluation Team.

Wisconsin: Summary of Evaluation Results Wisconsin: Summary of Evaluation Results y

Table 2-1d. All Programs: Tracked Energy Impacts -Annual First Year Program to Date (July 1, 2001–December 31, 2008)

Annual kWh Saved kW Reduction Annual Therms Saved Gross Verified Gross Verified Net Gross Verified Gross Verified Net Gross Verified Gross Verified Net The 18-month Contract Period (July 1, 2007–December 31, 2008) Total Saved 625,335,422 591,021,366 393,477,991 109,615 103,184 64,867 28,698,506 26,623,540 14,758,518 Business 433,653,604 403,836,023 240,728,194 91,286 85,387 50,340 22,145,786 20,248,021 10,796,685 Residential 179,864,369 180,197,195 148,799,826 16,583 16,606 13,824 4,143,595 4,146,014 3,278,792 Renewables 11,817,448 6,988,149 3,949,971 1,746 1,190 703 2,409,124 2,229,506 683,041 FY07 (July 1, 2006–June 30, 2007) Total Saved 260,254,357 238,189,767 162,763,097 43,911 41,008 27,979 15,914,328 13,694,194 8,520,818 Business 157,130,223 151,014,314 98,462,584 33,460 32,282 21,776 13,658,924 11,513,743 6,809,760 Residential 94,250,649 78,656,907 63,883,459 8,913 6,855 5,852 1,566,780 1,506,977 1,394,290 Renewables 8,873,486 8,518,546 417,054 1,538 1,871 351 688,625 673,475 316,767 FY06 (July 1, 2005–June 30, 2006) Total Saved 234,862,916 218,883,777 166,124,214 44,348 41,469 30,494 13,606,641 13,042,576 6,311,815 Business 131,911,853 131,863,071 91,697,275 28,366 28,309 19,042 9,674,445 9,418,597 4,365,774 Residential 90,185,421 73,961,454 72,078,061 14,066 11,286 11,066 1,858,471 1,573,064 1,489,822 Renewables 12,765,642 13,059,252 2,348,878 1,916 1,874 385 2,073,725 2,050,914 456,220 FY05 (July 1, 2004–June 30, 2005) Total Saved 276,013,670 214,892,987 152,928,433 41,103 35,913 22,850 9,615,956 9,175,217 5,232,659 Business 143,707,711 110,718,465 55,861,214 27,083 20,906 9,972 7,045,122 7,106,161 3,463,969 Residential 112,935,136 82,266,121 92,805,638 11,027 11,745 11,893 2,123,643 1,725,612 1,680,593 Renewables 19,370,823 21,908,401 4,261,581 2,993 3,262 985 447,192 343,443 88,097 FY04 (July 1, 2003–June 30, 2004) Total Saved 282,836,318 228,323,107 168,974,239 44,697 37,696 26,986 13,556,180 14,467,224 12,436,908 Business 156,470,715 137,364,604 79,935,444 28,607 23,544 13,187 11,639,771 12,613,286 10,614,752 Residential 125,867,708 90,474,549 88,589,694 15,890 13,932 13,604 1,824,949 1,640,105 1,622,408 Renewables 497,895 483,954 449,101 199 220 195 91,460 213,833 199,749 FY03 (July 1, 2002–June 30, 2003) Total Saved 230,845,735 221,681,741 145,617,986 38,144 35,777 23,213 7,935,747 8,086,332 5,523,996 Business 129,814,716 128,323,629 62,483,249 23,353 21,385 10,551 6,258,454 6,196,249 3,642,577 Residential 97,358,904 89,638,260 80,130,947 14,066 13,788 12,184 1,677,293 1,890,084 1,881,419 Renewables 3,672,115 3,719,852 3,003,790 725 604 478 0 0 0 FY02 (July 1, 2001–June 30, 2002) Total Saved 54,507,968 56,484,523 39,048,723 10,935 11,705 8,163 3,673,296 2,652,791 1,616,168 Business 30,559,935 30,532,158 18,764,493 6,966 7,036 4,397 2,760,541 1,740,729 785,725 Residential 23,948,033 25,952,365 20,284,229 3,969 4,668 3,766 912,755 912,062 830,443 Note: The numbers in this table are annual first year (not lifecycle).

27

Source: State of Wisconsin, Public Service Commission of Wisconsin – Focus on Energy Evaluation, Evaluation Calendar Year 2009, Detailed Evaluation Plans. Final: April 16, 2009. Prepared by: The Focus on Energy Evaluation Team.

California: Gross Energy Evaluation Rigor Levels California: Gross Energy Evaluation Rigor Levels gy g gy g

Rigor Level Minimum Allowable Methods for Gross Energy Evaluation

• 1. Simple Engineering Model (SEM) with M&V equal to IPMVP Option A and meeting all requirements in the M&V Protocol for

this method. Sampling according to the Sampling and Uncertainty Protocol. 2 Normalized Annual Consumption (NAC) using pre- and post-program participation consumption from utility bills from the

Basic

• 2. Normalized Annual Consumption (NAC) using pre- and post-program participation consumption from utility bills from the

appropriate meters related to the measures undertaken, normalized for weather, using identified weather data to normalize for heating and/or cooling as is appropriate to measures included. Twelve (12) months pre-retrofit and twelve (12) months post- retrofit consumption data is required. Sampling must be according to the Sampling and Uncertainty Protocol.

• 1. A fully specified regression analysis of consumption information from utility bills with inclusion/adjustment for changes and

y p

g y

p y j g background variables over the time period of analysis that could potentially be correlated with the gross energy savings being

• measured. Twelve (12) months post-retrofit consumption data are required. Twelve (12) months pre-retrofit consumption data

are required, unless program design does not allow pre-retrofit billing data, such as in new construction. In these cases, well- matched control groups and post-retrofit consumption analysis is allowable.1 Sampling must be according to the Sampling and Uncertainty Protocol utilizing power analysis as an input to determining required sample size(s). 2

Building energy simulation models that are calibrated as described in IPMVP Option D requirements in the M&V Enhanced

• 2. Building energy simulation models that are calibrated as described in IPMVP Option D requirements in the M&V
• Protocols. If appropriate, may alternatively use a process-engineering model (e.g., AirMaster+ ) with calibration as described in

the M&V Protocols. Sampling according to the Sampling and Uncertainty Protocol.

• 3. Retrofit isolation engineering models as described in IPMVP Option B requirements in the M&V Protocols. Sampling

according to the Sampling and Uncertainty Protocol.

• 4. Experimental design established within the program implementation process, designed to obtain reliable net energy

savings based upon differences between energy consumption between treatment and non-treatment groups from consumption data.2 Sampling must be according to the Sampling and Uncertainty Protocol.

1 Post-retrofit only billing collapses the analysis from cross-sectional time-series to cross-sectional. Given this, even more care and examination is expected with regard to controlling for cross-sectional issues that could potentially bias the

savings estimate.

2 The overall goal of the Direct Impact Protocols is to obtain reliable net energy and demand savings estimates. If the methodology directly estimates net savings at the same or better rigor than the required level of rigor, then a gross savings

and participant net impact analysis is not required to be shown separately.

28

Source: State of California PUC, California Energy Efficiency Evaluation, April 2006

California: M&V Options California: M&V Options

M&V Option How Savings are Calculated Typical Applications Option A: Partially Measured Retrofit I solation

Savings are determined by partial field measurement of the energy Engineering calculations using short Lighting retrofit where power draw is Savings are determined by partial field measurement of the energy use of the system(s) to which an ECM was applied; separate from the energy use of the rest of the facility. Measurements may be either short-term or continuous. Partial measurement means that some but not all parameter(s) may be stipulated, if the total impact

• f

possible stipulation error(s) is not significant to the resultant savings Careful review of Engineering calculations using short term

• r

continuous post-retrofit measurements and stipulations. Lighting retrofit where power draw is measured periodically. Operating hours of the lights are assumed to be one half hour per day longer than store open hours error(s) is not significant to the resultant savings. Careful review of ECM design and installation will ensure that stipulated values fairly represent the probable actual value. Stipulations should be shown in the M&V Plan along with analysis of the significance of the error they may introduce.

Option B: Retrofit I solation

Savings are determined by field measurement of the energy use of the systems to which the ECM was applied; separate from the Engineering calculations using short term or continuous measurements Application of controls to vary the load on a constant speed pump using a variable speed the systems to which the ECM was applied; separate from the energy use of the rest of the facility. Short-term or continuous measurements are taken throughout the post-retrofit period. constant speed pump using a variable speed

• drive. Electricity use is measured by a kWh

meter installed on the electrical supply to the pump motor. In the base-year this meter is in place for a week to verify constant loading. The meter is in place throughout the post- retrofit period to track variations in energy use.

Option C: Whole Building

Savings are determined by measuring energy use at the whole facility level. Short-term or continuous measurements are taken throughout the post-retrofit period. Analysis of whole facility utility meter or sub-meter data using techniques from simple comparison to regression analysis. Multifaceted energy management program affecting many systems in a building. Energy use is measured by the gas and electric utility meters for a twelve month base-year period and throughout the post-retrofit period.

Option D: Calibrated Simulation

Savings are determined through simulation of the energy use of components or the whole facility. Simulation routines must be demonstrated to adequately model actual energy performance measured in the facility. This option usually requires considerable skill in calibrated simulation. Energy use simulation, calibrated with hourly or monthly utility billing data and/or end-uses metering. Multifaceted energy management program affecting many systems in a building but where no base-year data are available. Post- retrofit period energy use is measured by the gas and electric utility meters. Base-year energy use is determined by simulation using a model calibrated by the post-retrofit period tilit d t

29

utility data.

Source: State of California PUC, California Energy Efficiency Evaluation, April 2006

California: M t d V ifi ti O ti California: M t d V ifi ti O ti Measurement and Verification Options Measurement and Verification Options

Measure Type

Basic Rigor Level Enhanced Rigor Level Appliances

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Appliances

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Commissioning and O&M programs

Option D: Calibrated Simulation Option D: Calibrated Simulation

Comprehensive

Option D: Calibrated Simulation Option D: Calibrated Simulation

Envelope

Option D: Calibrated Simulation Option D: Calibrated Simulation

Food Service

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

HVAC Controls

Option D: Calibrated Simulation Option D: Calibrated Simulation

HVAC Equipment Efficiency

Option A: Partially Measured Retrofit Isolation Option D: Calibrated Simulation

Lighting Controls

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Lighting Efficiency

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

New Construction

Option D: Calibrated Simulation Option D: Calibrated Simulation

Non-HVAC Motor Controls

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Non-HVAC Motor Efficiency

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Process

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Refrigeration

Option A: Partially Measured Retrofit Isolation Option D: Calibrated Simulation

Water Heating

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

Water Pumping/Treatment

Option A: Partially Measured Retrofit Isolation Option B: Retrofit Isolation

30

Source: State of California PUC, California Energy Efficiency Evaluation, April 2006

California: Summary of M&V Protocol for Basic Level of Rigor California: Summary of M&V Protocol for Basic Level of Rigor g

• E.g., Appliance Programs
• E.g., Appliance Programs

Provision Requirement Physical inspection of installation to verify correct measure installation and installation quality Verification IPMVP Option Option A Source of Sti l t d D t [Technical Reference Manual] assumptions, program work papers, engineering references, manufacturers catalog data, on-site survey data Stipulated Data g , y Baseline Definition Consistent with program baseline definition. May include federal or [state] appliance standards effective at date of equipment manufacture, [state] building standards in effect at time of building permit; existing equipment conditions or common replacement or design practices as defined by the program Monitoring Strategy and Duration Spot or short-term measurements depending on measure type Weather dependent measures: normalize to long-term average weather data as directed by the Impact Weather Adjustments p g g y p Evaluation Protocol Calibration Criteria Not applicable Additional Provisions None

31

Source: State of California PUC, California Energy Efficiency Evaluation, April 2006

California: Summary of M&V Protocol

• f Enhanced Level of Rigor

California: Summary of M&V Protocol

• f Enhanced Level of Rigor

g

Provision Requirement Physical inspection of installation to verify correct measure installation and installation

• E.g., Commercial lighting Controls, New Construction

Verification

• quality. Review of commissioning reports or functional performance testing to verify correct
• peration

IPMVP Option Option B or Option D Source of Sti l t d D t [Technical Reference Manual] assumptions, program work papers, engineering references, f t t l d t it d t Stipulated Data manufacturers catalog data, on-site survey data Baseline Definition Consistent with program baseline definition. May include federal or [state] appliance standards effective at date of equipment manufacture, [state] building standards in effect at time of building permit; existing equipment conditions or common replacement or design practices as defined by the program p y p g Monitoring Duration Sufficient to capture all operational modes and seasons Weather Adjustments Weather dependent measures: normalize to long-term average weather data as directed by the Impact Evaluation Protocol by the Impact Evaluation Protocol Calibration Criteria Option D building energy simulation models calibrated to monthly billing or interval demand

• data. Optional calibration to end-use metered data

Additional Provisions Hourly building energy simulation program compliant with ASHRAE Standard 140-2001

32

Source: State of California PUC, California Energy Efficiency Evaluation, April 2006

Cost-Effectiveness Evaluation: C i l Cost-Effectiveness Evaluation: C i l Connecticut Example Connecticut Example

$0.0250 $0.0300 2006 Program Adminstrator Cost per Lifetime kWh Saved (Costs Include Direct Costs and Performance Incentives) $0.0250 $0.0300 2006 Program Adminstrator Cost per Lifetime kWh Saved (Costs Include Direct Costs and Performance Incentives)

• Calculated program

administrator cost per lifetime kWh saved

• Calculated program

administrator cost per lifetime kWh saved

$0 0050 $0.0100 $0.0150 $0.0200 $0 0050 $0.0100 $0.0150 $0.0200

lifetime kWh saved

• Comparatively assess to
• ther similar programs to

lifetime kWh saved

• Comparatively assess to
• ther similar programs to

$0.0000 $0.0050 $0.0000 $0.0050

determine relative cost- effectiveness

• Data Requirements:

determine relative cost- effectiveness

• Data Requirements:
• Data Requirements:

– Lifetime kWh savings – Administrator costs

• Data Requirements:

– Lifetime kWh savings – Administrator costs

33

Data Collection and Reporting Data Collection and Reporting Data Collection and Reporting Data Collection and Reporting

• Collect raw data from EDCs- including project,
• Collect raw data from EDCs- including project,

g p j , participant, program and aggregate data as well as associated data dictionaries and relationship diagrams. EDC ill l d d t t GDS S FTP (SFTP) it g p j , participant, program and aggregate data as well as associated data dictionaries and relationship diagrams. EDC ill l d d t t GDS S FTP (SFTP) it

• EDCs will upload data to GDS Secure FTP (SFTP) site

to protect participant and other confidential data

• Read only access to EDC data tracking systems (where
• EDCs will upload data to GDS Secure FTP (SFTP) site

to protect participant and other confidential data

• Read only access to EDC data tracking systems (where

Read only access to EDC data tracking systems (where possible)

• Raw data allows for verification of aggregate data as

ll t h ki f i di id l j t Read only access to EDC data tracking systems (where possible)

• Raw data allows for verification of aggregate data as

ll t h ki f i di id l j t well as spot checking of individual projects well as spot checking of individual projects

34

Reporting Website Reporting Website Reporting Website Reporting Website

• Can be a link from the PUC website or embedded in
• Can be a link from the PUC website or embedded in

the site itself.

• Public reports section and capability for password

t t d ti f EDC d PUC the site itself.

• Public reports section and capability for password

t t d ti f EDC d PUC protected section for EDC and PUC use.

• Capability to export data to Microsoft Excel (.xls),

Adobe Acrobat (.pdf), .csv, .tiff, etc. protected section for EDC and PUC use.

• Capability to export data to Microsoft Excel (.xls),

Adobe Acrobat (.pdf), .csv, .tiff, etc. Adobe Acrobat (.pdf), .csv, .tiff, etc.

• Reports examples include Program Year to date

savings and costs, savings by program, by EDC, etc. Adobe Acrobat (.pdf), .csv, .tiff, etc.

• Reports examples include Program Year to date

savings and costs, savings by program, by EDC, etc.

35

Timetable for Development of h di l Timetable for Development of h di l the Audit Plan the Audit Plan

• Kick-off meeting: September 23rd
• Kick-off meeting: September 23rd

g p

• Draft “straw man” proposal for M&V – October 7th
• Draft “straw man” proposal – impact evaluation –

g p

• Draft “straw man” proposal for M&V – October 7th
• Draft “straw man” proposal – impact evaluation –
• Draft straw man proposal

impact evaluation October 7th

• Draft “straw man” proposal – process evaluation –
• Draft straw man proposal

impact evaluation October 7th

• Draft “straw man” proposal – process evaluation –

October 7th

• First draft of Statewide Evaluation Audit Plan –

November 1st October 7th

• First draft of Statewide Evaluation Audit Plan –

November 1st November 1s

• Final Statewide Evaluation Audit Plan – December 2nd

November 1s

• Final Statewide Evaluation Audit Plan – December 2nd

36

Part 5 Development of Sector and Part 5 Development of Sector and Development of Sector and Program Specific Evaluation Development of Sector and Program Specific Evaluation Plans Plans

37

I mpact Evaluation Process I mpact Evaluation Process I mpact Evaluation Process I mpact Evaluation Process

38

Programs Programs

Residential Residential Commercial/ I ndustrial Commercial/ I ndustrial /

• Energy Efficiency Behavior and Education
• Energy Star and High Efficiency Appliance
• Appliance Turn-In/Recycling
• CFL Rewards
• Energy Efficiency Behavior and Education
• Energy Star and High Efficiency Appliance
• Appliance Turn-In/Recycling
• CFL Rewards
• Energy Audits and Assessments
• HVAC Efficiency/Tune-up
• Lighting Efficiency
• Equipment Rebates/Prescriptive

C t T h l A li ti

• Energy Audits and Assessments
• HVAC Efficiency/Tune-up
• Lighting Efficiency
• Equipment Rebates/Prescriptive

C t T h l A li ti

• Home Performance/Home Audits/Whole-

Building/Weatherization

• HVAC Efficiency
• Solar/Renewable Rebates
• New Construction
• Home Performance/Home Audits/Whole-

Building/Weatherization

• HVAC Efficiency
• Solar/Renewable Rebates
• New Construction
• Custom Technology Applications
• New Construction
• “Performance Contracting”
• Government (State and Local)/Institutional (Schools)
• Government/Non-Profit Lighting Efficiency
• Custom Technology Applications
• New Construction
• “Performance Contracting”
• Government (State and Local)/Institutional (Schools)
• Government/Non-Profit Lighting Efficiency
• New Construction
• Low-Income Home Performance Check-up

Audit/Appliance Replacement/Weatherization

• Low-Income Joint Utility Usage Management
• Low-Income Room AC Replacement
• New Construction
• Low-Income Home Performance Check-up

Audit/Appliance Replacement/Weatherization

• Low-Income Joint Utility Usage Management
• Low-Income Room AC Replacement
• Federal Facilities
• Street and Traffic Lighting Efficiency
• Federal Facilities
• Street and Traffic Lighting Conversion
• Programmable Controllable Thermostat
• Federal Facilities
• Street and Traffic Lighting Efficiency
• Federal Facilities
• Street and Traffic Lighting Conversion
• Programmable Controllable Thermostat
• Efficiency Rewards Rate
• Super Peak Time of Use (TOU)
• Home Energy Incentives
• Multifamily Housing
• Direct Load Control
• Efficiency Rewards Rate
• Super Peak Time of Use (TOU)
• Home Energy Incentives
• Multifamily Housing
• Direct Load Control
• Pay Ahead (Smart) Service Rate
• Motors and Drives
• Custom Application
• Customer Load Response
• Distributed Generation
• Pay Ahead (Smart) Service Rate
• Motors and Drives
• Custom Application
• Customer Load Response
• Distributed Generation
• Direct Load Control
• Direct Load Control
• Direct Load Control
• Contracted Demand Response
• Critical Peak Rebate (CPR) Rate
• Time of Use (TOU) w/ Critical Peak Pricing Rate
• Hourly Pricing Option (HPO) Rate
• Direct Load Control
• Contracted Demand Response
• Critical Peak Rebate (CPR) Rate
• Time of Use (TOU) w/ Critical Peak Pricing Rate
• Hourly Pricing Option (HPO) Rate

39

Hourly Pricing Option (HPO) Rate

• Renewable Resources
• Permanent Load Reduction
• Conservation Voltage Reduction

Hourly Pricing Option (HPO) Rate

• Renewable Resources
• Permanent Load Reduction
• Conservation Voltage Reduction

TRM Matrix – Residential TRM Matrix – Residential

TRM Measures with Deemed Savings TRM Measures with Measurement and/ or TRM Guidelines TRM Notes: Simulation Required Energy Efficiency Behavior and Education Energy Star and High Efficiency Appliance

• ES Refrigerators
• ES Clothes Washers
• ES Dishwashers
• Dehumidifiers
• ES Room AC
• ES Freezer

Appliance Turn-I n/ Recycling

• Refrigerator/Freezer
• Based on number of units and

fixed deemed savings per unit CFL Rewards/ Lighting

• ES CFL Bulbs
• ES Torchieres
• ES Indoor Fixtures
• ES Outdoor Fixture
• ES Ceiling Fan w/ Lights

Home Performance/ Home Audits/ Whole-

• ES Windows
• HomeCheck Software

Home Types –

• No algorithms or

Building/ Weatherization

• Home Performance w/ ES

Windows:

• Heat Pump
• Electric

Heat/CAC

• Electric Heat

Only information provided for ES/Generic Home Audits HVAC Efficiency

• Central AC

Ai S HP

• Air Source HP
• Ground Source HP
• DSHP Desuperheater
• Furnace – High E Fan

Solar/ Renewable Rebates

40

Residential Continued Residential Continued

TRM Measures with Deemed Savings TRM Measures with Measurement and/ or TRM Guidelines TRM Notes: Deemed Savings Measurement and/ or Simulation Required Guidelines New Construction

• Insulation Up-Grades
• High E Windows
• Air Sealing
• High E HVAC Equipment
• Specify REM/Rate User Defined

Reference Home

• Duct Sealing
• Lighting & Appliances
• Ventilation Equipment

Low-I ncome Home Performance Check- p A dit/ Appliance up Audit/ Appliance Replacement/ Weatherization Low-I ncome Joint Utility Usage Management Low-I ncome Room AC Replacement p Efficiency Rewards Rate Super Peak Time of Use (TOU) Home Energy I ncentives Multifamily Housing Direct Load Control

• Air Conditioning Cycling
• Pool Pump Load Control

41

TRM Matrix – Commercial/ I ndustrial TRM Matrix – Commercial/ I ndustrial

TRM Measures with Deemed Savings TRM Measures with Measurement and/ or Simulation TRM Guidelines TRM Notes: Savings Measurement and/ or Simulation Required Guidelines Energy Audits and Assessment HVAC Efficiency/ Tune-up

• Central Air Conditioners
• Room AC
• Air Cooled DX
• Split Systems
• Packaged Terminal Systems

Packaged Terminal Systems

• Air Source Heat Pumps
• Water Source Heat Pumps
• Ground Water/Source Heat Pumps
• Electric Chillers

Lighting Efficiency

• CFLs
• Low Taby T-5/T-8
• High Bay T-5/T-8
• High E Fluorescent
• Pulse Start Metal Halide
• Low Bay LED
• Fluorescent Lighting Fixture
• LED Exit Signs
• Lighting Controls – Occupancy

Sensor/Controls/Daylight Dimmer

• 20% Lighting Power Density (LPD)

Reduction Eq ipment Rebates/ P esc ipti e Equipment Rebates/ Prescriptive Custom Technology Applications New Construction “Performance Contracting” Government (State and Local)/ I nstitutional (Schools) Government/ Non-Profit Lighting Efficiency Federal Facilities Street and Traffic Lighting Conversion

• LED Round Traffic Signals
• LED Turn Arrows
• LED Pedestrian Signs

42

Commercial/ I ndustrial Continued Commercial/ I ndustrial Continued

TRM Measures with Deemed Savings TRM Measures with Measurement and/ or Simulation Required TRM Guidelines TRM Notes: Programmable Controllable Thermostats Pay Ahead (Smart) Service Rate Motors and Drives

• Variable Frequency Drives
• Air Compressor with VFD
• Motors: No qualitative

description or applicability

• guidelines. Baseline Motor

Efficiencies are specified. Custom Application Customer Load Response

• All Demand Response treated

independently Distributed Generation Direct Load Control

• All Demand Response treated

independently Contracted Demand Response

• All Demand Response treated

d d l independently Critical Peak Rebate (CPR) Rate Time of Use (TOU) w/ Critical Peak Pricing Rate Hourly Pricing Option (HPO) Rate Renewable Resources Permanent Load Reduction Conservation Voltage Reduction

43