[PPT] - Davis-Besse Nuclear Power Station August 15, 2002 1 Introduction PowerPoint Presentation

SLIDE 1

1

Davis-Besse Nuclear Power Station

August 15, 2002

SLIDE 2

2

FENOC Chief Operating Officer

Introduction

SLIDE 3

3

Introduction
Lew Myers, FENOC Chief Operating Officer
Management and Human

Performance Root Causes

Steve Loehlein, Root Cause Analysis Team Leader
Corrective Actions and Summary
Lew Myers, FENOC Chief Operating Officer

SLIDE 4

4

Discuss the root causes:

Management Oversight
Corrective Action Program
Technical Rigor
Program Compliance

And the key corrective actions to be taken

SLIDE 5

5

Earlier Root Cause investigation and the NRC

Augmented Inspection Team report both concluded that management had ineffectively implemented processes, and thus failed to detect and address plant problems as

pportunities arose
Root Cause Analysis Team was tasked to determine

WHY the significance of the conditions in the plant was not recognized

Introduction

SLIDE 6

6

Team Composition

Technical Root Cause Analysis Team Leader
Root Cause Qualified FENOC Employees
FENOC Oversight Support
Process Expertise (Conger & Elsea)
Davis-Besse Management Personnel

Industry Input

Root Cause Process Experts
Organizational Effectiveness Experts
Organizational Development Consultants

Introduction

SLIDE 7

7

Root Cause Analysis Team

Lead: Steve Loehlein

(Beaver Valley)

Bill Babiak (Perry)
Mario DeStefano (Perry)
Randy Rossomme

(Beaver Valley)

Lesley Wildfong

(Conger & Elsea)

Bill Mugge (Davis-Besse)
Joe Sturdavant (Davis-Besse)
Bobby Villines (Davis-Besse)
Dick Smith (Conger & Elsea)
Dorian Conger and Ken Elsea (C&E)
Spyros Traiforos

Introduction

Root Cause Team

SLIDE 8

8

Root Cause Analysis Team Leader

SLIDE 9

9

Understand why, over a period of years, Davis-Besse personnel failed to identify corrosion of the Reactor Pressure Vessel Head base metal

Problem Statement

SLIDE 10

10

There was less than an adequate nuclear safety focus – There was a focus on production, established by management, combined with taking minimum actions to meet regulatory requirements, that resulted in the acceptance of degraded conditions

Root Cause Statement

SLIDE 11

11

SLIDE 12

12

Approach

Scope Development

– Technical Root Cause results provided clues

Errors occurred over several years, and in several areas * Program effectiveness needed to be assessed * The potential for boric acid to cause damage was an issue for the plant in 1998/99, but actions taken then did not result in detection of head corrosion

SLIDE 13

13

In-Depth Evaluations

Event and Causal Factors chart and Barrier Analysis

techniques used

Management Oversight and Risk Tree (MORT) technique

used

MORT Analysis Sections

– Technical Information Systems – Corrective Action Program – Hazard Analysis Process – Task Performance Errors – Management Support / Oversight

Corrective Actions based on conclusions of data analysis

SLIDE 14

14

Technical Root Cause Analysis Report
Interviews (more than 120)
Documents (approximately 700)
Over 20 years of Data

Data Sources

SLIDE 15

15

SLIDE 16

16

1. Boric Acid Corrosion Control and Inservice Inspection

(ISI) Programs

2. Handling of Technical Information
3. Corrective Action Program
4. Hazard Assessment Process
5. Management Oversight/Risk Assessment

Presentation Sequence of Data Analysis/Results

SLIDE 17

17

Boric Acid & In-Service Inspection Programs

Hazard-Barrier-Target Analysis evaluated the Boric Acid

Corrosion Control and Inservice Inspection (ISI) Programs Model assumes that boric acid is the hazard, and the RPV head is the target Barriers included design, training, inspection for leaks and corrosion, cleaning, and corrective actions (nearly 50 in all)

SLIDE 18

18

Conclusions

The failure to clean the RPV head (a failed barrier in the

analysis) prevented the team from analyzing the behavior that would have resulted had a bare head inspection been preformed

The RPV head was not a focus in the process
None-the-less, had the programs been followed as required,

they were adequate to have prevented the serious head damage

SLIDE 19

19

Handling Technical Information

Evaluated using MORT analysis technique
Evaluated the process the plant used to ensure that

technical information was properly assessed and incorporated

SLIDE 20

20

Conclusion

The process for disseminating and incorporating technical

information was adequate

Personnel failed to correctly apply key industry information

and plant knowledge about the potential harmful effects of boric acid

SLIDE 21

21

Corrective Action Program

Utilized MORT technique and Change Analysis
Examined critical steps in process
Initiation
Operability Review/Categorization
Cause Analyses
Corrective Actions
Trending/Effectiveness Reviews

SLIDE 22

22

Conclusion

Davis-Besse adequately identified and documented

nonconforming conditions

Personnel at ALL levels did not effectively implement the

Corrective Action Process Operability/operational impact underestimated Categorization did not recognize significance Shallow cause analyses Inadequate corrective actions Inadequate trending of recurring equipment problems

SLIDE 23

23

Hazard Assessment Process

Evaluated using MORT analysis technique
Examined how the organization used it to recognize and

evaluate nuclear safety risks

A key focus was on the application of the process for

evaluating issues subject to 10CFR50.59, Safety Evaluations

SLIDE 24

24

Conclusions

Hazard Analyses process contained the necessary elements to

ensure the design and licensing basis was maintained

Process that required detailed analysis became less restricted
ver time

Result was that in later years, the process was not applied to Containment Air Coolers, Radiation Monitor Filters, or boric acid on RPV head

SLIDE 25

25

Started from Technical Root Cause Analysis Report

– Timeline of Key Events (Figure 26) from that report provides insight – Plant conditions provided clues that problem existed * Reactor Coolant System unidentified leak rate * Containment Radiation Monitor Filter plugging * Containment Air Cooler cleaning frequency * Boric acid accumulations on the head

Management Oversight/Risk Assessment

SLIDE 26

26

Examined 22 Condition Reports (PCAQRs & CRs)

– Operability/operational impact underestimated by engineering and operations – Low categorization, with no root cause analysis required – Shallow cause analyses with focus on resolving symptoms rather than cause – Corrective Actions deferred resolution or treated symptoms – No collective significance recognized or evaluated; no visible senior management sponsorship

Management Oversight/Risk Assessment (continued)

SLIDE 27

27

P C AQR 9 6-0551 V arying siz es of bori c acid mounds scatter e d in var iou s areas: Head no t clea ned.

Cyc le 1 1 Cyc le 1 0 1996 199 7 1998 199 9 2000 2001 200 2 1995 Cy cle 12. A Cycle 12.B Cyc le 13

5 17 2 1 w eek 1 -2 dy Monthl y replace ment for preventive maint ena nce Monthl y replace ment for preventive maint ena nce 2 w k s 1 wk 1 -2 dy 1 wk 2 w k s 1 wk 1 -2 dy 2 w k s 2 w k s 1 3RFO 1 1RFO 1 0RFO 1 2RFO

Numbe r of Air Cooler Clean ings Op eratin g Cycles Calendar Co n tainmen t Monitor Filte r Repl acemen t AA BA

P ress urize r Re lief Va lve Leak a g e

0. 1 0. 3 0. 4 0. 2 0. 8 0. 7 0. 6 0. 5 0. 9 1. Monthly Average U n identified L e akage (gpm ) 0. 10 30 20 40 50

1. Numb e

r of Nozzles Not Visibl e in V isual Inspec tion s 4 PC A QR 96-055 1 d ue to inability t o comply w it h N G-E N-

00324. B&WOG

Safety E valua t ion t ake s credit fo r visual inspec t ion. PCAQ R 96-0 551 boric acid

n the head

is dow ngraded a nd closed aft e r 2 years 9 month s. CR writ ten on RCS lea kage ass

ciated w

it h data scatt e r . 3/01 Low flow ala r m d ue to b

r on

bu i ld

u

p - CR initiated. 3/99 S ampl e po int is swa pped in an att empt t o redu ce th e bor on bu ild-u

p. 9/01

First C AC cleanings since 0 2/24/92. 11/98 CR is s ued based on CAC plenum pr e ssure d ecreas ing aft er s ig ni fica n t work during

12RFO. 6/00

Root caus e on cle aning of the CA C s no t per fo rmed u nt il 13 R FO.

Reactor Co

lant 2 Valve

6 P CAQRs in 5 month s. (1st r oot cau s e) Return to power, valve is r e injected . 10/98 3rd root caus e on RC

2 4/00

Ru p ture disc bu r st TM 10/ 17/98 CR 99-1300 closed w it hout d et e rmination o f c

ause. 9/99

P ressu r ize r cod e sa f e t y valves repaired w hen le akage is .8 gpm. 5/99 C R w r itt e n on fluctuating R C S leakag e .25 gpm. 7/01 Filter p a per c

n tains ir on
xide

, cr e dited Mag naf lu

x. C
nt a

in me n t su mp s amples have cor rosio n products. 7/99 RP V h e ad is no t consid ered as a source of ir on

xide

. E ng. memo to Operation s Four HEP As inst a ll e d fo r iron oxide removal. ( TM ) 8/99 Plant Engineering belie ves the cau se

f

cor rosi on is CAC plenu m or pip ing

r

conduit corr osion . 7/99 I odine c ar tr idges are removed du e t o d et e cto r sa t urations . 11/01 C NR B is to ld t he run times b et w een c leaning will increase bec ause Magna f lux leve l is decrea s ing. 7/99 V alve is leaking

n st a

rtup, decision to use a s-is. (No sa f e t y eva luation ) 5/98 Inspe ct ion eve r y 30 d ays - de cis ion not t o

repack. Plant is

in f orced

u

t a ge. 10/98 One nut is missing, valve is f unc t ional. Se cond nut is missing, valve is inoperable - PCAQR 98- 1681. Valve is in je

cted. 9/98

Work Or d er t o re p la ce yoke and repack the valve, BACC Program is no t add r essed. 5/98 C

llective

s ignificance

f

manageme n t issu e on RC 2 - PCAQ R 98- 0020 (2nd r oot cau s e) 12/98 RC 2 packing leak, no CR E ffectiveness review

f CR

98-00 20 de t e r mines action s w er e ef fec t ive. 10/00 P C AQR 98- 0767 Most of t he he ad is covere d in an uneven layer along w it h sever al "fist" sized lumps: Not c

mpletely clea

ned. C R 00-1 037 Lar g e deposit s of b

ron and lava-

like flow t hrough the mo u se holes. Cont ainment Sump lea kage >1 gpm RCS U /I lea kage increasing; L e aks not identif ied . 10/01 Independ ent Laborator y res u lt s of parti c ulate s ample indica t e the likely

r ig

in to be a steam source . 11/99 PCAQ R w r itt e n t o d

cument step in CAC

p le num pr essure . 11/98 C R w r itt e n t o eva luate imp a ct o f c

leaning. 12/00

C R w r itt e n t o docu m e nt s t e p d r op in CAC plenum pres s

ure. 1/01

At tributed to CR D f la nge leaka ge

SLIDE 28

28

Why did the organization not recognize the significance of the plant condition?

Conclusions from other analyses were used to develop an
verall understanding of failure to recognize significance of

the plant condition

Additional MORT analysis was done in Management

Policy/Incentives

Added numerous interviews insights

SLIDE 29

29

Conclusions

Beginning in the mid-1990s, management focus was on

production concerns Rigor in assessing issues for their potential impact on nuclear safety diminished: Taking minimum actions to meet regulatory requirements was interpreted to be adequate for nuclear safety Management style was less directly involved, and relied on subordinates to escalate concerns

SLIDE 30

30

Conclusions (continued)

Results were:

Plant was restarted and run for extended periods with some degraded components Personnel performed with the philosophy that issues were not serious unless they were proven to be Rigor in processes declined at the same time that the threat of head damage increased

SLIDE 31

31

Root Causes - Management Oversight

A less than adequate nuclear safety focus and a production

focus, combined with minimum actions to meet regulatory requirements

SLIDE 32

32

Root Cause - Corrective Action Program

Inadequate implementation of the Corrective Action Program:

Addressing symptoms rather than causes Categorization did not recognize significance Less than adequate cause determinations Less than adequate corrective actions Poor equipment trending

SLIDE 33

33

Root Cause - Technical Rigor

Failure to integrate and apply key industry information and

site knowledge/experience; and to compare new information to baseline knowledge

SLIDE 34

34

Root Cause - Program Compliance

Some steps in the Boric Acid Corrosion Control Procedure

were not followed

SLIDE 35

35

Root Cause - Contributing Causes

Some decisions were made without considering the need for a

safety analysis

Corrective Action Program was not state-of-the-art

SLIDE 36

36

Alloy 600 material used in the original design of the

CRDM nozzles was susceptible to cracking/leakage; the original gaskets in the CRDM flanges were susceptible to leakage

Training was not provided to individuals performing

inspections for boric acid

Inspections activities and corrective action were not

coordinated through the Boric Acid Corrosion Control (BACC) Coordinator

BACC Procedure did not specifically reference the

CRDM nozzles as one of the probable locations of leakage

Other Key Observations

SLIDE 37

37

Condition Reports associated with the boric acid

issues tended to stay unresolved until significant degradation occurred

There was little evidence of QA’s involvement and

the documented findings by QA were of mixed quality

Monetary incentive program rewards production

more than safety at senior levels

Written policies do not support a strong safety focus
Operations had minimal involvement
Management had minimal entries into containment

Other Key Observations (continued)

SLIDE 38

38

FENOC Chief Operating Officer

SLIDE 39

39

Extent of Condition

Davis-Besse Building Blocks ensure adequacy of systems, programs, and the organization to support safe and reliable operation. Specifically:

The System Health Assurance Plan provides for rigorous

system reviews

The Management and Human Performance Excellence

Plan will ensure a strong and sustained safety focus

The Program Compliance Plan ensures programs meet

industry high standards of performance

SLIDE 40

40

Reactor Head Reactor Head Resolution Plan Resolution Plan Bob Schrauder Bob Schrauder Program Compliance Program Compliance Plan Plan Jim Powers Jim Powers Containment Health Containment Health Assurance Plan Assurance Plan Randy Fast Randy Fast Restart Test Plan Restart Test Plan Randy Fast Randy Fast Management and Management and Human Performance Human Performance Excellence Plan Excellence Plan Lew Myers Lew Myers System Health System Health Assurance Plan Assurance Plan Jim Powers Jim Powers Restart Action Plan Restart Action Plan Lew Lew Myers Myers

Restart Overview Restart Overview Panel Panel Return to Service Plan Return to Service Plan

SLIDE 41

41

Corrective Actions

New Senior Management Team with high standards
Implement the Management and Human Performance

Excellence Plan

Case Study Training and Rebaselining of Standards (how the event

happened, what barriers broke down, and what needs to be different in the future)

Safety Conscious Work Environment Survey and Assessment
Organization Effectiveness
Four Cs (Compliments, Communications, Concerns, and Changes)

Management (Structured Approach)

Management Observation Program
Evaluation of Directors and Managers

SLIDE 42

42

Corrective Actions

Complete a Program Compliance Plan which includes a

detailed latent issues review of Corrective Action Program by

utside consultants
Strengthen the Corrective Action Review Board to enforce

higher standards for cause evaluations and effective corrective action (chaired by Plant Manager or another director-level individual)

Ensure criteria used for categorization of significant or repeat

equipment failures are appropriate and utilized by plant personnel

Routinely perform assessments of categorization

SLIDE 43

43

Corrective Actions

Repeat conditions are to be treated as Significant Conditions

Adverse to Quality (SCAQ)

Review existing long-standing issues for possible SCAQ

categorization and use root cause evaluation techniques to obtain resolution

Require the use of formal cause determination techniques for

root and basic cause evaluations to ensure analytical rigor is applied

Define and implement training for cause evaluations
Improve guidance on reviews of the effectiveness of

corrective actions (focus on verifying causes have been fixed and provide training on revised guidance)

Implement an effective site-wide equipment trending program

SLIDE 44

44

Corrective Actions

Rebaseline Standards and Expectations in each FENOC Group
Establish Engineering Assessment Board to reinforce standards
Establish FENOC hierarchy of documents for consistent

standards for analyses of safety issues

Establish a Periodic System Walk-down Program
Establish a Periodic Engineering Program Review Process

SLIDE 45

45

Corrective Actions

Provide training to applicable personnel and managers that

includes:

Need to remove boric acid from components
Inspect for signs of corrosion
Perform inspections for signs of boric acid in component

internals

Reinforce standards and expectations for procedure compliance and

the need for work-practice rigor

Implement Management Observation Program with weekly

schedules (used at Perry and Beaver Valley)

Perform independent assessments of procedure compliance
Discuss procedure compliance regularly at morning meeting

SLIDE 46

46

Corrective Actions

Establish the FENOC decision-making process at Davis-

Besse, including hazard analyses

Perform Corrective Action Procedure Benchmarking

SLIDE 47

47

Other Relevant Corrective Actions and Improvements

Design of New RPV Head
BACC Procedure Revision to include CRDM Nozzles
Training on BACC Procedure
Coordination of Boric Acid Control Activities
Timely Corrective Action Resolutions
Reviews of Quality Assurance Audits and Surveillances

Adequacy

Realign Incentive Program to Increase Focus on Safety
Establish Policies to Support Safety
Operations involvement
Management presence in the field

SLIDE 48

48

VICE PRESIDENT - FENOC OVERSIGHT

L. William Pearce

FENOC CHIEF OPERATING OFFICER Lew W. Myers EXECUTIVE VICE PRESIDENT - FENOC Gary R. Leidich MANAGER - OPERATIONS EFFECTIVENESS Michael J. Ross VICE PRESIDENT - NUCLEAR Lew W. Myers PLANT MANAGER

J. Randel Fast

DIRECTOR - MAINTENANCE Michael J. Stevens DIRECTOR - SUPPORT SERVICES Robert W. Schrauder DIRECTOR - NUCLEAR ENGINEERING James J. Powers, III

MANAGER - CHEMISTRY AND RADIATION PROTECTION Robert W. Pell MANAGER - PLANT OPERATIONS Michael J. Roder MANAGER - NUCLEAR TRAINING William A. Mugge MANAGER - MAINTENANCE Peter D. Roberts MANAGER - WORK CONTROL

C. David Nelson

MANAGER - OUTAGE MANAGEMENT

C. David Nelson (acting)

MANAGER - REGULATORY AFFAIRS Patrick J. McCloskey MANAGER - PLANT ENGINEERING Joseph W. Rogers MANAGER - LEARNING ORGANIZATION Dave Gudger MANAGER - QUALITY SERVICES Linda M. Dohrmann MANAGER - SECURITY Gary A. Skeel MANAGER - HUMAN RESOURCES Deanna L. Haskins MANAGER - DESIGN BASIS ENGINEERING John J. Grabnar

New to Position

SLIDE 49

49

CEO of FirstEnergy has set the standard of returning Davis-Besse back to service in a safe and reliable manner, and doing the job right the first time. We are committed to meeting this challenge.