Remote Control of VMT Final Report Review Contract: 507.12.01 - - PowerPoint PPT Presentation

Remote Control of VMT

Final Report Review Contract: 507.12.01 PWSRCAC Board Meeting January 23-24, 2014

Hisey and Associates, LLC

∗ Scope of Contract ∗ Definition of Key Terms ∗ Review Process ∗ Applicable Regulations and Standards ∗ Alyeska documents ∗ Employee interviews ∗ Other publicly available documents ∗ Summary of Findings ∗ Recommendations

Overview

1. Identify assets at the VMT that are subject to remote control and those subject to local

• control. Include an analysis of the interface between the two classes of assets.

2. Analyze command and control design, capability, visibility and effectiveness in managing

normal and abnormal operations at the VMT.

3. Review the training requirements and procedures for OCC operators relative to VMT-

specific activities. Evaluate conformance to operating procedures and requirements.

4. Assess how the four key processes at the VMT (Tank Storage, Vessel Loading, Power

Generation, and Ballast Water Treatment) are controlled at the VMT and/or the OCC and how operations and system status are coordinated and communicated between the VMT and OCC controllers.

5. Assess the extent to which moving the OCC from the VMT to the Anchorage location

changed the limited control capabilities the OCC has for four VMT processes identified in #4 above.

6. Provide Alyeska with recommendations as appropriate for improving the control of

assets at the Valdez Marine Terminal during both routine and abnormal operations.

7. Provide reasonable amount of consultation to third parties regarding results, findings,

and recommendations associated with work performed under this agreement.

Scope of Contract

∗ OCC – Operations Control Center (Anchorage) ∗ SCADA – Supervisory Control and Data Acquisition

• DCS – Distributed Control System
• PLC – Programmable Logic Controller
• HMI – Human Machine Interface
• UCOS – User Configurable Open System (TAPS control system

software)

∗ Controller – Person who operates system from console ∗ Alarm Management – proper design, analysis, rationalization, implementation, monitoring and improvement of control system alarms

Definition of Key Terms

Operations Control Center (Anchorage)

• Review applicable regulations, standards and industry best-practice
• Review company policies, procedures, requirements, documents and

records

• Interview knowledgeable employees responsible for implementing

procedures, etc.

• Validate with documented evidence
• Assess company practices and employee understanding against all

requirements

• Document findings and recommendations

Typical System Review Process

∗ Alyeska provided many of their standards, policies and procedures, but provided few documents to demonstrate conformance and very limited employee interviews ∗ Scope extended to consider publicly available Alyeska documents involving control system issues

∗ FERC Tariff Litigation ∗ State Assessment Review Board (SARB) hearings

∗ Evaluate possible VMT control system related issues or similar risks

Modified Review Process

∗ Federal Regulations (49 CFR Part 195) DOT/PHMSA Liquid Pipelines.

∗ Same control system for Pipeline and VMT

∗ Industry Standard Practices

• API RP 1165 – SCADA Displays
• API RP 1167 – SCADA Alarm Management
• API RP 1168 – Control Room Management
• ANSI/ISA – 18.2 Alarm Management

Applicable Regulations and Standards

∗ Many documents requested were deemed “out of scope” by Alyeska and not provided ∗ All documents provided by Alyeska were identified as confidential ∗ Pipeline references were redacted by Alyeska ∗ Many documents (forms) were provided blank ∗ Largely represent policies, procedures and forms exist ∗ Do not provide evidence that policies and procedures are being followed

Alyeska Documents

∗ 10-12 Interviews originally planned were subsequently denied ∗ Interviewed one automation engineer and one Valdez technician ∗ All questions were provided to Alyeska in advance of interviews ∗ Individuals were largely unfamiliar with subject matter of questions, deferring responses to others who were not made available ∗ OCC Manager responded to one of nine sets of questions

• posed. Responses were extremely brief.

∗ Unable to test/validate conformance to standards

Employee Interviews

∗ FERC TAPS Tariff Case - Pipeline Strategic Reconfiguration (prudence challenge)

∗ Case also addressed issues with implementation of control system changes made since 2007 ∗ Searched public documents re: OCC and control system ∗ Assessed potential for similar issues related to VMT

∗ 2013 State (tax) Assessment Review Board hearings

∗ Project Work Request (PWR) involving critical control system communication building replacement at VMT

Other Publicly Available Documents

∗ Some control system documents do not contain sufficient detail ∗ BE-20 (Berth Operators Manual) provides appropriate detail ∗ PS-9 Crude Tank Overfill Incident findings are relevant to control of VMT ∗ VMT Communications Hub / SCADA Building remains in a “failed state” ∗ Key Performance Indicators (KPI) for Control System Alarm Management are inadequate

Summary of Findings

∗ Alyeska did not demonstrate having completed Documentation and Rationalization (D&R) of the VMT control system alarms as required by regulations and company standards ∗ Response requirements for a berth vapor control building exhaust fan alarm appear inadequate ∗ Status of Alyeska commitment to complete a Reliability Centered Maintenance Review of the Control System is unknown

Summary of Findings (continued)

∗ PS09 Tank Overfill Incident root causes and contributing factors included:

∗ Design less than adequate (LTA) ∗ Previous Incident Management Action Plans (MAP) and Lessons Learned LTA ∗ Situational Awareness LTA [OCC Controllers and field personnel] ∗ Safe Operating Committees (SOC) LTA ∗ Standards, Policies and Administrative Controls (Procedures) LTA

∗ To the extent these systemic issues remain, the same factors could contribute to unnecessary risk at VMT. Resolution of these risks needs to be validated by external risk management

• rganization or regulatory agency.

Key Findings and Related Recommendations

∗ Alyeska has stated publicly they are evaluating Strategic Reconfiguration type of changes to the VMT. ∗ To the extent proposed configuration changes may include a significant increase in process instrumentation and status under the OCC, PWSRCAC should be aware of not only their design, but the specifics of the implementation process, taking into account the type of problems encountered when making the significant changes during the pipeline SR projects

Recommendation

∗ Alyeska has a Project Work Request (PWR – Z556) in process to replace a critical communication structure at VMT that houses control system hardware ∗ PWR submitted as evidence by Alyeska in 2013 state tax hearings ∗ The original 2008 PWR states the structure is in a “failed state” ∗ Alyeska now says the term “failed state” was in error and will be corrected ∗ PWSRCAC should request a review/update on the status of PWR Z556

Finding and Recommendation

Text images from PWR-Z556

VMT Communications Building Houses “critical communications infrastructure” at site of original Operations Control Center

∗ Alyeska should establish clear measureable goals or Key Performance Indicators (KPIs) to track and monitor their control system alarms.

Recommendation

∗ Alyeska did not demonstrate having completed Documentation and Rationalization (D&R) of the VMT control system alarms as required by regulations and company standards ∗ PWSRCAC should request documentation from Alyeska to establish the status of the Master Alarm Database (MADB) and D&R for VMT control system alarms

Finding and Recommendation

∗ OCC Terminal Controller training is critical to safe

• peration of VMT

∗ Training records for VMT Controllers were not provided for review ∗ PWSRCAC should request Alyeska provide a status of VMT Controller training

Finding and Recommendation

∗ Alyeska committed to the Joint Pipeline Office (JPO) in 2001-2002 to complete a Reliability Centered Maintenance (RCM) Review on the TAPS Control System ∗ PWSRCAC should ask if Alyeska has completed an RCM analysis of the control system, and if it was updated after the relocation of the OCC from Valdez to Anchorage

Finding and Recommendation