Design Review for Improvements to the Access Process Dan Sexton, - PowerPoint PPT Presentation

Design Review for Improvements to the Access Process Dan Sexton, Henry Robertson, and Jerry Kowal Safety Systems Group Electrical Engineering Department March 25, 2015

Outline of Presentation • Introduction of Purpose • Summary of Event • Standard Sequence of a Controlled Access • Issues Requiring Attention • Proposed Change Options • Review of Potential Solutions • Logic Change Proposal • Procedural and Training Changes • Process Changes • Documentation Status • Conclusions and Outcomes • References • Questions/Concerns/Discussion D. Sexton & H. Robertson, Design Review for Improvements to Access Process 2

Introduction of Purpose • On September 12, 2014 there was a procedural violation of the controlled access process resulting in an unauthorized entry to a controlled area. Notable Event – ACC-14-0912 • As a result of this a CATS item was generated: NE-2014-08-01: Accelerator Operations and EES Dept. shall evaluate the PSS to implement an engineering solution that eliminates the potential vulnerability of unauthorized access. • This presentation is to highlight the proposed solutions, the current testing results, and implementation schedule. D. Sexton & H. Robertson, Design Review for Improvements to Access Process 3

Summary of Event • The PSS state of the Injector segment was Controlled Access (CA) where employee #1 was making multiple trips in and out of the Injector. • The on-duty Safety System Operator (SSO) had to step out of the control room so the on-duty Crew Chief (CC) was fulfilling dual roles of acting SSO and CC. • After completion of one of these accesses for employee #1 the acting SSO forgot to ensure both door #1 and #2 were locked. • During this time while in CA an unauthorized entry occurred by employee #2 while employee #1 was in the Injector • Employee #2 failed to check the PSS state message display when entering the Injector through the access room where both door #1 and #2 were unlocked at the same time. • Employee #2 completed their tasks in the Injector and then exit through door #1 and #2 which were both unlocked. • Employee #1 completed their tasks and entered the access room through door #2 for out processing of the CA procedure. • At this time it was brought to the attention of the SSO that there was another individual previously in the Injector. • Once this was realized the area was dropped to Restricted Access, the appropriate individuals were contacted, and an investigation began. D. Sexton & H. Robertson, Design Review for Improvements to Access Process 4

Standard Sequence of a Controlled Access – Entry • All Prerequisites met – Rad-Survey, Devices off, Stoppers Inserted, and Segment Key set to “Controlled Access” • Individual(s) wanting access approaches segment access room. • Contacts Safety System Operator (SSO) from outside Door 1(Outer Door) • SSO answers call & if appropriate unlocks Door 1 for individual(s) to enter access room • SSO Locks Door 1 behind individual(s) • Individual(s) gaining access contact SSO • SSO releases Exchange Key, if these are the first individual(s) in the segment • Individual(s) remove the Exchange Key and place into the master slot of the key bank. • Each individual removes their own key to carry with them and give that information along with an affirmation of ODH training and Dosimeter. • SSO verifies and records each individuals: name, associated key number, ODH training, dosimeter, and the time into the tunnel segment. • Unlock Door 2 and allow individual(s) to enter • Lock Door 2 behind individual(s) D. Sexton & H. Robertson, Design Review for Improvements to Access Process 5

Standard Sequence of a Controlled Access – Exit • Individual(s) contact Safety System Operator (SSO) and identify area wanting to exit. • Unlock Door 2 and allow individual(s) to enter access room • Door 2 is then locked • Individual(s) place key(s) back into key bank • SSO records the out the time of each individual exiting the segment • Last person places the Exchange Key if all of the keys are back into the key bank. • SSO unlocks Door 1 • Individual(s) exit access room. • SSO locks Door 1 • Controlled Access Complete D. Sexton & H. Robertson, Design Review for Improvements to Access Process 6

Issues Requiring Attention Several issues arose relating to human factors that contributed to the event. • Employee #2 failed to observe the PSS segment message display to determine the status. **Area to examine for improvement • The acting SSO wasn’t focused solely on SSO duties since he was filling in. – SSO’s main focus is the interface with the PSS. – Multi-tasking contributed; The acting SSO was also the on-duty CC **Addressed with reinforced training • Current access logic allows the SSO to leave both doors unlocked. – The interlocks are tied to the actual door switches not the locking feature. – Door locks are considered depth in defense controls **Area to examine for improvement D. Sexton & H. Robertson, Design Review for Improvements to Access Process 7

Proposed Change Options Many options were considered for ways to address what went wrong.  Should additional indications of machine state be present outside of the access rooms?  Add beacons or change the message display for more awareness?  Are the audible announcements intelligible?  Is an additional speaker needed in the hut?  Should the SSO access control panel physically be changed?  Look at ergonomics of the safety console.  How to deal with the various states of the PSS where both doors can be unlocked?  Should the access control process be modified?  Design the access control logic for each segment to ensure compliance with the defined procedure. D. Sexton & H. Robertson, Design Review for Improvements to Access Process 8

Review of Potential Solutions Should Additional Indications be Present Outside of the Access Rooms? - Adding beacons outside of the access rooms. - Could potentially add confusion in Sweep and Controlled Access. - Ensure all of the message displays are operational before Summer/Fall 2015 Certification. - Most units currently in operation are from the original installation which are no longer available or supported. - Failed units are being replaced with a new model from a different vendor. - Test audible announcements for all areas for functionality and clarity. - This was accomplished during the most recent certification. D. Sexton & H. Robertson, Design Review for Improvements to Access Process 9

Review of Potential Solutions Should the SSO access controls panel physically be changed? - Different styles of switches and indicators were reviewed for improved usability. - Over complicates the panel design and user interface D. Sexton & H. Robertson, Design Review for Improvements to Access Process 10

Review of Potential Solutions Should the access control process be modified? - Utilize the status of the door locks, lock controls, and the PSS segment state within the segment PLCs to generate a state machine. - Disallow improper door locking/unlocking sequence and provide indications of improper sequence. - Address concerns noted by MCC Ops - Enhance the door locking logic by removing “switch memory” - Currently when dropping out of an exclusion state, the last push button state issued is maintained. - Ex: If while in PP, the SSO mistakenly pushes the segment door lock button the logic will allow the door to unlock when the segment state is lowered to CA. D. Sexton & H. Robertson, Design Review for Improvements to Access Process 11

Logic Change Proposal What are we solving? The capability for two doors to be in an unlocked state at the same time during Sweep and Controlled Access. – Change of door control paradigm - The approach is to have the default door lock state as locked rather than unlocked. - Transitions up in state apply more restrictive controls on the door locks - Transitions down retain locked condition of previous state - Unlocking requires action by SSO which is validated by the logic – Solution was developed and pre-tested in SSG lab – Tested in the BSY and Tagger segments during latest certification D. Sexton & H. Robertson, Design Review for Improvements to Access Process 12

Logic Change Proposal D. Sexton & H. Robertson, Design Review for Improvements to Access Process 13

Procedural and Training Changes  Update the PSS Controlled Access Procedure to include a step to have SSO ensure all doors are locked (ex. visually inspect the indicator buttons)  Train the SSOs on all of the interim procedural changes. • Train the SSOs on the new engineering and process changes D. Sexton & H. Robertson, Design Review for Improvements to Access Process 14

Process Changes - Implement the proposed logic changes in the LERF PSS logic prior to start of certification on March 31, 2015. – Verify with the actual hardware and interface panel for the FEL – HCO scheduled for the first day to run through the various states. - Implement the logic changes in all PSS segments during the CEBAF PSS certifications scheduled for August of 2015. – This change is delayed because once implemented a re-certification will be required for all segments. D. Sexton & H. Robertson, Design Review for Improvements to Access Process 15