Roman Gorbonosov on behalf of the Beams Department Controls Group - PowerPoint PPT Presentation

Roman Gorbonosov on behalf of the Beams Department Controls Group Based on the input from M.Arruat, V.Baggiolini, JC.Bau, M.Buttner, P.Charrue, S.Deghaye, E.Hatziangeli, G.Kruk, M.Lamont, A.Radeva, U.Raich, C.Roderick, J.Serrano, W.Sliwinski, J.Wozniak CERN – Geneva - Switzerland Accelerators and Technology Sector Beams Department - Controls Group

 LHC control system requirements  Philosophy of development  Overview of the architecture  Key components  Quality Assurance (QA)  Outlook towards the Future 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 2

Presentation PUBLIC ETHERNET Layer NETWORK Core Control GUIs Fixed Displays DB access OPERATOR FIXED OPERATOR Core Control GUIs Fixed Displays DB access CONSOLES DISPLAYS CONSOLES Core Control GUIs Fixed Displays DB access TCP/IP communication services Java RMI, JMS Role Based Access (RBAC) Business CERN GIGABIT ETHERNET Layer Software FILE APPLICATION SCADA TECHNICAL NETWORK SERVERS SERVERS SERVERS CTRL CTRL Diagnostics Sequencer Interlock DB LHC Monitoring DB System Settings & Software DB (DIAMON) Logging Settings & Logging Architecture TCP/IP communication services Settings & & Post Logging TIMING GENERATION Oasis Logging Alarms Mortem T T T T Timing Controls MiddleWare (CMW) RT Lynx/OS WORLDFIP PLC System VME Front Ends Front Ends Role Based Access (RBAC) – Critical Settings Management TCP/IP communication services Front End Layer FESA servers PVSS FESA servers PVSS T T FESA servers PVSS T T T 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 3

Presentation PUBLIC ETHERNET Layer NETWORK Core Control GUIs Fixed Displays DB access OPERATOR FIXED OPERATOR Core Control GUIs Fixed Displays DB access CONSOLES DISPLAYS CONSOLES Core Control GUIs Fixed Displays DB access TCP/IP communication services Java RMI, JMS Role Based Access (RBAC) Business CERN GIGABIT ETHERNET Layer Software FILE APPLICATION SCADA TECHNICAL NETWORK SERVERS SERVERS SERVERS CTRL CTRL Diagnostics Sequencer Interlock DB LHC Monitoring DB System Settings & Software DB (DIAMON) Logging Settings & Logging Architecture TCP/IP communication services Settings & & Post Logging TIMING GENERATION Oasis Logging Alarms Mortem T T T T Timing Controls MiddleWare (CMW) RT Lynx/OS WORLDFIP PLC System VME Front Ends Front Ends Role Based Access (RBAC) – Critical Settings Management TCP/IP communication services Front End Layer FESA servers PVSS FESA servers PVSS T T FESA servers PVSS T T T 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 4

Alarms Summary of problems in single place Huge number of SW processes Huge number of devices Diagnostics Monitoring 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 5

 Monitors controls infrastructure  Computers (front-ends, servers, consoles)  Network  Software applications  Provides overview of infrastructure state 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 6

14 October 2013 Academic Training Lectures - CERN Beams Controls Group 7

DiaMon GUI DiaMon JMS Server JMS DiaMon JMX SNMP DAQ DAQ DAQ JMS DiaMon DiaMon DiaMon JMX SNMP agent agent agent Consoles Front-Ends Servers Processes HW 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 8

 Helps finding the root cause of the problem  Provides evolution history  Allows certain actions  restart system  restart process 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 9

 Software alarms system  Problem => notification  Does not deal with human/equipment safety  Notifies about problems requiring human intervention 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 10

14 October 2013 Academic Training Lectures - CERN Beams Controls Group 11

DiaMon Alarms Alarms GUI Console X Console Y Alarms JMS Server DiaMon JMS JMS Server JMS DiaMon JMX SNMP DAQ DAQ DAQ JMS DiaMon DiaMon DiaMon Alarms JMX SNMP agent agent agent API Consoles Front-Ends Servers Processes HW … 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 12

 Relevant alarms only threshold Alarm? time 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 13

 Relevant alarms only threshold Alarm? time 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 14

 Relevant alarms only threshold time ON delay 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 15

 Relevant alarms only threshold time OFF delay 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 16

 Relevant alarms only threshold Dead band time 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 17

 Relevant alarms only Context threshold time 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 18

 Relevant alarms only  Quality of raised alarms  Dependent on clients: ex. different accelerators  Alarms history  Maps alarms to people and possible actions  Alarms priority 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 19

 Diagnostic Monitoring  Metrics from >2000 computers  ~10 M updates / day  Alarms  ~200 K alarm definitions (80’000 for LHC)  ~150 alarm events / minute 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 20

 Extend monitoring to process internals  JMX metrics  Periodic sanity checks  Automatic alarms analysis  Frequent, oscillating alarms => contact expert  Long-standing alarms => reconsider alarm 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 21

Presentation PUBLIC ETHERNET Layer NETWORK Core Control GUIs Fixed Displays DB access OPERATOR FIXED OPERATOR Core Control GUIs Fixed Displays DB access CONSOLES DISPLAYS CONSOLES Core Control GUIs Fixed Displays DB access TCP/IP communication services Java RMI, JMS Role Based Access (RBAC) Business CERN GIGABIT ETHERNET Layer Software FILE APPLICATION SCADA TECHNICAL NETWORK SERVERS SERVERS SERVERS CTRL CTRL Diagnostics Sequencer Interlock DB LHC Monitoring DB System Settings & Software DB (DIAMON) Logging Settings & Logging Architecture TCP/IP communication services Settings & & Post Logging TIMING GENERATION Oasis Logging Alarms Mortem T T T T Timing Controls MiddleWare (CMW) RT Lynx/OS WORLDFIP PLC System VME Front Ends Front Ends Role Based Access (RBAC) – Critical Settings Management TCP/IP communication services Front End Layer FESA servers PVSS FESA servers PVSS T T FESA servers PVSS T T T 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 22

Software Interlock System Hardware Machine Protection 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 23

 Surveys the state of key LHC components  Acts if necessary  abnormal situation  beam dump  Part of overall Machine Protection 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 24

External Data Processing 1Hz Systems Beam Interlock Timing Data Acquisition Alarms etc. Controls MiddleWare Front End Layer 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 25

Data Processing Permit G nodes AND Logical E F nodes OR AND Alarm Elementary A B C D Ignore/mask nodes 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 26

14 October 2013 Academic Training Lectures - CERN Beams Controls Group 27

 Domain conditions representation  Complex condition logic  Provides the operations with condition calculation diagnostics  Extensible  Deterministic and highly reliable 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 28

 Deployed practically for all accelerators  SIS for LHC has  ~2700 subscriptions  ~5200 elementary / ~800 logical / 8 permits  SIS for SPS is used to save energy  Up to 200’000 euro/year 14 October 2013 Academic Training Lectures - CERN Beams Controls Group 29