The LHCb Experiment Control System Clara Gaspar, May 2016 Control - PowerPoint PPT Presentation

The LHCb Experiment Control System Clara Gaspar, May 2016

Control System Scope ❚ In charge of the Control and Monitoring of all areas of the experiment Detector & General Infrastructure (Power, Gas, Cooling, etc.) Experiment Control System Detector Channels Trigger TFC Front End Electronics Readout Boards High Level Trigger (Farm) Storage Monitoring DAQ External Systems (LHC, Technical Services, Safety, etc.) Clara Gaspar, May 2016 2

Control System Architecture ECS INFR. DCS TFC HLT LHC DAQ Status & Alarms Commands … … SubDet1 SubDet2 SubDetN SubDet1 SubDet2 SubDetN DCS DCS DCS DAQ DAQ DAQ SubDet1 SubDet1 SubDet1 SubDet1 SubDet1 Legend : LV TEMP GAS FEE RO Control Unit … … Device LV LV LV Unit FEE FEE FEE Dev1 Dev2 DevN Dev1 Dev2 DevN Clara Gaspar, May 2016 3

LHC Experiments: JCOP ❚ The JCOP Framework is based on: ❙ SCADA System – WinCC-OA for: ❘ Device Description (Run-time Database) ❘ Device Access (OPC, Profibus, drivers) Device Units ❘ Alarm Handling (Generation, Filtering, Masking, etc) ❘ Archiving, Logging, Scripting, Trending Control Units ❘ User Interface Builder ❘ Alarm Display, Access Control, etc. ❙ SMI++ providing: ❘ Abstract behavior modeling (Finite State Machines) ❘ Automation & Error Recovery (Rule based system) Clara Gaspar, May 2016 4

Device Units Device Unit ❚ Provide access to “real” devices: ❙ The Framework provides (among others): ❘ “Plug and play” modules for commonly used equipment. For example: 〡 CAEN or Wiener power supplies (via OPC) 〡 LHCb CCPC and SPECS based electronics (via DIM) ❘ A protocol (DIM) for interfacing “home made” devices. For example: 〡 Hardware devices like a calibration source 〡 Software devices like the Trigger processes (based on LHCb’s offline framework – GAUDI) ❘ Each device is modeled as a Finite State Machine Clara Gaspar, May 2016 5

Hierarchical control Control Unit ❚ Each Control Unit: ❙ Is defined as one or more Finite State Machines ❙ Can implement rules based on its children’s states ❙ In general it is able to: ❘ Summarize information (for the above levels) ❘ “Expand” actions (to the lower levels) ❘ Implement specific behaviour DCS & Take local decisions … 〡 Sequence & Automate operations Tracker Muon DCS DCS 〡 Recover errors ❘ Include/Exclude children (i.e. partitioning) Muon Muon 〡 Excluded nodes can run is stand-alone LV GAS ❘ User Interfacing 〡 Present information and receive commands Clara Gaspar, May 2016 6

Hierarchical Control Tools OFF Switch_ON Switch_OFF ON Recover ERROR ❚ Build FSM hierarchy across different machines ❚ Dynamically generated Operation UIs ❙ Embedded Partitioning: Include, Exclude, etc. Clara Gaspar, May 2016 7

LHCb Operations ❚ Main Tools: ❙ RunControl ❘ Handles the DAQ & Dataflow ❘ Allows to: ❘ Configure the system ❘ Start & Stop runs ❙ AutoPilot ❘ Knows how to start and keep a run going from any state. ❙ BigBrother ❘ Based on the LHC state: ❘ Controls SD Voltages ❘ VELO Closure ❘ RunControl ❙ AlarmScreen Clara Gaspar, May 2016 8

Run Control ❚ Matrix Domain x Sub-detector ❚ Activity Used to configure all sub-system Clara Gaspar, May 2016 9

Alarm Screen Clara Gaspar, May 2016 10

Other Monitoring Tools Clara Gaspar, May 2016 11

ECS: Some numbers ❚ Size of the Control Tree: ECS ❙ Distributed over ~200 PCs ❘ Mostly Linux (VMs) HV DCS TFC DAQ HLT LHC ❘ Some Windows ❙ >8000 Control Units DCS … DAQ … SubDet1 SubDetN SubDet1 SubDetN ❙ >50000 Device Units DCS DAQ ❚ Run Control Timing ❙ Cold Start to Running: 4 minutes ❘ Configure all Sub-detectors, Start & Configure ~50000 HLT processes (always done well before PHYSICS) ❙ Stop/Start Run: 6 seconds Clara Gaspar, May 2016 12

Conclusions ❚ LHCb has designed and implemented a coherent and homogeneous control system ❚ The complete experiment: ❙ Is operated by only 1 person ❙ Is almost completely automated (basically only confirmations from Operator) ❚ Thanks to the use of the JCOP Framework (and it’s many features, tools and components) ❙ The manpower needs are very low (and can be shared between sub-systems) ❙ The development time is quite short ❙ Training and support available and large user community Clara Gaspar, May 2016 13

Backup Clara Gaspar, May 2016 14

FSM Operation Domains ❚ DCS Domain ❚ HV Domain Equipment’s operation related to a Equipment’s operation related to the LHC State (Ex: High Voltages) running period (Ex: GAS, Cooling) Recover Switch_OFF Recover ERROR NOT_READY ERROR NOT_READY OFF OFF Go_STANDBY1 RAMPING_STANDBY1 Switch_ON Switch_OFF READY STANDBY1 Go_STANDBY2 ❚ DAQ Domain RAMPING_STANDBY2 STANDBY2 Equipment’s operation related to a Go_READY RAMPING_READY “RUN” (Ex: RO board, HLT process) READY Recover ERROR UNKNOWN NOT_READY ❚ LHCb FSM Templates Configure CONFIGURING Provided to all Sub-systems Reset READY Start Stop RUNNING Clara Gaspar, May 2016 15