The LHCb Experiment Control System Clara Gaspar, May 2016 Control - - PowerPoint PPT Presentation

Clara Gaspar, May 2016

The LHCb Experiment Control System

Clara Gaspar, May 2016

❚ In charge of the Control and Monitoring of all areas of the experiment

2

Control System Scope

Detector Channels Front End Electronics Readout Boards High Level Trigger (Farm) Storage Trigger

Experiment Control System

DAQ

Detector & General Infrastructure (Power, Gas, Cooling, etc.) External Systems (LHC, Technical Services, Safety, etc.) TFC Monitoring

Clara Gaspar, May 2016

3

Control System Architecture

LV Dev1 LV Dev2 LV DevN

DCS

SubDetN DCS SubDet2 DCS SubDet1 DCS SubDet1 LV SubDet1 TEMP SubDet1 GAS

… …

Commands

Control Unit Device Unit

DAQ

SubDetN DAQ SubDet2 DAQ SubDet1 DAQ SubDet1 FEE SubDet1 RO FEE Dev1 FEE Dev2 FEE DevN

… …

Legend:

INFR. TFC LHC ECS HLT

Status & Alarms

Clara Gaspar, May 2016

4

❚ The JCOP Framework is based on:

❙ SCADA System – WinCC-OA for:

❘ Device Description (Run-time Database) ❘ Device Access (OPC, Profibus, drivers) ❘ Alarm Handling (Generation, Filtering, Masking, etc) ❘ Archiving, Logging, Scripting, Trending ❘ User Interface Builder ❘ Alarm Display, Access Control, etc.

❙ SMI++ providing:

❘ Abstract behavior modeling (Finite State Machines) ❘ Automation & Error Recovery (Rule based system)

LHC Experiments: JCOP

Device Units Control Units

Clara Gaspar, May 2016

5

Device Units

❚ Provide access to “real” devices:

❙ The Framework provides (among others):

❘ “Plug and play” modules for commonly used

• equipment. For example:

〡 CAEN or Wiener power supplies (via OPC) 〡 LHCb CCPC and SPECS based electronics (via DIM)

❘ A protocol (DIM) for interfacing “home made” devices. For example:

〡 Hardware devices like a calibration source 〡 Software devices like the Trigger processes (based on LHCb’s offline framework – GAUDI)

❘ Each device is modeled as a Finite State Machine

Device Unit

Clara Gaspar, May 2016

6

Hierarchical control

❚ Each Control Unit:

❙ Is defined as one or more Finite State Machines ❙ Can implement rules based on its children’s states ❙ In general it is able to:

❘ Summarize information (for the above levels) ❘ “Expand” actions (to the lower levels) ❘ Implement specific behaviour & Take local decisions

〡 Sequence & Automate operations 〡 Recover errors

❘ Include/Exclude children (i.e. partitioning)

〡 Excluded nodes can run is stand-alone

❘ User Interfacing

〡 Present information and receive commands

DCS

Muon DCS Tracker DCS

…

Muon LV Muon GAS Control Unit

Clara Gaspar, May 2016

❚ Build FSM hierarchy across different machines ❚ Dynamically generated Operation UIs

❙ Embedded Partitioning: Include, Exclude, etc. 7

Hierarchical Control Tools

ON OFF ERROR Switch_ON Switch_OFF Recover

Clara Gaspar, May 2016

LHCb Operations

8

❙ RunControl

❘ Handles the DAQ & Dataflow ❘ Allows to: ❘ Configure the system ❘ Start & Stop runs

❙ AutoPilot

❘ Knows how to start and keep a run going from any state.

❙ BigBrother

❘ Based on the LHC state: ❘ Controls SD Voltages ❘ VELO Closure ❘ RunControl

❙ AlarmScreen

❚ Main Tools:

Clara Gaspar, May 2016

Run Control

9

❚ Matrix

Domain x Sub-detector

❚ Activity

Used to configure all sub-system

Clara Gaspar, May 2016

10

Alarm Screen

Clara Gaspar, May 2016

11

Other Monitoring Tools

Clara Gaspar, May 2016

12

ECS: Some numbers

DCS

SubDetN DCS SubDet1 DCS …

DAQ

SubDetN DAQ SubDet1 DAQ …

HV TFC LHC HLT ECS

❚ Size of the Control Tree:

❙ Distributed over ~200 PCs

❘ Mostly Linux (VMs) ❘ Some Windows

❙ >8000 Control Units ❙ >50000 Device Units

❚ Run Control Timing

❙ Cold Start to Running: 4 minutes

❘ Configure all Sub-detectors, Start & Configure ~50000 HLT processes (always done well before PHYSICS)

❙ Stop/Start Run: 6 seconds

Clara Gaspar, May 2016

❚ LHCb has designed and implemented a coherent and homogeneous control system ❚ The complete experiment:

❙ Is operated by only 1 person ❙ Is almost completely automated (basically only confirmations from Operator)

❚ Thanks to the use of the JCOP Framework

(and it’s many features, tools and components) ❙ The manpower needs are very low (and can be shared between sub-systems) ❙ The development time is quite short ❙ Training and support available and large user community

13

Conclusions

Clara Gaspar, May 2016

14

Backup

Clara Gaspar, May 2016

15

FSM Operation Domains

❚ DCS Domain

Equipment’s operation related to a running period (Ex: GAS, Cooling)

❚ HV Domain

Equipment’s operation related to the LHC State (Ex: High Voltages)

❚ DAQ Domain

Equipment’s operation related to a “RUN” (Ex: RO board, HLT process)

READY STANDBY1 OFF ERROR Recover STANDBY2 RAMPING_STANDBY1 RAMPING_STANDBY2 RAMPING_READY NOT_READY Go_STANDBY1 Go_STANDBY2 Go_READY RUNNING READY NOT_READY Start Stop ERROR UNKNOWN Configure Reset Recover CONFIGURING READY OFF ERROR NOT_READY Switch_ON Switch_OFF Recover Switch_OFF

❚ LHCb FSM Templates Provided to all Sub-systems