Monitoring of the DAQ2 system DAQ2 Shift Tutorial 2 cDAQ group - - PowerPoint PPT Presentation

▶
monitoring of the daq2 system
SMART_READER_LITE
LIVE PREVIEW

Monitoring of the DAQ2 system DAQ2 Shift Tutorial 2 cDAQ group - - PowerPoint PPT Presentation

Nov 05, 2023 408 likes •758 views

DAQ2 Shift Tutorial 1 cDAQ group Monitoring of the DAQ2 system DAQ2 Shift Tutorial 2 cDAQ group Monitoring tools RCMS/LVL0 interface 1. Has been covered by Hannes aDAQMon 2. Overview screen to see at a glance the CMS running

slide-1
SLIDE 1

DAQ2 Shift Tutorial cDAQ group 1

Monitoring of the DAQ2 system

slide-2
SLIDE 2

DAQ2 Shift Tutorial cDAQ group 2

Monitoring tools

1.

RCMS/LVL0 interface

 Has been covered by Hannes

2.

aDAQMon

 Overview screen to see at a glance the CMS running configuration and rates.

3.

DAQView

 Most comprehensive monitoring tool for issues with data flow. Here you can

monitor the data from FEDs to BUs.

4.

Elastic Search / Filter Farm monitoring (File Merging)

 Shows the progress of file merging before being sent to T0. Important monitor of

transfer system. Also shows the state of the Filter farm. 5.

CPM controller

 Central Partition Manager for the TCDS system. Good place to see rates, state of

detector inputs, etc. 6.

HotSpot

 Central display for sentinel messages for errors from all processes.

slide-3
SLIDE 3

DAQ2 Shift Tutorial cDAQ group 3

aDAQmon – DAQ Summary

http://cmsonline.cern.ch/daqStatusSCX/DAQstatusGre.html Status bar gives a quick overview of the DAQ

slide-4
SLIDE 4

DAQ2 Shift Tutorial cDAQ group 4 Main systems (LHC, DCS,...) status FED-RU data stream FED RU configuration Box color: Sub-Sys ID RU/BU box color: CPU 100% FED IN FED OUT RU bandwidth plot BU bandwidth plot # Ev. in BU BU RAM disk % BU OUT disk % DAQ Sub-Sys configuration RU/BU box RED frame: flash data not updated Event storage summary

slide-5
SLIDE 5

DAQ2 Shift Tutorial cDAQ group 5

DAQView

slide-6
SLIDE 6

DAQ2 Shift Tutorial cDAQ group 6

DAQView

Status & navigation FED Builder FEROL/FMM Event Builder RU/EVM FFF Appliances BU & FU Age of monitor data

slide-7
SLIDE 7

DAQ2 Shift Tutorial cDAQ group 7

DAQView - Navigation

Stop refreshing page Switch pages between FEDbuilder, FFF, and all You only need cDAQ

Start DAQView if it is not running

Current run Duration and start time of run (or last restart of DAQView) Last update of page must be current!

If it is stale, you need to restart DAQView

slide-8
SLIDE 8

DAQ2 Shift Tutorial cDAQ group 8

DAQView – FED builder

TTC partition name & no. Current TTS state of partition %warning, %busy in TTS partition FEROL PC

(link to hyperdaq page)

FED information (see next page) min/max # fragments received by FEROL. Highlighted in yellow if different to trigger. Min is only displayed if not equal to max. FED builder name Confused? Try the table help button!

slide-9
SLIDE 9

DAQ2 Shift Tutorial cDAQ group 9

DAQView – FEROL and FMM

 Entries are of form

 FRL_geoslot: FEDSourceID or  FRL_geoslot: FEDSourceID1, FEDSourceID2 or  FEDSourceID

 For a pseudo-FED (=TTS link only, but no data is read out by DAQ)

 Additional info may be displayed next to the FEDSourceID

(from left to right)

 Percentage of time during which FED was in Warning ( ) or Busy ( ) during

the last 3 seconds (if non-zero)

 Current state of TTS if other than Ready  FEDSourceID (expected) 601

 Grey if FRL input not enabled (FMM not enabled in case of pseudo-FED)  Highlighted in color of current TTS state if other than Ready

 Percentage of time with DAQ backpressure

during last update interval (5s) if non-zero

 Warnings

 Received source ID different to expected  FED or SLINK CRC errors  Number of fragments received by FRL if no data is flowing and this FRL is lagging “behind”

Use this to judge whether a FED is creating dead-time because of a FED problem or because of DAQ-backpressure

W:9.9% B:0.2% W <6.9%

#FCRC=69 9605

slide-10
SLIDE 10

DAQ2 Shift Tutorial cDAQ group 10

DAQView – RU/EVM Information

EVM/RU host

(link to hyperdaq page)

First row is TCDS / EVM Rate (kHz) # fragments built by RU/EVM since start of run # incomplete fragments

>> 1 indicates a problem on the RU

Throughput (MB/s) Super-fragment size (kB) # events currently in RU

>>1 indicates problem in IB

# events requested by BU

normal EVM >> 1 && RUs < 100

Each row is one FEDbuilder Shaded values mean FEDbuilder is not in readout

slide-11
SLIDE 11

DAQ2 Shift Tutorial cDAQ group 11

DAQView – FFF/BU

BU host

(link to hyperdaq page)

Rate per BU (kHz) Throughput (MB/s) Event size (kB) Confused? Try the table help button! Events built since start of run # events being built Resource information (see next page) # files written # LS for which there is a file Current LS number Each line is one Appliance

slide-12
SLIDE 12

DAQ2 Shift Tutorial cDAQ group 12

DAQView – BU Resources

BU resources are used for requesting events

Each resource corresponds to multiple events

Less resources mean less event requests to EVM

Load balancing between independent appliances

Backpressure mechanism if FFF/HLT cannot keep up

Each BU has a number of resources (#resources)

Resources can be blocked (#blocked)

RAM disk becomes full

Not enough FU CPU cores are available to process data

FU processing lags behind

Resources for which no event data has been received are counted under #requests

If #requests > 0, the BU is able to accept new events

slide-13
SLIDE 13

DAQ2 Shift Tutorial cDAQ group 13

DAQView – Running, or not?

LVL0: DAQ is running No, rate is 0 kHz None of the HF FEDs has sent any events No fragments in RU Many events requested No data flow as HF has not sent any data  Talk to HF expert

slide-14
SLIDE 14

DAQ2 Shift Tutorial cDAQ group 14

DAQView – Who Blocks the Run?

ECAL is 100% in Warning Rate is 0 kHz FED 602 is in warning and last event is 9605 There’s backpressure from DAQ RU waits for data from FED 59 FED 59 has not sent any data FED 59 is the culprit  Talk to Tracker expert

slide-15
SLIDE 15

DAQ2 Shift Tutorial cDAQ group 15

DAQView – DAQ backpressure

ECAL is 50% in Warning There’s backpressure from DAQ Very few events requested by BUs All BUs are “blocked” or “throttled” RAM disk is full All resources blocked RAM disk is nearly full 25/32 resources blocked No FU cores available All resources blocked Only a few FU cores available 26/32 resources are blocked FFF is blocked  Try to figure out what is wrong (and call DAQ oncall) The rate is 10 kHz

slide-16
SLIDE 16

DAQ2 Shift Tutorial cDAQ group 16

F3 Monitor

slide-17
SLIDE 17

DAQ2 Shift Tutorial cDAQ group 17

Storage & Transfer System

1 7

Aggregate files (event data, DQM histograms & metadata) as they appear

Micro-merger on each FU aggregates the data from all processes on the FU Mini-merger on the BU aggregates the data from all FUs Mega-merger(s) aggregate the data from all BUs

Data and meta-data are aggregated per luminosity section

Each luminosity section and stream treated independently If previous step has completed successfully, input data can be deleted

slide-18
SLIDE 18

DAQ2 Shift Tutorial cDAQ group 18

F3 Monitor http://cmsdaq0/daqfff/ecd/

Nice demo available at http://cmsdaq0/daqfff/ecd/doc/presentation/ List of recent runs Access old runs Active run Both boxes must be green Time chart of HLT activity

Confused? Try the guide!

Stream rates vs LS Stream names

(click to hide them)

Completeness of data

Alert DAQ oncall when multiple boxes are not green (this situation is okay)

slide-19
SLIDE 19

DAQ2 Shift Tutorial cDAQ group 19

Central Partition Manager

slide-20
SLIDE 20

DAQ2 Shift Tutorial cDAQ group 20

TCDS

 Combines the pre-LS1:

 Trigger Control System (TCS) The conductor of all CMS triggering and data-taking  Trigger Timing and Control (TTC) The distributor of clock, L1As, and synchronisation signals  Trigger Throttling System (TTS) The feedback of readiness states from FEDs to TCS

 Many-legged creature:

 The ‘head’ is the Central Partition Manager (controlled by central DAQ)  Many different legs (i.e., partitions) across the different subsystems (controlled by the

subsystems)

slide-21
SLIDE 21

DAQ2 Shift Tutorial cDAQ group 21

TCDSCentral tcds-control-central.cms:2000/urn:xdaq-application:lid=100

slide-22
SLIDE 22

DAQ2 Shift Tutorial cDAQ group 22

TCDSCentral tcds-control-central.cms:2000/urn:xdaq-application:lid=100

TTC machine interface applications Provide the connection between the LHC RF and timing signals and CMS.

slide-23
SLIDE 23

DAQ2 Shift Tutorial cDAQ group 23

TCDSCentral tcds-control-central.cms:2000/urn:xdaq-application:lid=100

Central Partition Manager (CPM) Drives CMS. Controls triggers, calibration sequence, timing and synchronisation, … This application should tell you what and how many triggers are flowing,

  • r why not.
slide-24
SLIDE 24

DAQ2 Shift Tutorial cDAQ group 24

CPMControllertcds-control-central.cms:2050/urn:xdaq-application:lid=100

Running state shows if triggers are flowing or why not: Stopped Running Blocked by TTS Blocked by DAQ backpressure etc.

Hardware status tab

slide-25
SLIDE 25

DAQ2 Shift Tutorial cDAQ group 25

CPMControllertcds-control-central.cms:2050/urn:xdaq-application:lid=100

Running state: Stopped Running Blocked by TTS Blocked by DAQ backpressure etc. shows what can/will block triggers TTS and trigger blockers tab

slide-26
SLIDE 26

DAQ2 Shift Tutorial cDAQ group 26

CPMControllertcds-control-central.cms:2050/urn:xdaq-application:lid=100

Running state: Stopped Running Blocked by TTS Blocked by DAQ backpressure etc. This shows which partition is not TTS-READY TTS and trigger blockers tab

slide-27
SLIDE 27

DAQ2 Shift Tutorial cDAQ group 27

CPMControllertcds-control-central.cms:2050/urn:xdaq-application:lid=100

This tab shows:

  • What rate of triggers are flowing, per type
  • What rate of triggers are being suppressed, per type
  • What the deadtime is, per source
  • How much time each partition spends in TTS not-READY

(at the bottom)

Rates and deadtimes tab

slide-28
SLIDE 28

DAQ2 Shift Tutorial cDAQ group 28

CPMControllertcds-control-central.cms:2050/urn:xdaq-application:lid=100

Add random triggers

Input sources

slide-29
SLIDE 29

DAQ2 Shift Tutorial cDAQ group 29

HotSpot

Make sure that it updates (pulsates) Check regularly for Errors or Fatal by clicking on corresponding button

slide-30
SLIDE 30

DAQ2 Shift Tutorial cDAQ group 30

HotSpot

Click on error Analyze the error and take appropriate action You can use HTML to copy it into the elog Acknowledge understood errors

slide-31
SLIDE 31

DAQ2 Shift Tutorial cDAQ group 31

Handsaw

 Running in a terminal on the shifter console

 You need an account in the online cluster to start it

 Scrolling display of error messages from DAQ

 All messages (and more) are in HotSpot or LVL0  Handsaw is often quicker to find the most relevant message

slide-32
SLIDE 32

DAQ2 Shift Tutorial cDAQ group 32

What to do if it does not work

 Don’t panic! Keep cool.

 Not always easy, especially during stable beams  Think before clicking!  GUIs are sometimes slow in reacting. Be patient…

 Look for error messages (LVL0, HotSpot, Handsaw)  Look at DAQView for anything suspicious

 Figure out what subsystem is causing problems  Be aware that one subsystem might get backpressure from DAQ due to other

issues  Talk to the shift leader and other shifters

 They might be aware of problems affecting DAQ  E.g. if a subsystem lost power, DAQ will go into error

(you might be the first to realize it!)

 If you are unsure or stuck, don’t hesitate to call the DAQ oncall

anytime (76600)

slide-33
SLIDE 33

DAQ2 Shift Tutorial cDAQ group 33

Documentation and Resources

 DAQ2 shifters guide twiki page

 https://twiki.cern.ch/twiki/bin/view/CMS/ShiftPourNuls2014

 The left bar of the DAQ2 shifters guide has many valuable links:

DAQ shifter bulletin board: read before every shift. DAQ shifter hypernews: subscribe to this! All DAQ shift related announcements are sent here DAQ ELOG: Link to DAQ area of the ELOG DAQ Shift Tutorial: link to slides from shift tutorial Glossary of DAQ Terms: definition of all the DAQ acronyms. Expert on call: link to DAQ DOC area of shift tool Expert List: link to list of DAQ and HLT experts DAQ shift schedule: link to DAQ shifters area of shift tool P5 shuttle: link to shuttle schedule