ProtoDUNE/DUNE timing system and CCM Stoyan Trilov, University of - - PowerPoint PPT Presentation
ProtoDUNE/DUNE timing system and CCM Stoyan Trilov, University of Bristol CCM meeting 13/11/2019 1 Outline ProtoDUNE I (overall) startup Timing system control layers CCM Control software Firmware/hardware Behaviour of
1
▪ ProtoDUNE I (overall) startup ▪ Timing system control layers
– CCM – Control software – Firmware/hardware
▪ Behaviour of timing system: current (ProtoDUNE I) → future (ProtoDUNE II/DUNE)
– Timing master – Timing endpoint – GPS signal loss, and recovery – Timing master hot-swap – Commands for synchronisation signal 13/11/2019 2
13/11/2019 3
▪ Getting ProtoDUNE I to “RUNNING” state
requires interplay between all systems
– Timing – DAQ – Trigger – Front-end electronics
▪ Should the timing system live outside the FSM in
ProtoDUNE II/DUNE?
– i.e. almost like infrastructure, e.g. power – Not currently the case in ProtoDUNE
13/11/2019 4
CCM Timing system control software Hardware/firmware Debug and development
Commands Information Debug information/commands
13/11/2019 5 ▪ Holds timing system configuration – How many endpoints (with their UID) are expected to be found? – Are all expected endpoints supposed to be functioning correctly? – Delay value for each endpoint ▪ Sends commands to timing system via intermediate control layer – Configure these set of endpoints right now ▪ Receives distilled information from timing system – Status of GPS signal lock – Number of found endpoints and their status ▪ Common interface for SP and DP timing systems?
CCM Timing system control software Hardware/ firmware Debug and development
13/11/2019 6 ▪ Has knowledge of low level hardware/firmware API – Can query and report low level information from timing system – Send low level commands to timing master, and endpoints (via master?) – Useful for lower level debugging and during development/commissioning ▪ Able to aggregate status information of timing system – Condense multiple datapoints into single statuses, e.g. OK, WARNING, ERROR – Automate certain functionality, e.g. recovery of lost endpoints, periodic delay adjustments for end points, timing master hot-swapping ▪ Receive/execute commands from CCM and debug interface (humans) – Reset and configure particular endpoint(s)
CCM Timing system control software Hardware/ firmware Debug and development
13/11/2019 7 ▪ Receives/executes commands from the control software – e.g. commands implemented in firmware ▪ Low level state machine – Different and invisible to CCM, e.g. waiting for CDR lock
➢ However information still available via debugging interface
▪ Performs delay alignment – Measures correct delay to be used, and applies it
CCM Timing system control software Hardware/ firmware Debug and development
▪ ProtoDUNE I timing master implemented as an AIDA TLU
– It does not use a GPS oscillator – Derives 10 MHz signal from DP WR system (WR-LEN)
▪ Before entering “RUNNING” state, the master needs to
– Load correct configuration, IP address, etc. – Receive relevant commands. e.g. “start”
▪ Possible causes of malfunction
– Loss of WR 10 MHz signal → switch to internal oscillator
➢ System continues to function, but clock will drift from GPS time
13/11/2019 8
9 ▪ Envisage timing system in ProtoDUNE II/DUNE in an
“always on” state
– i.e. if there is power, there should be an active timing system – Master/spare selection done by software (DUNE only)
▪ On startup either from cold start or reset
– Wait for stable 10 MHz from GPS/and subsequent lock?
➢ GPS oscillator startup and monitoring?
– Power on/Initialise fanout AMC/FMCs? – Signal lock for each unit? – Configure endpoints
13/11/2019 10 Startup sequence State Description 0x0 Standing by 0x1 Waiting SFP for signal 0x2 Waiting CDR lock 0x3 Waiting for good frequency check 0x4 Waiting for comma alignment 0x5 Waiting for 8b10 decoder good packet 0x6 Waiting for time stamp initialisation Waiting for delay setup 0x8 Ready Error states 0xc Error in Rx 0xd Error in time stamp check 0xe Error in physical layer after lock ▪ Firmware FSM for an endpoint shown on the left – Full version would be hidden to CCM? – Only error and condensed status states propagated upwards?
▪ In case of an endpoint malfunction
– Board state machine moves to error
▪ In theory to recover an endpoint, only actions against the endpoint
are required – e.g. reset, or firmware reload – Not actually implemented in ProtoDUNE due to the lack of two-way communication with the endpoints
➢ Photon system laser issue
13/11/2019 11
▪ In case of an endpoint failure, an error state is raised
– Flag propagated up to CCM by endpoint – CCM keeps track for how long the endpoint is down
▪ Automated recovery procedure
– Attempt recovery
➢ Reset, firmware reload, etc.
– What happens if automated recovery procedure failure?→ manual intervention, …
▪ Use mock endpoints to diagnose endpoint issue
– Commissioning
▪ ProtoDUNE II
– It is envisaged to have its own GPS disciplined oscillator providing 10 MHz and time signals to the timing system – GPS signal loss may not necessarily mean clock drift, depends on duration
▪ DUNE
– It will have 2 redundant GPS disciplined oscillators providing 10 MHz and time signals to 2 redundant timing systems (hot swap) – What happens in case of GPS signal loss in active system? May not have to trigger a swap immediately. – Use Rb oscillator + software to cross-check lock of the 2 GPS oscillators
▪ GPS oscillator is certainly going to output a host of monitoring information (SNMP?)
– GPS signal lock – Clock frequency information – FSM? 13/11/2019 12
13/11/2019 13
▪ The two redundant timing systems transmit into a passive combiner
– Only one system transmits at any one time, one is active, one is in standby – Allows firmware upgrades/tests without downtime
▪ How do you decide if the redundant system should take over?
– Software control system which monitors both systems and detects faults? – Cross-communication between the two systems? – What level of fault triggers the swap? Warnings vs errors? – Redundant systems is not a new concept, principles and protocols already developed → research
▪ Testing hot-swap functionality will be a major and important task
13/11/2019 14
▪ ProtoDUNE I
– Regular timestamp messages sent once ~1s – Sync signals are currently enabled following reset
➢ done via (pdtbutler), i.e. pdtbutler io PDTS_TERTIARY reset
▪ ProtoDUNE II/DUNE
– When are sync commands enabled? – Add/remove commands from current list? 13/11/2019 15
Sync commands TimeSync 0x0 Echo 0x1 SpillStart 0x2 SpillStop 0x3 RunStart 0x4 RunStop 0x5 WibCalib 0x6 SSPCalib 0x7 FakeTrig0 0x8 FakeTrig1 0x9 FakeTrig2 0xa FakeTrig3 0xb BeamTrig 0xc NoBeamTrig 0xd ExtFakeTrig 0xe
▪ Endpoint calibration and monitoring testing
– Requires fix of PDS laser issue – Can also be tested in an lab environemnt
▪ Development/test of software allowing endpoints to dropout and rejoin DAQ ▪ GPS oscillator setup and testing
– Prototyping DUNE-like system. mTCA, etc. 13/11/2019 16