NP04 Powercut Recovery and Cold Starts Pengfei Dine, Bonnie King, Geoff Savage DUNE DAQ meeting 29 July 2019
Power cut #1 July 23 • Power cut with subsequent cooling failure brought down all servers except np04-srv-001 and np04-srv-004 (on UPS) 2
Power cut #1 July 23 recovery • Manually pressed power button on servers without IPMI configured/cabled (and IPMI head nodes) • Issued power on commands via IPMI where possible with no particular order ● cronjob to delay reboot in place and mounts came back correctly ● had to restart supervisord where it came up before NFS mount ● had to mount CIFS mount manually 3
Power cut #2 • Had to power down servers again due to water pressure drop • This time, non-critical servers were gracefully shut down and IPMI head nodes kept up • Most servers came back except srv-0[03, 04, 10. 11, 12, 21, 22, 24] due to known issue (keystroke required to boot) 4
Power cut #2 recovery • Some RAID volumes (not on UPS) got upset after the power cut ● recovered mounts on np04-srv-003 and np04-srv-004 by manually assembling volumes ● raw data was written to some mount areas while mount was missing, filling up / ● moved data to correct area Need to redirect boot loader and kernel init to serial console for IPMI access (plan to do this later today) 5
Planned improvements • Get serial console redirection to Serial Over Lan working during boot (can send keystrokes remotely) • Configure supervisord with correct startup dependency • Audit ansible playbooks; remove outdated playbooks, create playbook to provision new node from scratch • Alerting in Prometheus ● disk usage, missing mounts, etc etc • Ansible roles for work done in test period • np04-onl-XXX cabled for IPMI? • np04-srv-007 IPMI cable 6
Recommend
More recommend
