35 Ton Run 2 Hardware and Operational experiences Alan Hahn 4/14/16 Dune Friday Phone Meeting 1
Organization • Overall Run • Cryo • Detector “Hardware” Categories – TPC • Including HV – Cold Electronics – Photon Detectors – DAQ – Systems 4/14/16 Dune Friday Phone Meeting 2
Run: 35T Cooldown and Filling Top-Off from TRAILER Membrane Temperature sensors XFer from GAr xfer from LAPD (over weekend) ~1.5 tons LAPD LAPD Pump trip 4/14/16 Dune Friday Phone Meeting 3
Run: e - Lifetime PrM2 & 3, both long PrMs midway heights in Cryostat(these lifetimes may drop a bit (~10%) using PrM0 as a constraint) Sitewide Power Outage Bad News Period 3 ms Pump Start LN2 Cooling Loss Tubing Break 4/14/16 Dune Friday Phone Meeting 4
Run: Purity Stratification near start of Purification PrM 1 (near Cryostat bottom) PrM 4 (near surface) O2 Gas Analyzer, ~(~80 cm from bottom) 4/14/16 Dune Friday Phone Meeting 5
Geometry of 35 Ton Cryostat (from E.Voirin recent 35 Ton FEA calculations Dune DocDb 1156) Pump outflow 4 purity monitors in this corner (Geometry Not in CFD model) PrM1 Pump intake 4/14/16 Dune Friday Phone Meeting 6
New constraint Discussion of Results / Lessons Learned Phase separator cools inlet flow to approach saturation temperature, which is below the “bulk temperature” of the argon. This cooler, more dense layer of argon tends to stay at the bottom, and not mix well with the rest of the argon. Back pressure control at the phase separator could allow us to flow in warmer argon, which would then mix well with the bulk. Pump Discharge Pipe with Phase Separator on Top, LAr flows down, and impinges on cryostat floor. Pump Suction RTD measurements previous slide verifies this temperature stratification 4/14/16 Dune Friday Phone Meeting 7
Run: Stratification Explanation PrM1&4 with 3 Precision RTDs Opportunity provided by Power Outage Pump is off Pump is on 3 RTD offsets Barometric Pressure made to match 12 hrs after Outage Uppermost RTD PrM 1&4 show PrM 1&4 lifetimes Lowest RTD same lifetime diverge 4/14/16 Dune Friday Phone Meeting 8
Run: Beware the Ides of March • Three consecutive “events” ~one week apart and all on weekends 1. Site-wide power outage stopped our recirculation and dropped the LAr Purity (3/4—Friday Night) • See the loss of purity due to outage on previous slides. 2. Loss of LN 2 Cooling (broken valve) caused a loss of LAr (~1/2 ton through boil off) (3/12--Saturday) • Amount of loss because Autodialer Alarm had not been reset since the previous week’s Power Outage—Doh! – Only noticed on Sunday and fixed in afternoon • Dropped LAr level below desired HV-precaution level 3. Unrecoverable Contamination of LAr (3/17-Saturday 2 AM) • Small tube attached to GAr Bellows Pump that was being used to transfer GAr between LAPD and 35Ton broke- pulling in air and injecting it into the LAr stream. – Estimate it would take ~20 weeks of filtering/filter regeneration to remove just the O 2 , N 2 would not be removed. • Went from 100 ppt to 30 ppm O 2 in about 3/4 hr. 4/14/16 Dune Friday Phone Meeting 9
This broken tubing is on the suction side of pump that was pulling GAr out of LAPD, compressing it and injecting it into the LAr coming from 35T just before the purification filters. This was how we were raising the LAr level in the Cryostat Bellows Pump 4/14/16 Dune Friday Phone Meeting 10
Cryo Issues • Pumping – Had problems establishing the 35Ton Internal LAr Pumping – Pumps had been rebuilt after Phase 1, but had never been run in. – Finally putting 3 phase 208 AC on pump motor broke it free. • Then normal AC Drive worked fine. – Lesson learned—nothing goes into cryostat without a functionality test • Even if it worked fine last time (e.g. Phase 1). • Need a better interface into Controls and Alarms – Autodialer not working cost us LAr level when LN2 valve stopped functioning. • Did it help lead us a bit quicker into the Tubing break?? (Speculative) – Painful to use the Cryo Interface at time – incredible slow downs. • Discourages one from looking at trends if it takes forever to get up and run. • Massive Contamination of LAr – I believe there will be a review of exactly what happened and that will suggest several lessons to be learned from • How did we get into this situation in the first place? • Recognition of how fast you can completely contaminate a sizeable amount of LAr • How to identify other similar modes that might cause same problem. • Perhaps a way to quickly and automatically respond to an event since there is almost no time for people to recognize what is going on. 4/14/16 Dune Friday Phone Meeting 11
TPC • Detector Installation in Cryostat was a well planned and effectively carried out enterprise – Thanks Jack Fowler, Dan Wenman, Bo Yu, Lee Scott, Bob Kubinski – Demonstrates that careful planning actually works! • Field Cage, Cathode, HV – 60 kV (@Cathode) in “clean” LAr • Wanted to get track measurements made at this lower field before “risking” raising the HV higher and potentially breaking something. • Did see small spikes, and did have two apparent breakdowns (one while ramping back up after first breakdown). • These breakdowns happened just ~12 hrs before the LAr contamination – Large disappointment with LAr contamination is that we weren’t able to test Cathode HV at 120 kV in pure LAr – However we did continue the tests in what we had • Held 120 kV for 1 week without breakdown • Held 135 kV for 1 day with breakdown. • A summary of the HV studies will be made. – Most info is in elog 4/14/16 Dune Friday Phone Meeting 12
HV History over Run 150kV 150kV These are Voltages at Power Supply , ~10% higher 135kV 133kV than at cathode. breakdowns 67kV PowerOutage Cooling Loss contamination 4/14/16 Dune Friday Phone Meeting 13
Cold Electronics • As is well known, the FEMBs + the APAs have been plagued by relatively high noise levels – These swamp the original specs that were based on thermal noise figures, as measured with capacitive loads – Clearly see noise influences that correlate to the individual wire planes 4/14/16 Dune Friday Phone Meeting 14
Recent data from Cold Run17366 3/16) Test done by Bo Y. and Matt W. at BNL with 3 Bd APA. DocDB 9439, 7/2014 Data from 4 Bd taken at DAB (7/15) 4/14/16 Dune Friday Phone Meeting 15
Noise (Continued) • Our “low noise” state includes this correlated noise. • In addition, we are occasionally/often triggered into an extremely high noise state that makes it impossible to use the data for anything. • Much work is being done now by Marvin Johnson, Brian Kirby, Linda Bagby, Steve Chappa… in trying to characterize this noise – Hope that understanding can be fed into the electronics/APA design/??? for ProtoDune 4/14/16 Dune Friday Phone Meeting 16
Online Event Display -Collection Plane Top view of APA Very High Noise State “Normal” Noise State Long drift APA Short drift 4/14/16 Dune Friday Phone Meeting 17
Photon Detectors • Detectors – One Detector left unplugged during cabling • Observed Low noise level for those channels wasn’t recognized as the serious problem it was. – PD Signals seem to be impacted by noise—not positive if this is noise being generated by the APAs • SSPs – After fixing an issue with the NOvA timing chain (back at DAB), these have been reliable – We have had very good response from ANL when we have had any problems • Calibration Module – Working fine, able to calibrate the PDs 4/14/16 Dune Friday Phone Meeting 18
DAQ • Flange Board – All signals/electrical services to APA/PD pass through this Board. – Had serious issues with initial version – Second version much better • Issue with Bias lines—leakage currents quite high, and very non-linear with voltage • Eventually we bypassed the FB and went directly to SHV bulkhead connectors on own flanges. • PennTrigger Board – Slow start, but eventually it wsa providing our triggers • Cosmic Ray Counters – Allowed us to trigger on tracks that are useful for looking at the APA features. • Fiber Links – Very problematic at DAB and early at PC4. Often had to “reseat” the optical transceivers, or reset the links. – Settled down at some point and were less of a problem for rest of run. • RCE systems – Once things calmed down, they appear to be working fine. • Spares were an issue, since a failure would take down ½ the TPC APAs 4/14/16 Dune Friday Phone Meeting 19
Recommend
More recommend
