LSE-78 LSST Observatory Network Design Ron Lambert LSST Network Architect LSST Project and Community Workshop 2018 August 13, 2018 #lsst2018 #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 1
LSE-78 LSST Observatory Network Design • Covers the Long-Haul Network for LSST to enable the mission. Has been dramatically amended over the years of development from 2005 • to 2018 to design the flow of traffic from the summit on Cerro Pachon through Santiago to Miami to NCSA. The links and distances include • Summit to Base 85Km • La Serena to Santiago 700Km • Santiago to Miami 7000Km • Miami to NCSA 2300Km • These are the necessary links to satisfy the data flows required from • summit to NCSA Does not include details of Local Area Networks in Summit and Base • Facilities (refer to LSE-309) #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 2
Networks and Path Diversity between Sites and Centers LSST Data Facility HQ Site French Site Archive Center HQ Facility Satellite Processing Alert Production Observatory Management Data Release Production Center Science Operations Calibration Products Production Education and Public Data Release Production EPO Infrastructure Outreach Long-term Storage (copy 2) Data Access Center Data Access and User Services Summit Site Summit Facility Base Site Telescope and Camera Data Acquisition Base Center Crosstalk Correction Long-term storage (copy 1) Data Access Center Data Access and User Services #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 3
Nightly International Data Flows 60 second 10M Alerts requirement (average) stops here per night VOEvent Broker Server (Non-DM) LDF Site DM Data (up to 39 Gbps) Northbound Southbound Crosstalk Images Calibration Images Raw Images DIAObject Catalog Wavefront Images DIASource Catalog Raw Calibration Images OCS/TCS Data (up to 1 Gbps) Engineering Facility DB Data Quality Metrics Atmospheric Telescope Images and Spectra Raw and Crosstalk Corrected Images 24 - 30 TB / night VOEvent 6.4 GB per image Broker Server in La Serena Base Site (Non-DM) #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 4
Non-Nightly International Data Flows LDF Site French Site Calibration products (satellite processing center) US DAC L2 Data Products EPO Center L2 Data Products Eastbound Westbound Raw images 1/2 of L2 Data Products Calibration products Engineering Facility Database 1/2 of L2 Data Products Chilean DAC (coadds, catalogs, SDQA) Calibration products L2 Data Products Backup: Physical Shipments Data Release: LDF Site to Base Site Coadds Hardware Object catalog New and Replacement Source catalog Hardware (annually) Forced Source catalog Base Site Data Release loaded (secondary method) #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 5
LSST Baseline Long Haul Network Links CC-IN2P3 Tucson 10Gb Lyon Amlight/internet2 10Gb ESnet SLAC, 10Gb IN2P3 Cerro (institutional) Other DOE Pachon Champaign Chicago 40x10Gb AURA Boca 10x10Gb 2 * 100Gb Santiago (Science 2x 100Gb Raton Non-LSST NCSA/ Data) Esnet/FLR/i2 AURA ICCN (Fail-over) 100Gb 20x10Gb 100Gb (spectrum) AURA (spectrum) AmLight/RNP/RedCLARA (OCS/TCS Amlight/Angola Cable (via Buenos Aires) Control) Miami Sao La Serena Paolo Santiago 2 x 100Gb 1 x 100Gb Amlight/ANSP/ 1 x 40 –100 Gb RedCLARA REUNA Primary Path Panama Secondary Path #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 6
Baseline Allocation of Bandwidth NOTES: [1] Guaranteed reservation [S] Spectrum [2] Expected use, Prioritized/QoS [3] Carried on AURA circuits, not LSST. Fail-over from AURA to LSST is not baselined. [4] Combinations of Cases use numbers from lowest performance case [5] All numbers are stated in line speed reservation, efficiency may lower actual delivered BW "[6] Cases of total network failure"" on a link, i.e. all paths connecting two points are not shown. Those cases rely on data buffering and catch-up." #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 7
Network Segments Baseline - Segment 1: “Mountain-Base” • From Cerro Pachon to La Serena computing facility, 6 x 100 Gbs (plus 2x100Gbs AURA shared) - Primary circuit on LSST DWDM - Secondary “fail-over” backup circuit on AURA DWDM • Two fiber sub-segments, single path - Summit-Gatehouse (dedicated fibers on AURA property) = 30km - Gatehouse-La Serena (dedicated fibers in public cable) = 55km - Segment 2: La Serena-Santiago • From La Serena computing facility to connection point in Santiago with international links - Level 3 facility, exploring redundancy in Santiago Primary link is dedicated fibers in public cable (100 Gbps) • • Secondary link is shared leased capacity 100Gbs likely, 40Gbs worst case #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 8
Network Segments Baseline (cont.) - Segment 3 : International links • Provided by Amlight/RNP/CLARA (100 Gbps each) - Santiago – Panama – Miami west coast - Santiago – Sao Paolo – Miami east coast - Santiago – Boca Raton – Miami - Segment 4: US links • Utilizing Internet-2 and ESnet • Agreement with Esnet for 2 x 100 Gbps Atlanta – Chicago in work • NCSA provides Chicago – Champaign (LDF) #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 9
AMPATH International Links #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 10
Bandwidth Timeline: LDM-142 NOTES: NxM notation indicates N connections of M speed. All bandwidths are in gigabits/second (Gb/s). All allow burst up to unused capacity [1] best/lworst case presumes continued IRNC, international REN investments, and industrial partnerships through AMPATH/AMLIGHT likely case presumes only continued IRNC investment, worst case assumes only LSST support. [2] This is proposed. IN2P3 provides this and will match the other links in capacity to the degree possible. [3] Prior baseline funding for 200 Gbps was sufficient to acquire 600 Gbps for same cost This is for LSST side only, AURA side is also available for fail-over 200 Gbps [4] Requires Telefonica to upgrade equipment on this link. This is their main trunk on this route, so likely will be done, but no schedule. [5] AMLight will have 2x100 Gbps fiber links. LSST to buy more 1x100 line and 10x10 client interfaces in the DWDM equipment at REUNA on this link to go beyond 10x10. [6] Gap between Commissioning and Operations is schedule contingency for Construction/Commisssioning. Operations will start sooner if this contingency is not used. [7] Leased/shared capacity [8] Lambda/dedicated capacity [9] Bandwidth via RNP/CLARA [10] LSST DWDM is 6 x 100, AURA DWDM is 2x 100 [11] Via Boca Raton [12] B/W is to clients, users. Wired LAN connections in computer rooms are 10G from switches to servers, 100G between switches. WAPs to access switches are 5G, between acess switches 10G. #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 11
Recent Accomplishments - Accepted fibers • Summit – AURA Gatehouse • AURA Gatehouse – La Serena • La Serena - Santiago - Installed Dense Wave Division Multiplex (DWDM) equipment on Cerro Pachon, in La Serena, and in Santiago - Conducted First Optic Light Demonstration - Activated 100 Gbps Ring Santiago – Miami - Installed temporary network and phone service on summit #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 12
Work in Progress - Fail-over testing of Cisco ACI-based Summit Network - Connect Summit Network to DWDM - Update obsolete Microwave equipment • Voice, web, mail backup, not for data, control - DWDM procurement for 100 Gbps Spectrum/activate link - Design of Base Data Center in La Serena - Agreement with Esnet to provide 2 x 100 Gbps links Atlanta – Chicago - Maintenance agreement for fiber on AURA property #lsst2018 LSST Project and Community Workshop 2018 • Tucson • August 13 - 17 13
Recommend
More recommend
