IT Team and IT Related Meetings Jeffrey Kantor LSST IT Sr. Manager - - PowerPoint PPT Presentation

1

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

IT Team and IT Related Meetings Jeffrey Kantor LSST IT Sr. Manager LSST Project and Community Workshop 2018 August 13, 2018

2

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• IT Team Background and Context
• IT-related coordination Teams and Meetings
• Summit – Base Information Technology and Communications Tiger Team (SBTT)
• Information Technology Services Committee (ITSC)
• Information Technology Security Team (ITSec)
• LSST Network Engineering Team (NET)
• NCOA IT Working Group (NCOA ITWG)
• Slides cover (not all will be presented):
• Charter and members
• Accomplishments
• Future
• Ways to interact with the team

Agenda

3

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• “We are therefore evolving the existing IT reporting structure so that the teams in Tucson and in La Serena report to a single manager who will

report to Victor Krabbendam. This new IT manager will be responsible for the traditional Project IT services in both the North and South as well as the IT development work for both the Telescope and Site and the Data Management groups. This single IT group will be well positioned to transition into operations and to interface efficiently with the NCOA IT department forming soon. Jeff Kantor will be appointed the Interim Manager of the LSST IT group. Iain Goodenow will continue to lead the group in Tucson and Luis Corral will lead the group in La Serena.”

• Single functional organization, process, standards
• Consult/Consent, Inform/Act process
• JIRA (IT Support (Help Desk), IT Projects)
• Confluence (documentation …)
• Consistent execution of LSST/AURA processes with IT tasks (onboarding, IT

Security, …) irrespective of site

• Responsible for all LSST project office-provided, supported services irrespective
• f site
• Weekly coordination meetings on Fridays 9am PT

IT Team Integration – Steve Kahn email 20180409

4

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Need vehicle(s) to support LSST MREFC Project on “all things ITC”
• Act as “consultant” to PMO and project (hardware and software

deliverables remain responsibility of subsystems)

• Task 1: Ensure Summit – Base Complex is optimally designed/engineered

to avoid ITC gaps, redundancies, inefficiencies (SBTT)

• Task 2: Ensure consistency of life cycle management and support of all IT

Services during MREFC project and as foundation for operations (ITSC)

• Address IT Security

PMO Task – Victor Krabbendamemail 20160524

5

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• We need to engineer the Summit – Base Complex as one ITC entity, with

a consistent, coherent design

• This cuts across subsystems that contribute facilities, equipment,

software, operational policies and procedures

• Many point-to-point communications and some workshops had tried to

address various aspects of this as the subsystems have tried to do the work required in their “silos”, but…

• We needed a more integrated approach to achieve the goal in time for

early integration and through end of commissioning.

• This was chartered in the form of a cross-subsystem “tiger team”, with a

defined set of deliverables, a plan/schedule to provide them, and a T/CAM to track this and report on it

Task 1: Summit – Base Complex Integrity

6

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• We agreed we need Summit – Base ITC Design document(s) that are

subsystem agnostic in terms of who delivers what and where it resides, and focus on:

• End-to-end data flows and processes
• Partitioning/allocation of responsibilities for above into logical nodes (e.g.

“commissioning cluster, diagnostic cluster, image buffer, etc.)

• Full definition of interfaces between nodes (expand ICDs as necessary)
• Mapping of logical nodes to physical ITC infrastructure(DAQ, DWDM switch,

etc.) and location (Summit, Base, both)

• Designs for cross-cutting services (e.g. System Time, Security, Network(s), etc.)
• AND ONLY AFTER THAT
• Mapping of the above to WBS elements and subsystems providing the

implementation of the design

Summit – Base ITC Design

7

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The SBTT is a cross-subsystem team empowered to make technical decisions to optimize

Summit Base ITC as an integrated system, i.e. not silo'ed by WBS.

• Members:
• Gregory Dubois-Felsmann(DM/SE)
• Mike Huffer (Camera DAQ)
• Tony Johnson (Camera CCS)
• Jeff Kantor (PMO TCAM)
• Ron Lambert (DM/TS)
• German Schumacher (TS OCS/TCS/Integration)
• Kian-Tat Lim (DM/SE)
• Michelle Butler (DM)
• Alexander Withers (ISO)
• Iain Goodenow (PMO/ITS)
• Chuck Claver (SE/Commissioning)
• Dave Mills (TS OCS/TCS)
• Andres Villalobos (DM/TS)
• Luis Corral (DM/TS)
• Note: Members are the only people who are given assignments; they are expected to reach out to others on the project as

necessary to achieve their assignments.

SBTT Charter and members

8

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• A wide range of design elements of the Summit – Base ITC design have been addressed as captured in

Confluence, JIRA, and numerous documents

• Many active development tasks (configuration management with Puppet for SB complex and

Pathfinders, integration with IT Security and Identity Management, …)

• All of the TT proceedings are captured in weekly meeting minutes
• The overall design decisions/recommendations are documented in LSE-309 Summit – Base ITC Design

Document

• LSE-309 was approved to establish initial baseline version
• The current version is stable with respect to the overall complex, the Summit, and the Summit –

Base Network, which are already in construction/equipment procurement and deployment

• An LCR will be prepared by the end of CY18 to expand to full coverage of the Base, to support

FY18 -19 construction/equipment procurement and deployment

• LSE-299 Summit Computer Room Requirements was developed, imported into Magic Draw, updated,

and baselined

• Subsequent LCRs approved to update LSE-299 based on construction experience, events
• LSE-239 Base Facility Data Center Requirements was imported into Magic Draw, updated, and baselined
• Subsequent LCRs approved to update LSE-239 based on construction experience, events

SBTT Accomplishments

9

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The Tiger Team started in the last quarter of FY16, and has continued

through FY18 to accomplish Task 1

• The Tiger Team has fulfilled its primary mission when:
• LSE-309 is baselined with the entire complex covered
• The methods and timeline for implementation and verification are

defined and accepted by the PMO, SE, and the subsystems

• The current intent is to disband the Tiger Team at that time and transfer

responsibility to System Engineering and Commissioning Teams

• Any future architecture and design work and modifications to this

document are expected to be a function of the LSST System Engineering team.

SBTT Future

10

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• We have a slack channel
• We publish weekly meeting agendas and minutes in

confluence

• We invite topical experts to meetings when something of

broad interest is covered

• You can contact us via any project-standard vehicle,

including JIRA issues, weekly management meetings, etc.

• We all represent our respective project organizations, feel

free to contact your SBTT member by any means you find convenient

Ways to interact with the SBTT

11

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• We have many entities (subsystems, SE, SQuARE, PMO) selecting,

purchasing, deploying, supporting, upgrading, and retiring IT Services for the MREFC project

• We don’t have good vehicles for keeping everyone informed about all the

services: who provides them, how to get support, are other groups planning something similar or have done something that can be leveraged, etc.

• There are no internal “service levels” defining what level of service and

support can be expected

• Many point-to-point communications and “local” channels have been set up

to to address various aspects, but…

• We needed a more integrated approach to achieve the goal of optimal

management and support

• This was chartered in the form of a broad representative committee and a

process

Task 2: IT Services Life Cycle Management

12

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

ITSC Ecosystem and Roles

Service Users

LSST PO LSST Subsystems LSST Corporation LSST Science Collaborations LSST Community

Service Providers

LSST ITS LSST SQuaRE NCSA

• NOAO (N & S)

University of Arizona Commercial Service Providers

Strategy Governance Coordination

13

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• To advise what ITS the project provides/uses
• To make sure interoperability exists among ITS where possible
• To combine, reuse, recycle ITS when possible
• To prevent ITS from becoming stagnant or security hazards
• Act as an agent of the project to keep project advised of what is

going on at all spectrums of ITS

• To make recommendation whether a particular IT Service

continues, when to retire/replace

• To make recommendations how the Project Office ITS (mail,

web, issues, etc.) will transition into commissioning and

• perations

ITSC Charter

14

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The Project Office has final authority over any action or decision made

by the IT Services Committee, especially if it results in a budgetary increase or has other consequences to project or observatory operations

• Actions or Decisions cannot violate existing Information Technology

Security Program and Compliance Policies

• LPM 121 – LSST Master Information Security Plan
• LPM 123 – Acceptable Use Policy
• Subsystem IT Security Plans: LPM , LDM, LTS, LCAM
• LPM 141 – Compliance Policy

Governance & Transparency

15

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The IT Services Council (ITSC) consists of:
• Core Members
• Representative Members

ITSC Membership

16

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The Core Members consist of representatives from entities that are

key providers of IT Services to the project

• Members:
• PMO : Daniel Calabrese (chair), Jeff Kantor (ex officio)
• Tucson IT Services Team: Iain Goodenow
• La Serena IT Services Team: Luis Corral
• LSST Information Security Officer: Alex Withers
• NCSA: Michelle Butler
• DM: Wil O’Mullane
• Non-voting
• LSST Compliance Officer: Veronica Kinnison

ITSC Core Members

17

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The Representative Members represent a cross-section of the various project

groups that have a responsibility for delivery consisting of and/or a vested interest in:

• IT Assets (including Services)
• Accessing IT Infrastructure
• Impacting IT Security
• Other IT related topics
• Members
• Systems Engineering: Michael Reuter
• Data Management: Wil O’Mullane (also Core Member)
• Telescope and Site: Harini Sundararaman
• Camera: Tony Johnson
• EPO: Ben Emmons

ITSC Representative Members

18

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Reference White Paper entitled: “General Architecture

Principles” Version 1 – 08.11.2016

• LPM-51 update to to include ITSC role
• Services Database with comprehensive inventory of current

services and initial inventory of planned services

• Clarification of Use of Third Party Tools: Only Whitelist is

Supported

• IT Requests for Consensus (RFCs) Process
• 16 processed, 14 Implemented, 1 Adopted, 1 Withdrawn
• Search JIRA issues, Project = “IT Request For Comments”

ITSC Accomplishments

19

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The ITSC’s role of coordination and recommendation on ITS

is currently chartered only in construction and commissioning

• It is even more important that this role be performed in

Operations

• It is expected the ITSC or something like it will be chartered

by the Observatory Director and will be folded into the IT team in NCOA

ITSC Future

20

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Your Representative Member is your primary interface to the ITSC
• Responsible for making you aware of others IT Services plans,

proposals, etc.

• Responsible for making ITSC aware of your IT Services plans,

proposals, etc.

• Meetings are open to anyone, agendas and minutes are public
• ITRFCs in JIRA are public (use Project = ITRFC to search issues)
• Services database is currently accessible by ITSC, will be public in future

Ways to interact with the ITSC

21

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• To develop, publicize, maintain and update, and verify compliance with

the LSST Information Technology Security Program

• LPM 121 – LSST Master Information Security Plan
• LPM 123 – Acceptable Use Policy
• Subsystem IT Security Plans: LPM , LDM, LTS, LCAM
• Prevent IT Security Program from becoming stagnant (annual refresh)
• Identify, notify, log, and provide incident analysis on IT Security incidents
• Proactively identify and address security vulnerabilities
• Act as an agent of the project to keep project advised of what is going on

at all spectrums of IT Security

• Make recommendation whether a particular IT Security mechanisms

should be added, updated, retired

ITSec Charter

22

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The IT Security Team (ITSec) consists of:
• Alex Withers (NCSA and LSST IT Security Program Officer)
• Jeff Kantor (PMO)
• Daniel Calabrese (PMO)
• Iain Goodenow(IT Tucson)
• Luis Corral (IT La Serena)

ITSec Membership

23

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• LSST Information Technology Security Program developed IAW

best practices defined by NSF-funded Center for Trustworthy Science (CTSC)/TrustedCI

• Baselined in FY15, updated annually since
• LPM 121 – LSST Master Information Security Plan
• LPM 123 – Acceptable Use Policy
• Subsystem IT Security Plans: LPM , LDM, LTS, LCAM
• Multiple Security Policy/Plan documents (e.g. Mobile Device

Security, Password Policy, …)

• Monthly IT Security Team Meetings
• Numerous identified vulnerabilities addressed
• Numerous IT Security incidents addressed

ITSec Accomplishments

24

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Any Member is your primary interface to the ITSC
• Meetings are open to anyone, agendas and minutes are public
• IT Security incidents must be reported immediately via most expedient means (phone, email, in person, …)
• The LSST ISO (iso@lsst.org) should be informed of all incidents
• Also inform your Local Responder:
• LSST PMO/Tucson
• Primary Responder: Information Security Officer, iso@lsst.org
• Secondary Responder: Senior System Administrator, ssa@lsst.org
• LSST at NCSA
• NCSA Help Desk, help+security@ncsa.illinois.edu, is a 24x7 help desk that dispatches incident response. The NCSA Help Desk can be called at 1-217-

244-0710 and requested to page the security incident response team.

• LSST at La Serena
• Primary Responder: Information Security Officer, iso@ctio.noao.edu
• Secondary Responder: Junior Information Security Officer, jiso@ctio.noao.edu
• Partner sites where LSST operates have existing incident response policies and procedures. LSST IT Security sub-plan managers arrange with existing

site organization incident response systems so that sub-plan managers are notified when local sites discover an issue affecting LSST. Similarly, LSST personnel will notify relevant site personnel if LSST personnel discover an issue affecting the site. LSST sub-plan managers will verify existing incident response policies meet their requirements. LSST ISO will notify LSST Management if there is significant business impact. Assigning severity and

• categorizing incidents are addressed by the site primary responder.
• LSST Information Security Incident Response Policy and Procedures LPM-124 (rel2.0) Latest Revision 2017-10-25

Ways to interact with the ITSec

25

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Per the LSST Network Operations and Management Plan, the
• verall architecture, engineering, testing, operations and

maintenance of the LSST Networks are assigned to the Network Engineering Team (NET) under the direction of the LSST Network Architects and LSST Management.

Network Engineering Team (NET) Charter

26

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• The NET is formed of project/observatory and network provider technical
• staff. The current members of the NET are:
• Ron Lambert (AURA/LSST Network Architect)
• Jeronimo Bezerra (FIU/Amlight Network Architect)
• Matt Kollross (NCSA Network Architect)
• Humberto Galiza (FIU/Amlight Network Engineer)
• Albert Astudillo (REUNA Network Engineer)
• Shahram Sobhani (AURA/LSST Network Engineer)
• Managers/Executive Sponsors:
• Sandra Jaque (REUNA Manager of Technology)
• Julio Ibarra (FIU Assistant VP)
• Jeff Kantor (LSST International Networks and Base Site TCAM)
• Chris Morrison (Gemini)

NET Membership

27

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Documents
• LSE-78 LSST Observatory Network Design defines the overall network design

for all long-haul networks

• LDM-142 Network Sizing Model defines the budgeted cost and schedule of

deployment of links and bandwidth, and their utilization

• LSST Network End-to-End Test Plan defines the plan for development testing

and monitoring the LSST networks

• LSST Observatory Network Verification Plan defines plan for formal

verification of the LSST networks. EARLY DRAFT

• LSST Observatory Network Verification Matrix defines requirements and

methods for formal verification of the LSST networks. EARLY DRAFT

• LSST Network Operations and Management Plan defines the plan for
• perating and maintaining the LSST networks as a single integrated process.
• Test and verification tools and processes
• perfSonar, Data Transfer Nodes, Quality of Service (QoS) test bed
• Live demonstrations, including First Optic Light Demonstration (12/2017)

NET Accomplishments

28

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Contact any member of the NET
• LSST NET mailing list
• International Communications Confluence page is jump-off

point for all NET matters, including links to monthly meeting pages

• The NET meets monthly at 0800 PT in an online meeting on

Blue Jeans

Ways to interact with the NET

29

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• AURA has proposed, and the NSF has approved, the merger of the AURA ground-

based astronomy centers under the name NCOA

• Gemini, NOAO/CTIO under single cooperative agreement as of 10/1/19
• LSST enters once it starts operations
• Centers become programs, all supported by matrixed support organizations for

IT, HR, Business/Admin, Science

• The NCOA IT WG is early working team to establish NCOA IT support organization
• Organization, processes, standards, infrastructure
• Many document deliverables
• Early scope (through FY19) and later
• NCOA (Interim) IT lead will be hired in FY18
• ITWG will segue to NCOA IT

NCOA IT Working Group (IT WG) Charter

30

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Chris Morrison – Gemini (Chair)
• Ryan Richmond – AURA IT
• Adam Bolton – NOAO/CTIO and NCOA Program team
• Jeff Kantor - LSST

NCOA IT WG Membership

31

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Earlier iteration of IT WG (pre-NCOA) focused on

infrastructure integration in Chile

• The documents produced by this IT WG are archived and

formed the basis for work of current NCOA IT WG

• Many documents have since been developed or are being

developed

• The overall list is in NCOA IT Integration Work Items for FY18

and Early FY19

NCOA IT WG Accomplishments

32

LSST Project and Community Workshop 2018 • Tucson • August 13 - 17

#lsst2018

• Contact any member of the NCOA IT WG
• NCOA IT WG Sharepoint space
• Requires login
• Includes archived documents, meeting notes
• NCOA IT WG google documents (works in progress)
• NCOA IT WG meets at varying intervals on zoom

Ways to interact with the NCOA IT WG