management cdlm
play

Management (CDLM) for Petascale Projects Arun Jagatheesan - PowerPoint PPT Presentation

Jan 27, 2023 •407 likes •603 views

Collaborative Data Life-cycle Management (CDLM) for Petascale Projects Arun Jagatheesan iRODS.org, DICE, SDSC/UCSD Agenda Introductions LSST as use case CDLM Attributes of CDLM History behind the story MDAS (Massive Data

  1. Collaborative Data Life-cycle Management (CDLM) for Petascale Projects Arun Jagatheesan iRODS.org, DICE, SDSC/UCSD

  2. Agenda • Introductions • LSST as use case • CDLM • Attributes of CDLM

  3. History behind the story • MDAS (Massive Data Analysis System) • Support data-intensive applications that manipulate very large data sets by building upon object-relational database technology and archival storage technology • 1995 by DARPA • SDSC SRB (Storage Resource Broker) • iRODS • Flexible license for our community • Flexible rules for users • Flexible data management

  4. My role in iRODS Community • Large-scale usage and adoption of iRODS • Research and Analysis of large-scale use-cases • Design requirements for large-scale users • Consult on iRODS-based storage infrastructure • Community Growth • Tutorials, dissemination • iROD-Chat (2006), SRB-Chat (2003) • Academic and Industrial users

  5. Large Scale Synoptic Survey • Survey entire sky every 3 nights • Dark Energy, Dark Matter, Near Earth Asteroids, and more • World’s largest digital camera (3 billion pixels) • Images 3000 times wider than Hubble • Data from Chile to US and rest of the world • 15 TB/night, over hundred(s) petabytes • www.youtube.com/watch?v=LtMJ_WwvBb8

  6. Data Products • Releases • Cataloged database • Provenance Info QuickTime™ and a • Metadata TIFF (Uncompressed) decompresso are needed to see this picture. • Processed Data Sets • Raw Images

  7. LSST Data Infrastructure Layout QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompr are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture.

  8. LSST Data Train QuickTime™ and a and iRODS TIFF (Uncompressed) decompressor are needed to see this picture. /file1..10.fits /file1..10.fits QuickTime™ and a /nobel.event TIFF (Uncompressed) decompressor are needed to see this picture. /file1..10.fits /catalog1.db /catalog1.db UK or IN2P3 /file1..10.fits /file1..10.fits /catalog1.db /catalog1.db

  9. LSST CDLM Problem Statement • LSST data-lifecycle management infrastructure for: • Performance oriented data storage sub-systems • Capacity oriented data storage sub-systems • Data (usage oriented) distribution networks • [Provenance and archive storage systems] • Confluence of three major storage dimensions • HPC data processing (pipelines to produce our data) • Datacenter sharing (data centers that host our data) • Data delivery and distribution (usage of our data)

  10. CDLM • Collaborative Data Lifecycle Management • Multiplexing of a single data life-cycle amongst more than one autonomous partner • Attributes of data-lifecycle is shared • Varying levels of autonomy and inter- dependence

  11. Multiplexing a Data Life-cycle • Data Creation (Raw data) • Data Processing (Derived data) • Data Analysis (Data warehouse, ..) • Data Namespace • Data Dissemination • Data Provenance • Data Archival

  12. Levels of Collaboration • Collaboration on Data Life-cycle not necessarily mean collaboration of businesses • Some types of CDLM • Symbiotic - All partner businesses benefit from CDLM • Neutral - No effect on businesses due to CDLM • Competitive - partners of CDLM are actually competitors of the resulting business process (forced to have a common platform to compete) • Hybrid - Multiple or transient partner relationships

  13. Autonomy & Inter-dependence at right levels for CDLM to work

  14. LSST Data Layout QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompr are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompresso are needed to see this picture.

  15. ALMA data flow QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) d are needed to see this QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) de are needed to see this

  16. LSST SC-2008 Prototype QuickTime™ and a TIFF (Uncompressed) d are needed to see this QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decompre are needed to see this picture. QuickTime™ and a TIFF (Uncompressed) decom are needed to see this pictu QuickTime™ and a TIFF (Uncompressed) decompressor are needed to see this picture.

  17. CDLM Infrastructure Design • Requirements, Expectations and Performance Management • Minimize dependencies (without affecting cost) • Reduce individual autonomy into hierarchical groups (that can remain autonomous) • Hierarchical rules and community rules

  18. iRODS enabling CDLM • Global Namespace • Resource allocation and service levels as policies/rules • Hierarchical rules and access controls • Highly Flexible System

  19. Similar projects? Let’s talk • The power of the community • Not necessarily “large” scale • Symbiotic • arun@diceresearch.org

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Scoring, term weighting, the vector space model Giorgio Gambosi Course of Information Retrieval

Scoring, term weighting, the vector space model Giorgio Gambosi Course of Information Retrieval

Why ranked retrieval? Term frequency tf-idf weighting The vector space model Scoring, term weighting, the vector space model Giorgio Gambosi Course of Information Retrieval CdLM in Computer Science University of Rome Tor Vergata Derived

769 views • 51 slides
MANUFACTURING MANAGEMENT CAPACITY MANAGEMENT What is Capacity management The management of

MANUFACTURING MANAGEMENT CAPACITY MANAGEMENT What is Capacity management The management of

MANUFACTURING MANAGEMENT CAPACITY MANAGEMENT What is Capacity management The management of the limits of an organization's resources, such as its labor force, manufacturing and office space, technology and equipment, raw materials, and

617 views • 26 slides
Resource Resource Management Management RESOURCE MANAGEMENT RESOURCE MANAGEMENT We have a

Resource Resource Management Management RESOURCE MANAGEMENT RESOURCE MANAGEMENT We have a

Moreno Baricevic Stefano Cozzini CNR-IOM DEMOCRITOS Trieste, ITALY Resource Resource Management Management RESOURCE MANAGEMENT RESOURCE MANAGEMENT We have a pool of users and a pool of resources, then what? some software that controls

538 views • 22 slides
Market Risk Management Gen Ins. Risk Management Life Ins. Risk Management Op.

Market Risk Management Gen Ins. Risk Management Life Ins. Risk Management Op.

Market Risk Management Gen Ins. Risk Management Life Ins. Risk Management Op. Risk Management Board of Enterprise Risk Senior Management Committees Directors Management Credit Risk Cross-Border Exposure Country Rating

453 views • 6 slides
Configuration management Configuration management Configuration management Configuration

Configuration management Configuration management Configuration management Configuration

Configuration management Configuration management Configuration management Configuration management Configuration management Field of management that focuses on establishing and maintaining consistency of a systems attributes

647 views • 6 slides
Medical Organization Needs SOA Patients Service Oriented Architecture Finance Doctors Data

Medical Organization Needs SOA Patients Service Oriented Architecture Finance Doctors Data

Management of Patient Records Management of Medical Records Management of Services Price management Relations with Insurance Company Financial Management and Accounting Medical Supplies Management Auxiliary

952 views • 9 slides
What is Risk Management? What is Risk Management? A (great) management tool Focus

What is Risk Management? What is Risk Management? A (great) management tool Focus

Proudly Presents How To Establish Functional Risk Management in Your Public Entity R. J oy J ackson, City of London Tina Gardiner, Region of York RIMS, Ottawa, 2011 What is Risk Management? What is Risk Management? A (great)

463 views • 24 slides
Self Management VS. Professional Management ADVANTAGES OF SELF MANAGEMENT $$$ The main

Self Management VS. Professional Management ADVANTAGES OF SELF MANAGEMENT $$$ The main

Self Management VS. Professional Management ADVANTAGES OF SELF MANAGEMENT $$$ The main reason to manage your own home is cost savings. This may be a valid enough reason to undertake self management. However, there are some items to

393 views • 17 slides
Change Management Change Management Change Management Appreciate the balance within Change

Change Management Change Management Change Management Appreciate the balance within Change

Slide 2 Purpose of this Session Objectives Understanding of Change Management principles within RBS Understand why CM is important / necessary Change Management Change Management Change Management Appreciate the balance within

249 views • 4 slides
Financial Management Review of Financial Management Financial Management System Internal

Financial Management Review of Financial Management Financial Management System Internal

Financial Management Review of Financial Management Financial Management System Internal Controls Accuracy of Report Information Program Income Salaries And Wages OMB Circular A-133 Single Audits Financial Management

461 views • 12 slides
Value Management Overview Doug Cantrell, PE, PMP AGC Conference March 14, 2018 Value

Value Management Overview Doug Cantrell, PE, PMP AGC Conference March 14, 2018 Value

Value Management Overview Doug Cantrell, PE, PMP AGC Conference March 14, 2018 Value Management Team State Value Management Engineer Jessica Kuse, PE, CPM Value Management Value Management Value Management Value Management Program

922 views • 44 slides
IT Management, Simplified Real-&me IT management solu&ons for the

IT Management, Simplified Real-&me IT management solu&ons for the

IT Management, Simplified Real-&me IT management solu&ons for the new speed of business Product Overview Who we are A Real-Time IT Management Company Enterprise IT management software

370 views • 19 slides
MANAGEMENT CONCEPTS and FUNDAMENTALS Dr. M.Thenmozhi Professor Department of Management

MANAGEMENT CONCEPTS and FUNDAMENTALS Dr. M.Thenmozhi Professor Department of Management

MANAGEMENT CONCEPTS and FUNDAMENTALS Dr. M.Thenmozhi Professor Department of Management Studies Indian Institute of Technology Madras Chennai 600 036 E-mail: mtm@iitm.ac.in WHAT IS MANAGEMENT ? 1.Field of Study - Management principles,

286 views • 16 slides
Key Management Lifecycle Key Management Lifecycle Cryptographic key management encompasses the

Key Management Lifecycle Key Management Lifecycle Cryptographic key management encompasses the

Key Management Lifecycle Key Management Lifecycle Cryptographic key management encompasses the entire lifecycle of cryptographic keys and other keying material. Basic key management guidance is provided in [SP800-21]. A single item of keying

522 views • 18 slides
Portfolio Management 101 Portfolio Management 101 Portfolio Management Institute 2006 PMI Forum

Portfolio Management 101 Portfolio Management 101 Portfolio Management Institute 2006 PMI Forum

Portfolio Management 101 Portfolio Management 101 Portfolio Management Institute 2006 PMI Forum Chicago, IL Jeff Gerson Al Derosa, CFA Business Practice Mgmt Fixed Income Kurt Meyer Paul Davis Equities Marketing Running a Successful PM

772 views • 65 slides
Lecture 7: Bu ff er Management 1 / 46 Bu ff er Management Administrivia To accommodate

Lecture 7: Bu ff er Management 1 / 46 Bu ff er Management Administrivia To accommodate

Bu ff er Management Lecture 7: Bu ff er Management 1 / 46 Bu ff er Management Administrivia To accommodate students who faced challenges with setting up the virtual machine and / or getting familiar with C ++ , we are bumping up the number of

973 views • 46 slides
Concussion in road cycling Dr Neil Heron, F.FSEM (UK), GP , PhD Consultant in Sport and

Concussion in road cycling Dr Neil Heron, F.FSEM (UK), GP , PhD Consultant in Sport and

Concussion in road cycling Dr Neil Heron, F.FSEM (UK), GP , PhD Consultant in Sport and Exercise Medicine Clinical Lecturer Team Physician, Team Ineos Sports related concussion (SRC) SRC gaining wider public recognition

693 views • 21 slides
Accelerate Cycle-Level Multi-Core RISC-V Simulation with Binary Translation Xuan Guo, Robert

Accelerate Cycle-Level Multi-Core RISC-V Simulation with Binary Translation Xuan Guo, Robert

Accelerate Cycle-Level Multi-Core RISC-V Simulation with Binary Translation Xuan Guo, Robert Mullins Department of Computer Science and Technology Both the paper and the slides are made available under CC BY 4.0 Motivation We want to

790 views • 22 slides
ShortestNontrivialCycles inDirectedSurfaceGraphs JeffErickson SOCG2011

ShortestNontrivialCycles inDirectedSurfaceGraphs JeffErickson SOCG2011

ShortestNontrivialCycles inDirectedSurfaceGraphs JeffErickson SOCG2011 Drink! 2 Theproblem Givenanedgeweighted directed graph G embeddedonasurface,

801 views • 45 slides
Is ENSO a cycle or a series of events? William S. Kessler NOAA / Pacific Marine Environmental

Is ENSO a cycle or a series of events? William S. Kessler NOAA / Pacific Marine Environmental

Is ENSO a cycle or a series of events? William S. Kessler NOAA / Pacific Marine Environmental Laboratory, Seattle Early ideas (Wyrtki) saw El Nio as an isolated disturbance. The advent of simple and intermediate coupled models s u

459 views • 10 slides
On s -fully cycle extendable line graphs Yehong Shao Ohio University Southern, Ironton, OH 45638

On s -fully cycle extendable line graphs Yehong Shao Ohio University Southern, Ironton, OH 45638

On s -fully cycle extendable line graphs Yehong Shao Ohio University Southern, Ironton, OH 45638 p. 1/15 Definitions p. 2/15 fully cycle extendable A graph G is said to be fully cycle extendable if every vertex of G lies on a

697 views • 41 slides
TSNsched: Automated Schedule Generation for Time Sensitive Networking Aellison Cassimiro T. dos

TSNsched: Automated Schedule Generation for Time Sensitive Networking Aellison Cassimiro T. dos

TSNsched: Automated Schedule Generation for Time Sensitive Networking Aellison Cassimiro T. dos Santos * , Ben Schneider , Vivek Nigam * * Universidade Federal da Paraba, fortiss FMCAD 2019 Oct 23, 2019 SUMMARY 1. TSN:

687 views • 27 slides
Distributed Queuing in Dynamic Networks Costas Busch Louisiana State University (Joint work

Distributed Queuing in Dynamic Networks Costas Busch Louisiana State University (Joint work

Distributed Queuing in Dynamic Networks Costas Busch Louisiana State University (Joint work with Gokarna Sharma) FOMC 2013 1 Dynamic Networks Fabian Kuhn, Nancy Lynch, Rotem Oshman STOC 2010 Communication Round (Broadcast Step) Each

1.21k views • 71 slides
Qu Quality Imp mproveme ment 2.0: PDS PDSA Cy Cycles in A les in Actio tion n Marci

Qu Quality Imp mproveme ment 2.0: PDS PDSA Cy Cycles in A les in Actio tion n Marci

Qu Quality Imp mproveme ment 2.0: PDS PDSA Cy Cycles in A les in Actio tion n Marci Brewer, Breastfeeding Program Manager Anna Corona, Program Manager Louisiana Department of Health Child and Adolescent Health Jennifer Young, Lynda

963 views • 41 slides

More recommend