Enabling Enabling Data- -Intensive Science Intensive Science - - PowerPoint PPT Presentation

Enabling Enabling Data Data-

• Intensive Science

Intensive Science with Tactical Storage Systems with Tactical Storage Systems

• Prof. Douglas Thain
• Prof. Douglas Thain

University of Notre Dame University of Notre Dame http:// http://www.cse.nd.edu/~dthain www.cse.nd.edu/~dthain

The Cooperative Computing Lab The Cooperative Computing Lab

Our model of computer science research: Our model of computer science research:

– – Understand Understand how users with complex, large how users with complex, large-

• scale

scale applications need to interact with computing systems. applications need to interact with computing systems. – – Design Design novel computing systems that can be applied novel computing systems that can be applied by many different users == basic CS research. by many different users == basic CS research. – – Deploy Deploy code in real systems with real users, suffer code in real systems with real users, suffer real bugs, and learn real lessons == applied CS. real bugs, and learn real lessons == applied CS.

Application Areas: Application Areas:

– – Astronomy, Bioinformatics, Biometrics, Molecular Astronomy, Bioinformatics, Biometrics, Molecular Dynamics, Physics, Game Theory, ... ??? Dynamics, Physics, Game Theory, ... ???

External Support: NSF, IBM, Sun External Support: NSF, IBM, Sun http:// http://www.cse.nd.edu/~ccl www.cse.nd.edu/~ccl

Abstract Abstract

Users of distributed systems encounter many Users of distributed systems encounter many practical barriers between their jobs and the practical barriers between their jobs and the data they wish to access. data they wish to access. Problem: Users have access to many Problem: Users have access to many resources resources (disks), but are stuck with the (disks), but are stuck with the abstractions abstractions (cluster NFS) provided by administrators. (cluster NFS) provided by administrators. Solution: Tactical Storage Systems allow any Solution: Tactical Storage Systems allow any user to create, reconfigure, and tear down user to create, reconfigure, and tear down abstractions without bugging the administrator. abstractions without bugging the administrator.

Transparent Distributed Filesystem

shared disk

The Standard Model The Standard Model

The Standard Model The Standard Model

Transparent Distributed Filesystem

shared disk

Transparent Distributed Filesystem

shared disk

private disk private disk private disk private disk FTP, SCP, RSYNC, HTTP, ...

Problems with the Standard Model Problems with the Standard Model

Users encounter partitions in the WAN. Users encounter partitions in the WAN.

– – Easy to access data inside cluster, hard outside. Easy to access data inside cluster, hard outside. – – Must use different mechanisms on diff links. Must use different mechanisms on diff links. – – Difficult to combine resources together. Difficult to combine resources together.

Different access modes for different purposes. Different access modes for different purposes.

– – File transfer: preparing system for intended use. File transfer: preparing system for intended use. – – File system: access to data for running jobs. File system: access to data for running jobs.

Resources go unused. Resources go unused.

– – Disks on each node of a cluster. Disks on each node of a cluster. – – Unorganized resources in a department/lab. Unorganized resources in a department/lab.

A global file system can A global file system can’ ’t satisfy everyone! t satisfy everyone!

What if... What if...

Users could easily access any storage? Users could easily access any storage? I could borrow an unused disk for NFS? I could borrow an unused disk for NFS? An entire cluster can be used as storage? An entire cluster can be used as storage? Multiple clusters could be combined? Multiple clusters could be combined? I could reconfigure structures without root? I could reconfigure structures without root?

– – (Or bugging the administrator daily.) (Or bugging the administrator daily.)

Solution: Tactical Storage System (TSS) Solution: Tactical Storage System (TSS)

Outline Outline

Problems with the Standard Model Problems with the Standard Model Tactical Storage Systems Tactical Storage Systems

– – File Servers, Catalogs, Abstractions, Adapters File Servers, Catalogs, Abstractions, Adapters

Applications: Applications:

– – Remote Database Access for Remote Database Access for BaBar BaBar Code Code – – Remote Dynamic Linking for CDF Code Remote Dynamic Linking for CDF Code – – Logical Data Access for Bioinformatics Code Logical Data Access for Bioinformatics Code – – Expandable Database for MD Simulation Expandable Database for MD Simulation

Improving the OS for Grid Computing Improving the OS for Grid Computing

Tactical Storage Systems (TSS) Tactical Storage Systems (TSS)

A TSS allows any node to serve as a file A TSS allows any node to serve as a file server or as a file system client. server or as a file system client. All components can be deployed without All components can be deployed without special privileges special privileges – – but with security. but with security. Users can build up complex structures. Users can build up complex structures.

– – Filesystems Filesystems, databases, caches, ... , databases, caches, ...

Two Independent Concepts: Two Independent Concepts:

– – Resources Resources – – The raw storage to be used. The raw storage to be used. – – Abstractions Abstractions – – The organization of storage. The organization of storage.

file transfer file system file system file system file system file system file system file system Central Filesystem App Distributed Database Abstraction Adapter App Distributed Filesystem Abstraction Adapter App Cluster administrator controls policy on all storage in cluster

UNIX UNIX UNIX UNIX UNIX UNIX UNIX

Workstations owners control policy on each machine. file server file server file server file server file server file server file server

UNIX UNIX UNIX UNIX UNIX UNIX UNIX

??? Adapter 3PT

Components of a TSS: Components of a TSS:

1 1 – – File Servers File Servers 2 2 – – Catalogs Catalogs 3 3 – – Abstractions Abstractions 4 4 – – Adapters Adapters

1 1 – – File Servers File Servers

Unix Unix-

• Like Interface

Like Interface

– – open/close/read/write

• pen/close/read/write

– – getfile/putfile getfile/putfile to stream whole files to stream whole files – – opendir

• pendir/stat/rename/unlink

/stat/rename/unlink

Complete Independence Complete Independence

– – choose friends choose friends – – limit bandwidth/space limit bandwidth/space – – evict users? evict users?

Trivial to Deploy Trivial to Deploy

– – run server + run server + setacl setacl – – no privilege required no privilege required – – can be thrown into a grid system can be thrown into a grid system

Flexible Access Control Flexible Access Control

file server A file server B Chirp Protocol file system

• wner of

server A

• wner of

server B

Related Work Related Work

Lots of file services for the Grid: Lots of file services for the Grid:

– – GridFTP GridFTP, , NeST NeST, SRB, RFIO, SRM, IBP, ... , SRB, RFIO, SRM, IBP, ... – – Adapter interfaces with many of these! Adapter interfaces with many of these!

Why have Why have another another file server? file server?

– – Reason 1: Must have precise Unix semantics! Reason 1: Must have precise Unix semantics!

Apps distinguish ENOENT Apps distinguish ENOENT vs vs EACCES EACCES vs vs EISDIR. EISDIR. FTP always returns error 550, regardless of error. FTP always returns error 550, regardless of error.

– – Reason 2: TSS focused on easy deployment. Reason 2: TSS focused on easy deployment.

No privilege required, no No privilege required, no config config files, no rebuilding, files, no rebuilding, flexible access control, ... flexible access control, ...

Access Control in File Servers Access Control in File Servers

Unix Security is not Sufficient Unix Security is not Sufficient

– – No global user database possible/desirable. No global user database possible/desirable. – – Mapping external credentials to Unix gets messy. Mapping external credentials to Unix gets messy.

Instead, Make External Names First Instead, Make External Names First-

• Class

Class

– – Perform access control on remote, not local, names. Perform access control on remote, not local, names. – – Types: Types: Globus Globus, Kerberos, Unix, Hostname, Address , Kerberos, Unix, Hostname, Address

Each directory has an ACL: Each directory has an ACL:

globus globus:/O= :/O=NotreDame NotreDame/CN= /CN=DThain DThain RWLA RWLA kerberos:dthain@nd.edu kerberos:dthain@nd.edu RWL RWL hostname:*. hostname:*.cs.nd.edu cs.nd.edu RL RL address:192.168.1.* RWLA address:192.168.1.* RWLA

Problem: Shared Namespace Problem: Shared Namespace

file server globus:/O=NotreDame/* RWLAX a.out test.c test.dat cms.exe

Solution: Reservation (V) Right Solution: Reservation (V) Right

file server O=NotreDame/CN=* V(RWLA) /O=NotreDame/CN=Monk RWLA mkdir a.out test.c /O=NotreDame/CN=Monk mkdir /O=NotreDame/CN=Ted RWLA a.out test.c /O=NotreDame/CN=Ted mkdir only!

2 2 -

• Catalogs

Catalogs

catalog server catalog server periodic UDP updates HTTP XML, TXT, ClassAds

3 3 -

• Abstractions

Abstractions

An abstraction is an organizational layer built on An abstraction is an organizational layer built on top of one or more file servers. top of one or more file servers. End Users End Users choose what abstractions to employ. choose what abstractions to employ. Working Examples: Working Examples:

– – CFS: Central File System CFS: Central File System – – DSFS: Distributed Shared File System DSFS: Distributed Shared File System – – DSDB: Distributed Shared Database DSDB: Distributed Shared Database

Others Possible? Others Possible?

– – Distributed Backup System Distributed Backup System – – Striped File System (RAID/Zebra) Striped File System (RAID/Zebra)

CFS: Central File System CFS: Central File System

file server adapter adapter adapter appl appl appl file file file CFS CFS CFS

ptr ptr ptr

DSFS: Dist. Shared File System DSFS: Dist. Shared File System

file server appl appl file server file server file file file file file file file file file file adapter adapter DSFS DSFS lookup file location access data

DSDB: Dist. Shared Database DSDB: Dist. Shared Database

adapter adapter appl appl file server file server file file file file file file file file file file database server file index query direct access insert create file DSDB DSDB

system calls trapped via ptrace tcsh cat vi

tcsh cat vi

file table process table

Like an OS Kernel Like an OS Kernel

– – Tracks Tracks procs procs, files, etc. , files, etc. – – Adds new capabilities. Adds new capabilities. – – Enforces owner Enforces owner’ ’s policies. s policies.

Delegated Delegated Syscalls Syscalls

– – Trapped via Trapped via ptrace ptrace interface. interface. – – Action taken by Parrot. Action taken by Parrot. – – Resources Resources chrgd chrgd to Parrot. to Parrot.

User Chooses User Chooses Abstr Abstr. .

– – Appears as a Appears as a filesystem filesystem. . – – Option: Timeout tolerance. Option: Timeout tolerance. – – Option: Cons. semantics. Option: Cons. semantics. – – Option: Servers to use. Option: Servers to use. – – Option: Auth mechanisms. Option: Auth mechanisms.

4 4 -

• Adapter

Adapter

Adapter - Parrot Abstractions: CFS – DSFS - DSDB

HTTP, FTP, RFIO, NeST, SRB, gLite ???

file transfer file system file system file system file system file system file system file system Central Filesystem App Distributed Database Abstraction Adapter App Distributed Filesystem Abstraction Adapter App Cluster administrator controls policy on all storage in cluster

UNIX UNIX UNIX UNIX UNIX UNIX UNIX

Workstations owners control policy on each machine. file server file server file server file server file server file server file server

UNIX UNIX UNIX UNIX UNIX UNIX UNIX

??? Adapter

Performance Summary Performance Summary

Nothing comes for free! Nothing comes for free!

– – System calls: order of magnitude slower. System calls: order of magnitude slower. – – Memory bandwidth overhead: extra copies. Memory bandwidth overhead: extra copies. – – TSS can drive network/switch to limits. TSS can drive network/switch to limits.

Compared to NFS Protocol: Compared to NFS Protocol:

– – TSS slightly better on small operations. (no lookup) TSS slightly better on small operations. (no lookup) – – TSS much better in network bandwidth. (TCP) TSS much better in network bandwidth. (TCP) – – NFS caches, TSS doesn NFS caches, TSS doesn’ ’t (today), mixed blessing. t (today), mixed blessing.

On real applications: On real applications:

– – Measurable slowdown, typically 5 percent. Measurable slowdown, typically 5 percent. – – Benefit: far more flexible and scalable. Benefit: far more flexible and scalable.

Outline Outline

Problems with the Standard Model Problems with the Standard Model Tactical Storage Systems Tactical Storage Systems

– – File Servers, Catalogs, Abstractions, Adapters File Servers, Catalogs, Abstractions, Adapters

Applications: Applications:

– – Remote Database Access for Remote Database Access for BaBar BaBar Code Code – – Remote Dynamic Linking for CDF Code Remote Dynamic Linking for CDF Code – – Logical Data Access for Bioinformatics Code Logical Data Access for Bioinformatics Code – – Expandable Database for MD Simulation Expandable Database for MD Simulation

Improving the OS for Grid Computing Improving the OS for Grid Computing

Remote Database Access Remote Database Access

script Parrot TSS file server file system DB data libdb.so sim.exe WAN CFS

HEP Simulation Needs Direct DB Access HEP Simulation Needs Direct DB Access

– – App linked against Objectivity DB. App linked against Objectivity DB. – – Objectivity accesses Objectivity accesses filesystem filesystem directly. directly. – – How to distribute application How to distribute application securely securely? ?

Solution: Remote Root Mount via TSS: Solution: Remote Root Mount via TSS:

parrot parrot – –M /=/chirp/fileserver/ M /=/chirp/fileserver/rootdir rootdir DB code can read/write/lock files directly. DB code can read/write/lock files directly.

GSI Auth GSI

Credit: Sander Klous @ NIKHEF

Remote Application Loading Remote Application Loading

appl Parrot ld.so HTTP server file system liba.so libb.so libc.so WAN

Credit: Igor Sfiligoi @ Fermi National Lab

HTTP

Modular Simulation Needs Many Libraries Modular Simulation Needs Many Libraries

– – Devel

• Devel. on workstations, then ported to grid.

. on workstations, then ported to grid. – – Selection of library depends on analysis tech. Selection of library depends on analysis tech. – – Constraint: Must use HTTP for file access. Constraint: Must use HTTP for file access.

Solution: Dynamic Link with TSS+HTTP: Solution: Dynamic Link with TSS+HTTP:

– – /home/ /home/cdfsoft cdfsoft -

• > /http/

> /http/dcaf.fnal.gov/cdfsoft dcaf.fnal.gov/cdfsoft

select several MB from 60 GB of libraries

Technical Problem Technical Problem

HTTP is not a HTTP is not a filesystem filesystem! (No directories) ! (No directories)

– – Advantages: Firewalls, caches, Advantages: Firewalls, caches, admins admins. .

Appl Parrot HTTP Module HTTP Server root etc home bin alice cms babar

• pendir(/home)
• pendir(/home)

GET /home HTTP/1.0 <HTML> <HEAD> <H1>

Technical Problem Technical Problem

Solution: Turn the directories into files. Solution: Turn the directories into files.

– – Can be cached in ordinary proxies! Can be cached in ordinary proxies!

Appl Parrot HTTP Module HTTP Server root etc home bin alice cms babar

• pendir(/home)
• pendir(/home)

GET /home/.dir HTTP/1.0 .dir .dir make httpfs alice babar cms

Logical Access to Bio Data Logical Access to Bio Data

Many databases of biological data in different Many databases of biological data in different formats around the world: formats around the world:

– – Archives: Swiss Archives: Swiss-

• Prot,

Prot, TreMBL TreMBL, NCBI, etc... , NCBI, etc... – – Replicas: Public, Shared, Private, ??? Replicas: Public, Shared, Private, ???

Users and applications want to refer to data Users and applications want to refer to data

• bjects by logical name, not location!
• bjects by logical name, not location!

– – Access the nearest copy of the non Access the nearest copy of the non-

• redundant protein

redundant protein database, don database, don’ ’t care where it is. t care where it is.

Solution: EGEE data management system maps Solution: EGEE data management system maps logical names ( logical names (LFNs LFNs) to physical names ( ) to physical names (SFNs SFNs). ).

Credit: Christophe Blanchet, Bioinformatics Center of Lyon, CNRS IBCP, France http://gbio.ibcp.fr/cblanchet, Christophe.Blanchet@ibcp.fr

Logical Access to Bio Data Logical Access to Bio Data

BLAST

Parrot

RFIO gLite HTTP FTP Chirp Server FTP Server gLite Server EGEE File Location Service Run BLAST on LFN://ncbi.gov/nr.data

• pen(LFN://ncbi.gov/nr.data)

Where is LFN://ncbi.gov/nr.data? Find it at: SFN://ibcp.fr/data/NR nr.data nr.data nr.data RETR nr.data

• pen(SFN://ibcp.fr/nr.data)

Appl Appl: Distributed MD Database : Distributed MD Database

State of Molecular Dynamics Research: State of Molecular Dynamics Research:

– – Easy to run lots of simulations! Easy to run lots of simulations! – – Difficult to understand the Difficult to understand the “ “big picture big picture” ” – – Hard to systematically share results and ask questions. Hard to systematically share results and ask questions.

Desired Questions and Activities: Desired Questions and Activities:

– – “ “What parameters have I explored? What parameters have I explored?” ” – – “ “How can I share results with friends? How can I share results with friends?” ” – – “ “Replicate these items five times for safety. Replicate these items five times for safety.” ” – – “ “Recompute Recompute everything that relied on this machine. everything that relied on this machine.” ”

GEMS: Grid Enabled Molecular Sims GEMS: Grid Enabled Molecular Sims

– – Distributed database for MD Distributed database for MD siml siml at Notre Dame. at Notre Dame. – – XML database for indexing, TSS for storage/policy. XML database for indexing, TSS for storage/policy.

GEMS Distributed Database GEMS Distributed Database

database server catalog server catalog server XML -> host1:fileA host7:fileB host3:fileC

A C B Y Z X

XML -> host6:fileX host2:fileY host5:fileZ data XML+ Temp>300K Mol==CH4 Credit: Jesus Izaguirre and Aaron Striegel, Notre Dame CSE Dept. host5:fileZ host6:fileX

DSFS

Adapter

Active Recovery in GEMS Active Recovery in GEMS

GEMS and Tactical Storage GEMS and Tactical Storage

Dynamic System Configuration Dynamic System Configuration

– – Add/remove servers, discovered via catalog Add/remove servers, discovered via catalog

Policy Control in File Servers Policy Control in File Servers

– – Groups can Collaborate within Constraints Groups can Collaborate within Constraints – – Security Implemented within File Servers Security Implemented within File Servers

Direct Access via Adapters Direct Access via Adapters

– – Unmodified Simulations can use Database Unmodified Simulations can use Database – – Alternate Web/ Alternate Web/Viz Viz Interfaces for Users. Interfaces for Users.

Outline Outline

Problems with the Standard Model Problems with the Standard Model Tactical Storage Systems Tactical Storage Systems

– – File Servers, Catalogs, Abstractions, Adapters File Servers, Catalogs, Abstractions, Adapters

Applications: Applications:

– – Remote Database Access for Remote Database Access for BaBar BaBar Code Code – – Remote Dynamic Linking for CDF Code Remote Dynamic Linking for CDF Code – – Logical Data Access for Bioinformatics Code Logical Data Access for Bioinformatics Code – – Expandable Database for MD Simulation Expandable Database for MD Simulation

Improving the OS for Grid Computing Improving the OS for Grid Computing

OS Support for Grid Computing OS Support for Grid Computing

Grid computing in general suffers because Grid computing in general suffers because

• f limitations in the operating system.
• f limitations in the operating system.

Security and permissions: Security and permissions:

– – No No ACLs ACLs -

• > hard to share data

> hard to share data – – Root can Root can setuid setuid -

• > hard to secure services.

> hard to secure services.

Resource allocation: Resource allocation:

– – Cannot reserve space Cannot reserve space -

• > jobs crash

> jobs crash – – Hard to clean up Hard to clean up procs procs -

• > unreliable systems

> unreliable systems

student root alice httpd visitor kerberos bob visitor anon1 anon2

These two users are completely different: root:kerberos:alice:visitor root:kerberos:bob:visitor The web server can create distinct anonymous accounts. No need for global nobody.

kerberos given to the login server. alice created by krb5 login. student created at run-time.

Tactical Storage Systems Tactical Storage Systems

Separate Separate Abstractions Abstractions from from Resources Resources Components: Components:

– – Servers, catalogs, abstractions, adapters. Servers, catalogs, abstractions, adapters. – – Completely user level. Completely user level. – – Performance acceptable for real applications. Performance acceptable for real applications.

Independent but Cooperating Components Independent but Cooperating Components

– – Owners of file servers set policy. Owners of file servers set policy. – – Users must work within policies. Users must work within policies. – – Within policies, users are free to build. Within policies, users are free to build.

Parting Thought Parting Thought

Many users of the grid are constrained by Many users of the grid are constrained by functionality functionality, not performance. , not performance. TSS allows end users to build the TSS allows end users to build the structures that they need for the moment structures that they need for the moment without involving an admin. without involving an admin. Analogy: building blocks Analogy: building blocks for distributed storage. for distributed storage.

Acknowledgments Acknowledgments

Science Collaborators: Science Collaborators:

– – Christophe Christophe Blanchet Blanchet – – Sander Sander Klous Klous – – Peter Peter Kunzst Kunzst – – Erwin Laure Erwin Laure – – John John Poirer Poirer – – Igor Igor Sfiligoi Sfiligoi

CS Collaborators: CS Collaborators:

– – Jesus Jesus Izaguirre Izaguirre – – Aaron Aaron Striegel Striegel

CS Students: CS Students:

– – Paul Brenner Paul Brenner – – James Fitzgerald James Fitzgerald – – Jeff Jeff Hemmes Hemmes – – Paul Madrid Paul Madrid – – Chris Chris Moretti Moretti – – Phil Snowberger Phil Snowberger – – Justin Wozniak Justin Wozniak

For more information... For more information...

Cooperative Computing Lab Cooperative Computing Lab http://www.cse.nd.edu/~ccl http://www.cse.nd.edu/~ccl Cooperative Computing Tools Cooperative Computing Tools http:// http://www.cctools.org www.cctools.org Douglas Thain Douglas Thain

– – dthain@cse.nd.edu dthain@cse.nd.edu – – http:// http://www.cse.nd.edu/~dthain www.cse.nd.edu/~dthain