[PPT] - SRM Space Tokens SRM Space Tokens SRM Space Tokens SRM Space PowerPoint Presentation

SLIDE 1

SRM Space Tokens SRM Space Tokens SRM Space Tokens SRM Space Tokens

Scalla/xrootd

Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University Stanford University 27-May-08 http://xrootd slac stanford edu http://xrootd.slac.stanford.edu

SLIDE 2

O tli Outline

Introduction

SRM Static Space Token Refresher

S T k i th S ll A hit t Space Tokens in the Scalla Architecture

Disk partitions as a space token paradigm

How it was done

Space usage and quotas by space token

New Stuff

Proxies unlimited Announcements

Conclusion & Future Outlook

27-May-08 2: http://xrootd.slac.stanford.edu

Conclusion & Future Outlook

SLIDE 3

SRM St ti S T k SRM Static Space Tokens

Encapsulate fixed space characteristics

Type of space

yp p

E.g., Permanence, performance, etc.

Imply a specific quota

Using a particular arbitrary name

E.g., data, dq2, mc, etc E.g., data, dq2, mc, etc

Typically used to create new files

Think of it as a space profile

27-May-08 3: http://xrootd.slac.stanford.edu

Think of it as a space profile

SLIDE 4

SRM S T k & P th SRM Space Tokens & Paths

Static Space Tokens may be redundant

True if exclusive correspondence exists

p

Token

Path

dq2

/atlas/dq2/….

mc

/atlas/mc/….

Space tokens useful for overlapping namespaces

Makes space token utility non-obvious

But I digress, so let’s move on….

27-May-08 4: http://xrootd.slac.stanford.edu

SLIDE 5

S T k & td Space Tokens & xrootd

Space attribute concept already part of xrootd

Embodied by notion of cgroup

y g p

A cgroup is a logical name for one or more file systems

Implemented in the standard oss plug-in p p g

Used by default libXrdOfs.so plug-in

The real work was to support SRM concepts The real work was to support SRM concepts

Largely in the area of virtual quotas

Opportunity to greatly improve the implementation

27-May-08 5: http://xrootd.slac.stanford.edu

Opportunity to greatly improve the implementation

SLIDE 6

Wh D S T k A l ?

h i i

Where Do Space Tokens Apply?

authentication (gsi, krb5, etc)

P t l (1 f ) Protocol Driver

(XRD) lfn2pfn

prefix encoding

authorization

(name based)

Protocol (1 of n) (xrootd) Storage System

(oss, drm/srm, etc)

(name based)

File System

(ofs, sfs, alice, etc) (oss, drm/srm, etc)

Clustering

(ofs, sfs, alice, etc)

SRM Static Space Tokens Implemented Here 27-May-08 6: http://xrootd.slac.stanford.edu

Clustering

(cmsd)

SLIDE 7

P i i S T k P di Partitions as a Space Token Paradigm

Disk partitions map well to SRM space tokens

A set of partitions embody a set of space attributes

Performance, quota, etc.

A static space token defines a set of space attributes

P titi d t ti t k i t h bl

Partitions and static space tokens are interchangeable

xrootd already supports multiple partitions

Real as

ell as irt al partitions

Real as well as virtual partitions

Can leverage this support for SRM space token support

So on to xrootd partition management

27-May-08 7: http://xrootd.slac.stanford.edu

So, on to xrootd partition management

SLIDE 8

P titi A hit t Partition Architecture

N real partitions can be aggregated

Each aggregation is called a virtual partition

gg g p

Uniform name space across all partitions

Real partitions are space load balanced Reduces the granularity of failure

Implemented via symlinks from a name space

Name space itself resides in a real partition

li k li k li k li k Name Space Disk Space Disk Space Disk Space Disk Space

27-May-08 8: http://xrootd.slac.stanford.edu

symlinks symlinks symlinks symlinks

SLIDE 9

Vi t l R l P titi Virtual vs Real Partitions

Vi l R l h bli / t 1 Virtual Partitions Real Partitions

ss.cache public /store1 xa
ss.cache dq2 /store2 xa
ss.cache data /store3 xa

Simple two step process

Define your real partitions (one or more)

These are file system mount-points

What’s this? What’s this?

Map virtual partitions on top of real ones

Virtual partitions can share real partitions

By convention, virtual partitions equal static token names

Yields implicit SRM space token support

27-May-08 9: http://xrootd.slac.stanford.edu

p p pp

SLIDE 10

I t d i P titi Introducing xa xa Partitions

Original oss partition architecture was limited

Simplistic symlink target names

Constrained file path length to 255 or less Could not automatically track assigned space tokens

The xa xa option introduced for SRM support The xa xa option introduced for SRM support

Supports paths up to 1024 characters Automatically tracks assigned space token

y g p

Tracks usage for real and virtual partitions

Both supported for backward compatibility

27-May-08 10: http://xrootd.slac.stanford.edu

The xa

xa version is now preferred in all cases

SLIDE 11

P titi A ti Partition Aggregation

Name Space Partition

/atlas/mydatafile

p Holds Name Space Can also optionally hold data files

Independent Filesystems Mount Points

/store1/public

symlink

hold data files

Mount Points

Symlink conventions simplify audits &

/store1 /store2

symlink

simplify audits & name space recovery

/store3

/store2/dq2/00/1234567890% q /store2/dq2/00/1234567890%.pfn

ss.cache public /store1 xa
ss.cache dq2 /store2 xa
ss cache data

/store3 xa #

Virtual partitions dq2 & data now share same Real partition

27-May-08 11: http://xrootd.slac.stanford.edu /store3/data

ss.cache data /store3 xa

/store2/data

#

ss.cache data /store2 xa

Real partition

SLIDE 12

P titi S l ti Partition Selection

Partitions selected by virtual partition name

Configuration file:

ss.cache public /store1 xa
ss.cache dq2 /store2 xa
ss.cache data /store3 xa

New files “cgi-tagged” with virtual partition name

root://host:1094//atlas/mydatafile?cgroup=dq2

The default is “public” The default is public

File allocated in a real partition associated with

the named virtual partition

27-May-08 12: http://xrootd.slac.stanford.edu

By convention, the name is the SRM space token name

SLIDE 13

R l Vi t l P titi Real vs Virtual Partitions

A real partition represents a hard quota

Non-overlapping virtual partitions are real

pp g p

Simple and very effective

Typically not efficiently utilized

Shared real partitions

Overlapping virtual partitions are virtual Overlapping virtual partitions are virtual Provide better space utilization, but…

Need usage tracking and quota management

27-May-08 13: http://xrootd.slac.stanford.edu

Need usage tracking and quota management

SLIDE 14

P titi U T ki Partition Usage Tracking

Usage is tracked by partition

Automatic for real partitions

p

Configurable for virtual partitions

ss.usage {nolog | log dirpath}

As Virtual Partition ⇔ SRM Space Token

Usage is also automatically tracked by space token Usage is also automatically tracked by space token

POSIX getxattr() returns usage information

See Linux man page

27-May-08 14: http://xrootd.slac.stanford.edu

See Linux man page

SLIDE 15

P titi Q t M t Partition Quota Management

Quotas applied by partition

Automatic for real partitions

p

Configurable for virtual partitions

ss.usage quotafile filepath

POSIX getxattr() returns quota information

Used by Fuse/xrootd to enforce quotas Used by Fuse/xrootd to enforce quotas

Fuse has view of the complete cluster

Using the cluster name space daemon 27-May-08 15: http://xrootd.slac.stanford.edu

SLIDE 16

Th Q t Fil The Quota File

Lists quota for each virtual partition

Hence, also a quota for each static space token Simple multi-line format

vpname nnnn[k | m | g | t]\n

Re-read whenever it changes Re-read whenever it changes

Useful only in the context of the cnsd xrootd

Quotas need to apply to the whole cluster Quotas need to apply to the whole cluster

Investigating native integration with the redirector

Currently, only FUSE/xrootd enforces quotas

27-May-08 16: http://xrootd.slac.stanford.edu

Currently, only FUSE/xrootd enforces quotas

SLIDE 17

Oth C id ti Other Considerations

Files cannot be easily reassigned space tokens

Must manually “move” file across partitions

y p

Partitions 1-to-1 correspondence with space tokens

Can always get original space token name y g g p

Use file-specific getxattr() call

Quotas for virtual partitions are “soft” Quotas for virtual partitions are soft

Time causality prevents hard limit

Use real partitions if hard limit needed

27-May-08 17: http://xrootd.slac.stanford.edu

Use real partitions if hard limit needed

SLIDE 18

P i U li it d Proxies Unlimited

Cl i P S Classic Proxy Server

Restricted to a very specific role

I d i FUSE FUSE Introducing FUSE FUSE as a proxy

All cluster features available

xrootd xrootd

We are still investigating this exciting concept

cmsd cmsd xrootd xrootd

Proxy plugin Proxy plugin

xrootd xrootd

FUSE FUSE

Fuse/xrootd

xrootd xrootd ff ti d

27-May-08 18: http://xrootd.slac.stanford.edu

xroot cluster proxy node effective proxy node

SLIDE 19

FUSE FUSE P T f R t FUSE FUSE Proxy Transfer Rates

SLAC #Streams BNL MB/s CERN MB/s scp 1 0.7 0.4 idft 1 1 4 0 4 gridftp 1 1.4 0.4 xrdcp 1 2.2 0.6 xrdcp 5 5.6 2.2 xrdcp 11 10.0 4.8 xrdcp 15 10.1 6.1 We don’t have any explanations yet. We don’t have any explanations yet. We need to do more tests. We need to do more tests.

(especially with multi (especially with multi stream gridftp) stream gridftp)

FUSE FUSE

Fuse/xrootd

xrootd xrootd

27-May-08 19: http://xrootd.slac.stanford.edu

(especially with multi (especially with multi-stream gridftp) stream gridftp)

effective proxy node

SLIDE 20

S D t Sh i A hi bl Secure Data Sharing Achievable

FUSE FUSE

xrootd xrootd

FUSE FUSE

xrootd xrootd BNL BNL WISC WISC

FUSE FUSE

Fuse/xrootd

effective proxy node

FUSE FUSE

Fuse/xrootd

effective proxy node xrootd xrootd xrootd xrootd UTA UTA SLAC SLAC US Atlas US Atlas

FUSE FUSE

Fuse/xrootd

xrootd xrootd ff ti d

FUSE FUSE

Fuse/xrootd

xrootd xrootd ff ti d

Data Data

Redirectors Redirectors

27-May-08 20: http://xrootd.slac.stanford.edu

effective proxy node effective proxy node

Data Data everywhere everywhere

SLIDE 21

A t ! A t ! Announcements! Announcements!

The CERN-based Scalla web page is online!

http://savannah.cern.ch/projects/xrootd/

p p j

Scalla CVS repository is going public!

Will be located in afs with unrestricted read access Will be located in afs with unrestricted read access Planning on providing web access

27-May-08 21: http://xrootd.slac.stanford.edu

SLIDE 22

C l i & F t O tl k Conclusion & Future Outlook

Scalla/xrootd is living up to our expectations

Relatively easy to add SRM space token support

More improvement are in the pipeline

Kernel memory direct data transfer (a.k.a. sendfile)

Significant reduction in CPU/Memory usage

Directed Support Services (DSS DSS) Architecture

A t k ff M lti i l fil

A take-off on Multics special files

Currently supporting Just-In-Time Alice Data Analysis

Bandwidth management during on-demand data transfers

27-May-08 22: http://xrootd.slac.stanford.edu

Framework for simple intra-cluster resource management

SLIDE 23

A k l d t Acknowledgements

Current software Collaborators

Andy Hanushevsky, Fabrizio Furano Root: Fons Rademakers, Gerri Ganis (security), Bertrand Bellenot (windows) Alice: Derek Feichtinger, Andreas Peters, Guenter Kickinger STAR/BNL: Pavel Jackl, Jerome Lauret SLAC: Jacek Becla, Tofigh Azemoon, Wilko Kroeger

, g , g

Operational collaborators

BNL, CERN, CNAF, FZK, INFN, IN2P3, GSI, RAL, SLAC

SLAC Funding SLAC Funding

US Department of Energy

Contract DE-AC02-76SF00515 with Stanford University

27-May-08 23: http://xrootd.slac.stanford.edu