T he B2 SE RVICE SUIT E www e uda t e u/ se rvic e s www.e uda - - PowerPoint PPT Presentation

T he B2 SE RVICE SUIT E

www e uda t e u/ se rvic e s www.e uda t.e u/ se rvic e s

T his wo rk is lic e nse d unde r the Cre a tive Co mmo ns CC BY 4 0 lic e nc e

E UDAT re c e ive s funding fro m the E uro pe a n Unio n's Ho rizo n 2020 pro g ra mme - DG CONNE CT e -I nfra struc ture s. Co ntra c t No . 654065

www.e uda t.e u Co mmo ns CC-BY 4.0 lic e nc e . Attrib utio n: E UDAT – www.e uda t.e u

B2 SE RVICE SUIT E

E UDAT

• ffe rs a c omple te se t of r

e se a r c h

E UDAT

• ffe rs a c omple te se t of r

e se a r c h da ta se r vic e s, e xpe r tise a nd te c hnolog y solutions to a ll (E

uro pe a n) sc ie ntists a nd

solutions to a ll (E

uro pe a n) sc ie ntists a nd re se a rc he rs. T he se sha r

e d se r vic e s a nd stor a g e r e sour c e s

T he se sha r

e d se r vic e s a nd stor a g e r e sour c e s

a re distr

ibute d a c r

• ss 15 E

ur

• pe a n

c ountr ie s c ountr ie s.

Da ta a re sa fe ly sto re d a lo ng side so me o f

E ’ t f l t E ur

• pe ’s most powe r

ful supe r c ompute r s.

B2 Se r vic e s

3

B2 SE RVICE SUIT E is pa rt of E UDAT ...

a pa n-E uro pe a n initia tive b uilding a susta ina b le

p

a pa n E uro pe a n initia tive b uilding a susta ina b le c ro ss-disc iplina ry a nd c ro ss-na tio na l da ta infra struc ture pro viding a se t o f sha re d se rvic e s fo r a c c e ssing a nd pre se rving re se a rc h da ta suppo rting multiple re se a rc h c o mmunitie s b y ki l l ith th t d li th t h i l wo rking c lo se ly with the m to de live r the se te c hnic a l se rvic e s a s pa rt o f the E UDAT Co lla b o ra tive Da ta I nfra struc ture (CDI ) I nfra struc ture (CDI )

A tr uly pa n- E ur

• pe a n Infr

a str uc tur e y p p

E UDAT

• ffe rs

c o mmo n da ta se rvic e s, suppo rting multiple re se a rc h c o mmunitie s a s we ll a s individua ls multiple re se a rc h c o mmunitie s a s we ll a s individua ls, thro ug h a g e o g ra phic a lly distrib ute d, re silie nt ne two rk

• f 35 E

uro pe a n o rg a nisa tio ns Our visio n is to e na b le E uro pe a n re se a rc he rs a nd pra c titio ne rs fro m a ny re se a rc h disc ipline to pre se rve , find, d d t i a c c e ss, a nd pro c e ss da ta in a truste d e nviro nme nt, a s pa rt o f a Co lla b o ra tive Da ta I nfra struc ture I nfra struc ture

Community- Drive n Solutions y

E U DAT se rvic e s are de signe d, built and imple me nte d base d o n use r c o mmunity re quire me nts

BIOMEDICAL &

c o mmunity re quire me nts.

MATERIALS & BIOMEDICAL & MEDICAL SCIENCES ANALYTICAL FACILITIES

MAPPER

PHYSICAL SCIENCES & ENGINEERING

Why Should I Use it?

Outsourc e yo ur da ta ma na g e me nt ne e ds…

y

y g L e ss e ffo rt tha n b uilding yo ur o wn infra struc ture Mo re time a nd mo ne y fo r the re se a rc h … to a g ro up o f re spe c te d a nd truste d da ta c e ntre s... With ti l d i t ti l it With a na tio na l a nd inte rna tio na l re mit Ha ve a lo ng pa st a nd future Who ha ve a we a lth o f e xpe rie nc e suppo rting ... Who ha ve a we a lth o f e xpe rie nc e suppo rting

da ta ma na g e me nt fo r sc ie nc e.

Ma ny o f who m suppo rt lo ng te rm c ura tio n o r a re Ma ny o f who m suppo rt lo ng te rm c ura tio n o r a re wo rking to suppo rt it

Suppor t thr

• ughout the life c yc le

b2drop.e uda t.e u

pp g y

CREATING DATA PROCESSING DATA RE‐USING DATA ANALYSING DATA GIVING ACCESS TO DATA PRESERVING DATA

E UDAT DAT A DOMAINS

PUBLISHED DATA DOMAIN

Discovery of Discovery of Linking Linking Discovery of Digital Objects Discovery of Digital Objects g Digital Objects

• Publications

g Digital Objects

• Publications

REGISTERED DATA DOMAIN

Register Register Stage Stage

WORKSPACE ( )

Digital Objects Digital Objects Digital Objects Digital Objects

(TEMPORARY ‐ TRANSIENT)

Sync a nd E xc ha ng e Re se a rc h Da ta

B2DROP

E UDAT ’ s Pe rso na l Clo ud Sto ra g e Se rvic e E UDATs Pe rso na l Clo ud Sto ra g e Se rvic e

B2DROP is a se c ure a nd truste d da ta e xc ha ng e se rvic e fo r re se a rc he rs a nd sc ie ntists to ke e p the ir re se a rc h da ta re se a rc he rs a nd sc ie ntists to ke e p the ir re se a rc h da ta

sync hronize d a nd up-to -da te a nd to e xc ha ng e with o the rs.

b2drop.e uda t.e u www.e uda t.e u

b2drop.e uda t.e u

An ide a l solution fo r re se a rc he rs a nd sc ie ntists to: Store a nd e xc ha ng e da ta with

c o lle a g ue s a nd te a m me mb e rs, inc luding re se a rc h da ta no t fi li d f b li hi fina lize d fo r pub lishing

sha re da ta with fine -g ra ine d

a c c e ss c o ntro ls

sync hronize multiple ve rsions o f

da ta a c ro ss diffe re nt de vic e s

F e a ture s: e a u e s:

20GB sto ra g e pe r use r L iving o b je c ts, so no PI Ds Ve rsio ning a nd o ffline use Ve rsio ning a nd o ffline use De skto p sync hro nisa tio n

Store a nd Sha re Re se a rc h Da ta

B2SHARE

B2SHARE is a use r- frie ndly, re lia ble a nd tr

ustwor thy wa y fo r

re se a rc he rs, sc ie ntific c o mmunitie s a nd sc ie ntists to store a nd

sha re sma ll-sc a le re se a rc h da ta fro m dive rse c o nte xts.

b2sha r e .e uda t.e u www.e uda t.e u

b2sha re .e uda t.e u

A winning so lutio n fo r re se a rc he rs, sc ie ntists a nd c ommunitie s to :

store da ta sa fe ly a t a truste d

a nd c e rtifie d da ta c e ntre

pre se rve da ta to g ua ra nte e

l t i t lo ng -te rm pe rsiste nc e

c ontrol a c c e ss a nd sha re da ta with c o lle a g ue s a nd the

wo rld wo rld

F e a ture s:

me ta da ta ma na g e me nt pe rma ne nt PI Ds p Ope n Ac c e ss suppo rt

Re plic a te Re se a rc h Da ta Sa fe ly

B2SAF E

B2SAF E is a robust, sa fe a nd hig hly a va ila ble se rvic e whic h a llo ws c o mmunity a nd de pa rtme nta l re po sito rie s to imple me nt

da ta ma na g e me nt polic ie s on re se a rc h da ta a c ro ss multiple da ta ma na g e me nt polic ie s on re se a rc h da ta a c ro ss multiple

a dministra tive do ma ins in a trustwo rthy ma nne r.

e uda t.e u/ b2sa fe www.e uda t.e u

e uda t.e u/ b2sa fe

T he ide a l solution fo r c o mmunitie s with no fa c ility for a rc hiva l to: re plic a te re se a rc h da ta into

se c ure da ta sto re s

hi d

h

a rc hive a nd pre se rve re se a rc h

da ta in the lo ng -te rm b ring da ta c lo se to powe rful

c ompute re sourc e s c o- loc a te da ta with diffe re nt

c o mmunitie s b e ne fit fro m e c onomie s of sc a le

F e a ture s:

la rg e -sc a le sto ra g e ro b ust a nd hig hly a va ila b le g y pe rma ne nt PI Ds

B2SAF E Use Ca se from CL ARIN E RIC Re plic a tion of L ing uistic Da ta Re plic a tion of L ing uistic Da ta

iRODS PID PID

SAMQFS

iRODS iRODS

GPFS

O S

dCache

O S

GPFS dCache

HPSS DMF

16

Ge t Da ta to Computa tion

B2ST AGE

B2ST AGE is a re lia ble , e ffic ie nt, lig ht- we ig ht a nd e a sy- to- use se rvic e to tra nsfe r re se a rc h da ta se ts b e twe e n E UDAT sto ra g e re so urc e s a nd hig h-pe rfo rma nc e c o mputing (HPC) wo rkspa c e s

e uda t.e u/ b2sta g e www.e uda t.e u

e uda t.e u/ b2sta g e

move la rg e a mounts of da ta F a c ilita ting c ommunitie s to: move la rg e a mounts of da ta

b e twe e n da ta sto re s a nd hig h- pe rfo rma nc e c o mpute re so urc e s

re - ing e st c omputa tiona l re sults re - ing e st c omputa tiona l re sults

b a c k into E UDAT

de posit la rg e da ta se ts o nto E

UDAT re so urc e s fo r lo ng te rm pre se rva tio n re so urc e s fo r lo ng -te rm pre se rva tio n

F e a ture s:

hig h-spe e d tra nsfe r re lia b le a nd lig ht-we ig ht g g ma na g e s pe rma ne nt PI Ds

F ind Re se a r c h Da ta

B2F IND

B2F I ND is a simple , use r-frie ndly me ta da ta c a ta log ue of

re se a rc h da ta c olle c tions sto re d in E

UDAT da ta c e ntre s a nd o the r re po sito rie s.

b2find.e uda t.e u www.e uda t.e u

b2find.e uda t.e u

A me ta da ta c a ta log ue se rvic e to: se e k da ta obje c ts a nd c olle c tions

using po we rful me ta da ta se a rc he s

c a ta log ue c ommunity da ta b y c a ta log ue c ommunity da ta b y

me a ns o f se le c te d me ta da ta

browse throug h multi- disc iplina ry da ta c o lle c tio ns filte re d b y c o nte nt da ta c o lle c tio ns filte re d b y c o nte nt,

pro ve na nc e a nd te mpo ra l ke ywo rds

F e a ture s:

simple to use sta nda rds-b a se d c o mpre he nsive c a ta lo g ue

IDM Inte g ra tion

https:/ / b 2a c c e ss.e uda t.e u

Primary Primary Identities

p / /

B2SHARE (O th 2) B2SAFE (X 509)

OpenID IdPs from RIs

e g ORCID (Oauth 2) B2STAGE (X.509)

B2ACCESS IdP

e.g. ESGF, ENES ORCID ResearchID Scopus B2DROP (X.509)

User Profile

Social IdP

Linkedin Google Facebook Linkedin Access Token B2DROP (SAML) M lti P t l OAuth 2 authorization server

OpenID P

eduGAIN X.509 B2HANDLE (SAML) Data Project Multi‐Protocol Identity Management, LoA support EUDAT CA SAML

SAML Id

RIs e.g. SAML j Coordination Portal Helpdesk TTS Site & Service powered by Unity IDM EUDAT federation database KI CLARIN

9 IdP

PRACE EGI PRACE EGI EUDAT Service Endpoints Site & Service Registry

21

B2ACCESS AAI functions database PK

X.509

EGI WLCG EGI WLCG

Roa dma p Da ta Ac c e ss a nd Re - Use Se rvic e Are a

b2drop.e uda t.e u

M12‐M18

Mar 2016 – Aug 2016

M19‐M24

Sep 2016 – Feb 2017

+M24

Mar 2017 – Feb 2018

B2SHARE

• Release of B2SHARE 2.0
• Pilot: integration with

B2NOTE

• Support for cloud

storage services like DropBox and Google

• Integration with the

EUDAT Generic Execution Framework

• Pilot: B2SHARE as EUDAT

CDI Metadata Store

• Integrate with DTR for pilot

users Drive

• Support for more

storage back‐end solutions, e.g. cloud‐ (GEF)

• Support for metadata

extraction and data exploration via the storage and object stores

• Support versioning
• Support Digital Object

GEF Identifiers (DOIs)

B2DROP

• Integrated with B2SHARE
• Integrated with B2ACCESS
• Deployment with Puppet

Deployment with Puppet and Docker

Roa dma p Da ta Ac c e ss a nd Re - Use Se rvic e Are a

b2drop.e uda t.e u

M12‐M18

Mar 2016 – Aug 2016

M19‐M24

Sep 2016 – Feb 2017

+M24

Mar 2017 – Feb 2018

B2FIND

• Continued community

integration

• Improved user experience
• Customisation of GUI for

communities

• Prototype of SRU interface
• SRU interface in

production

• Integrate with
• Resolve granularity issues
• Integration with B2NOTE
• Performance and scalability

improvements

• Extend harvesting

methodes (OGC / CSW )

• Improved search

functionalities for EUDAT CDI Metadata Store

• Integration with

Data Type Registry hierarchical search and taxonomies

• Improved semantic

mapping

Registry Services

• Pilot instance of the Data

Type Registry

• Collaborate with pilot

communities on further

• Further development of

the DTR based on feedback from communities and

• ther EUDAT services
• Evaluate DTR pilot
• Bring DTR into

production

• Integrate DTR with

communities on further evaluation and adaptation

• f the DTR
• Integrate the DTR pilot with

B2ACCESS for Federated AAI

• ther EUDAT services
• Integration of the DTR with

B2NOTE Integrate DTR with

• ther EUDAT

services based on pilot evaluation B2ACCESS for Federated AAI

• Integration of the pilot DTR

with B2SHARE and B2FIND

Roa dma p Da ta Ac c e ss a nd Re - Use Se rvic e Are a

b2drop.e uda t.e u

M12‐M18

Mar 2016 – Aug 2016

M19‐M24

Sep 2016 – Feb 2017

+M24

Mar 2017 – Feb 2018

B2ACCESS

• B2SHARE integrated with

B2ACCESS for authentication

• Support for integration

with external community sites

• Complete integration

with external community sites

• Integration with Data Type

Registry pilot

• Integration with EUDAT CDI

common HTTP REST API

• Integration with PRACE
• Integration with EGI
• Distributed authorisation
• Complete integration

with PRACE

• Complete integration

with EGI

• Integration with B2SAFE &

DPM

• Integration with B2DROP
• Integration with B2STAGE
• Improvements based
• n change requests

from communities and partners

• Integration with Data

Project Coordination Portal

• Installation packages &

distributed setup

• Pilot for XACML based

authorisation solution

E UDAT Pr

• duc tion E

nvir

• nme nt

Helpdesk & Support

Data Management Project Enabling

p pp Security Team

• Network, Configuration
• Compute Resources

Service Hosting, Service on Demand Service Deployment

• Storage, Storage Services

Operational & Central Services Service and Resource Provisioning & Coordination 14 generic centres, 15PB committed, 5‐10Gb/s per site (potential of > 1000 PB aggregated)

25

Inte r a c tions With Use r s

Da ta Pilo ts 5 b io me d a nd life sc ie nc e 7 e a rth sc ie nc e , e ne rg y a nd e nviro nme nt 5 h i l i d i i 5 physic a l sc ie nc e s a nd e ng ine e ring 5 So c ia l Sc ie nc e a nd Huma nitie s I nte ra c tio ns thro ug h da ta pro je c t c o -o rdina tio n po rta l I nte ra c tio ns thro ug h da ta pro je c t c o -o rdina tio n po rta l Allo ws use r to spe c ify the se rvic e s (Qo S) tha t the y wa nt https:/ / dp.e uda t.e u/ c o mmunitie s/ E UDAT / e xa mple

• o f-a -da ta -pilo t-de sc riptio n/ vie w

And re q ue st pre fe re nc e s fo r site s And re q ue st pre fe re nc e s fo r site s … b ut still ne e ds 1 to 1 disc ussio ns o n ma ny o c c a sio ns

Ma king it RE AL L Y use ful

RE ST API a nd pytho n lib ra ry

g

RE ST API a nd pytho n lib ra ry So me se rvic e s ha ve we ll suppo rte d API (B2DROP (we b DAV) a nd B2SHARE (RE ST ) ( ) ( ) Othe r RE ST API s unde r de ve lo pme nt

E UDAT python libra ry (ong oing )

e uda t.e u/ b2sta g e

U py

• b a y (o

g o g )

A prog ra mma ble a nd fle xible A prog ra mma ble a nd fle xible

python API libra ry to c re a te a nd e xe c ute wor kflows c o mpo sing

p g diffe re nt E UDAT se rvic e s

L e t’s ma ke E UDAT sc r ita ble !

• Enable collaborations with other projects

(PRACE, EGI..)

• First draft of the library:

○ find datasets by community names or ○ find datasets by community names or tags using B2FIND ○ Manage transfers using Globus Online API (and FTS3 later on);

E UDAT python libra ry: e xa mple s

e uda t.e u/ b2sta g e

U py

• b a y: e

a p e s

Data search:

from eudat import b2find # Get “aleph” dataset datasets = b2find get dataset source(community='aleph') datasets = b2find.get_dataset_source(community= aleph ) print 'First dataset URI = ' + datasets[0]

PID resolution:

from eudat import b2handle # Get physical URL of the object resolving PID URL = b2handle get url by pid("/11100/33ac01fc‐6850‐11e5‐b66e‐e41f13eb32b2")

Data transfer:

URL = b2handle.get_url_by_pid( /11100/33ac01fc‐6850‐11e5‐b66e‐e41f13eb32b2 ) print "Returned URL: " + URL from eudat import b2stage # Initialize the Globus client globus = b2stage.ClientGlobus(resource_file_path=RESOURCES_FILE, debug=True) # Perform a data transfer between 2 endpoints (third party transfer) task_id = globus.transfer(src_endpoint, dst_endpoint, src, dst, recursive=args['-r'])

F

• r more info:

htt / / b 2d d t https:/ / b 2dro p.e uda t.e u https:/ / e uda t.e u/ se rvic e s/ use rdo c / b 2dro p https:/ / b 2sha re .e uda t.e u https:/ / e uda t.e u/ se rvic e s/ use rdo c / b 2sha re https:/ / e uda t.e u/ se rvic e s/ use rdo c / b 2sa fe https:/ / e uda t.e u/ se rvic e s/ use rdo c / b 2sta g e p g http:/ / b 2find.e uda t.e u https:/ / e uda t e u/ se rvic e s/ use rdo c / b 2find https:/ / e uda t.e u/ se rvic e s/ use rdo c / b 2find