Globus integration in the NCAR RDA data portal: Recent enhancements - - PowerPoint PPT Presentation

▶
globus integration in the ncar rda data portal recent
SMART_READER_LITE
LIVE PREVIEW

Globus integration in the NCAR RDA data portal: Recent enhancements - - PowerPoint PPT Presentation

Oct 19, 2022 383 likes •536 views

NCAR/CISL/Research Data Archive Globus integration in the NCAR RDA data portal: Recent enhancements Thomas Cram Shortened presentation title NCAR, Computational Information Systems Laboratory (CISL) Shortened presentation title GlobusWorld

slide-1
SLIDE 1

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

Globus integration in the NCAR RDA data portal: Recent enhancements

GlobusWorld 2018

April 25, 2018

Thomas Cram

NCAR, Computational Information Systems Laboratory (CISL)

NCAR/CISL/Research Data Archive

slide-2
SLIDE 2

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

About the RDA

  • Purpose

– Support climate & weather research

at NCAR and UCAR universities with reference datasets

  • Data collections

– Ocean & atmospheric observations,

analyses, reanalyses, operational NWP products

– 700+ datasets, 8M files, 1.8 PB

  • Free and open access
  • Worldwide usage
  • Science educated staff

2

rda.ucar.edu

slide-3
SLIDE 3

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

Supported data access pathways

3

  • HTTPS (wget, cURL, REST

API, etc.)

  • OPeNDAP
  • GridFTP (Globus)
  • HPSS staged-to-disk
  • Direct access on GLADE

filesystem (NCAR users)

  • Subsetting (spatial,

parameter, temporal, etc.)

  • Data format conversion
slide-4
SLIDE 4

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

How does Globus simplify data management and improve our user services? Use cases:

  • 1. Curated file lists
  • 2. Automated transfers of delayed mode

products

* Facilitated by:

  • Globus Auth
  • Browse Endpoint helper
  • Python SDK

4

slide-5
SLIDE 5

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

Use case #1: Curated file lists * BEFORE GLOBUS Integration*

5

  • Data discovery on RDA

web app

slide-6
SLIDE 6

Shortened presentation title Shortened presentation title NCAR RDA Globus integration 6

Use case #1: Curated file lists * BEFORE GLOBUS Integration *

  • Data discovery on RDA

web app

  • Redirect to Globus UI
  • Blind search for files
slide-7
SLIDE 7

Shortened presentation title Shortened presentation title NCAR RDA Globus integration 7

  • Data discovery on RDA

web app

  • Redirect to Globus UI
  • Blind search for files
  • Request timeouts for

long dir listings

Use case #1: Curated file lists * BEFORE GLOBUS Integration *

slide-8
SLIDE 8

Shortened presentation title Shortened presentation title NCAR RDA Globus integration 8

Use case #1: Curated file lists * AFTER GLOBUS Integration *

  • Discovery, file list

curation (RDA portal)

slide-9
SLIDE 9

Shortened presentation title Shortened presentation title NCAR RDA Globus integration 9

Use case #1: Curated file lists * AFTER GLOBUS Integration *

  • Discovery, file list

curation (RDA portal)

  • Select destination

endpoint (Browse Endpoint helper API)

slide-10
SLIDE 10

Shortened presentation title Shortened presentation title NCAR RDA Globus integration 10

  • Discovery, file list

curation (RDA portal)

  • Select destination

endpoint (Browse Endpoint helper API)

  • Receive Auth code,

exchange for token

  • Submit transfer on

user’s behalf

Use case #1: Curated file lists * AFTER GLOBUS Integration *

code = form.getvalue('code’) tokens = client.oauth2_exchange_code_for_tokens(code) ... ... ... task_id = transfer.submit_transfer(transfer_data)['task_id']

slide-11
SLIDE 11

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

BEFORE GLOBUS Integration:

  • User submits request
  • Server side processing

(delayed mode)

  • E-mail download link to user
  • Request Globus share on

download page

11

Example 2: Delayed mode request transfers

slide-12
SLIDE 12

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

With GLOBUS integration:

  • User requests Globus transfer during

form submission

  • User selects destination endpoint
  • Receive Auth code, exchange for

token, store in session

  • Finish processing, e-mail notification

sent to user

– Data already on its way

12

Example 2: Delayed mode request transfers

slide-13
SLIDE 13

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

Looking ahead

  • HTTPS access
  • Big data challenges

– Campaign storage – Cloud-hosted environments – Support required for a large community of NCAR users

13

slide-14
SLIDE 14

Shortened presentation title Shortened presentation title NCAR RDA Globus integration

Contact us

rda.ucar.edu rdahelp@ucar.edu

14

ncarrda.blogspot.com @NCAR.RDA @NCAR_RDA