Making the Grid and the Virtual Observatory Interoperable Dr. - - PowerPoint PPT Presentation

▶
making the grid and the virtual observatory interoperable
SMART_READER_LITE
LIVE PREVIEW

Making the Grid and the Virtual Observatory Interoperable Dr. - - PowerPoint PPT Presentation

Aug 29, 2022 137 likes •540 views

Enabling Grids for E-sciencE Making the Grid and the Virtual Observatory Interoperable Dr. Giuliano Taffoni (INAF Trieste, Italy) www.eu-egee.org INFSO-RI-508833 Data Avalanche Enabling Grids for E-sciencE Astronomy is Facing a Major Data

slide-1
SLIDE 1

INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

Making the Grid and the Virtual Observatory Interoperable

  • Dr. Giuliano Taffoni (INAF Trieste, Italy)
slide-2
SLIDE 2

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Data Avalanche

  • Astronomy is Facing a Major Data Avalanche:

– Multi-Terabyte Sky Surveys and Archives (Soon: Multi-Petabyte), Billions

  • f Detected Sources, Hundreds of Measured Attributes per Source …

2

slide-3
SLIDE 3

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

New Trends in Astronomy

  • Astronomical Data is growing exponentially

–

3

  • P. Quinn
slide-4
SLIDE 4

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

New Trends in Astronomy

  • Astronomical Data is growing exponentially

–

3

  • P. Quinn

~ 100 Gb/night

slide-5
SLIDE 5

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

The Challenging Aspects...

  • Large digital sky surveys are becoming the dominant

source of data in astronomy: currently > 100 TB in major archives, and growing rapidly

  • Typical sky survey today: ~ 10 TB of image data, ~

109 detected sources, ~ 102 measured attributes per source

  • Data sets orders of magnitude larger, more complex,

and more homogeneous than in the past

  • Roughly 1+ TB/Sky/band/epoch

– NB: Human Genome is ~ 1 TB, Library of Congress ~ 20 TB

  • Spanning the full range of wavelengths, radio

through x-ray: a panchromatic, less biased view of the universe –

4

slide-6
SLIDE 6

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

...and a panchromatic view

5

Radio Far-Infrared Visible

slide-7
SLIDE 7

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

The Breakdown in Making Science

  • Understanding of Complex Astrophysical Phenomena

Requires Complex and Information-Rich Data Sets, and the Tools to Explore them...

  • This will lead to a change in the nature of the

Astronomical Discovery Process...

  • which requires a novel research environment for

Astronomy

6

The Virtual Observatory

slide-8
SLIDE 8

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

The Virtual Observatory concept

7

  • A response of the astronomical community to

the scientific and technological challenges posed by massive data sets

  • Federate the existing and forthcoming large

digital sky surveys and archives, and provide the tools for their scientific exploitation

  • A dynamical, interactive, web-based research

environment for the new astronomy with massive data sets

  • Technology-enabled, but science-driven
slide-9
SLIDE 9

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

The Virtual Observatory concept

7

  • A response of the astronomical community to

the scientific and technological challenges posed by massive data sets

  • Federate the existing and forthcoming large

digital sky surveys and archives, and provide the tools for their scientific exploitation

  • A dynamical, interactive, web-based research

environment for the new astronomy with massive data sets

  • Technology-enabled, but science-driven
  • What is a Virtual Observatory?

– Dynamic collection of hardware, data and software working in harmony to solve arbitrarily large and complex astronomical

  • problems. The VObs is the middleware and tools for Astronomers.
slide-10
SLIDE 10

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

So what...?

  • Vobs opened new perspectives in Astronomy:

– Web: all documents of the world inside your computer – VO: all astronomical databases in the world inside your

computer

  • Concrete example:
  • Find all the observations of a given source available in all

astronomical archives in a given wavelength range

  • Tell me which ones are in raw or processed form
  • Allow me to retrieve them
  • If raw, give me access to the tools to reduce them on-the-fly

Very time consuming, if at all possible, at present

8

slide-11
SLIDE 11

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

So what...?

  • Vobs opened new perspectives in Astronomy:

– Web: all documents of the world inside your computer – VO: all astronomical databases in the world inside your

computer

  • Concrete example:
  • Find all the observations of a given source available in all

astronomical archives in a given wavelength range

  • Tell me which ones are in raw or processed form
  • Allow me to retrieve them
  • If raw, give me access to the tools to reduce them on-the-fly

Very time consuming, if at all possible, at present

8

What is missing?

slide-12
SLIDE 12

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Virtual Observatory Grid

  • The Virtual Observatory is a Grid;
slide-13
SLIDE 13

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Virtual Observatory Grid

  • The Virtual Observatory is a Grid;

9

slide-14
SLIDE 14

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Virtual Observatory Grid

  • The Virtual Observatory is a Grid;

9

Objection, objection, objection, objection

Thats not what we meant by a GRID!!!!!

slide-15
SLIDE 15

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Grid Essential

10

  • “You can't be a real country

unless you have a beer and an

  • airline. It helps if you have some

kind of a football team, or some nuclear weapons, but at the very least you need a beer”. » Frank Zappa

  • You can't be a real Grid unless

you have a commodity and a discovery mechanism. It helps if you have some kind of middleware or some supercomputers, but at the very least you need a commodity.

slide-16
SLIDE 16

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Virtual Observatory is a Grid

11

Commodity Discovery Middleware Computational resources Supercomputers

IVO application Registry DAL

slide-17
SLIDE 17

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

What is it missing?

  • I need to make complex data miming calculations on my

data.

  • I need to compare Observational data with the result of

my Code...

  • ...but I need to run it somewhere
  • I need to make some data reduction.
  • Theoretical Virtual Observatory

– Theoretical data produced on the fly – Comparison between theory and observation

12

slide-18
SLIDE 18

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

What is it missing?

  • I need to make complex data miming calculations on my

data.

  • I need to compare Observational data with the result of

my Code...

  • ...but I need to run it somewhere
  • I need to make some data reduction.
  • Theoretical Virtual Observatory

– Theoretical data produced on the fly – Comparison between theory and observation

12

Maybe I can use the Grid!

slide-19
SLIDE 19

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

EuroVO: the VObs in Europe

  • The idea of the Euro-VO is to make it feel as if all the astronomical

data and tools are available on the astronomers desktop, even though they are actually located on systems spread out over the whole of Europe and even the rest of the world.

  • EuroVO TECH

– responsible for completing the design work and feasibility studies on the backbone software components that will make the Euro-VO possible.

  • EuroVO DCA

– Coordinate and assist European Data Centers; – Produce a knowledge GRID (data + services) – Coordinate with national and international GRID projects

13

slide-20
SLIDE 20

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

EuroVO DCA

  • Interest area: massive and distributed

computing, Grid computing;

  • Promote coordination between GRID(s) and

VObs;

  • Point of view of Data Centers;
  • GRID(s) through Data Centers.

14

slide-21
SLIDE 21

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

In practice

  • How can Data Centers benefit of of GRID

computing?

  • How can Astronomers can benefit of Grid

computing?

15

slide-22
SLIDE 22

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Partners

16

Name Affiliation Project

  • G. Taffoni

INAF DCA

  • M. Sponza

INAF DCA

  • P. Osuna

ESAC DCA

  • R. Alvarez Timon

ESAC DCA

  • G. Lemson

MPG DCA

  • J. Zuther

MPG DCA

  • H. Enke

IAP AstroGrid-D

  • E. Solano

INTA DCA

  • J. Santander Vela

IAA-CISC Spanish Grid A.Schaaff CNRS DCA

  • K. Noddle

LU DCA

  • G. Rixon

IAC AstroGrid

  • E. Valentyn

NOVA DCA

  • A. Belikov

NOVA DCA

EGEE VO: dca.euro-vo.org

slide-23
SLIDE 23

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Coordination

  • Keywords

– Interoperability – Usability – Re-usability

  • Useful Informations from GRIDs:

– tools and services already developed! – problems already faced – dead-end already encountered

  • Why not to use them instead of re-inventing?

– the SSO example (see later)

17

slide-24
SLIDE 24

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Making VObs and Grid interoperate

18

  • Auth & Auth
  • Data Management
  • Job Management
  • Information Systems
  • Single-sign-on
  • VOSpace
  • Workflow
  • Registries
slide-25
SLIDE 25

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

A first problem

19

slide-26
SLIDE 26

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

20

slide-27
SLIDE 27

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

20

  • Authentication and authorization

mechanisms:

– VOMS – Shibboleth – etc...

slide-28
SLIDE 28

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

20

  • Authentication and authorization

mechanisms:

– VOMS – Shibboleth – etc...

  • Single-sign-on
slide-29
SLIDE 29

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

21

  • Authentication and authorization

mechanisms:

– VOMS

slide-30
SLIDE 30

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

21

  • Authentication and authorization

mechanisms:

– VOMS – Shibboleth

slide-31
SLIDE 31

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

21

  • Authentication and authorization

mechanisms:

– VOMS – Shibboleth – etc...

slide-32
SLIDE 32

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

21

  • Authentication and authorization

mechanisms:

– VOMS – Shibboleth – etc...

  • Single-sign-on-double-functionality
slide-33
SLIDE 33

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Auth & Auth

21

  • Authentication and authorization

mechanisms:

– VOMS – Shibboleth – etc...

  • Single-sign-on-double-functionality
  • User certificate, host certificate,

application certificate...

slide-34
SLIDE 34

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

IVObs Delegation Service

22

Client

POST the creation of a proxy certificate with a predefined identity get CSR, signs it, add VOMS extensions returns it to delegation service. Can also delete, renew and query previous proxies

Server

IVO delegation

create key pair store key pairs, store the VOMS extensions, renew VOMS extensions, create proxy certificate at usage

RESTful service

HTTPS transport POST/DELETE GET

VOMS Server

IVO service Grid serivce

get credential get credential exchange credentials

slide-35
SLIDE 35

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Data Management

  • VOspace is the Data access protocol for VObs
  • Web service interface
  • The problem:

– share data from Grids and VObs. – we have EU Grid infrastructure that provides storage space for the Astronomical Data

23

slide-36
SLIDE 36

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

VObs and the Grid: WMS

  • Work in progress to interface the prototype of workflows

systems with gLite WMS.

  • NOTICE:

– different approach: it is not the user that submits its job but the Data centre that provides a service (...an application)

24

slide-37
SLIDE 37

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Conclusions

  • To build the bridge between the Grid and the VObs it is

necessary to make interoperable the suite of standards and web services of the VObs with tools and services of the Grid

  • The work in progress impacts some key aspects like:

– authentication and authorization mechanisms to gain access to VObs resources (data) and Grid resources through a single authentication transaction (single sign-

  • n);

– access to both VObs resources and Grid resources simultaneously, in a transparent way to the final user and in both directions (from the VObs to the Grid and from the Grid to the VObs)

  • two approaches foresee the ability to provide “wrapped”

science applications, either legacy code or new, as services in an application server (from the VObs to the Grid) or the ability to federate VObs components (astronomical databases) as embedded resources of the Grid

25

slide-38
SLIDE 38

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Conclusion

  • Interoperation is a two way procedure

– we are taking care that VObs adopts Grid (de jure/de facto) standards – The Grid should account for VObs requirements.

  • We should meet and discuss:

26

slide-39
SLIDE 39

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Conclusion

  • Interoperation is a two way procedure

– we are taking care that VObs adopts Grid (de jure/de facto) standards – The Grid should account for VObs requirements.

  • We should meet and discuss:

26

EuroVO and Grid workshop 9-11 April 2008 Garching (Munich) http://wwwas.si.inaf.it/eurovow2008

slide-40
SLIDE 40

ISSGC05, 18 July 2005

Enabling Grids for E-sciencE

INFSO-RI-508833

Acknowledgements

  • Prof P. Padovani
  • Prof. P. Queen
  • Prof. F. Genova
  • and the IVOA (www.ivoa.net)

27