Migrating from Grid to Cloud: Case Study from GEO Grid National - - PowerPoint PPT Presentation

▶
migrating from grid to cloud case study from geo grid
SMART_READER_LITE
LIVE PREVIEW

Migrating from Grid to Cloud: Case Study from GEO Grid National - - PowerPoint PPT Presentation

Nov 29, 2023 45 likes •297 views

Migrating from Grid to Cloud: Case Study from GEO Grid National Institute of Advanced Industrial Science and Technology Yoshio Tanaka What is the GEO Grid ? The GEO (Global Earth Observation) Grid is aiming at providing a Cyber

slide-1
SLIDE 1

Migrating from Grid to Cloud: Case Study from GEO Grid

National Institute of Advanced Industrial Science and Technology

Yoshio Tanaka

slide-2
SLIDE 2

What is the GEO Grid ?

  • The GEO (Global Earth Observation) Grid is aiming at providing a Cyber

Infrastructure for worldwide Earth Sciences communities to accelerate GEO sciences based on the concept that relevant data and computation are virtually integrated with a certain access control and ease-of-use interface those are enabled by a set of Grid and Web service technologies.

Geo* Contents Applications Satellite Data Map

Geology

GIS data Field data

Environment

Resources Grid Technologies

Disaster mitigation

AIST: OGF Gold sponsor (a founding member) AIST: OGC Associate member (since 2007)

slide-3
SLIDE 3

Example: Flood simulation

Digital Elevation Map Created by ASTER Satellite Images Rain Sensors Neural-Map based Data Mining / Landslide Simulation on Large Cluster Computers Visualize Hi-Resolution Hazard Map GIS Data

ASTER GEO MET

slide-4
SLIDE 4

W hy Grid? – federation of distributed resources -

security

High performance CSW

Federation of distributed DBs computing

SSO

slide-5
SLIDE 5

GEO Grid Security: GSI + VOMS

slide-6
SLIDE 6

TDRS Terra/ASTER ERSDIS/NASA APAN/TransPAC GEO Grid Cluster

L0 L0 L0 L0 L0 L0 L0 L0 L0 L0 L0 L0

Data GIS server WFS WCS Maps map server WMS Meta data catalogue/ metadata server CSW OGSA DAI GRAM GridFTP gateway server Storage (DEM) portal server GSI + VOMS GSI + VOMS GSI + VOMS Account DB account (GAMA) server VO DB VO (VOMS) server

user login credential GET query exec

slide-7
SLIDE 7

Demo Environments in 2007 - SIMS (ASTER+MODIS+Formsat2)

Database Server (Sybase)

FORMOSAT-2

Application Server

OGSA- DAI Globus

SQL w/ JDBC

NSPO@TW

Database Server (PostgreSQL) ASTER MODIS OGSA- DAI

SQL w/ JDBC

OGSA- DAI Globus

AIST@JP

AIST OGSA-DAI Client Integration Framework with OGSA-DAI Java Program

SQL SQL SQL SQL SQL SIMS portlet

  • query data
  • create web page which

shows thumbnail images VOMS VOMS

slide-8
SLIDE 8

SIMS – Search Results

MODIS FORMOSAT-2 ASTER

slide-9
SLIDE 9

GEO Grid Service Exam ples

  • Satellite data archive and processing
  • ASTER, PALSAR, MODIS, etc.
  • Satellite data application
  • Application of Satellite-Field data Integrator (SFI) for aerosol monitoring

Description http://fon.geogrid.org/aerosol/

  • SDCP (Science Degree Confluence Project) –Community validation tool for

global land-cover & digital elevation models http://eco.geogrid.org/sdcp/

  • Hazard information
  • QuiQuake (Quick Estimation System for Earthquake Maps Triggered by

Observation Records) http://qq.ghz.geogrid.org/QuakeMap/index.en.html

  • Volcanic Gravity Flow Simulations on Volcanic Area

http://volcano.geogrid.org/applications/EnergyCone/

  • Geoscience data
  • Geological maps, Active fault data, etc.
slide-10
SLIDE 10

Migration from Grid to Cloud

slide-11
SLIDE 11

Motivation for m igrating to Cloud

  • Deployment of applications is not easy

– Procedures for including new resources (deployments of applications) are troublesome.

  • Need easy-to-use.
  • Write once, run everywhere!
  • Do we need Grid protocols?

– Do we need Grid Security?

  • Delegation is necessary for third-party file transfer.
  • But key management is burden for end users.
  • Installation/ configuration of VOMS is not easy.

– Do we need Grid protocol (e.g. GRAM)?

  • GEO Grid applications use not Grid middleware/ protocol but the other

standards (e.g. OGC).

  • Need to adapt the direction for wider use

– GEO Grid system is stably in operation, but not extendable (elastic).

  • Data server and computing server are tightly coupled.
  • It’s hard to use resources outside organization.

– Is GEO Grid Design appropriate for use by business partners? – Japanese government has a plan of promoting use of satellite data for wide use.

slide-12
SLIDE 12

Goals of and approaches by PRAGMA

  • Enable Specialized Applications to run easily
  • n distributed resources

– Build once, run everywhere!!

  • Investigate Virtualization as a practical

mechanism

– Supporting Multiple VM Infrastructures (Xen, KVM, OpenNebula, Rocks, WebOS, EC2)

  • Share VM images in PRAGMA VM repository

so that we can boot our application VMs at any site by any PRAGMA colleagues.

– Discussed in PRAGMA 20 workshop @ HK, March 3rd and 4th, 2011, 1 week before the big earthquake in Japan…

slide-13
SLIDE 13

2011 Tohoku Earthquake changed our R& D environments

slide-14
SLIDE 14

Satellite Data Flow and Services Prior to March 11

14

NASA AIST ERSDAC JAXA Terra/ASTER

  • Archive (tape, B-ray)
  • Archive (on-Disk)
  • Processing
  • WMS
  • Data providing
  • Portal

ALOS/PALSAR 70 GB/day (ASTER) 360 GB/day (PALSAR)

  • ASTER data: NASA→ERSDAC→AIST
  • PALSAR data: JAXA→ERSDAC→AIST

(processing, WMS, portal site, and data providing by AIST)

TDRS

slide-15
SLIDE 15

Data Flow and Services from March 11 till April 20

15

NASA (AIST) ERSDAC JAXA Terra/ASTER

  • Processing
  • WMS

ALOS/PALSAR

  • ASTER data: NASA→ERSDAC→(AIST)→
  • PALSAR data: JAXA→ERSDAC→(AIST)→

(processing and WMS by Orkney, portal site by Google)

Orkney Google TDRS

  • Portal
slide-16
SLIDE 16

Data Flow and Services from April 21

NASA (AIST) ERSDAC JAXA TDRS Terra/ASTER

  • Processing
  • WMS

ALOS/PALSAR

  • ASTER data: NASA→ERSDAC→(AIST)→
  • PALSAR data: JAXA→ERSDAC→(AIST)→ (processing by NCHC, SDSC,

and OCCI, WMS by NCHC, portal site by Google)

Google

  • Portal

UCSD

  • Processing
  • Processing

OCCI NCHC

slide-17
SLIDE 17

I nsights

  • Fortunately, we already had VM images for

satellite data processing.

– We have prepared for using cloud.

  • Need to make it routine use!
  • PRAGMA members had disasters/ accidents.

– Japan earthquake – Thailand flooding – California power outage

  • PRAGMA members has common

interests/ needs to build a sustainable infrastructure which could be used to support each other in case of emergency.

– We accelerated the development/ deployment of PRAGMA Cloud.

slide-18
SLIDE 18

PRAGMA Grid/Clouds

26 institutions in 17 countries/regions, 23 compute sites, UZH Switzerland NECTEC KU Thailand UoHyd India MIMOS USM Malaysia HKU HongKong ASGC NCHC Taiwan HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam AIST OsakaU UTsukuba Japan MU Australia KISTI KMU Korea JLU China SDSC USA UChile Chile CeNAT-ITCR Costa Rica BESTGrid New Zealand CNIC China LZU China UZH Switzerland LZU China ASTI Philippines IndianaU USA UValle Colombia

Slide by courtesy of PRAGMA

slide-19
SLIDE 19

Deploy Three Different Softw are Stacks on the PRAGMA Cloud

  • QuiQuake

– Simulator of ground motion map when earthquake occurs – Invoked when big earthquake occurs

  • HotSpot

– Find high temperature area from Satellite – Run daily basis (when ASTER data arrives from NASA)

  • WMS server

– Provides satellite images via WMS protocol – Run daily basis, but the number of requests is not stable.

All these applications run as Condor workers

slide-20
SLIDE 20

AIST HotSpot + Condor gFS gFS gFS gFS gFS SDSC (USA) Rocks Xen NCHC (Taiwan) Ezilla/OpenNebula KVM LZU (China) Rocks KVM AIST (Japan) OpenNebula KVM IU (USA) Rocks Xen Osaka (Japan) Rocks Xen gFC gFC gFC gFC gFC gFC gFS gFS gFS gFS gFS

GFARM Grid File System (Japan)

AIST QuickQuake + Condor AIST Geogrid + Bloss AIST Web Map Service + Condor UCSD Autodock + Condor NCHC Fmotif

= VM deploy Script

VM Image copied from gFarm VM Image copied from gFarm VM Image copied from gFarm VM Image copied from gFarm VM Image copied from gFarm

Condor Master

VM Image copied from gFarm

S S S S S S S

gFC gFS

= Grid Farm Client = Grid Farm Server

slave slave slave slave slave slave

Put all together

Store VM images in Gfarm systems Run vm-deploy scripts at PRAGMA Sites Copy VM images on Demand from gFarm Modify/start VM instances at PRAGMA sites Manage jobs with Condor Slide by courtesy of PRAGMA

slide-21
SLIDE 21

Essential Steps

  • 1. AIST/ GEO Grid creates their VM image
  • 2. Image made available in “centralized”

storage (currently Gfarm is used)

  • 3. PRAGMA sites copy GEO Grid images to

local clouds

  • 1. Assign IP addresses
  • 2. What happens if image is in KVM and site is

Xen?

  • 4. Modified images are booted
  • 5. GEO Grid infrastructure now ready to use

Slide by courtesy of P. Papadopoulos, UCSD

slide-22
SLIDE 22

PRAGMA Compute Cloud

UoHyd India MIMOS Malaysia NCHC Taiwan AIST OsakaU Japan SDSC USA CNIC China LZU China LZU China ASTI Philippines IndianaU USA JLU China

Cloud Sites Integrated in GEO Grid Executio Pool

Slide by courtesy of P. Papadopoulos, UCSD

slide-23
SLIDE 23

New Security Model ( in progress)

OpenID Portal Provider Service Provider AuthZ Server

Resource (e.g. satellite data)

browser Resource Owner (e.g. data owner) Yahoo Google, etc. Web Portal

scope OAuth AuthZ Server AuthN

Resource Server End users

Request services AuthZ Resources Request Services

OpenID (accountable) Biglobe Use OpenID/OAuth for AuthN/AuthZ Planning to use OpenID Connect

AuthN

slide-24
SLIDE 24

Sum m ary

  • We learned a lot through Grid

experiments.

  • Migrating from Grid to Cloud

– Virtualization technologies is useful for making distributed infrastructure easy to use. – Better for business use.

  • Still have many research issues.

– Data – Network virtualization – Resource managements – Security – Making it routine-use

slide-25
SLIDE 25

25

Global Earth Observation Grid

http://www.geogrid.org/

Thank you very m uch for your attention !