Migrating from Grid to Cloud: Migrating from Grid to Cloud: - - PowerPoint PPT Presentation

▶
migrating from grid to cloud migrating from grid to cloud
SMART_READER_LITE
LIVE PREVIEW

Migrating from Grid to Cloud: Migrating from Grid to Cloud: - - PowerPoint PPT Presentation

Jun 19, 2023 354 likes •544 views

Migrating from Grid to Cloud: Migrating from Grid to Cloud: Migrating from Grid to Cloud: Migrating from Grid to Cloud: Case Study from GEO Grid Case Study from GEO Grid Kyoung-Sook Kim Data Science Research Group Information Technology

slide-1
SLIDE 1

Migrating from Grid to Cloud: Migrating from Grid to Cloud: Migrating from Grid to Cloud: Migrating from Grid to Cloud: Case Study from GEO Grid Case Study from GEO Grid

Kyoung-Sook Kim

Data Science Research Group Information Technology Research Institute

slide-2
SLIDE 2

Interoperability and Integration Interoperability and Integration

Satellite image Geology

Geo* Contents Applications

Individual interfaces & protocols

for information exchange

Satellite image

for information exchange It is not easy to

  • Access and (re)use
  • Manage and control

GIS data

Environment & Energy

  • Manage and control
  • Integrate

geospatial contents and services Cooperate with different

Sensor data

http://www.geofabrik.de/data/shapefiles.html

Disaster Response

  • Cooperate with different
  • rganizations

High cost of application

Sensor data

High cost of application development and management

Geospatial World F

  • rum 2014 @

Geneve, S witzerland 2014/5/15 2

slide-3
SLIDE 3

GEO Grid: Overview GEO Grid: Overview

*Global Earth Observation

e-Science infrastructure

Science infrastructure for global earth observation systems with heterogeneous geospatial datasets

Geology

Geo* Contents Applications

Standard Standard -based interoperability & Integration

Satellite image Geology

  • Ease-of-use interfaces (search, access, process data)
  • Low integration efforts

Standard Standard

  • based interoperability & Integration

WMS

WFS Web Web

GIS data Environment & Energy

WPS

WMS

WFS SOS WCS

CS-W

Web

Web

Services Services

Sensor data

http://www.geofabrik.de/data/shapefiles.html

Disaster Response

  • Security management

Sensor data Disaster Response

  • Security management
  • Scalability of resources

(computing and storage)

Geospatial World F

  • rum 2014 @

Geneve, S witzerland 2014/5/15 3

slide-4
SLIDE 4

GEO Grid: GEO Grid: Overview Overview

e-Science infrastructure

Science infrastructure for global earth observation systems with heterogeneous geospatial datasets

Geology

Geo* Contents Applications

Standard Standard -based interoperability & Integration

Satellite image Geology

  • Ease-of-use interfaces (search, access, process data)
  • Low integration efforts

Standard Standard

  • based interoperability & Integration

WMS

WFS Web Web

GIS data Environment & Energy

WPS

WMS

WFS SOS WCS

CS-W

Web

Web

Services Services

Grid Computing Technologies &

Sensor data

http://www.geofabrik.de/data/shapefiles.html

Disaster Response

Grid Computing Technologies & Standards

Sensor data Disaster Response

for sharing of geographically distributed resources

Geospatial World F

  • rum 2014 @

Geneve, Switzerland

2014/5/15 4

for sharing of geographically distributed resources and controlling resource sharing rules

slide-5
SLIDE 5

GEO Grid: GEO Grid: Overview Overview

e-Science infrastructure

Science infrastructure for global earth observation systems with heterogeneous geospatial datasets

Geology

Geo* Contents Applications

Standard Standard -based interoperability & Integration

Satellite image Geology

  • Ease-of-use interfaces (search, access, process data)
  • Low integration efforts

Standard Standard

  • based interoperability & Integration

WMS

WFS Web Web

GIS data Environment & Energy

WPS

WMS

WFS SOS WCS

CS-W

Web

Web

Services Services

Grid Computing Technologies &

Sensor data

http://www.geofabrik.de/data/shapefiles.html

Disaster Response

Grid Computing Technologies &

Sensor data Disaster Response

Distributed Distributed and Parallel Processing Metadata Management (LOD/RDF) Security and User Management Storage Grids Heterogeneous Heterogeneous and Distributed Database Federation

Geospatial World F

  • rum 2014 @

Geneve, Switzerland

2014/5/15 5

Processing

Processing

(LOD/RDF)

Grid and Cloud Infrastructure Grid and Cloud Infrastructure

Management Federation

Federation

slide-6
SLIDE 6

GEO GEO Grid: Grid: System Architecture System Architecture

Outcomes

Outcomes Outcomes Outcomes

Decision Making Support Decision Making Support

Sci-Tech

R&D

Sci-Tech

R&D

Business Business Share Disaster

Information Share Disaster Information Support Support Information Information

Application Application Services Services

ASTER

DEM

QuiQuake QuiQuake

Flood Simulation Flux Monitoring Hotspot Detection Others Science DCP

Services Services OGC standard web services OGC standard web services

Heterogeneous

WMS

WCS

CS-W

WPS

SOS WFS

Grid and Cloud Grid and Cloud Infrastructure Infrastructure GeoSpatial GeoSpatial Resources Resources

Storage Grids Security and User

Management Heterogeneous and Distributed Database Federation Computing Grid and Cloud

Our HPC cluster

GeoSpatial GeoSpatial Resources Resources

Our HPC cluster

Data Archive Services

Satellite Data Geological Data

40m 20m 10m 30m
  • 0.01
  • 0.05 -0.1
  • 0.2
  • 0.5
40m 20m 10m 30m
  • 0.01
  • 0.05 -0.1
  • 0.2
  • 0.5
40m 20m 10m 30m
  • 0.01
  • 0.05 -0.1
  • 0.2
  • 0.5

sensors

ASTER satellite archive ~= 2million scenes ~= 1PB MODIS, JERS-1 PALSAR,LANDSAT Geological Map of Japan CfreeVG10,SRTM, Other geospatial info

2014/5/15 6 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

sensors

PALSAR,LANDSAT geospatial info

resources

slide-7
SLIDE 7

An An Example Example of GEO Grid

  • f GEO Grid Public Services

Public Services

  • QuiQuake (https://gbank.gsj.jp/QuiQuake/index.en.html)
  • Quick estimation system for
  • Quick estimation system for

earthquake maps triggered by

  • bservation records
  • Provides wide-ranging and detailed

(250m-grid) strong ground motion maps for quick disaster response.

  • Produces the results soon after the
  • ccurrence of an earthquake.
  • Archiving all data of past earthquake
  • ccurrences.

2014/5/15 7 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

slide-8
SLIDE 8

QuiQuake QuiQuake Realization Realization in GEO Grid in GEO Grid

Share Disaster

Information

Share Disaster

Information Sensor Data Decision Making Support Decision Making Support Information Information High Performance Provide the results as OGC services and formats Geospatial Data Data Integration Performance Computing

Outcome Outcome Application Application Services Services Grid and Cloud Grid and Cloud Infrastructure Infrastructure GeoSpatial GeoSpatial Resources Resources

2014/5/15 8 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

slide-9
SLIDE 9

GEO Grid Security GEO Grid Security

  • GSI (Grid Security Infrastructure)

– S ecure communication (authenticated and confidential) between Grid elements – Security across organizational boundaries – Security across organizational boundaries – ”Single sign-on" for users of the Grid in multiple resources and/or sites

  • VO (Virtual Organization)
  • VO (Virtual Organization)

Management

A dynamic group of individuals, groups, or

  • rganizations who define
  • rganizations who define

the conditions and rules for sharing resources

2014/5/15 9 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

slide-10
SLIDE 10

Migration from Grid to Cloud Migration from Grid to Cloud

2011 T

  • hoku E

arthquake

GE O Grid at AIS T (~200km away) was damaged by the earthquake.

Cannot operate GEO Grid for 2 months by recovery works & subsequent power Cannot operate GEO Grid for 2 months by recovery works & subsequent power restriction request.

BUT s atellite imageswere very important to understand situations and provide useful information for helping the disaster response activities.

2014/5/15 10 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

provide useful information for helping the disaster response activities.

slide-11
SLIDE 11

ASTER Comparison Before and After Tsunami ASTER Comparison Before and After Tsunami

  • AS

TE R :

– Japanese sensor (optical, 15m) on NASA Terra

satellite

satellite

– Can create 3D model by making stereo-matching – AIST can produce true-color images.

11

Mouth of Kitakami-river

slide-12
SLIDE 12

GEO Grid Disaster GEO Grid Disaster Responses Responses

  • Rapid Evacuation :

– Moved our minimum environment to oversea volunteer servers

  • Using VM/Cloud technology

–

Changed our daily network (data transfer) workflow to use these servers

–

Changed our daily network (data transfer) workflow to use these servers

  • To get/receive latest images from the satellite(ASTER)

ALOS/PALSAR TDRS

Until March 11

  • Data providing
  • Portal

70 GB/day (ASTER) NASA AIST

ERSDAC JAXA

360 GB/day (PALSAR)

  • Archive (tape, B-ray)
  • Archive (on-Disk)
  • Processing
  • WMS
  • AS

TER data: NAS A ERSDAC AIS T

2014/5/15 12 Geospatial World F

  • rum 2014 @

Geneve, Switzerland

  • AS

TER data: NAS A ERS DAC AIS T

  • P

ALS AR data: J AXA ERS DAC

AIST

(AIST: processing, WMS, portal site, and data archive)

slide-13
SLIDE 13

What We DID What We DID

Terra/ASTER ALOS/PALSAR TDRS

Data Flow and Services from March 11 to April 20

  • Portal

NASA (AIST) ERSDAC JAXA

Orkney Google

  • Processing
  • WMS
  • WMS
  • ASTER data: NASA

ERSDAC (AIST)

(Orkney: processing and WMS

, Google: portal site)

2014/5/15 13 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

  • AS

TE R data: NAS A E R S DAC (AIS T)

  • P

ALS AR data: J AXA E R S DAC (AIS T)

(Orkney: processing and WMS

, Google: portal site)

slide-14
SLIDE 14

What We DID What We DID

TDRS Terra/ASTER ALOS/PALSAR

Data Flow and Services From April 21

  • Portal
  • Processing

NASA (AIST) ERSDAC JAXA Google UCSD OCC UCSD

  • Processing

NCHC WMS Server

  • Processing
  • WMS
  • AS

TER data: NAS A ERS DAC (AIS T)

NCHC@TW, SDSC@USA, and OCC@USA: processing

WMS Server QuiQuake

High performance (long-running) computing using

  • versea servers for creating scientific information

2014/5/15 14 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

  • AS

TE R data: NAS A E R S DAC (AIS T)

  • P

ALS AR data: J AXA E R S DAC (AIS T)

NCHC@TW, SDSC@USA, and OCC@USA: processing NCHC@TW: WMS

Google: portal site

slide-15
SLIDE 15

Time Time-line of our Activity line of our Activity

Date Events and Activities Descriptions

Mar 11 Occurrence of Earthq.

GEO Grid services stop

Mar 14

GEO Grid disaster TF Establishment Support by private companies, research inst., universities worldwide worldwide

Mar 15 Start of data transport and migration of

processing codes ASTER and PALSAR from ERSDAC Computer and WMS in Orkney Portal site in Google site

Mar 19 HP open and WMS start

ASTER and PALSAR, damage interpretation, QuiQuake(manual)

Mar 19 HP open and WMS start

ASTER and PALSAR, damage interpretation, QuiQuake(manual)

Mar 21 Start of data transport Formosat-2 by NSPO (via JAXA) Mar 25 Geological maps in WMS Seamless geological map, active fault, and geochemical map Mar 31 Value-added-ASTER Natural color, orthorectification, automation

Rough Numbers from 3.11 Rough Numbers from 3.11 1 week to evacuate and to start providing analyzed data to public 1 month to recover the service using oversea sites 2 months to recover/resume our environment Rough Numbers from 3.11 Rough Numbers from 3.11 1 week to evacuate and to start providing analyzed data to public 1 month to recover the service using oversea sites 2 months to recover/resume our environment

Mar 31 Value-added-ASTER Natural color, orthorectification, automation Mar 31

CS-W deployment

WMS list Apr 1 QuiQuake open QuiQuake (automatic) Apr 20 Value-added-PALSAR Crustal deformation by InSAR

2 months to recover/resume our environment Yet another 1to2 month(s) to get back the service from oversea sites 2 months to recover/resume our environment Yet another 1to2 month(s) to get back the service from oversea sites

Apr 20 Value-added-PALSAR Crustal deformation by InSAR Apr 21 Services move abroad NCHC (NARL-Taiwan), SDSC (UCSD), OCC (Univ.Chicago) Apr 28 GEO Grid cluster resume May 24 ASTER/AIST service resume

Data publication to res earch us ers

May 27 QuiQuake/AIST resume

2014/5/15 15 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

May 27 QuiQuake/AIST resume June 30

PALSAR/AIST service resume

slide-16
SLIDE 16

Towards Cloud Computing Towards Cloud Computing

  • Fortunately, we had VM images of satellite data processing

– Ready to provisioning in a cloud environment – Connected to HPC international grid testbed (PRAGMA) – Connected to HPC international grid testbed (PRAGMA)

  • Sharing our VM images in PRAGMA VM repository
  • We can boot our application VMs at any site by any PRAGMA colleagues
  • However,
  • However,

– Manual deployment at each site one by one – Lots of manual configuration due to heterogeneity and for security – Lots of manual configuration due to heterogeneity and for security – Tightly coupled data and computing servers – Not real-time – Heavy data formats and integration (OGC standards) – Heavy data formats and integration (OGC standards) – …

2014/5/15 16 Geospatial World F

  • rum 2014 @

Geneve, S witzerland

slide-17
SLIDE 17

GEO Grid for Cloud Computing GEO Grid for Cloud Computing

  • Virtualization as a Practical Mechanism

– Supporting Multiple VM Infrastructures (Xen, KVM,

OpenNebula, CloudStack, Rocks, EC2)

  • Data Intensity
  • On-demand self-service
  • Broad network access
  • Resource pooling

Cloud [NIST definition for using computing resources]

  • Data Intensity

–

Data services to support data discovery, access, processing, and delivery on demand with minimal transmission (ex., radiation monitoring service)

  • Resource pooling
  • Rapid elasticity
  • Measured service

Cloud Characteristics transmission (ex., radiation monitoring service)

  • Complex Workflows

–

Machine learning platform based on Hadoop and MapReduce

–

Lavatube (visual workflow engine)

  • Software as a Service
  • Platform as a Service
  • Infrastructure as a

Service

Service Models

–

Lavatube (visual workflow engine)

–

Real-time analytics

  • High-performance Database for Linked Data

–

Distributed and parallel LOD processing

  • Public
  • Private
  • Hybrid

Deployment Models

–

Distributed and parallel LOD processing

  • Cloud Security

–

OpenID/OAuth for AuthN/AuthZ

  • GeoSocial Media
  • Hybrid
  • Community

Models

The NIS T Definition of C loud C

  • mputing, http://csrc.nist.gov/publications/nistpubs/800-

145/SP800-145.pdf

  • GeoSocial Media

– New data integration method

2014/5/15 17 Geospatial World F

  • rum 2014 @

Geneve, Switzerland

slide-18
SLIDE 18

Global E arth Observation Grid Global E arth Observation Grid Global E arth Observation Grid Global E arth Observation Grid

http://www.geogrid.org/ http://www.geogrid.org/