Ninf-G - PowerPoint PPT Presentation

Ninf-G チュ ート リ アル （ 初～中級向け） 谷 村 村 勇 輔 輔 谷 勇 (yusuke ( yusuke. .tanimura tanimura@ @aist aist.go. .go.jp jp) ) Grid Technology Research Center, AIST Grid Technology Research Center, AIST National Institute of Advanced Industrial Science and Technology

Outline What is GridRPC GridRPC? ? What is Overview v.s. MPI Typical scenarios Overview of Ninf Ninf- -G and G and GridRPC GridRPC API API Overview of Ninf-G: Overview and architecture GridRPC API Ninf-G API How to develop Grid applications using Ninf How to develop Grid applications using Ninf- -G G Build remote libraries Develop a client program Run Recent activities/achievements in Ninf Ninf project project Recent activities/achievements in

What is GridRPC? Programming model on Grid based on Grid Remote Procedure Call (GridRPC)

Layered Programming Model/Method Easy but Portal / PSE inflexible GridPort, HotPage, GPDK, Grid PSE Builder, etc… High-level Grid Middleware MPI (MPICH-G2, PACX-MPI, …) MPI MPI GridRPC (Ninf-G, NetSolve, …) Low-level Grid Middleware Globus Toolkit Primitives Difficult Socket, system calls, … but flexible

Some Significant Grid Programming Models/Systems Data Parallel Data Parallel MPI - MPICH-G2, GridMPI, PACX-MPI, … Task Parallel Task Parallel GridRPC – Ninf, Netsolve, DIET, OmniRPC, … Distributed Objects Distributed Objects CORBA, Java/RMI, … Data Intensive Processing Data Intensive Processing DataCutter, Gfarm, … Peer- Peer -To To- -Peer Peer Various Research and Commercial Systems UD, Entropia, JXTA, P3, … Others … Others …

GridRPC Utilization of remote Supercomputers ② Return result Internet user ① Call remote libs Call remote (special) libraries Use as backend of Large-scale distributed computing using portals / ASPs multiple computing resources on Grids Suitable for implementing task-parallel applications (compute independent tasks on distributed resources)

GridRPC Model Client Component Client Component Caller of GridRPC ． Manages remote executables via function handles Remote Executables Remote Executables Callee of GridRPC. Dynamically generated on remote servers. Information Manager Information Manager Manages and provides interface information for remote executables. ・ ・ ・ ・ ・ ・ ・ ・ Server Func. Handle ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ Server Client ・ ・ ・ ・ ・ ・ ・ ・ Compuer ・ ・ ・ ・ ・ ・ ・ ・ Client Component Server ・ ・ ・ ・ ・ ・ ・ ・ 遠隔実行プログラム Remote Executables Info. Manager

GridRPC: RPC “ tailored ” for the Grid Medium to Coarse- -grained calls grained calls Medium to Coarse Call Duration < 1 sec to > week Task Task- -Parallel Programming on the Grid Parallel Programming on the Grid Asynchronous calls, 1000s of scalable parallel calls Large Matrix Data & File Transfer Large Matrix Data & File Transfer Call-by-reference, shared-memory matrix arguments Grid- -level Security level Security (e.g., (e.g., Ninf Ninf- -G with GSI) G with GSI) Grid Simple Client- -side Programming & Management side Programming & Management Simple Client No client-side stub programming or IDL management Other features Other features … …

GridRPC v.s. MPI GridRPC GridRPC MPI MPI parallelism parallelism task parallel data parallel task parallel data parallel model model client/server client/server SPMD SPMD API API GridRPC API API MPI GridRPC MPI co- -allocation allocation co dispensable indispensable dispensable indispensable fault tolerance fault tolerance good good poor (fatal) poor (fatal) private IP nodes private IP nodes available unavailable available unavailable resources resources can be dynamic can be dynamic static ＊ static ＊ others others easy to gridify gridify well known easy to well known existing apps. existing apps. seamlessly move to seamlessly move to Grid Grid * May be dynamic using process spawning

Typical scenario 1: desktop supercomputing Utilize remote supercomputers from your Utilize remote supercomputers from your desktop computer desktop computer Reduce cost for maintenance of libraries Reduce cost for maintenance of libraries server ASP- -like approach like approach ASP client arguments results Numerical Libraries Applications

Typical scenario 2: parameter surevey Compute independent tasks Compute independent tasks on distributed resources on distributed resources eg. combinatorial optimization problem solvers Fault nodes can be Fault nodes can be discarded/retried discarded/retried Dynamic allocation / release Dynamic allocation / release of resources is possible of resources is possible Client Servers

Typical scenario 3: GridRPC + MPI Coarse- Coarse -grained independent grained independent parallel (MPI) programs are parallel (MPI) programs are executed on distributed clusters executed on distributed clusters Combine coarse- -grained parallelism grained parallelism Combine coarse (by GridRPC GridRPC) and fine ) and fine- -grained grained (by parallelism (by MPI) parallelism (by MPI) Dynamic migration of MPI jobs is Dynamic migration of MPI jobs is possible possible Practical approach for large- -scale scale Practical approach for large computation computation Client Servers

Sample Architecture and Protocol of GridRPC System – Ninf - Server side Client side Server side setup Call remote library Call remote library � Build Remote Library Retrieve interface Executable information � Register it to the Ninf Server Invoke Remote Numerical Library Executable IDL file Library It Calls back to the client IDL Compiler Client Connect back Interface Generate Invoke Interface Reply Executable Request Remote Library fork Ninf Server Executable Register

GridRPC: based on Client/Server model Server- -side setup side setup Server Remote libraries must be installed in advance Write IDL files to describe interface to the library Build remote libraries Syntax of IDL depends on GridRPC systems e.g. Ninf-G and NetSolve have different IDL Client- -side setup side setup Client Write a client program using GridRPC API Write a client configuration file Run the program

Ninf-G Overview and Architecture Overview and Architecture National Institute of Advanced Industrial Science and Technology

What is Ninf-G? A software package which supports programming and A software package which supports programming and execution of Grid applications using GridRPC GridRPC. . execution of Grid applications using The latest version is 2.4.0. The latest version is 2.4.0. The version 4.0 beta based on WS-components of Globus is also available. Ninf- Ninf -G is developed using G is developed using Globus Globus C and Java APIs. C and Java APIs. Uses GSI, GRAM, MDS, (GASS), and Globus-IO Ninf- Ninf -G includes G includes C/C++, Java APIs, libraries for software development IDL compiler for stub generation Shell scripts to compile client program build and publish remote libraries sample programs and manual documents

Globus Toolkit Defacto standard as low-level Grid middleware

Requirements for Grid Security Security authentication, authorization, message protection, etc. Information services Information services Provides various information available resources (hw/sw), status, etc. resource management resource management process spawning on remote computers scheduling scheduling data management, data transfer data management, data transfer usability usability Single Sign On, etc. others others accounting, etc …

What is the Globus Toolkit? A Toolkit which makes it easier to develop A Toolkit which makes it easier to develop computational Grids computational Grids Developed by the Globus Alliance and many others Developed by the Globus Alliance and many others all over the world (mainly ANL and USC/ISI) all over the world (mainly ANL and USC/ISI) Defacto standard as a low level Grid middleware Defacto standard as a low level Grid middleware Most Grid testbeds are using the Globus Toolkit Three versions are exist Three versions are exist 2.4.3 (GT2 / Pre-WS) 3.2.1 (GT3 / OGSI) 4.0.1 (GT4 / WSRF) GT2 component is included in GT3/GT4 GT2 component is included in GT3/GT4 Pre-WS components

GT2 components GSI: Single Sign On + delegation GSI: Single Sign On + delegation MDS: Information Retrieval MDS: Information Retrieval Hierarchical Information Tree (GRIS+GIIS) GRAM: Remote process invocation GRAM: Remote process invocation Three components: Gatekeeper Job Manager Queuing System (pbs, sge, etc.) Data Management: Data Management: GridFTP Replica management GASS (excluded in GT4) Globus XIO Globus XIO GT2 provides C/Java APIs and Unix commands GT2 provides C/Java APIs and Unix commands for these components for these components

GRAM: Grid Resource Allocation Manager g a t e k e e p e r Job request J o b - m a n a g e r J o b - m a n a g e r Job control, monitor Error notification Q u e u i n g s y s t e m process process process process