High Availability iRODS System (HAIRS) Yutaka Kawai, KEK Adil - - PowerPoint PPT Presentation

▶
high availability irods system hairs
SMART_READER_LITE
LIVE PREVIEW

High Availability iRODS System (HAIRS) Yutaka Kawai, KEK Adil - - PowerPoint PPT Presentation

Oct 14, 2022 228 likes •545 views

KEK H IGH E NERGY A CCELERATOR R ESEARCH O RGANIZATION High Availability iRODS System (HAIRS) Yutaka Kawai, KEK Adil Hasan, ULiv March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 1 Outline

slide-1
SLIDE 1

HIGH ENERGY ACCELERATOR RESEARCH ORGANIZATION

KEK

High Availability iRODS System (HAIRS)


Yutaka Kawai, KEK Adil Hasan, ULiv

March 25th, 2010 1 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv)

slide-2
SLIDE 2

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 2

slide-3
SLIDE 3

Introduction

▸ Replication enables high availability (HA) system for catalog service

▹ Replicate by back-end, i.e. iRODS ▹ Replicate by front-end;

▪ i.e. AMGA (ARDA[1] Metadata Grid Application)

▫ Metadata Catalogue of EGEE’s gLite 3.1 Middleware ▫ Back-end : Oracle, PostgreSQL, MySQL, SQLite ▫ http://amga.web.cern.ch/amga/

▸ The current iRODS HA is implemented by replicating ICAT DB with PgPool tool [2]

▹ A problem when iRODS server fails ▹ Solve the problem by using Director

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 3

slide-4
SLIDE 4

The Current iRODS HA

▸ ICAT DB replication by Pgpool

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 4

ICAT ICAT

Pgpool Pgpool iRODS Server PostgreSQL iRODS Client Change the server info in .irodEnv A B

slide-5
SLIDE 5

Problem of the current HA

▸ Even if the iRODS server fails, clients still continue to access the same server without noticing the failure.

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 5

ICAT ICAT

Pgpool Pgpool iRODS Server PostgreSQL iRODS Client

?

Need to change server info in .irodEnv A B

slide-6
SLIDE 6

Solution by using Director

▸ Place a Director between Client and Server

▹ Monitor the iRODS server statuses ▹ Load balance to the iRODS servers

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 6

ICAT ICAT

Pgpool Pgpool iRODS Server PostgreSQL iRODS Client Director Director A B

slide-7
SLIDE 7

How to Implement Director?

▸ UltraMonkey [3]

▹ Linux based director ▹ Low cost but not so high speed ▹ Need some steps to setup

▸ Hardware Director

▹ High cost and high speed ▹ Easy to setup (?) ▹ Cisco, HP, etc.

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 7

slide-8
SLIDE 8

UltraMonkey

▸ UltraMonkey consists of 3 components

▹ Linux Virtual Server (LVS) : Load balancing ▹ ldirectord : Monitoring real servers ▹ Linux-HA (LHA) : Monitoring directors

▸ LVS and ldirectord are used here

▹ LVS : Provide Virtual IP for load balance ▹ ldirectord : Monitoring iRODS service ▹ LHA : Future use for director redundancy

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 8

slide-9
SLIDE 9

Virtual IP for load balance

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 9

iRODS Real Severs iRODS Real Severs 192.168.2.0/24 .102 .101 .240 Linux Director Linux Director 192.168.1.0/24

VIP 192.168.1.200 VIP 192.168.1.200

.240 iRODS Client iRODS Client .100 iRODS Client can specify only this VIP in .irodsEnv Gateway of Real Servers is Director

slide-10
SLIDE 10

Monitoring iRODS service

▸ ldirector monitors iRODS real servers

▹ Polling server status via iRODS control port

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 10 <MsgHeader_PI> <type>RODS_VERSION RODS_VERSION</type> <msgLen>182</msgLen> <errorLen>0</errorLen> <bsLen>0</bsLen> <intInfo>0</intInfo> </MsgHeader_PI> <Version_PI> <status>-4000</status> <relVersion>rods2.1</relVersion> <apiVersion>d</apiVersion> <reconnPort>0</reconnPort> <reconnAddr></reconnAddr> <cookie>0</cookie> </Version_PI>

Director Director iRODS Server Port# 1247 req: any string ack: iRODS MsgHeader

slide-11
SLIDE 11

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 11

slide-12
SLIDE 12

Large File Transfer

▸ iRODS uses parallel ports to transfer a large file.

▹ Smaller than 32MB file is transferred through iRODS control port #1247.

▸ iRODS catalog server directs a server to

  • pen parallel ports to transfer a large file

▹ iRODS clients can directly connect with the server through the parallel ports.

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 12

slide-13
SLIDE 13

Process of Large File Transfer

▸ Steps to transfer a large file in iRODS

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 13

ICAT

iRODS Client iRODS Server iRODS Server w/o ICAT

Physical Data

Start service for Parallel I/O (4) PostgreSQL (1) (3) iput a large file

Find physical location to store

(2) A C File Transfer via Parallel I/O

slide-14
SLIDE 14

Large File Transfer w/ Director

▸ Need to confirm whether Director interferes in transferring a large file or not ▸ The physical storage should be located

  • ut of the local network of iRODS real

servers

▹ Director handles only iRODS catalog server IP ▹ Director cannot manage all of the parallel ports

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 14

slide-15
SLIDE 15

Process using Director

▸ Works as same as normal case

▹ Only one additional step between (1) and (2)

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 15

ICAT ICAT

Pgpool Pgpool iRODS Server PostgreSQL iRODS Client Director Director iRODS Server w/o ICAT

Physical Data

(1) (3) (4) A B C

(1)’

(2)

slide-16
SLIDE 16

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 16

slide-17
SLIDE 17

Speed Performance

▸ Test Program

▹ concurrent-test in iRODS package ▹ iput, imeta, iget, imv ▹ 1000 entries ▹ Servers are VMs (Xen) on same physical machine

▪ Client is located on the different machine

▸ Test Case

▹ Case1: Normal case.

▪ Client directly accesses one iRODS server.

▹ Case2: Using a director.

▪ Client accesses one iRODS server via Director.

▹ Case3: Load sharing case.

▪ Client accesses two iRODS servers via Director.

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 17

slide-18
SLIDE 18

Speed Performance (cont’d)

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 18

▸ Using a Director (Case2)

▹ About 10% slower than no Director (Case1) ▹ Reasonable to consider tradeoff between speed and availability

slide-19
SLIDE 19

Speed Performance (cont’d)

▸ Load sharing case (Case3)

▹ About 5% slower than Case2 ▹ The concurrent-test is not suitable under such a Load balanced system. ▹ Need a program using multi-clients or multi-threading methods.

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 19

slide-20
SLIDE 20

Opinions in this study

▸ Network limitation

▹ Director works as NAT. Difficult to place iRODS catalog servers in different subnets. ▹ But the problem depends on NAT technology. We hope some NAT vender can implement extensions.

▸ Speed Performance

▹ The “concurrent-test” consumes overhead. The result 10% slow is in one of the worst cases. We may see less than 10% in actual uses.

▸ PostgreSQL only?

▹ How about other DB services? They have the same tools as PgPool? ▹ Back-end replication is enough? Front-end replication should be considered for iRODS?

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 20

slide-21
SLIDE 21

Summary

▸ iRODS HA system

▹ The current approach using only PgPool ▹ The new approach using Director ▹ The new one can solve the current problem

▸ Large File Transfer

▹ iRODS large file transfer works well when using Director

▸ Speed Performance

▹ Director results in the speed performance of concurrent- test getting slower 10%

▸ Future works

▹ Apply this solution to other catalog services

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 21

slide-22
SLIDE 22

References

▸ [1] : ARDA is A Realization of Distributed Analysis for LHC, http://lcg.web.cern.ch/LCG/activities/arda/arda.html ▸ [2] : iRODS High Avaliability, https://www.irods.org/index.php/iRODS_High_Avaliability ▸ [3] : Ultra Monkey project, http://www.ultramonkey.org/ ▸ [4] : citation from abstract of “Resource Namespace Service Specification”, https://forge.gridforum.org/sf/go/doc8272 ▸ [5] : http://www.cs.virginia.edu/~vcgr/wiki/index.php/ Understanding_Your_Genesis_II_Distribution#RNS_Namespa ce

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 22

slide-23
SLIDE 23

Back up

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 23

slide-24
SLIDE 24

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Future works (apply to RNS application) ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 24

slide-25
SLIDE 25

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Future works (apply to RNS application) ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 25

slide-26
SLIDE 26

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Future works (apply to RNS application) ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 26

slide-27
SLIDE 27

Outline

▸ Introduction ▸ iRODS HA system with Director ▸ Large File Transfer ▸ Speed Performance ▸ Future works (apply to RNS application) ▸ Summary

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 27

slide-28
SLIDE 28

What is RNS ?

▸ RNS : Resource Namespace Service

▹ RNS offers a simple standard way of mapping names to endpoints within a grid or distributed network [4] ▹ The latest version is available here;

https://forge.gridforum.org/sf/go/doc8272

▸ Java based RNS application is being developed by Osaka University and Tsukuba University

▹ This application is similar to iRODS ▹ The other kind of RNS application is Grid Shell of Genesis II by The Virginia Center for Grid Research (VCGR) [5].

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 28

slide-29
SLIDE 29

Apply to RNS application??

▸ Derby can do replication?

▹ http://wiki.apache.org/db-derby/ReplicationWriteup ▹ No load-sharing in the above example

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 29

DB DB

Replication Replication RNS Server Derby RNS Client Director Director ?

slide-30
SLIDE 30

Issues in RNS application

▸ Several issues to be solved

▹ Derby is not enough to work replication as same as using PostgreSQL w/Pgpool ▹ Need some developments to replace Derby by PostgreSQL ▹ The catalog implementation in the current RNS application has specific IP addresses

March 25th, 2010 iRODS User Meeting @ UNC in Chapel Hill, USA -- Yutaka (KEK), Adil (ULiv) 30