The openais Architecture An inside look at an implementation of SA - - PowerPoint PPT Presentation

▶
the openais architecture
SMART_READER_LITE
LIVE PREVIEW

The openais Architecture An inside look at an implementation of SA - - PowerPoint PPT Presentation

Feb 01, 2024 751 likes •1.08k views

The openais Architecture An inside look at an implementation of SA Forum's AIS Steven Dake Presented by Tim Anderson 14 September 2004 1 History of openais Started life as cmgr in February 2002 Hotswap manager for ATCA

slide-1
SLIDE 1

14 September 2004 1

The openais Architecture

Steven Dake Presented by Tim Anderson

An inside look at an implementation of SA Forum's AIS

slide-2
SLIDE 2

14 September 2004 2

History of openais

Started life as “cmgr” in February 2002

Hotswap manager for ATCA

Converted to AIS in May 2003 AIS released as MontaVista product in Dec 2003 Rearchitected to use virtual synchrony Jan 2004 Released under Revised BSD license July 2004 Mark Haverkamp contributed EVT service Aug/Sept 2004

slide-3
SLIDE 3

14 September 2004 3

Setup and Configuration

Save /etc/ais/network.conf: bindnetaddr: 192.168.1.0 mcastaddr: 226.94.1.1 mcastport:6000 Read QUICKSTART in source package for more details Create shared key: linux# ./keygen OpenAIS Authentication key generator. Gathering 1024 bits for key from /dev/random. Writing openais key to /etc/ais/authkey.

slide-4
SLIDE 4

14 September 2004 4

The Architecture

Libais.a EVT

  • penais Executive

Group Messaging Interface Aispoll Interface Service Manager Nodes(1..16) Nodes(1..16), instances (1..n) IPC EVS CKPT AMF CLM EVT EVS CKPT AMF CLM

slide-5
SLIDE 5

14 September 2004 5

Definitions

  • Group Messaging

Sending messages from 1 sender to many receivers.

  • Processor

The entity responsible for executing group messaging and membership protocols.

  • Configuration

A view, or description, of the processors within a group.

  • Agreed Order

All processors agree upon delivery order of messages delivered using group messaging.

  • Virtual Synchrony

A model of group messaging whereby all messages within a configuration view are delivered in agreed order. Configuration changes are delivered in the same order relative to messages to every processor.

slide-6
SLIDE 6

14 September 2004 6

Group Messaging Interface

Implements Extended Virtual Synchrony Compile-time configuration of maximum

message size

Encryption and Authentication of all messages 4 Priority Levels Uses multicast Implemented using UDP Multipathing in progress

slide-7
SLIDE 7

14 September 2004 7

The Ring Protocol

Processor #1 Processor #2 Processor #3

  • Sequence Number
  • Retransmit List
  • flow control count
  • group arut

ORF Token

slide-8
SLIDE 8

14 September 2004 8

The Ring Protocol

Processor #1 Processor #2 Processor #3

  • Sequence Number
  • Retransmit List
  • flow control count
  • group arut

ORF Token Seq No #1

slide-9
SLIDE 9

14 September 2004 9

The Ring Protocol

Processor #1 Processor #2 Processor #3 Seq No #1 MCAST #1, #2

  • Sequence Number
  • Retransmit List
  • flow control count
  • group arut

ORF Token

slide-10
SLIDE 10

14 September 2004 10

Processor #1 Processor #2 Processor #3 Seq No #1 MCAST #1, #2 Seq No #3

The Ring Protocol

  • Sequence Number
  • Retransmit List
  • flow control count
  • group arut

ORF Token

slide-11
SLIDE 11

14 September 2004 11

Processor #1 Processor #2 Processor #3 Seq No #1 MCAST #1, #2 Seq No #3 Detects Missing #2 MCAST #3, #4, #5

The Ring Protocol

  • Sequence Number
  • Retransmit List
  • flow control count
  • group arut

ORF Token

slide-12
SLIDE 12

14 September 2004 12

Processor #1 Processor #2 Processor #3

  • Sequence Number
  • Retransmit List

(RTR)

  • flow control count
  • group arut

ORF Token Seq No #1 MCAST #1, #2 Seq No #3 Detects Missing #2 MCAST #3, #4, #5 Seq #6, RTR #2

The Ring Protocol

slide-13
SLIDE 13

14 September 2004 13

Processor #1 Processor #2 Processor #3 MCAST #1, #2 Seq No #3 Detects Missing #2 MCAST #3, #4, #5 Seq #6, RTR #2 MCAST #2, #6 Detects Missing #5

  • Sequence Number
  • Retransmit List

(RTR)

  • flow control count
  • group arut

ORF Token

The Ring Protocol

slide-14
SLIDE 14

14 September 2004 14

Processor #1 Processor #2 Processor #3 Seq No #3 Detects Missing #2 MCAST #3, #4, #5 Seq #6, RTR #2 MCAST #2, #6 Detects Missing #5 Seq #7, RTR #5

  • Sequence Number
  • Retransmit List

(RTR)

  • flow control count
  • group arut

ORF Token

The Ring Protocol

slide-15
SLIDE 15

14 September 2004 15

Processor #1 Processor #2 Processor #3 Detects Missing #2 MCAST #3, #4, #5 Seq #6, RTR #2 MCAST #2, #6 Detects Missing #5 Seq #7, RTR #5 MCAST #5

  • Sequence Number
  • Retransmit List

(RTR)

  • flow control count
  • group arut

ORF Token

The Ring Protocol

slide-16
SLIDE 16

14 September 2004 16

Why Virtual Synchrony

Integrated Membership Strong Membership Guarantees Agreed Ordering of Messages Self Delivery Use of multicast Group Wide Flow Control Performance

slide-17
SLIDE 17

14 September 2004 17

Why Virtual Synchrony – Integrated Membership

gmi_join (struct gmi_groupname *groupname, void (*deliver_fn) ( struct gmi_groupname *groupname, struct in_addr source_addr, struct iovec *iovec, int iov_len), void (*confchg_fn) ( struct sockaddr_in *member_list, int member_list_entries, struct sockaddr_in *left_list, int left_list_entries, struct sockaddr_in *joined_list, int joined_list_entries), gmi_join_handle *handle_out); Messages delivered with deliver_fn, configuration changes delivered with confchg_fn

slide-18
SLIDE 18

14 September 2004 18

Why Virtual Synchrony – Strong Membership Guarantees

Example: A bank stores money in a distributed fashion based how many banks are in its “network”. The account starts with $300 and $99 is deposited. One bank closed forever around the time of the

  • deposit. What could happen without strong membership guarantees?

Account = $100 Confchg from 4 to 3 Deposits $33 Account = $133 Bank #1 Account = $100 Deposits $25 Confchg from 4 to 3 Account = $125 Bank #2 Account = $100 Confchg from 4 to 3 Deposits $33 Account = $133 Bank #3

slide-19
SLIDE 19

14 September 2004 19

Why Virtual Synchrony – Agreed Ordering of Messages

Object Creation occurs on two seperate processors with the same “name”. What happens without agreed ordering of messages? Race conditions! Created Object “A” Send A to other processors Receives create, but already exists Created Object “A” Send A to other processors Receives create, but already exists Processor #1 Processor #2

slide-20
SLIDE 20

14 September 2004 20

Why Virtual Syncrhony – self delivery

Library Request Library Handler Group Messaging Interface Executive Handler Library Response Executive Request Executive Request Library Request Intelligence Here

slide-21
SLIDE 21

14 September 2004 21

Why Virtual Syncrhony – Use of Multicast

int gmi_mcast ( struct gmi_groupname *groupname, struct iovec *iovec, int iov_len, int priority);

slide-22
SLIDE 22

14 September 2004 22

The Service Handler

Service Manager Manages service handlers. Every Service has 1 or more service handlers. Handles requests from library connections. Handles requests from group messaging delivery. Handles partitions and merges. Initializes the service for a new library connection Exits the service for a departing library connection. Initializes the service for the first time.

slide-23
SLIDE 23

14 September 2004 23

The Service Handler – Details

struct service_handler { struct libais_handler *libais_handlers; int libais_handlers_count; int (**aisexec_handler_fns) (void *msg, struct in_addr source_addr); int aisexec_handler_fns_count; int (*confchg_fn) ( struct sockaddr_in *member_list, int member_list_entries, struct sockaddr_in *left_list, int left_list_entries, struct sockaddr_in *joined_list, int joined_list_entries); int (*libais_init_fn) (struct conn_info *conn_info, void *msg); int (*libais_exit_fn) (struct conn_info *conn_info); int *aisexec_init_fn) (void); }

slide-24
SLIDE 24

14 September 2004 24

Flow Control

Group Messaging Interface uses flow control on

network

Library can access executive much faster then

network can transmit requests

Library is flow controlled by group messaging

interface.

slide-25
SLIDE 25

14 September 2004 25

Flow Control – Details

int gmi_send_ok ( int priority, int msg_size); If gmi_send_ok is zero, library request receives SA_ERR_TRY_AGAIN. This support is handled by the service manager. The library handler is not concerned with flow control.

slide-26
SLIDE 26

14 September 2004 26

Performance / No Encryption or Auth Checkpoint Write from One Processor (100 mbit network)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 1 2 3 4 5 6 7 8 9 10

Throughput

KB / MSG MB / SEC

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 250 500 750 1000 1250 1500 1750 2000 2250

Transactions Per Second

KB / MSG TRANS / SEC

Total Available Bandwidth

slide-27
SLIDE 27

14 September 2004 27

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 1 2 3 4 5 6 7 8 9

Throughput

KB / MSG MB / SEC

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 200 400 600 800 1000 1200 1400 1600

Transactions Per Second

KB / MSG TRANS / SEC

Performance / With Encryption and Auth Checkpoint Write from One Processor (100 mbit network)

slide-28
SLIDE 28

14 September 2004 28

Performance / Group Messaging Scalability with more Processors

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10

Group Messaging Throughput

No Encryption Encryption

Processor Count MB / SEC

slide-29
SLIDE 29

14 September 2004 29

Project Statistics

Executive LOC: 16229 Library LOC: 5951 Include LOC: 2819 Total LOC: 24999 (wc -l) BK Changesets since openais inception: 65

slide-30
SLIDE 30

14 September 2004 30

Production Release Criteria

0.7 (stable) to be released in 2004 Includes AMF, CKPT, EVT, CLM, EVS At least 85% code coverage of every source file

except for files in test directory

Published valgrind analysis of any reported

memory errors or leaks

Code review of remaining uncovered code Initially support linux 2.4, linux 2.6 systems

slide-31
SLIDE 31

14 September 2004 31

Come Join In

We need developers to develop DLOCK and MSG services. We need developers to develop Linux man pages. We need developers to develop distro packaging. We need user reports of failures and successes. Web Address: http://developer.osdl.org/dev/openais Mailing List: openais@lists.osdl.org Download: http://developer.osdl.org/cherry/openais Bitkeeper: bk clone bk://bk.osdl.org:openais ~/openais