[PPT] - Concurrent Composition of I/O Redundancy Be- haviors in Go Klaus PowerPoint Presentation

SLIDE 1

n i e l s b o h r i n s t i t u t e

university of copenhagen

Concurrent Composition of I/O Redundancy Be- haviors in Go

Klaus Birkelund Jensen and Brian Vinter {birkelund,vinter}@nbi.ku.dk

eScience, X-Ray and Neutron Scattering Niels Bohr Institute, University of Copenhagen CPA 2017 Malta

August 22, 2017 Slide 1/24

SLIDE 2

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Outline

1 Redundant I/O Behaviors

A short introduction to redundant and fault-tolerant I/O.

2 Go Interfaces and I/O

Some background on interfaces in Go and the standard library io package.

3 Composable Stream Behaviors

Description of the streammux package.

4 Implementation Details 5 Conclusions

Slide 2/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 3

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Redundant I/O Behaviors

Redundancy through high availability and/or increased durability.

Hardware level; multipath I/O.
Software/hardware, well-known example for disks;

RAID: Redundant Array of Inexpensive Disks. Behaviors are inherently concurrent and parallel.

Slide 3/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 4

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID Levels

RAID types are divided into levels.

Slide 4/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 5

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID Levels

RAID types are divided into levels.

RAID-0; striping. Not a real RAID level, no
redundancy. Allows increased performance by

parallelizing I/O.

Slide 4/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 6

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID Levels

RAID types are divided into levels.

RAID-0; striping. Not a real RAID level, no
redundancy. Allows increased performance by

parallelizing I/O.

RAID-1; mirroring. Replicates writes to multiple

devices ⇒ increased durability.

Slide 4/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 7

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID Levels

RAID types are divided into levels.

RAID-0; striping. Not a real RAID level, no
redundancy. Allows increased performance by

parallelizing I/O.

RAID-1; mirroring. Replicates writes to multiple

devices ⇒ increased durability.

RAID-4/5/6; parity-based redundancy. Combines

striping with parity data to increase utilizating of devices.

RAID-4; striping with dedicated parity on one device.
RAID-5; striping with distributed or rotating parity.
RAID-6; RAID-5 with an extra parity block ⇒ increased

redundancy.

Slide 4/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 8

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID-0

Striping, data blocks (Ai) is split between N devices.

A split A1 A3 A2 A4

Fast reads and writes but cannot tolerate any failure.

Slide 5/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 9

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID-1

Mirroring, written data is replicated to N devices.

A copy A1 A2 A1 A2

Tolerates N − 1 failed devices. Low utilization.

Slide 6/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 10

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID-4

Striping with parity data stored on one device.

A, B, . . . split xor A1 B2 A2 B2 A1 ⊕ A2 B1 ⊕ B2

Tolerates 1 failure. High utilization.

Slide 7/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 11

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Composition

Levels can be composed. An example striped mirror (RAID-1+0).

copy A1 A3 A1 A3 copy A2 A4 A2 A4 split A

Slide 8/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 12

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID State Machines

O start D F R err err new done err err

Slide 9/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 13

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

RAID State Machines

O start D F R err err new done err err The state machine for stripes are a bit less interesting... O start F err

Slide 9/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 14

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

I/O Redundancy Behaviors

Because this work is focused on streams, we restate the RAID levels in terms of I/O redundancy behaviors.

Striping, RAID-0 (again, not really a redundancy

behavior)

Mirroring, RAID-1
Dedicated parity, RAID-4
Distributed parity, RAID-5/6 (not part of this work)

Slide 10/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 15

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Go Interfaces

Go allows any user defined type to implement interfaces that define behavior. The Go io package includes two very useful interfaces; the io.Reader and io.Writer.

type Reader i n t e r f a c e { Read ( p [ ] byte ) ( n int , e r r e r r o r ) } type Writer i n t e r f a c e { Write ( p [ ] byte ) ( n int , e r r e r r o r ) }

Slide 11/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 16

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Go Interfaces

Go allows any user defined type to implement interfaces that define behavior. The Go io package includes two very useful interfaces; the io.Reader and io.Writer.

type Reader i n t e r f a c e { Read ( p [ ] byte ) ( n int , e r r e r r o r ) } type Writer i n t e r f a c e { Write ( p [ ] byte ) ( n int , e r r e r r o r ) }

Used a lot in the standard library

$ guru implements ~/ l o c a l /go/ s r c / i o / i o . go:#3309 | wc −l 207 $ guru implements ~/ l o c a l /go/ s r c / i o / i o . go:#3800 | wc −l 184

Slide 11/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 17

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Go Interfaces

Interfaces are satisfied by implementing the corresponding methods.

type PanicReader s t r u c t {} func ( PanicReader ) Read ( p [ ] byte ) ( n int , e r r e r r o r ) { panic ( "im j u s t gonna crash " ) }

Note It is not explicitly stated that an interface is being implemented, but PanicReader can now be used whenever an io.Reader is expected by a function.

Slide 12/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 18

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

streammux

The streammux package provides redundancy behaviors for arbitrary I/O types.

Works with types implementing the

io.ReadWriteCloser interface (e.g. regular files and network connections).

Provides stripe, mirror and dedicated parity behaviors.
Supports spare pools.

Slide 13/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 19

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

streammux, basic abstraction

The member is the basic abstraction.

Wraps types implementing the io.ReadWriteCloser

interface.

func Open() State readies the device for use and
ptionally opens the underlying device if it implements

the streammux.Opener interface.

func write(idx int, p []byte, ch chan rwT) writes the

byte slice p to the underlying device and signals completion (including errors) on the ch channel.

func read(idx int, p []byte, ch chan rwT) reads

len(p) bytes from the underlying device and signals completion (including errors) on the ch channel.

type rwT s t r u c t { idx , n int , p [ ] byte e r r e r r o r }

Slide 14/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 20

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

streammux, behaviors

Each redundancy behavior implemented by streammux implements the io.ReadWriteCloser interface and thus allows building of arbitrary redundancy networks.

Striped mirrors
Mirrored stripes

Slide 15/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 21

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Striping and Mirroring

The striping behavior is built from other members directly

Slide 16/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 22

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Striping and Mirroring

The striping behavior is built from other members directly

s t r i p e := streammux . NewStripe ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sda " ) ,
penOrPanic ( "/ dev / sdb " ) ,

})

Slide 16/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 23

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Striping and Mirroring

The striping behavior is built from other members directly

s t r i p e := streammux . NewStripe ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sda " ) ,
penOrPanic ( "/ dev / sdb " ) ,

})

Similar for mirrors

m i r r o r := streammux . NewMirror ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sda " ) ,
penOrPanic ( "/ dev / sdb " ) ,

})

Slide 16/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 24

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Striping and Mirroring

The striping behavior is built from other members directly

s t r i p e := streammux . NewStripe ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sda " ) ,
penOrPanic ( "/ dev / sdb " ) ,

})

Similar for mirrors

m i r r o r := streammux . NewMirror ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sda " ) ,
penOrPanic ( "/ dev / sdb " ) ,

})

Composition

m i r r o r 2 := streammux . NewMirror ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sdc " ) ,
penOrPanic ( "/ dev / sdd " ) ,

}) comp := streammux . NewStripe ( [ ] i o . ReadWriteCloser { mirror , mirror2 , })

Slide 16/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 25

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Dedicated Parity

The dedicated parity behavior is made up from a single parity member and N stripe members.

dp := streammux . NewDedicatedParity (

penOrPanic ( "/ dev / sda " ) ,

[ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sdb " ) ,
penOrPanic ( "/ dev / sdc " ) ,

} , )

For both mirror and dedicated parity, rebuilding is an off-line process done when the explicitly replacing a member. On-line rebuilding is not supported because the members does not implement the io.Seeker interface.

Slide 17/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 26

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Spare Pool

Hot spares for fail-over can be added to arrays by creating a spare pool.

s p a r e s := streammux . NewSparePool ( [ ] i o . ReadWriteCloser {

penTCPConnOrPanic ( "some . domain :1337 " ) ,

streammux . NewStripe ( [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sdx " ) ,
penOrPanic ( "/ dev / sdy " ) ,

}) , })

The spare pool is attached when creating the I/O behavior

members := [ ] i o . ReadWriteCloser {

penOrPanic ( "/ dev / sda " ) ,
penOrPanic ( "/ dev / sdb " ) ,

} s := streammux . NewMirror ( members , streammux . WithSparePool ( s p a r e s ) , )

Slide 18/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 27

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Spare Pool

some.domain:1337 sdx/sdy stripe split

(a) Spare pool.

sda/sdb mirror copy A1 A2 A1 A2

(b) Pristine mirror.

sda/some.domain:1337 mirror copy A1 A2 A3 A3

(c) sdb fails and the connection to some.domain:1337 is inserted as spare. Figure: An example mirror with fail-over spares.

Slide 19/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 28

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Parallel I/O Execution

func ( b . . . ) Read ( p [ ] byte ) ( n int , e r r e r r o r ) { ch := make( chan rwT) var a c t i v e [ ] s t r u c t {} f o r i , r e a d e r := range b . i o s { i f r e a d e r . State () == FAILED { continue } go r e a d e r . read ( i , make ( [ ] byte , len ( p ) ) , ch ) a c t i v e = append ( a c t i v e , s t r u c t {}{}) } i f len ( a c t i v e ) == 0 { return 0 , s y s c a l l . EIO } f o r range a c t i v e { rc := < −ch i f rc . e r r != n i l && rc . e r r != i o .EOF { // b e h a v i o r s p e c i f i c e r r o r h a n d l i n g } // b e h a v i o r s p e c i f i c h a n d l i n g

f

the f i l l e d b u f f e r ( rc . p ) } return }

Slide 20/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 29

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Parallel I/O Execution

I/O is issued to members in parallel.

func (m ∗Member) read ( i d x int , p [ ] byte , ch chan rwT) { i f m. s t a t e == FAILED { ch < − rwT{ idx , p , 0 , s y s c a l l . EIO} return } // segment h an d l i n g

mitted

// . . . // perform the a c t u a l read n , e r r := m. rwc . Read ( p ) i f e r r != n i l && e r r != i o .EOF {

m. s t a t e = FAILED

}

m. pos += n

// send r e s u l t back to the b e h a v i o r read method ch < − rwT{ idx , p , n , e r r } }

Slide 21/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 30

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Synchronization/Rebuilding

func (m ∗ Mirror ) Sync () { f o r { // wait f o r a r e p l a c e d d r i v e i d x := < −

m. r e p l a c e d

// f i n d a r e a d e r that i s OK f o r _, r e a d e r := range m. i o s { i f r e a d e r . State () == OK {

m. i o s [ i d x ] . SetState (REBUILDING)

_written , e r r := i o . Copy (m. i o s [ i d x ] . rwc , r e a d e r . rwc ) i f e r r != n i l { break }

m. i o s [ i d x ] . SetState (OK)

break } }

m. Close ()

} }

Slide 22/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 31

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Use Cases

Slide 23/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 32

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Use Cases

Mirrored copy

Basic use, make redundant copies.

Slide 23/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 33

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Use Cases

Mirrored copy

Basic use, make redundant copies.

Off-site parity

A little esoteric. Storing parity on tertiary storage (e.g. tape off-site).

Slide 23/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 34

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Use Cases

Mirrored copy

Basic use, make redundant copies.

Off-site parity

A little esoteric. Storing parity on tertiary storage (e.g. tape off-site).

Tee analysis

Similar to mirrored copy, redirect data for on-line stream analysis.

Slide 23/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 35

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Use Cases

Mirrored copy

Basic use, make redundant copies.

Off-site parity

A little esoteric. Storing parity on tertiary storage (e.g. tape off-site).

Tee analysis

Similar to mirrored copy, redirect data for on-line stream analysis.

High-availability for high-throughput archives

Never run out of a place to put data. Just fail-over immediately at any sign of error.

Slide 23/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017

SLIDE 36

u n i v e r s i t y o f c o p e n h a g e n n i e l s b o h r i n s t i t u t e

Summary

streammux

implements striping, mirroring and dedicated parity
allows any types supporting reads, writes and closing to

be composed into redundancy components of arbitrary depth through concurrent composition

supports the use of hot spares for high availability

Slide 24/24 — Birkelund — Concurrent Composition of I/O Redundancy Behaviors in Go — August 22, 2017