[PDF] - An FPGA Platform for Hyperscalers - Slides Presentation August 2017 PDF Document

SLIDE 1

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/319346565

An FPGA Platform for Hyperscalers - Slides

Presentation · August 2017

CITATIONS READS

472

1 author: Some of the authors of this publication are also working on these related projects: cloudFPGA View project Francois Abel IBM

36 PUBLICATIONS 481 CITATIONS

SEE PROFILE

All content following this page was uploaded by Francois Abel on 01 September 2017.

The user has requested enhancement of the downloaded file.

SLIDE 2

H

t

I n t e r c

n

n e c t s 2 5 , S a n t a C l a r a , C A , A u g . 2 9

3

, 2 1 7

A n F P G A P l a t f

r

m f

r

H y p e r s c a l e r s

F . A b e l , J . We e r a s i n g h e , C . H a g l e i t n e r , B . We i s s , S . P a r e d e s

I B M R e s e a r c h – Z u r i c h , S w i t z e r l a n d

SLIDE 3

2

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

Wh a t i s a h y p e r s c a l e r ?

■ D

e f i n i t i

n

– C l

u

d

p

e r a t

r

t h a t r u n s s e v e r a l h u n d r e d s

f

t h

u

s a n d s

f

s e r v e r s – O p e r a t

r

w i t h mo r e t h a n U S $ [ 1 , 4 , 8 ] b i l l i

n

i n a n n u a l r e v e n u e

*

f r

m:
a

s e r v i c e s u c h a s I a a S , P a a S

r

S a a S ( e . g . , A WS , R a c k s p a c e , G

g

l e )

I

n t e r n e t , s e a r c h , s

c

i a l n e t w

r

k i n g ( e . g . , S a l e s f

r

c e , A D P , G

g

l e )

E
c
mme

r c e / p a y me n t p r

c

e s s i n g ( e . g . , A ma z

n

, A l i b a b a , e B a y )

■ H

y p e r s c a l e r s i n 2 1 7 :

– 2 4 h y p e r s c a l e c

mp

a n i e s

* *

p

e r a t i n g 3 + d a t a c e n t e r s

S

u

r c e s : * C i s c

G

l

b

a l C l

u

d I n d e x a n d * * S y n e r g y R e s e a r c h G r

u

p

SLIDE 4

3

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

H y p e r s c a l e s e r v e r – H

w

d

e

s i t l

k

?

M i c r

s
f

t ' s c loud server architecture F a c e b

k

' s Y

s

e mi t e s l e d V 1 w i t h f

u

r M

n
L

a k e s e r v e r s

S

u

r c e s : H

w

M i c r

s
f

t D e s i g n s i t s C l

u

d

S

c a l e S e r v e r s , M i c r

s
f

t , 2 1 4 a n d I n t r

d

u c i n g " Y

s

e mi t e " : t h e f i r s t

p

e n s

u

r c e mo d u l a r c h a s s i s f

r

h i g h

p
w

e r e d mi c r

s

e r v e r s , F a c e b

k

, 2 1 5 .

SLIDE 5

4

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

Wh y d

w

e n e e d a n e w p l a t f

r

m?

■ T

r a d i t i

n

a l b u s a t t a c h me n t → F P G A a s a c

p

r

c

e s s

r

C P U

F P G A

S e r v e r N

d

e

C P U

S e r v e r N

d

e

F P G A

F P G A N

d

e P C I e

S e r v e r h

mo

g e n e i t y I n c r e a s e d s e r v e r c

s

t & p

w

e r P e r f

r

ma n c e b

s

t N u mb e r

f

a c c e l e r a t

r

s p e r s e r v e r Wo r k l

a

d f l e x i b i l i t y a n d mi g r a t i

n

– – – – – – – – – – + + + – + + – –

SLIDE 6

5

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

D i s a g g r e g a t i

n
f

t h e F P G A f r

m

t h e s e r v e r

[ 1 ]

■ N

e t w

r

k a t t a c h me n t → F P G A a s a p e e r p r

c

e s s

r

D C N e t w

r

k C P U

S e r v e r N

d

e

F P G A

F P G A N

d

e P C I e

F P G A

F P G A N

d

e F P G A N

d

e E t h e r n e t S w i t c h

F P G A

[ 1 ] J . We e r a s i n g h e e t a l . , “ E n a b l i n g F P G A s i n h y p e r s c a l e d a t a c e n t e r s , ” i n 2 1 5 I E E E I n t ' l C

n

f .

n

C l

u

d a n d B i g D a t a C

mp

u t i n g , B e i j i n g , C h i n a , 2 1 5 .

+ + + S e r v e r h

mo

g e n e i t y + + + S e r v e r c

s

t & p

w

e r + + M a n a g e me n t + + + N u mb e r

f

F P G A s p e r s e r v e r + + P e r f

r

ma n c e b

s

t + Wo r k l

a

d f l e x i b i l i t y & mi g r a t i

n

+ + /

L

a r g e

s

c a l e d i s t r i b u t e d a p p l i c a t i

n

s

SLIDE 7

6

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

S t a n d a l

n

e n e t w

r

k

a

t t a c h e d F P G A

1 ) R e p l a c e P C I e I / F w i t h i n t e g r a t e d N I C ( i N I C ) 3 ) R e p l a c e t r a n s c e i v e r s w / b a c k p l a n e c

n

n e c t i v i t y

I O

F P G A M

d

u l e

B a c k p l a n e

F P G A

H P I O H P I O C f g M

n

J T A G

K i n t e x U l t r a S c a l e

E mi f

P S

C

P e r v a s i v e

M P S S E J T A G

F l a s h

B P I C

n

f i g 10GBASE KR (x6) C

n

n e c t

r

USBv2 JTAG x72 PCIe (x8) PCIe (x8) SATA (x2)

D D R 4 D D R 4

D R A M

x72 PMBus EN PG MON

P WR

3 M

T M

S P D 8 S e r i e s

GTY (x8)

x 3 2 H R

I O

x 4 8 H P

J T A G I2C PCIe (x4) G T H G T H G T H

P S

C

A R M C

r

t e x

M

3 2 ) T u r n F P G A c a r d i n t

a

s e l f

c
n

t a i n e d a p p l i a n c e

i N I C

~ 1 5 %

SLIDE 8

7

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

O n e c a r r i e r s l e d = 3 2 F P G A mo d u l e s

■ O

u r f i r s t F P G A mo d u l e u s e s a X i l i n x K i n t e x U l t r a s c a l e K U 6

– A mi d

r

a n g e F P G A w i t h h i g h p e r f

r

ma n c e / p r i c e a n d l

w

w a t t a g e

SLIDE 9

8

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

O n e c a r r i e r s l e d = 3 2 F P G A mo d u l e s

■ O

u r f i r s t F P G A mo d u l e u s e s a X i l i n x K i n t e x U l t r a s c a l e K U 6

– A mi d

r

a n g e F P G A w i t h h i g h p e r f

r

ma n c e / p r i c e a n d l

w

w a t t a g e

×1 6 ×1 6

4 G E x 8

6 4 G b / s E t h e r n e t s w i t c h

SLIDE 10

9

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

T w

c

a r r i e r s l e d s p e r c h a s s i s = 6 4 F P G A s

F i g u r a t i v e p i c t u r e

L e g e n d ( p e r s l i c e ) :

[ = = ] x 8 4 G b E u p l i n k s ( 3 2 G b / s ) [ – – ] x 3 2 1 G b E F P G A

t
S

w i t c h l i n k s ( 3 2 G b / s ) [ – – ] x 3 2 1 G b E r e d u n d a n t l i n k s [ – – ] x 3 2 1 G b E F P G A

t
F

P G A l i n k s [ ██] x 1 6 P C I e x 8 G e n 3 S P x 1 S e r v i c e P r

c

e s s

r

B a l a n c e d ( i . e . n

v

e r

s

u b s c r i p t i

n

) b e t w e e n t h e n

r

t h a n d s

u

t h l i n k s

f

t h e E t h e r n e t s w i t c h

SLIDE 11

10

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

S i x t e e n c h a s s i s p e r r a c k = 1 2 4 F P G A s

1 2 4 F P G A s → 2 . 8 M D S P s , 2 x 1

1 5

F i x e d

P
i

n t M u l t i p l y

A

c c u mu l a t e s / s 1 T b / s b i

s

e c . B w – 1 6 T B D D R 4 – 4 k W ma x .

SLIDE 12

11

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

S i x t e e n c h a s s i s p e r r a c k = 1 2 4 F P G A s

1 2 4 F P G A s → 2 . 8 M D S P s , 2 x 1

1 5

F i x e d

P
i

n t M u l t i p l y

A

c c u mu l a t e s / s 1 T b / s b i

s

e c . B w – 1 6 T B D D R 4 – 4 k W ma x . E x a mp l e

f

a t w

t

i e r C l

s

a r c h i t e c t u r e w i t h 4 9 6 F P G A s a n d f

u

r e d g e s w i t c h e s ( T O R s )

→ x 4 T O R s

SLIDE 13

12

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

C

mb

i n e d p a s s i v e a n d a c t i v e w a t e r c

l

i n g

R e n d e r i n g

f

a 2 U ❌ 1 9 ” c h a s s i s

C

l

i n g r a i l

Packaging technology – Courtesy of DOME project

SLIDE 14

13

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

P a s s i v e c

l

i n g w / h e a t s p r e a d e r

55 mm 7.6 mm 7.2 mm

S c h e ma t i c

f

b

a

r d a s s e mb l y

p

w

e r n

t

c h

Packaging technology – Courtesy of DOME project

SLIDE 15

14

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

P r

t
t

y p e i n t h e l a b

Courtesy of DOME project

SLIDE 16

15

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

S

f

t w a r e

d

e f i n e d mu l t i

F

P G A f a b r i c ( S D M F F )

E . g . , s e c u r i t y E . g . , t e x t p r

c

e s s i n g E . g . , d i s t r i b u t e d D N N E . g . , H P C

D a t a C e n t e r N e t w

r

k

F e w S e r v e r s + 1 F P G A s 1 S e r v e r + 1 F P G A

H y p e r s c a l e i n f r a s t r u c t u r e

C

u

r t e s y

f

R e d d i t

F e w S e r v e r s + 1 F P G A s 1 S e r v e r + 5 F P G A s

C P U

SLIDE 17

16

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

N e t w

r

k p e r f

r

ma n c e

■ C

mp

a r i s

n

w i t h b a r e

me

t a l s e r v e r s , v i r t u a l ma c h i n e s a n d L i n u x c

n

t a i n e r s

[ 2 ]

[ 2 ] J . We e r a s i n g h e e t a l . , “ D i s a g g r e g a t e d F P G A s : N e t w

r

k p e r f

r

ma n c e c

mp

a r i s

n

a g a i n s t b a r e

me

t a l s e r v e r s , v i r t u a l ma c h i n e s a n d L i n u x c

n

t a i n e r s , ” i n I E E E I n t ' C

n

f .

n

C l

u

d C

mp

u t i n g T e c h n

l
g

y a n d S c i e n c e , L u x e mb

u

r g , 2 1 6 .

R T L a t e n c y ( µ S ) U D P T p u t ( G b / s ) U D P R T L a t e n c y ( µ S ) U D P T C P i W A R P T p u t ( G b / s ) U D P T C P i W A R P

SLIDE 18

17

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

S D M F F A p p l i c a t i

n

■ D

i s t r i b u t e d t e x t a n a l y t i c

[ 3 ]

[ 3 ] J . We e r a s i n g h e e t a l . , “ N e t w

r

k

a

t t a c h e d F P G A s f

r

d a t a c e n t e r a p p l i c a t i

n

s , ” i n I E E E I n t e r n a t i

n

a l C

n

f e r e n c e

n

F i e l d

P

r

g

r a mma b l e T e c h n

l
g

y ( F P T ' 1 6 ) , X i a n , C h i n a , 2 1 6 .

https://uima.apache.org/ UIM A: Unstructured Information M anagement Architecture M N: M aster Node, SN: Slave Node

Structured Data Unstructured Data

MN: Collection Process Engine Collection Reader CAS Consumer

SN: Analysis Engine SN: Analysis Engine

Standard UIMA

C P U C P U C P U F P G A

P C I e

C P U F P G A

P C I e

F P G A F P G A

S N 1 = S N 2 =

L a t e n c y ( ms ) T p u t ( c h a r / s ) C

s

t ( $ )

SLIDE 19

18

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7 ■ x

4 T e s l a P 1 w / N V L i n k

– T

t

a l p e r f

r

ma n c e 4 2 . 4 T e r a F L O P S S i n g l e

P

r e c i s i

n

* 8 4 . 8 T e r a F L O P S H a l f

P

r e c i s i

n

*

■ x

2 [ P O WE R 8 C P U + 2 5 6 G B D R A M ]

■ P

w

e r c

n

s u mp t i

n

≈ 2 . 3 k W

■ x

6 4 X i l i n x K U 6 –

T

t

a l p e r f

r

ma n c e 5 3 T e r a F L O P S S i n g l e

P

r e c i s i

n

* * 1 6 T e r a F L O P S H a l f

P

r e c i s i

n

* * 4 2 4 T e r a O P S F i x e d

P
i

n t ( I N T 8 ) * * *

1

T B D D R 4 / P

w

e r c

n

s u mp t i

n

≈ 2 . 5 k W

* * C

mp

u t e d a s : 6 4 x ( # D S P s

p

e r

K

U 6 x F

M A X

) / ( # D S P s

p

e r

F

u s e d M u l t i p l y A d d ) * * * X i l i n x , WP 4 8 7 ( v 1 . ) J u n e 2 7 , 2 1 7 – 8

B

i t D

t
P

r

d

u c t A c c e l e r a t i

n

* h t t p : / / w w w . n v i d i a . c

m/
b

j e c t / t e s l a

p

1 . h t ml

C

mp

u t e d e n s i t y – S 8 2 2 L C

( a k a M i n s k y )

v s F P G A c h a s s i s

■ S

a me 2 U c h a s s i s a n d s i mi l a r p

w

e r c

n

s u mp t i

n

R e n d e r e d p i c t u r e

SLIDE 20

19

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

S u mma r y

■ A

p l a t f

r

m t

d

e p l

y

F P G A a t l a r g e s c a l e i n D C s

– I n t e g r a t e s F P G A s a t t h e d r a w e r / c h a s s i s l a y e r – C

mb

i n e s p a s s i v e a n d a c t i v e w a t e r

c
l

i n g – P r

v

i d e s h i g h d e n s i t y , e n e r g y e f f i c i e n c y a n d r e d u c e d c

s

t s

F

i t s 1 + F P G A s p e r D C r a c k

■ B

u i l d s

n

t h e d i s a g g r e g a t i

n
f

F P G A s f r

m

t h e s e r v e r s

– F P G A s c

n

n e c t t

t

h e D C n e t w

r

k

v

e r 1 / 4 G b / s E t h e r n e t

K

e y e n a b l e r f

r

l a r g e s c a l e d e p l

y

me n t

f

F P G A s i n D C s

F

P G A s g e n e r a t e a n d c

n

s u me t h e i r

w

n n e t w

r

k i n g p a c k e t s

– F P G A c a r d s b e c

me

s t a n d

a

l

n

e r e s

u

r c e s

D

e p l

y

e d F P G A s b e c

me

i n d e p e n d e n t

f

t h e n u mb e r

f

s e r v e r s

P

r

mo

t e s t h e u s e

f

me d i u m- a n d l

w
c
s

t F P G A s

■ M

a k e s F P G A s p l e n t i f u l i n D C s

– U s e r s c a n r e n t a n d l i n k t h e m i n a n y t y p e

f

t

p
l
g

y

SLIDE 21

20

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

A c k n

w

l e d g me n t s

■ T

h i s w

r

k w a s c

n

d u c t e d i n t h e c

n

t e x t

f

t h e j

i

n t A S T R O N a n d I B M D O M E p r

j

e c t a n d w a s f u n d e d b y t h e N e t h e r l a n d s O r g a n i z a t i

n

f

r

S c i e n t i f i c R e s e a r c h ( N WO ) , t h e D u t c h M i n i s t r y

f

E L & L , a n d t h e P r

v

i n c e

f

D r e n t h e , t h e N e t h e r l a n d s .

■ S

p e c i a l t h a n k s t

M

a r t i n S c h ma t z , R

n

a l d L u i j t e n a n d A n d r e a s D

e

r i n g w h

i

n i t i a t e d t h i s n e w p a c k a g i n g c

n

c e p t f

r

t h e n e e d s

f

t h e i r mi c r

s

e r v e r D O M E p r

j

e c t .

SLIDE 22

21

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

T h a n k y

u

M a y t h e F P G A b e w i t h y

u

h t t p s : / / w w w . z u r i c h . i b m. c

m/

c c i / c l

u

d F P G A /

SLIDE 23

22

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

B a c k u p

SLIDE 24

23

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

T h e b a s e b

a

r d c a r r i e r a t s c a l e

SLIDE 25

24

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

D i s a g g r e g a t e d s w i t c h mo d u l e

Intel SeacliffTrail – Intel SeacliffTrail – Reference System Reference System

3 2 x 1 G b E + 8 x 4 G b E 4 8 x 1 G b E + 4 x 4 G b E

f r

m

7 , 9 3 8 * * c m

3

Switch Module SM6000 Switch Module SM6000

* * 4 1 x 4 4 x 4 . 4 c m * 1 4 x 6 x 4 . 5 c m

t

3

7 8 * c m

3

1 / 2 1

SLIDE 26

25

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

O t h e r c l

u

d F P G A mo d u l e s

N V M e M

d

u l e S e r v i c e p r

c

e s s

r

( T 4 2 4 ) U S B H u b M

d

u l e P

w

e r c

n

v e r t e r

SLIDE 27

26

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

x 4 T e s l a P 1 i n M i n s k y c h a s s i s – A i r c

l

e d

SLIDE 28

27

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

x 4 T e s l a P 1 i n M i n s k y c h a s s i s – Wa t e r c

l

e d

SLIDE 29

28

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

M i n s k y c h a s s i s v s c l

u

d F P G A c h a s s i s

h t t p : / / w e b . a r c h i v e .

r

g / w e b / 2 1 6 1 1 6 7 1 5 4 7 / h t t p s : / / w w w . x i l i n x . c

m/

s u p p

r

t / d

c

u me n t a t i

n

/ i p _ d

c

u me n t a t i

n

/ r u / f l

a

t i n g

p
i

n t . h t ml # k i n t e x u

SLIDE 30

29

H

t

I n t e r c

n

n e c t s 2 5 – A u g . 2 1 7

R e l a t e d Wo r k – O t h e r l a r g e

s

c a l e F P G A d e p l

y

me n t s

T h i s w

r

k ( F P G A s / R a c k : 1 2 4 ) E C 2 F 1

View publication stats View publication stats