Building the Drupal Borg BADCamp October 23nd 2011 Thursday, - - PowerPoint PPT Presentation

▶

Aug 23, 2022 307 likes •817 views

Building the Drupal Borg BADCamp October 23nd 2011 Thursday, January 12, 12 In The Beginning Scaling used to be far more complex. Thursday, January 12, 12 The Late 90s Ease of impleme ntation Java v1 Use-Cases Covered Thursday,

SLIDE 1

Building the Drupal Borg

BADCamp October 23nd 2011

Thursday, January 12, 12

SLIDE 2

In The Beginning

Scaling used to be far more complex.

Thursday, January 12, 12

SLIDE 3

The Late 90’s

Ease of impleme ntation Use-Cases Covered

Java v1

Thursday, January 12, 12

SLIDE 4

The Late 90’s

Ease of impleme ntation Use-Cases Covered

J2EE Java v1

Thursday, January 12, 12

SLIDE 5

The Late 90’s

Ease of impleme ntation Use-Cases Covered

J2EE Java v1 Microsoft Stack

Thursday, January 12, 12

SLIDE 6

The Late 90’s

Ease of impleme ntation Use-Cases Covered

J2EE LAMP Java v1 Microsoft Stack

Thursday, January 12, 12

SLIDE 7

A Dark Time

All apps written from scratch
Network databases still immature
“Publishing HTML” still the goal
Remember the web in the 90’s?

(sorta sucked)

Thursday, January 12, 12

SLIDE 8

LAMP Shines a Light

Share-nothing runtime
Network-connected database
Free and Open Source
Runs on any hardware

Thursday, January 12, 12

SLIDE 9

And so we grew

And so the web grew

Thursday, January 12, 12

SLIDE 10

“Web Scale”

C10k Challenge
Evented I/O wins
Nginx and Lighttpd
Memcache
Distributed reads

Thursday, January 12, 12

SLIDE 11

But the Architecture

Thursday, January 12, 12

SLIDE 12

By 2005...

http://www.danga.com/words/

LiveJournal Backend: Today

Roughly.

User DB Cluster 1 uc1a uc1b User DB Cluster 2 uc2a uc2b User DB Cluster 3 uc3a uc3b User DB Cluster 4 uc4a uc4b User DB Cluster 5 uc5a uc5b Memcached

mc4 mc3 mc2 mc12 ... mc1

mod_perl

web4 web3 web2 web50 ... web1

BIG-IP

bigip2 bigip1

perlbal (httpd/proxy)

proxy4 proxy3 proxy2 proxy5 proxy1

Global Database

slave1 master_a master_b slave2 ... slave5

MogileFS Database

mog_a mog_b

Mogile Trackers

tracker2 tracker1

Mogile Storage Nodes

... sto2 sto8 sto1

net.

Thursday, January 12, 12

SLIDE 13

Infrastructure

Ease of impleme ntation Use-Cases Covered

Shared Hosting VPS HA Web Cluster Dedicated Server Custom Cluster

Thursday, January 12, 12

SLIDE 14

Infrastructure

Ease of impleme ntation Use-Cases Covered

Shared Hosting VPS HA Web Cluster Dedicated Server Custom Cluster

Cost is a real Barrier

Thursday, January 12, 12

SLIDE 15

Infrastructure

Ease of impleme ntation Use-Cases Covered

Shared Hosting VPS HA Web Cluster Dedicated Server Custom Cluster

Expertise is a real barrier

Thursday, January 12, 12

SLIDE 16

The Units of Scale

Entire hardware servers is the wrong unit to scale. Tons of capacity at a chunk.

Thursday, January 12, 12

SLIDE 17

The Challenge

This isn’t just about scaling

ne website

Thursday, January 12, 12

SLIDE 18

The Challenge

This is about scaling the web

Thursday, January 12, 12

SLIDE 19

The Challenge

Limited developer talent limits Drupal adoption

Thursday, January 12, 12

SLIDE 20

The Challenge

Expertise in infrastructure is even more limited

Thursday, January 12, 12

SLIDE 21

The Challenge

Luckily infrastructure is friendly to automation

Thursday, January 12, 12

SLIDE 22

Bring on the Borg!

Thursday, January 12, 12

SLIDE 23

Bring on the Borg!

Thursday, January 12, 12

SLIDE 24

A Brief History of Borg

Google / AppEngine CfEngine, Puppet, Chef Heroku

Thursday, January 12, 12

SLIDE 25

This is how we do it

Thursday, January 12, 12

SLIDE 26

This is how we do it

Thursday, January 12, 12

SLIDE 27

Pantheon is the best of both worlds

Thursday, January 12, 12

SLIDE 28

Pantheon will un-block Drupal adoption on infrastructure

Thursday, January 12, 12

SLIDE 29

Inside Pantheon

The High Level View

Thursday, January 12, 12

SLIDE 30

Routing Mesh

Request Varnish Firewall “Styx”

Thursday, January 12, 12

SLIDE 31

Routing Mesh

Thursday, January 12, 12

SLIDE 32

Routing Mesh

Routing Mesh Yggdrasil DNS

Thursday, January 12, 12

SLIDE 33

Routing Mesh

Routing Mesh Yggdrasil DNS Cassandra

Thursday, January 12, 12

SLIDE 34

Routing Mesh

Routing Mesh Yggdrasil DNS Cassandra

Thursday, January 12, 12

SLIDE 35

Routing Mesh

Routing Mesh Yggdrasil DNS Cassandra

Thursday, January 12, 12

SLIDE 36

Routing Mesh

Routing Mesh Yggdrasil

Thursday, January 12, 12

SLIDE 37

Routing Mesh

Routing Mesh Yggdrasil DROPs Matrix

Thursday, January 12, 12

SLIDE 38

DROPs

Provision, Isolate and Scale at the per-process level.

Thursday, January 12, 12

SLIDE 39

DROPs

Provision, Isolate and Scale at the per-process level.

Thursday, January 12, 12

SLIDE 40

DROPs

We can scale this too

Thursday, January 12, 12

SLIDE 41

DROPs

We can scale this too

Thursday, January 12, 12

SLIDE 42

DROPs

We can scale this too

Thursday, January 12, 12

SLIDE 43

Anatomy of a DROP

Nginx PHP-FPM APC Cache

Service Bindings

Thursday, January 12, 12

SLIDE 44

Valhalla

Twisted DAV

Dev Test Live

Thursday, January 12, 12

SLIDE 45

Valhalla

Cassandra

Twisted DAV

Dev Test Live

Thursday, January 12, 12

SLIDE 46

Other Service Bindings

MySQL MemCache Solr Redis? Cassandra?

Thursday, January 12, 12

SLIDE 47

Drush Access

Custom SSH Server

drush @pantheon cc all

Thursday, January 12, 12

SLIDE 48

Drush Access

Yggdrasil DNS Custom SSH Server

drush @pantheon cc all

Thursday, January 12, 12

SLIDE 49

Magic On The Horizon

Thursday, January 12, 12

SLIDE 50

Have questions? Contact Zack Rosen zack@getpantheon.com 1-415-992-7531

Thursday, January 12, 12