LiveJournal: Behind The Scenes Scaling Storytime June 2007 USENIX - - PowerPoint PPT Presentation
LiveJournal: Behind The Scenes Scaling Storytime June 2007 USENIX Brad Fitzpatrick brad@danga.com danga.com / livejournal.com / sixapart.com This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To
http://danga.com/words/
June 2007 USENIX
This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/1.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.
1
http://danga.com/words/
Refer to previous presentations for more
http://danga.com/words/
Questions anytime! Yell. Interrupt. Part 0:
Part I:
Part II:
2
http://danga.com/words/
User DB Cluster 1 uc1a uc1b User DB Cluster 2 uc2a uc2b User DB Cluster 3 uc3a uc3b User DB Cluster N ucNa ucNb Job Queues (xN) jqNa jqNb Memcached
mc4 mc3 mc2 mcN ... mc1
mod_perl
web4 web3 web2 webN ... web1
BIG-IP
bigip2 bigip1
perlbal (httpd/proxy)
proxy4 proxy3 proxy2 proxy5 proxy1
Global Database
slave1 master_a master_b slave2 ... slave5
MogileFS Database
mog_a mog_b
Mogile Trackers
tracker3 tracker1
Mogile Storage Nodes
... sto2 sto8 sto1
djabberd
djabberd djabberd
gearmand
gearmand1 gearmandN
“workers”
gearwrkN theschwkN slave1 slaveN
3
http://danga.com/words/
college hobby project, Apr 1999 4-in-1:
10M+ accounts Open Source!
4
http://danga.com/words/
memcached
− distributed caching
MogileFS
− distributed filesystem
Perlbal
− HTTP load balancer, web
gearman
− LB/HA/coalescing low-
TheSchwartz
− reliable, async job
djabberd
− the super-extensible
..... OpenID federated identity
(all production, open source)
5
http://danga.com/words/
NIH? (Not Invented Here?) Are we reinventing the wheel?
6
http://danga.com/words/
We build wheels.
7
http://danga.com/words/
We build wheels.
7
http://danga.com/words/
We build wheels.
7
http://danga.com/words/
We build wheels.
(yes, arguably tires. sshh..)
7
http://danga.com/words/
8
http://danga.com/words/
1 server to hundreds... you can do all this with just 1 server!
9
http://danga.com/words/
Scaling:
Fast still matters,
that’s some good money
10
http://danga.com/words/
“Cluster”
load balancing (LB), high availablility (HA)
Load Balancing? High Availability? Venn Diagram time!
11
http://danga.com/words/
12
http://danga.com/words/
13
http://danga.com/words/
14
http://danga.com/words/
Simple:
mysql apache
15
http://danga.com/words/
mysql apache
16
http://danga.com/words/
Two single points of failure! No hot or cold spares Site gets slow again.
17
http://danga.com/words/
3 webs, 1 db Now we need to load-balance!
LVS, mod_backhand, whackamole, BIG-IP,
18
http://danga.com/words/
Now I/O bound... ... how to use another database?
19
http://danga.com/words/
We buy a new DB MySQL replication Writes to DB (master) Reads from both
20
http://danga.com/words/
21
http://danga.com/words/
mod_perl
web4 web3 web2 web12 ... web1
BIG-IP
bigip2 bigip1
mod_proxy
proxy3 proxy2 proxy1
Global Database
slave1 slave2 ... slave6 master
22
http://danga.com/words/
“This don't scale...”
DB master is SPOF Adding slaves doesn't scale
− only spreads reads, not writes!
23
http://danga.com/words/
databases eventual only writing
400 write/s 3 reads/s 400 write/s 3 r/s 400 write/s 3 reads/s 400 write/s 3 r/s 400 write/s 3 reads/s 400 write/s 3 r/s 400 write/s 3 reads/s 400 write/s 3 r/s 400 write/s 3 reads/s 400 write/s 3 r/s 400 write/s 3 reads/s 400 write/s 3 r/s 400 write/s 3 reads/s 400 write/s 3 r/s
24
http://danga.com/words/
Our database machines already did RAID We did backups So why put user data on 6+ slave machines?
25
http://danga.com/words/
Spread your databases out, into “roles”
− roles that you never need to join between
different users or accept you'll have to join in app Each user assigned to a numbered HA cluster Each cluster has multiple machines
− writes self-contained in cluster (writing to 2-3 machines, not
26
http://danga.com/words/
27
http://danga.com/words/
SELECT userid, clusterid FROM user WHERE user='bob'
27
http://danga.com/words/
SELECT userid, clusterid FROM user WHERE user='bob' userid: 839 clusterid: 2
27
http://danga.com/words/
SELECT userid, clusterid FROM user WHERE user='bob' userid: 839 clusterid: 2 SELECT .... FROM ... WHERE userid=839 ...
27
http://danga.com/words/
SELECT userid, clusterid FROM user WHERE user='bob' userid: 839 clusterid: 2 SELECT .... FROM ... WHERE userid=839 ... OMG i like totally hate my parents they just dont understand me and i h8 the world omg lol rofl *! :^- ^^; add me as a friend!!!
27
http://danga.com/words/
per-user numberspaces
Can move/upgrade users 1-at-a-time:
job server that coordinates, distributed long-
28
http://danga.com/words/
Turn pair of InnoDB machines into a cluster
One machine at a time mounting fs, running MySQL Heartbeat to move IP, {un,}mount filesystem, {stop,start}
filesystem repairs, innodb repairs, don’t lose any committed transactions. No special schema considerations MySQL 4.1 w/ binlog sync/flush options
29
http://danga.com/words/
Linux block device driver
− “Network RAID 1” − Shared storage without sharing! − sits atop another block device − syncs w/ another machine's
cross-over gigabit cable
InnoDB on DRBD: HA MySQL!
− can hang slaves off HA pair, − and/or, − HA pair can be slave of a
30
http://danga.com/words/
No magic bullet...
doesn’t scale with writes
special schemas
only HA, not LB
special-purpose
lots of options!
31
http://danga.com/words/
32
http://danga.com/words/
caching's key to performance
Where to cache?
memory waste (address space per apache child)
limited to single machine, same with Java/C#/
flushed per update, small max size
fixed length rows, small max size
33
http://danga.com/words/
our Open Source, distributed caching system implements a dictionary ADT, with network API run instances wherever free memory two-level hash
no “master node”, nodes aren’t aware of each
protocol simple, XML-free
popular, fast scalable
34
http://danga.com/words/
set, add, replace delete incr, decr
35
http://danga.com/words/
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
36
http://danga.com/words/
1GB
2GB
1GB
GET foo
36
http://danga.com/words/
1GB
2GB
1GB
GET foo
(response)
36
http://danga.com/words/
Up to client how to pick a memcached node Traditional way:
“Consistent hashing”
37
http://danga.com/words/
libevent
event-based, non-blocking design
slab allocator referenced counted objects
LRU all internal operations O(1)
38
http://danga.com/words/
39
http://danga.com/words/
BIG-IP, Alteon, Juniper, Foundry
− good for L4 or minimal L7 − not tricky / fun enough. :-)
Tried a dozen reverse proxies
− none did what we wanted or were fast enough
Wrote Perlbal
− fast, smart, manageable HTTP web server / reverse proxy / LB − can do internal redirects
and dozen other tricks
40
http://danga.com/words/
Perl parts optionally in C with plugins single threaded, async event-based
console / HTTP remote management
handles dead nodes, smart balancing multiple modes
plug-ins
41
http://danga.com/words/
42
http://danga.com/words/
perlbal to backends (mod_perls)
no complex load balancing logic: just use whatever's free.
clients persistent; not tied to a specific backend
42
http://danga.com/words/
perlbal to backends (mod_perls)
no complex load balancing logic: just use whatever's free.
clients persistent; not tied to a specific backend
42
http://danga.com/words/
perlbal to backends (mod_perls)
no complex load balancing logic: just use whatever's free.
clients persistent; not tied to a specific backend
42
http://danga.com/words/
perlbal to backends (mod_perls)
no complex load balancing logic: just use whatever's free.
clients persistent; not tied to a specific backend
42
http://danga.com/words/
connects to backends are often fast, but...
are you talking to the kernel’s listen queue? or apache? (did apache accept() yet?)
send OPTIONs request to see if apache is
Huge improvement to user-visible latency!
(and more fair/even load balancing)
#include <sys/socket.h> int listen(int sockfd, int backlog);
43
http://danga.com/words/
high, normal, low priority queues paid users -> high queue bots/spiders/suspect traffic -> low queue
44
http://danga.com/words/
large file serving w/ mod_perl bad...
45
http://danga.com/words/
internal redirects
either from disk, or from other URL(s)
one, clean URL mod_perl does auth, and is done. perlbal serves.
46
http://danga.com/words/
47
http://danga.com/words/
Now Perlbal can buffer uploads as well..
LifeBlog uploading
LiveJournal/Friendster photo uploads
on any of: rate, size, time blast at backend, only when full request is in
48
http://danga.com/words/
based on palette indexes, colors in URL,
49
http://danga.com/words/
50
http://danga.com/words/
51
http://danga.com/words/
our distributed file system open source userspace
based all around HTTP (NFS support now removed)
hardly unique
production-quality
52
http://danga.com/words/
alternatives at time were either:
new/uncommon/ unstudied on-disk formats because it was easy
53
http://danga.com/words/
files belong to classes,
tracks what disks files
keep replicas on devices
big, cheap disks
54
http://danga.com/words/
clients mogilefsd (does all real work) database(s) (MySQL, .... abstract) storage nodes
55
http://danga.com/words/
tiny text-based protocol Libraries available for:
tied filehandles MogileFS::Client
− my $fh = $mogc->new_file(“key”, [[$class], ...])
clients don't do database access
56
http://danga.com/words/
The Meat event-based message bus load balances client requests, world info process manager
Child processes:
interfaces client protocol w/ db(s), etc
57
http://danga.com/words/
Abstract as of Mogile 2.x
− MySQL − SQLite (joke/demo) − Pg/Oracle coming soon? − Also future:
wrapper driver, partitioning any above
− small metadata in one driver (MySQL Cluster?), − large tables partitioned over 2-node HA pairs
Recommend config:
− 2xMySQL InnoDB on DRBD − 2 slaves underneath HA VIP
1 for backups read-only slave for during master failover window 58
http://danga.com/words/
HTTP transport
− GET − PUT − DELETE
mogstored listens on 2 ports... HTTP. --server={perlbal,lighttpd,...} configs/manages your webserver of choice. perlbal is default. some people like apache, etc
− management/status:
iostat interface, AIO control, multi-stat() (for faster
files on filesystem, not DB
− sendfile()! future: splice() − filesystem can be any filesystem
59
http://danga.com/words/
60
http://danga.com/words/
60
http://danga.com/words/
60
http://danga.com/words/
61
http://danga.com/words/
62
http://danga.com/words/
63
http://danga.com/words/
system to load balance function calls... scatter/gather bunch of calls in parallel, different languages, db connection pooling, spread CPU usage around your network, keep heavy libraries out of caller code, ... ...
64
http://danga.com/words/
gearmand
workers.
clients
65
http://danga.com/words/
66
http://danga.com/words/
66
http://danga.com/words/
66
http://danga.com/words/
can_do(“funcA”) can_do(“funcA”) can_do(“funcB”)
66
http://danga.com/words/
can_do(“funcA”) can_do(“funcA”) can_do(“funcB”)
66
http://danga.com/words/
can_do(“funcA”) can_do(“funcA”) can_do(“funcB”)
66
http://danga.com/words/
can_do(“funcA”) can_do(“funcA”) can_do(“funcB”)
66
http://danga.com/words/
can_do(“funcA”) can_do(“funcA”) can_do(“funcB”)
66
http://danga.com/words/
efficient binary protocol No XML but also line-based text protocol for admin
67
http://danga.com/words/
Image::Magick outside of your mod_perls! DBI connection pooling (DBD::Gofer +
reducing load, improving visibility “services”
68
http://danga.com/words/
running code in parallel
running blocking code from event loops
spreading CPU from ev loop daemons calling between different languages, ...
69
http://danga.com/words/
Guarantees:
please wait for your results. if client goes away, no promises
but server will notify client(s) if working worker
No policy/conventions in gearmand
...
70
http://danga.com/words/
Don’t actually use it like this... but:
use strict; use DMap qw(dmap); DMap->set_job_servers("sammy", "papag"); my @foo = dmap { "$_ = " . `hostname` } (1..10); print "dmap says:\n @foo"; $ ./dmap.pl dmap says: 1 = sammy 2 = papag 3 = sammy 4 = papag 5 = sammy 6 = papag 7 = sammy 8 = papag 9 = sammy 10 = papag
71
http://danga.com/words/
Gearman is sexy.
Check it out!
oh crap, did I leak the secret?
72
http://danga.com/words/
73
http://danga.com/words/
Like gearman:
submitting jobs workers
But unlike gearman:
− fire & forget (as opposed to gearman, where you wait for
currently library, not network service
74
http://danga.com/words/
insert job “grab” job (atomic grab)
mark job done temp fail job for future
replace job with 1+ other jobs
...
75
http://danga.com/words/
backing store:
MySQL, Postgres, SQLite, ....
but HA: you tell it @dbs, and it finds one to
76
http://danga.com/words/
outgoing email (SMTP client)
− millions of emails per day − TheSchwartz::Worker::SendEmail − Email::Send::TheSchwartz
LJ notifications
− ESN: event, subscription, notification
one event (new post, etc) -> thousands of emails, SMSes,
pinging external services atomstream injection ..... dozens of users shared farm for TypePad, Vox, LJ
77
http://danga.com/words/
gearmand: not reliable, low-latency, no disks TheSchwartz: latency, reliable, disks In TypePad:
disks, but low-latency future: no disks, SSD/Flash, MySQL Cluster
78
http://danga.com/words/
79
http://danga.com/words/
Our Jabber/XMPP server
powers our “LJ Talk” service
S2S: works with GoogleTalk, etc perl, event-based (epoll, etc) done 300,000+ conns tiny per-conn memory overhead
80
http://danga.com/words/
everything is a hook
− auth, − roster, − vcard info (avatars), − presence, − delivery, − inter-node cluster delivery,
async hooks
81
http://danga.com/words/
82
http://danga.com/words/
User DB Cluster 1 uc1a uc1b User DB Cluster 2 uc2a uc2b User DB Cluster 3 uc3a uc3b User DB Cluster N ucNa ucNb Job Queues (xN) jqNa jqNb Memcached
mc4 mc3 mc2 mcN ... mc1
mod_perl
web4 web3 web2 webN ... web1
BIG-IP
bigip2 bigip1
perlbal (httpd/proxy)
proxy4 proxy3 proxy2 proxy5 proxy1
Global Database
slave1 master_a master_b slave2 ... slave5
MogileFS Database
mog_a mog_b
Mogile Trackers
tracker3 tracker1
Mogile Storage Nodes
... sto2 sto8 sto1
djabberd
djabberd djabberd
gearmand
gearmand1 gearmandN
“workers”
gearwrkN theschwkN slave1 slaveN
83
http://danga.com/words/
if extra time
84
http://danga.com/words/
Databases depend on fsync()
fsync() almost never works work
Solution: test! & fix
client/server spew writes/fsyncs, record intentions on alive machine,
85
http://danga.com/words/
connections == threads == memory
want connection/thread distinction in MySQL! w/ max-runnable-threads tunable
max threads
DBD::Gofer + Gearman
Data::ObjectDriver + Gearman
86