%w(map reduce).first A Tale About Rabbits, Latency and Slim - PowerPoint PPT Presentation

%w(map reduce).first A Tale About Rabbits, Latency and Slim Crontabs Paolo Negri thanks to: www.autoscout24.de

Summary:

Map http://www.matthiasdittrich.com/projekte/dliste/visualisations/index.html

rabbitMQ http://www.flickr.com/photos/myxi/448253580

crontab diet http://www.flickr.com/photos/tim_norris/2600843131/

Map Reduce “Programming model for processing and generating large data sets” (Google)

Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)

The problem Invoicing our clients

Is it as simple as... clients.map do |client| client.invoice end

No! Because the process is: • distributed • concurrent

Problems: • How many nodes? • How many workers? • Distribution mechanism to feed the workers?

What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue

Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Can be used to manage queues • Written in Erlang

Erlang? • Erlang is a general-purpose concurrent programming language designed by Ericsson • distributed • fault tolerant • soft real time • high availability

Install it • sudo apt-get install rabbitmq • sudo gem install !""#$%"&'

Do it - master node

Use it - worker node

What and where Master (ruby) RabbitMQ Worker TCP/IP (Erlang) (ruby) Worker (ruby)

Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers

Get for free • RabbitMQ can be clustered • Support of message acknowledgement • Queues can be persisted on disk (at a price) • low latency

Queue • Is an actual entity • has a name • can be inspected and managed

EventMachine

EventMachine Is an implementation of Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming

EventMachine

EventMachine amqp gem is built on EventMachine => you’re in a context where you can leverage concurrent programming

EM - Deferrables

EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “

EM - Deferrables

EM - Deferrables

Deferrables without deferrables with deferrables ClientStat ClientStat Arrears Arrears Time

Achieved so far • Easy distribution of tasks • Architecture that supports arbitrary number of workers (and masters) • Concurrency within the single worker

More rabbits Analogy with email system

Multicasting - producer

Multicasting - consumer

Multicasting Queue 1 Cons 1 Publisher Queue 2 Cons 2 Exchange msg A Cons 3 Queue3

Multicasting Queue 1 Cons 1 msg A Publisher Queue 2 Cons 2 Exchange msg A Cons 3 Queue3 msg A

Not only queues then Use messages distribution to build the nervous system of your app • communication across hosts, heterogeneous systems • low latency • clustering

Where to start? crontab -l 5 * * * * bin/do_the_quick_thing.rb 0 2 * * * bin/do_the_scary_thing.rb

Cron • Simple • Reliable • No maintenance • Status is not explicit • Locking? • Shot and forget

Queue • Distributed easily • Reliable • Can be inspected • Add/decrease workers • Makes you think! • Adds more complexity

On github - Projects • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin • famoseagle/carrot • celldee/bunny • ezmobius/nanite

Q&A ?

Thanks! Paolo Negri / hungryblank