%w(map reduce).first A Tale About Rabbits, Latency and Slim - - PowerPoint PPT Presentation

▶
w map reduce first a tale about rabbits latency and slim
SMART_READER_LITE
LIVE PREVIEW

%w(map reduce).first A Tale About Rabbits, Latency and Slim - - PowerPoint PPT Presentation

Jan 08, 2024 335 likes •780 views

%w(map reduce).first A Tale About Rabbits, Latency and Slim Crontabs Paolo Negri thanks to: www.autoscout24.de Summary: Map http://www.matthiasdittrich.com/projekte/dliste/visualisations/index.html rabbitMQ

slide-1
SLIDE 1

%w(map reduce).first A Tale About Rabbits, Latency and Slim Crontabs

Paolo Negri

thanks to: www.autoscout24.de

slide-2
SLIDE 2

Summary:

slide-3
SLIDE 3

Map

http://www.matthiasdittrich.com/projekte/dliste/visualisations/index.html

slide-4
SLIDE 4

http://www.flickr.com/photos/myxi/448253580

rabbitMQ

slide-5
SLIDE 5

crontab diet

http://www.flickr.com/photos/tim_norris/2600843131/

slide-6
SLIDE 6

Map Reduce

“Programming model for processing and generating large data sets” (Google)

slide-7
SLIDE 7

Map Reduce "Map" step

the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)

slide-8
SLIDE 8

The problem Invoicing our clients

slide-9
SLIDE 9

Is it as simple as...

clients.map do |client| client.invoice end

slide-10
SLIDE 10

No!

  • distributed
  • concurrent

Because the process is:

slide-11
SLIDE 11

Problems:

  • How many nodes?
  • How many workers?
  • Distribution mechanism to

feed the workers?

slide-12
SLIDE 12

What about queuing?

  • the master node takes the input, chops

it up into smaller sub-problems, and publishes them in a queue

  • workers independently consume the

content of the queue

slide-13
SLIDE 13

Here comes

  • RabbitMQ is an implementation of AMQP,

the emerging standard for high performance enterprise messaging

  • It’s opensource
  • Can be used to manage queues
  • Written in Erlang
slide-14
SLIDE 14

Erlang?

  • Erlang is a general-purpose concurrent

programming language designed by Ericsson

  • distributed
  • fault tolerant
  • soft real time
  • high availability
slide-15
SLIDE 15

Install it

  • sudo apt-get install rabbitmq
  • sudo gem install !""#$%"&'
slide-16
SLIDE 16

Do it - master node

slide-17
SLIDE 17

Use it - worker node

slide-18
SLIDE 18

What and where

RabbitMQ (Erlang) TCP/IP Master (ruby) Worker (ruby) Worker (ruby)

slide-19
SLIDE 19

Get for free

  • Decoupling master/worker
  • Workers take care of feeding themselves
  • Flexible number of workers
slide-20
SLIDE 20

Get for free

  • RabbitMQ can be clustered
  • Support of message acknowledgement
  • Queues can be persisted on disk

(at a price)

  • low latency
slide-21
SLIDE 21

Queue

  • Is an actual entity
  • has a name
  • can be inspected and managed
slide-22
SLIDE 22

EventMachine

slide-23
SLIDE 23

EventMachine

  • Non blocking IO and lightweight

concurrency

  • eliminate the complexities of high-

performance threaded network programming Is an implementation of Reactor Pattern

slide-24
SLIDE 24

EventMachine

slide-25
SLIDE 25

EventMachine

amqp gem is built on EventMachine => you’re in a context where you can leverage concurrent programming

slide-26
SLIDE 26

EM - Deferrables

slide-27
SLIDE 27

EM - Deferrables

“The Deferrable pattern allows you to specify

any number of Ruby code blocks that will be executed at some future time when the status

  • f the Deferrable object changes “
slide-28
SLIDE 28

EM - Deferrables

slide-29
SLIDE 29

EM - Deferrables

slide-30
SLIDE 30

Deferrables

ClientStat Arrears

without deferrables with deferrables Time

ClientStat Arrears

slide-31
SLIDE 31

Achieved so far

  • Easy distribution of tasks
  • Architecture that supports arbitrary

number of workers (and masters)

  • Concurrency within the single worker
slide-32
SLIDE 32

More rabbits

Analogy with email system

slide-33
SLIDE 33

Multicasting - producer

slide-34
SLIDE 34

Multicasting - consumer

slide-35
SLIDE 35

Multicasting

Exchange msg A Queue1 Queue3 Queue2 Cons1 Cons2 Cons3 Publisher

slide-36
SLIDE 36

Multicasting

Exchange Queue1 Queue3 Queue2 msg A msg A msg A Cons1 Cons2 Cons3 Publisher

slide-37
SLIDE 37

Not only queues then

  • communication across hosts,

heterogeneous systems

  • low latency
  • clustering

Use messages distribution to build the nervous system of your app

slide-38
SLIDE 38

Where to start? crontab -l

5 * * * * bin/do_the_quick_thing.rb 0 2 * * * bin/do_the_scary_thing.rb

slide-39
SLIDE 39

Cron

  • Simple
  • Reliable
  • No maintenance
  • Status is not explicit
  • Locking?
  • Shot and forget
slide-40
SLIDE 40

Queue

  • Distributed easily
  • Reliable
  • Can be inspected
  • Add/decrease workers
  • Makes you think!
  • Adds more complexity
slide-41
SLIDE 41

On github - Projects

  • eventmachine/eventmachine
  • tmm1/amqp
  • macournoyer/thin
  • famoseagle/carrot
  • celldee/bunny
  • ezmobius/nanite
slide-42
SLIDE 42

Q&A

?

slide-43
SLIDE 43

Thanks!

Paolo Negri / hungryblank