Breaking the Monolith Microservice Extraction at SoundCloud - - PowerPoint PPT Presentation

▶
breaking the monolith
SMART_READER_LITE
LIVE PREVIEW

Breaking the Monolith Microservice Extraction at SoundCloud - - PowerPoint PPT Presentation

Oct 27, 2023 153 likes •717 views

Breaking the Monolith Microservice Extraction at SoundCloud Soundcloud 11 hours uploaded every minute 150 million tracks 300 million users 2 History Rails 2.3 MySQL S3 3 What happened then? 4 Success! 5

slide-1
SLIDE 1

Breaking the Monolith

Microservice Extraction at SoundCloud

slide-2
SLIDE 2

Soundcloud

  • 11 hours uploaded every minute
  • 150 million tracks
  • 300 million users

2

slide-3
SLIDE 3

History

  • Rails 2.3
  • MySQL
  • S3

3

slide-4
SLIDE 4

What happened then?

4

slide-5
SLIDE 5

Success!

5

slide-6
SLIDE 6

History

  • Rails 2.3
  • MySQL
  • AWS
  • Cassandra
  • Hadoop
  • SolR
  • RabbitMQ

https://developers.soundcloud.com/blog/evolution-of-soundclouds-architecture

6

slide-7
SLIDE 7

Still not enough

  • More servers
  • Add caching layer
  • Defer long running tasks to workers

7

slide-8
SLIDE 8

Still not enough

  • Optimize database schema
  • Introduce read slaves
  • Dedicated databases for some models

8

slide-9
SLIDE 9

Monolith

9

slide-10
SLIDE 10

Major pain points

  • Testing, building and deploying
  • Dependency hell
  • “I’d rather not touch this”

10

slide-11
SLIDE 11

Rails problems I - No service layer

11

<% Category.all.each do |cat| %> <li><%= cat.name %></li> <% end %>

slide-12
SLIDE 12

Rails problems I - No service layer

⇒ Tight coupling with storage layer!

12

<% Category.all.each do |cat| %> <li><%= cat.name %></li> <% end %>

slide-13
SLIDE 13

Rails problems II - Active Record Magic

13

class User < ActiveRecord::Base validates_length_of :username, :within => 2..64 before_save :encrypt_password, :accept_terms_of_use has_many :comments, :dependent => :destroy # ... end

slide-14
SLIDE 14

Rails problems II - Active Record Magic

⇒ Easy to write, hard to maintain

14

class User < ActiveRecord::Base validates_length_of :username, :within => 2..64 before_save :encrypt_password, :accept_terms_of_use has_many :comments, :dependent => :destroy # ... end

slide-15
SLIDE 15

Ruby Problems

  • GIL
  • Native extensions
  • Dependency management can be painful

15

slide-16
SLIDE 16
slide-17
SLIDE 17

Extract features as Services

  • Less painful to maintain
  • Easy to replace
  • Fun to build

17

slide-18
SLIDE 18

An Example: Messages

  • 200 million messages
  • MySQL database on shared host
  • Features:

○ embedded sounds ○ email notifications ○ spam detection

18

slide-19
SLIDE 19

Migration Requirements

  • New schema to support upcoming features
  • Dedicated database
  • Zero downtime

19

slide-20
SLIDE 20

Chapter 1

The Database

slide-21
SLIDE 21

Migrating the Schema

21

slide-22
SLIDE 22

Migrating the Schema

22

slide-23
SLIDE 23

Migrating the Schema

Convenient Dependency Management with @Grapes

23

#!/usr/bin/env groovy @Grapes([ @Grab(group='org.yaml', module='snakeyaml', version='1.12'), @Grab(group='mysql', module='mysql-connector-java', version='5.1.24') ]) import groovy.sql.Sql import org.yaml.snakeyaml.Yaml

slide-24
SLIDE 24

Strategy

  • Import all messages
  • Setup cron job to get new messages
  • Listen to events for updates

24

slide-25
SLIDE 25

Chapter 2

The Application

slide-26
SLIDE 26

Creating a new service

26

slide-27
SLIDE 27

Convo

  • Scala
  • Twitter Finagle
  • Scalatra Framework

27

slide-28
SLIDE 28

Convo architecture

28

slide-29
SLIDE 29

Scala

  • Functional
  • OOP
  • Static but inferred typing

29

slide-30
SLIDE 30

Scala Joy = Options I

Good bye NullPointerException

30

val opt: Option[String] = params.get("id") val id: Int = opt.map(id => id.toInt).getOrElse(10)

slide-31
SLIDE 31

Scala Joy = Options II

Safe chaining with for comprehensions

31

for { id <- params.get("id") user <- users.lookup(id) count <- counts.forUser(user) } yield count

slide-32
SLIDE 32

Scala Joy = Pattern Matching

Expressive code with decomposition

32

Urn("soundcloud:users:20") match { case Urn(_, "tracks", _) => None, case Urn(_, "messages", "20") => None, case Urn(_, "users", id) => Some(id) }

slide-33
SLIDE 33

Scala Joy = Functional Goodness

Function arguments and references

33

delete("/playlist/:urn/likes")(destroy) def destroy(request: Request) = write(request, 200)(repo.deleteLike) def write (request: Request, statusCode: Int) (f: (UserSession, Urn) => Future[Like]) = { // ... }

slide-34
SLIDE 34

Futures!

slide-35
SLIDE 35

Finagle

  • Twitter rpc library on top of Netty
  • Support for multiple protocols
  • Future composition

35

slide-36
SLIDE 36

Futures

Instance API (excerpt)

36

class Future[A] { def get(): A def map[B](f: A => B): Future[B] def flatMap[B](f: A => Future[B]]): Future[B] def onSuccess(f : A => Unit): Future[A] }

slide-37
SLIDE 37

Futures

Object API (excerpt)

37

  • bject Future {

def value[A](a: A): Future[A] def exception[A](e: Throwable): Future[A] def collect[A](fs : Seq[Future[A]]): Future[Seq[A]] }

slide-38
SLIDE 38

Futures - Examples

Multiple transformations - The ugly way

38

service.getUsers().flatMap { users => service.tracksFor(users).flatMap { tracks => asJson(tracks) } }.onSuccess(json => log(s"found $json"))

slide-39
SLIDE 39

Futures - Example

Multiple transformations - The nice way

39

val response = for { users <- service.getUsers() tracks <- service.tracksFor(users) json <- asJson(tracks) } yield json response.onSuccess(json => log(s"found $json"))

slide-40
SLIDE 40

Scala Problems

  • Implicit conversions
  • Binary compatibility of libraries
  • Tooling still not perfect

40

slide-41
SLIDE 41

SBT

slide-42
SLIDE 42

IntelliJ

  • Code inspection
  • Debugging
  • SBT support

42

slide-43
SLIDE 43

Chapter 3

The Cutover

slide-44
SLIDE 44

Integrate Service

44

slide-45
SLIDE 45

Integration Risks

  • Service failure
  • Data loss after rolling back
  • Data loss caused by stale clients

45

slide-46
SLIDE 46

Integration Risks

  • Service failure → load testing, A/B testing
  • Data loss after rolling back
  • Data loss caused by stale clients

46

slide-47
SLIDE 47

Integration Risks

  • Service failure → load testing, A/B testing
  • Data loss after rolling back → prepare scripts, practice
  • Data loss caused by stale clients

47

slide-48
SLIDE 48

Integration Risks

  • Service failure → load testing, A/B testing
  • Data loss after rolling back → prepare scripts, practice
  • Data loss caused by stale clients → keep migration running

48

slide-49
SLIDE 49

Enable Feature

49

slide-50
SLIDE 50

Retire Old Database

50

slide-51
SLIDE 51

Convo

  • 500 million requests per day
  • 1000 qps during peak time
  • 5 instances

51

slide-52
SLIDE 52

Microservice Problems

  • Event bus dependency
  • Maintenance overhead
  • Distributed tracing

52

slide-53
SLIDE 53

Microservices

→ Not a silver bullet

53

slide-54
SLIDE 54
slide-55
SLIDE 55

Questions?

slide-56
SLIDE 56

Images

  • Slide 4,7 - Rails Logo http://en.wikipedia.org/wiki/File:Ruby_on_Rails.svg
  • Slide 6,51 - Party Cat http://ghostexist.deviantart.com/art/Party-Cat-logo-287986071
  • Silde 7 - MySQL Logo http://blogwifi.fr/?p=9990
  • Slide 7 - Hadoop Cop https://svn.apache.org/repos/asf/hadoop/logos/out_rgb/hadoop-security-logo.

jpg

  • Slide 10 - Hello, My Name Is: http://commons.wikimedia.org/wiki/File:Hello_my_name_is_sticker.

svg

  • Slide 14 - Sad Panda: http://www.whatsupyasieve.com/2012/09/17/lockout-blues/sad-panda-2/
  • Slide 16 - Exit Sign: http://logo-kid.com/emergency-exit-sign-left.htm
  • Slide 22 - Groovy Logo: http://groovy.codehaus.org/images/groovy-logo-medium.png
  • Slide 20, 25, 44 - Book Page: http://daviddiazolivares.deviantart.com/art/Old-Book-Page-

345869530

  • Slide 34 - Back to the future: http://i.huffpost.com/gen/1369403/thumbs/o-BACK-TO-THE-FUTURE-

facebook.jpg

  • Slide 42 - Tommy Lee Jones: http://persephonemagazine.com/2014/04/friday-news-bites-airline-

pranks-gabriel-garcia-marquez-pulitzers-more/film-title-no-country-for-old-men/

  • Slide 55 - That’s all folks: http://www.hd2wallpapers.com/view/thats_all_folks-1280x800.php

56