How SoundCloud scales Alexander Grosse @klangberater Freitag, 5. - - PowerPoint PPT Presentation

▶
how soundcloud scales
SMART_READER_LITE
LIVE PREVIEW

How SoundCloud scales Alexander Grosse @klangberater Freitag, 5. - - PowerPoint PPT Presentation

Jul 24, 2023 224 likes •906 views

How SoundCloud scales Alexander Grosse @klangberater Freitag, 5. April 13 Youtube for Audio Freitag, 5. April 13 Fakten Freitag, 5. April 13 Fakten Alexa Ranking (233) Freitag, 5. April 13 Fakten Alexa Ranking (233)

slide-1
SLIDE 1

How SoundCloud scales

Alexander Grosse @klangberater

Freitag, 5. April 13

slide-2
SLIDE 2

“Youtube for Audio”

Freitag, 5. April 13

slide-3
SLIDE 3

Fakten

Freitag, 5. April 13

slide-4
SLIDE 4
  • Alexa Ranking (233)

Fakten

Freitag, 5. April 13

slide-5
SLIDE 5
  • Alexa Ranking (233)
  • Employees (190) - 80 engineers

Fakten

Freitag, 5. April 13

slide-6
SLIDE 6
  • Alexa Ranking (233)
  • Employees (190) - 80 engineers
  • 4 office locations (Berlin, London, San Francisco,

Sofia)

Fakten

Freitag, 5. April 13

slide-7
SLIDE 7
  • Alexa Ranking (233)
  • Employees (190) - 80 engineers
  • 4 office locations (Berlin, London, San Francisco,

Sofia)

  • >35 million registered users

Fakten

Freitag, 5. April 13

slide-8
SLIDE 8
  • Alexa Ranking (233)
  • Employees (190) - 80 engineers
  • 4 office locations (Berlin, London, San Francisco,

Sofia)

  • >35 million registered users
  • partners (BBC, CNN)

Fakten

Freitag, 5. April 13

slide-9
SLIDE 9
  • Alexa Ranking (233)
  • Employees (190) - 80 engineers
  • 4 office locations (Berlin, London, San Francisco,

Sofia)

  • >35 million registered users
  • partners (BBC, CNN)
  • > 10.000 Apps

Fakten

Freitag, 5. April 13

slide-10
SLIDE 10

Freitag, 5. April 13

slide-11
SLIDE 11

How SoundCloud scales

Freitag, 5. April 13

slide-12
SLIDE 12

?

Freitag, 5. April 13

slide-13
SLIDE 13

Organization Technology

Freitag, 5. April 13

slide-14
SLIDE 14

What is the foundation?

Freitag, 5. April 13

slide-15
SLIDE 15

Freitag, 5. April 13

slide-16
SLIDE 16

Principles

Freitag, 5. April 13

slide-17
SLIDE 17

Top-Down vs. Bottom-Up

Principles

Freitag, 5. April 13

slide-18
SLIDE 18

Top-Down vs. Bottom-Up Trust

Principles

Freitag, 5. April 13

slide-19
SLIDE 19

Top-Down vs. Bottom-Up Trust How to react to mistakes

Principles

Freitag, 5. April 13

slide-20
SLIDE 20

Top-Down vs. Bottom-Up Trust How to react to mistakes Continuous Improvement

Principles

Freitag, 5. April 13

slide-21
SLIDE 21

Top-Down vs. Bottom-Up Trust How to react to mistakes Continuous Improvement Innovation

Principles

Freitag, 5. April 13

slide-22
SLIDE 22
  • Flat Hierarchies? Everybody has them...
  • Engineering teams responsible for architecture

(not architecture teams)

  • Influence on the product

Top-Down vs. Bottom-Up

Freitag, 5. April 13

slide-23
SLIDE 23
  • Deployment
  • Change Management
  • Process
  • Time for “technical debt”

Trust

Freitag, 5. April 13

slide-24
SLIDE 24
  • What will be changed if a mistake happens?

Reaction to mistakes

Freitag, 5. April 13

slide-25
SLIDE 25
  • R&D Department?
  • Only Product?

Innovation

Freitag, 5. April 13

slide-26
SLIDE 26

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-27
SLIDE 27
  • No dedicated QA roles

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-28
SLIDE 28
  • No dedicated QA roles
  • “You build it - you run it”

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-29
SLIDE 29
  • No dedicated QA roles
  • “You build it - you run it”
  • Team Building (Rotation, constant changes)

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-30
SLIDE 30
  • No dedicated QA roles
  • “You build it - you run it”
  • Team Building (Rotation, constant changes)
  • Hacker Time

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-31
SLIDE 31
  • No dedicated QA roles
  • “You build it - you run it”
  • Team Building (Rotation, constant changes)
  • Hacker Time
  • Continuous Improvement

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-32
SLIDE 32
  • No dedicated QA roles
  • “You build it - you run it”
  • Team Building (Rotation, constant changes)
  • Hacker Time
  • Continuous Improvement
  • Move fast and break things

SoundCloud‘s basic principles

Freitag, 5. April 13

slide-33
SLIDE 33

No QA

Does that mean we don‘t test?

Freitag, 5. April 13

slide-34
SLIDE 34

“You build it - you run it”

Freitag, 5. April 13

slide-35
SLIDE 35

Flat Hierarchy

Freitag, 5. April 13

slide-36
SLIDE 36

Innovation Hacker Time

Freitag, 5. April 13

slide-37
SLIDE 37

Freitag, 5. April 13

slide-38
SLIDE 38

Continuous Improvement Retrospectives Post Mortems (Projekte)

Freitag, 5. April 13

slide-39
SLIDE 39

Mistakes...

Freitag, 5. April 13

slide-40
SLIDE 40

Scrum? Kanban? Process?

Freitag, 5. April 13

slide-41
SLIDE 41

What is the most important thing you have to do so that this works?

Freitag, 5. April 13

slide-42
SLIDE 42

Freitag, 5. April 13

slide-43
SLIDE 43

???

Freitag, 5. April 13

slide-44
SLIDE 44

??? Initial Culture

Freitag, 5. April 13

slide-45
SLIDE 45

??? Initial Culture

Hiring!!!

Freitag, 5. April 13

slide-46
SLIDE 46

Freitag, 5. April 13

slide-47
SLIDE 47

SoundCloud Hiring Process

Freitag, 5. April 13

slide-48
SLIDE 48

How does it look like in reality?

Freitag, 5. April 13

slide-49
SLIDE 49

Organisation

Freitag, 5. April 13

slide-50
SLIDE 50

HTML5 Mobile (native)

  • ther

V1 System Engineering (Berlin / SF) Search T&S Payment Partner Tools Data Activ. Delivery Platform API

Freitag, 5. April 13

slide-51
SLIDE 51

How do we structure teams? cross functional?

Freitag, 5. April 13

slide-52
SLIDE 52

Do teams stay the same? Change!

Freitag, 5. April 13

slide-53
SLIDE 53

Technology

Freitag, 5. April 13

slide-54
SLIDE 54

Ruby/Scala/Go/Clojure/ Javascript

Freitag, 5. April 13

slide-55
SLIDE 55

Scale Horizontally/ Vertically?

Freitag, 5. April 13

slide-56
SLIDE 56

Rails->Mysql RabbitMQ->Cassandra

http://backstage.soundcloud.com/2012/08/ evolution-of-soundclouds-architecture/

Freitag, 5. April 13

slide-57
SLIDE 57
  • 7 Master, 9 slaves, 900GB data
  • Additional slaves in EC2
  • Master Peak is 15K IOPS writes, 45k reads
  • one slave lags intentionally 1 hour behind
  • Je nach Konsistenzanforderung wird Slave oder

Master gewählt (für reads - unterstützt read-only mode)

MySQL

Freitag, 5. April 13

slide-58
SLIDE 58
  • 3 Cluster
  • 88 Nodes
  • 15 TB of data
  • up to 70K reads/sec

Cassandra

Freitag, 5. April 13

slide-59
SLIDE 59

Hadoop Elastic Search

Proprietary Spam Detection

Freitag, 5. April 13

slide-60
SLIDE 60

ElasticSearch

80 Mio Docs (gaining 1 Mio every 6 days) 175 QPS (gaining 15 QPS every month)

Freitag, 5. April 13

slide-61
SLIDE 61

Hosted in S3/EC2 and physical DCs

Freitag, 5. April 13

slide-62
SLIDE 62

Outages

Freitag, 5. April 13

slide-63
SLIDE 63

Freitag, 5. April 13

slide-64
SLIDE 64

Post Mortems

Freitag, 5. April 13

slide-65
SLIDE 65
  • Move fast and break things
  • Don‘t get slow because you fear to make mistakes

Summary

Freitag, 5. April 13

slide-66
SLIDE 66

backstage.soundcloud.com

Freitag, 5. April 13

slide-67
SLIDE 67

Alexander Grosse alexandergrosse.com @klangberater We are hiring!

www.soundcloud.com/jobs

Freitag, 5. April 13