SLIDE 1
Andrew Godwin
Hi, I'm
Django core developer Senior Software Engineer at Used to complain about migrations a lot
Andrew Godwin Django core developer Senior Software Engineer at - - PowerPoint PPT Presentation
Andrew Godwin Django core developer Senior Software Engineer at - - PowerPoint PPT Presentation
Hi, I'm Andrew Godwin Django core developer Senior Software Engineer at Used to complain about migrations a lot Distributed Systems c = 299,792,458 m/s Early CPUs c = 60m propagation distance 5 MHz ~2cm Clock Modern CPUs c = 10cm
SLIDE 2
SLIDE 3
Distributed Systems
SLIDE 4
c = 299,792,458 m/s
SLIDE 5
Early CPUs
c = 60m propagation distance
Clock
~2cm
5 MHz
SLIDE 6
Modern CPUs
c = 10cm propagation distance 3 GHz
SLIDE 7
Distributed systems are made of independent components
SLIDE 8
They are slower and harder to write than synchronous systems
SLIDE 9
But they can be scaled up much, much further
SLIDE 10
Trade-offs
SLIDE 11
There is never a perfect solution.
SLIDE 12
Fast Good Cheap
SLIDE 13
SLIDE 14
Load Balancer WSGI Worker WSGI Worker WSGI Worker
SLIDE 15
Load Balancer WSGI Worker WSGI Worker WSGI Worker Cache
SLIDE 16
Load Balancer WSGI Worker WSGI Worker WSGI Worker Cache Cache Cache
SLIDE 17
Load Balancer WSGI Worker WSGI Worker WSGI Worker Database
SLIDE 18
CAP Theorem
SLIDE 19
Partition Tolerant Consistent Available
SLIDE 20
PostgreSQL: CP
Consistent everywhere Handles network latency/drops Can't write if main server is down
SLIDE 21
Cassandra: AP
Can read/write to any node Handles network latency/drops Data can be inconsistent
SLIDE 22
It's hard to design a product that might be inconsistent
SLIDE 23
But if you take the tradeoff, scaling is easy
SLIDE 24
Otherwise, you must find
- ther solutions
SLIDE 25
Read Replicas
(often called master/slave)
Load Balancer WSGI Worker WSGI Worker WSGI Worker Replica Replica Main
SLIDE 26
Replicas scale reads forever... But writes must go to one place
SLIDE 27
If a request writes to a table it must be pinned there, so later reads do not get old data
SLIDE 28
When your write load is too high, you must then shard
SLIDE 29
Vertical Sharding
Users Tickets Events Payments
SLIDE 30
Horizontal Sharding
Users 0 - 2 Users 3 - 5 Users 6 - 8 Users 9 - A
SLIDE 31
Both
Users 0 - 2 Users 3 - 5 Users 6 - 8 Users 9 - A Events 0 - 2 Events 3 - 5 Events 6 - 8 Events 9 - A Tickets 0 - 2 Tickets 3 - 5 Tickets 6 - 8 Tickets 9 - A
SLIDE 32
Both plus caching
Users 0 - 2 Users 3 - 5 Users 6 - 8 Users 9 - A Events 0 - 2 Events 3 - 5 Events 6 - 8 Events 9 - A Tickets 0 - 2 Tickets 3 - 5 Tickets 6 - 8 Tickets 9 - A User Cache Event Cache Ticket Cache
SLIDE 33
Teams have to scale too; nobody should have to understand eveything in a big system.
SLIDE 34
Services allow complexity to be reduced - for a tradeoff
- f speed
SLIDE 35
Users 0 - 2 Users 3 - 5 Users 6 - 8 Users 9 - A Events 0 - 2 Events 3 - 5 Events 6 - 8 Events 9 - A Tickets 0 - 2 Tickets 3 - 5 Tickets 6 - 8 Tickets 9 - A User Cache Event Cache Ticket Cache
User Service Event Service Ticket Service
SLIDE 36
User Service Event Service Ticket Service WSGI Server
SLIDE 37
Each service is its own, smaller project, managed and scaled separately.
SLIDE 38
But how do you communicate between them?
SLIDE 39
Service 2 Service 3 Service 1
Direct Communication
SLIDE 40
Service 2 Service 3 Service 1 Service 4 Service 5
SLIDE 41
Service 2 Service 3 Service 1 Service 4 Service 5 Service 6 Service 7 Service 8
SLIDE 42
Service 2 Service 3 Service 1
Message Bus
Service 2 Service 3 Service 1
SLIDE 43
A single point of failure is not always bad - if the alternative is multiple, fragile ones
SLIDE 44
Channels and ASGI provide a standard message bus built with certain tradeoffs
SLIDE 45
Backing Store
e.g. Redis, RabbitMQ
ASGI (Channel Layer) Channels Library Django
Django Channels Project
SLIDE 46
Backing Store
e.g. Redis, RabbitMQ
ASGI (Channel Layer) Pure Python
SLIDE 47
Failure Mode At most once
Messages either do not arrive, or arrive once.
At least once
Messages arrive once, or arrive multiple times
SLIDE 48
Guarantees vs. Latency Low latency
Messages arrive very quickly but go missing more
Low loss rate
Messages are almost never lost but arrive slower
SLIDE 49
Queuing Type First In First Out
Consistent performance for all users
First In Last Out
Hides backlogs but makes them worse
SLIDE 50
Queue Sizing Finite Queues
Sending can fail
Infinite queues
Makes problems even worse
SLIDE 51
You must understand what you are making
(This is surprisingly uncommon)
SLIDE 52
Design as much as possible around shared-nothing
SLIDE 53
Per-machine caches On-demand thumbnailing Signed cookie sessions
SLIDE 54
Has to be shared? Try to split it
SLIDE 55
Has to be shared? Try sharding it.
SLIDE 56
Django's job is to be slowly replaced by your code
SLIDE 57
Just make sure you match the API contract of what you're replacing!
SLIDE 58
Don't try to scale too early; you'll pick the wrong tradeoffs.
SLIDE 59
Thanks.
Andrew Godwin
@andrewgodwin
channels.readthedocs.io