scaling
play

Scaling Marty Weiner Yashh Nelapati Orodruin, Mordor The Shire - PowerPoint PPT Presentation

Jan 02, 2023 •515 likes •1.08k views

Scaling Marty Weiner Yashh Nelapati Orodruin, Mordor The Shire Friday, November 9, 12 Pinterest is . . . An online pinboard to organize and share what inspires you. Scaling Pinterest Friday, November 9, 12 Friday, November 9, 12 Friday,

  1. Scaling Marty Weiner Yashh Nelapati Orodruin, Mordor The Shire Friday, November 9, 12

  2. Pinterest is . . . An online pinboard to organize and share what inspires you. Scaling Pinterest Friday, November 9, 12

  3. Friday, November 9, 12

  4. Friday, November 9, 12

  5. Friday, November 9, 12

  6. Relationships Marty Weiner Grayskull, Eternia Scaling Pinterest Friday, November 9, 12

  7. Relationships Marty Weiner Grayskull, Eternia Yashh Nelapati Gotham City Scaling Pinterest Friday, November 9, 12

  8. Page Views / Day Mar 2010 Jan 2011 Jan 2012 Mar 2010 Jan 2011 Jan 2012 Scaling Pinterest Scaling Pinterest Friday, November 9, 12

  9. Page Views / Day · RackSpace · 1 small Web Engine Mar 2010 Jan 2011 Jan 2012 · 1 small MySQL DB · 1 Engineer Mar 2010 Jan 2011 Jan 2012 Scaling Pinterest Scaling Pinterest Friday, November 9, 12

  10. Page Views / Day Mar 2010 Jan 2011 Jan 2012 Scaling Pinterest Friday, November 9, 12

  11. Page Views / Day · Amazon EC2 + S3 + CloudFront · 1 NGinX, 4 Web Engines · 1 MySQL DB + 1 Read Slave · 1 Task Queue + 2 Task Processors · 1 MongoDB · 2 Engineers Mar 2010 Jan 2011 Jan 2012 Scaling Pinterest Friday, November 9, 12

  12. Page Views / Day Mar 2010 Jan 2011 Jan 2012 Mar 2010 Jan 2011 Jan 2012 Scaling Pinterest Scaling Pinterest Friday, November 9, 12

  13. · Amazon EC2 + S3 + CloudFront Page Views / Day · 2 NGinX, 16 Web Engines + 2 API Engines · 5 Functionally Sharded MySQL DB + 9 read slaves · 4 Cassandra Nodes · 15 Membase Nodes (3 separate clusters) · 8 Memcache Nodes · 10 Redis Nodes · 3 Task Routers + 4 Task Processors Mar 2010 Jan 2011 Jan 2012 Mar 2010 Jan 2011 Jan 2012 · 4 Elastic Search Nodes · 3 Mongo Clusters · 3 Engineers Scaling Pinterest Scaling Pinterest Friday, November 9, 12

  14. Lesson Learned #1 It will fail. Keep it simple. Scaling Pinterest Friday, November 9, 12

  15. Page Views / Day Mar 2010 Jan 2011 Jan 2012 Scaling Pinterest Friday, November 9, 12

  16. · Amazon EC2 + S3 + Akamai, ELB Page Views / Day · 90 Web Engines + 50 API Engines · 66 MySQL DBs (m1.xlarge) + 1 slave each · 59 Redis Instances · 51 Memcache Instances · 1 Redis Task Manager + 25 Task Processors · Sharded Solr Mar 2010 Jan 2011 Jan 2012 · 6 Engineers Scaling Pinterest Friday, November 9, 12

  17. Page Views / Day Mar 2010 Jan 2011 Jan 2012 Oct 2012 Scaling Pinterest Friday, November 9, 12

  18. · Amazon EC2 + S3 + Edge Cast, Akamai, Level 3 Page Views / Day · 180 Web Engines + 240 API Engines · 80 MySQL DBs (cc2.8xlarge) + 1 slave each · 110 Redis Instances · 200 Memcache Instances · 4 Redis Task Managers + 80 Task Processors · Sharded Solr Mar 2010 Jan 2011 Jan 2012 Oct 2012 · 40 Engineers Scaling Pinterest Friday, November 9, 12

  19. Why Amazon EC2/S3? · Very good reliability, reporting, and support · Very good peripherals, such as managed cache, DB, load balancing, DNS, map reduce, and more... · New instances ready in seconds Scaling Pinterest Friday, November 9, 12

  20. Why Amazon EC2/S3? · Very good reliability, reporting, and support · Very good peripherals, such as managed cache, DB, load balancing, DNS, map reduce, and more... · New instances ready in seconds · Con: Limited choice Scaling Pinterest Friday, November 9, 12

  21. Why Amazon EC2/S3? · Very good reliability, reporting, and support · Very good peripherals, such as managed cache, DB, load balancing, DNS, map reduce, and more... · New instances ready in seconds · Con: Limited choice · Pro: Limited choice Scaling Pinterest Friday, November 9, 12

  22. Why MySQL? · Extremely mature · Well known and well liked · Rarely catastrophic loss of data · Response time to request rate increases linearly · Very good software support - XtraBackup, Innotop, Maatkit · Solid active community · Very good support from Percona · Free Scaling Pinterest Friday, November 9, 12

  23. Why Memcache? · Extremely mature · Very good performance · Well known and well liked · Never crashes, and few failure modes · Free Scaling Pinterest Friday, November 9, 12

  24. Why Redis? · Variety of convenient data structures · Has persistence and replication · Well known and well liked · Consistently good performance · Few failure modes · Free Scaling Pinterest Friday, November 9, 12

  25. Clustering vs Sharding Scaling Pinterest Friday, November 9, 12

  26. Clustering · Data distributed automatically · Data can move · Rebalances to distribute capacity · Nodes communicate with each other Sharding Scaling Pinterest Friday, November 9, 12

  27. Clustering · Data distributed manually · Data does not move · Split data to distribute load · Nodes are not aware of each other Sharding Scaling Pinterest Friday, November 9, 12

  28. Why Clustering? · Examples: Cassandra, MemBase, HBase · Automatically scale your datastore · Easy to set up · Spatially distribute and colocate your data · High availability · Load balancing · No single point of failure Scaling Pinterest Friday, November 9, 12

  29. What could possibly go wrong? source: thereifixedit.com Scaling Pinterest Friday, November 9, 12

  30. Why Not Clustering? · Still fairly young · Fundamentally complicated · Less community support · Fewer engineers with working knowledge · Di ffi cult and scary upgrade mechanisms · And, yes, there is a single point of failure. A BIG one. Scaling Pinterest Friday, November 9, 12

  31. Clustering Single Point of Failure Scaling Pinterest Friday, November 9, 12

  32. Clustering Single Point of Failure Scaling Pinterest Friday, November 9, 12

  33. Clustering Single Point of Failure Scaling Pinterest Friday, November 9, 12

  34. Clustering Single Point of Failure Scaling Pinterest Friday, November 9, 12

  35. Clustering Single Point of Failure Cluster Management Algorithm Scaling Pinterest Friday, November 9, 12

  36. Cluster Manager · Same complex code replicated over all nodes · Failure modes: · Data rebalance breaks · Data corruption across all nodes · Improper balancing that cannot be fixed (easily) · Data authority failure Scaling Pinterest Friday, November 9, 12

  37. Lesson Learned #2 Clustering is scary. Scaling Pinterest Friday, November 9, 12

  38. Why Sharding? · Can split your databases to add more capacity · Spatially distribute and colocate your data · High availability · Load balancing · Algorithm for placing data is very simple · ID generation is simplistic Scaling Pinterest Friday, November 9, 12

  39. When to shard? · Sharding makes schema design harder · Waiting too long makes the transition harder · Solidify site design and backend architecture · Remove all joins and complex queries, add cache · Functionally shard as much as possible · Still growing? Shard. Scaling Pinterest Friday, November 9, 12

  40. Our Transition 1 DB + Foreign Keys + Joins 1 DB + Denormalized + Cache 1 DB + Read slaves + Cache Several functionally sharded DBs + Read slaves + Cache ID sharded DBs + Backup slaves + Cache Scaling Pinterest Friday, November 9, 12

  41. Watch out for... · Cannot perform most JOINS · No transaction capabilities · Extra e ff ort to maintain unique constraints · Schema changes requires more planning · Reports require running same query on all shards Scaling Pinterest Friday, November 9, 12

  42. How we sharded Scaling Pinterest Friday, November 9, 12

  43. Sharded Server Topology db00001 db00513 db03072 db03584 db00002 db00514 db03073 db03585 ....... ....... ....... ....... db00512 db01024 db03583 db04096 Initially, 8 physical servers, each with 512 DBs Scaling Pinterest Friday, November 9, 12

  44. High Availability db00001 db00513 db03072 db03584 db00002 db00514 db03073 db03585 ....... ....... ....... ....... db00512 db01024 db03583 db04096 Multi Master replication Scaling Pinterest Friday, November 9, 12

  45. Increased load on DB? db00001 db00002 ....... db00256 db00001 db00002 ....... db00257 db00512 db00258 ....... db00512 To increase capacity, a server is replicated and the new replica becomes responsible for some DBs Scaling Pinterest Friday, November 9, 12

  46. ID Structure 64 bits Type Shard ID Local ID · A lookup data structure has physical server to shard ID range (cached by each app server process) · Shard ID denotes which shard · Type denotes object type (e.g., pins) · Local ID denotes position in table Scaling Pinterest Friday, November 9, 12

  47. Lookup Structure {“sharddb001a”: ( 1, 512), “sharddb002b”: ( 513, 1024), “sharddb003a”: (1025, 1536), ... “sharddb008b”: (3585, 4096)} sharddb003a DB01025 users users ser-data 1 ser-data user_has_boards 2 boards ser-data 3 Scaling Pinterest Friday, November 9, 12

  48. ID Structure · New users are randomly distributed across shards · Boards, pins, etc. try to be collocated with user · Local ID’s are assigned by auto-increment · Enough ID space for 65536 shards, but only first 4096 opened initially. Can expand horizontally. Scaling Pinterest Friday, November 9, 12

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms

Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms

Rafael Oliveira University of Toronto Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms Three Step Analysis Generalization One More Application of Scaling Non-Negative Matrices &

604 views • 15 slides
Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent

Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent

Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent publications Insight Series Tool Animation The rationale for scaling As the SDGs require transformational change, scaling can provide: Reaching more

609 views • 37 slides
ETI se Scaling for many organizations Hierarchy Recursive vs T.to eRaokon traverses the

ETI se Scaling for many organizations Hierarchy Recursive vs T.to eRaokon traverses the

Discovery Naming or IN 5 Why name 172.217 11.14 or google.com Indirection Security Isafety concerns How name iEE f google.com Et Challenge Scaling Scaling for many clients Caching D y D ETI se Scaling for many organizations Hierarchy

215 views • 4 slides
Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large

Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large

Scaling Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Scaling-at-large Principles of Complex Systems Allometry Allometry Definitions Definitions Course 300, Fall, 2008 Examples Examples

568 views • 27 slides
So#ware Scaling Mo/va/on & Goals HW Configura/on & Scale Out So#ware Scaling

So#ware Scaling Mo/va/on & Goals HW Configura/on & Scale Out So#ware Scaling

So#ware Scaling Mo/va/on & Goals HW Configura/on & Scale Out So#ware Scaling Efforts System management Opera/ng system Programming environment PreAcceptance Work HW stabiliza/on & early scaling

390 views • 19 slides
UMBC A B M A L T F O U M B C I M Y O R T 1 (10/18/04) I E S R C E O V

UMBC A B M A L T F O U M B C I M Y O R T 1 (10/18/04) I E S R C E O V

Advanced VLSI Design Scaling CMPE 640 Scaling Technology scaling rate is approximately 13%/year, halving every 5 years. 100 10 m 1 0.1 1960 1970 1980 1990 2000 The size of the circuits also continues to increase. Besides increasing

362 views • 7 slides
Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of

Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of

Scaling Scaling Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of Complex Systems Allometry Allometry CSYS/MATH 300, Fall, 2010 Definitions Definitions Examples Examples History: Metabolism

250 views • 20 slides
Outline Introduction What are the cusp anomaly and generalized scaling function? Superstring in

Outline Introduction What are the cusp anomaly and generalized scaling function? Superstring in

Scaling and integrability from AdS 5 S 5 Riccardo Ricci Imperial College London DAMTP Cambridge 14th October Work in collaboration with S. Giombi, R.Roiban, A. Tseytlin and C.Vergu Scaling and integrability from AdS 5 S 5 Riccardo Ricci

893 views • 50 slides
Communication Avoiding Power Scaling Power Scaling Derivatives of Algorithmic Communication

Communication Avoiding Power Scaling Power Scaling Derivatives of Algorithmic Communication

Communication Avoiding Power Scaling Power Scaling Derivatives of Algorithmic Communication Complexity John D. Leidel, Yong Chen Parallel Programming Models & Systems for High End Computing (P2S2 2015) Sept 1, 2015 1 Overview Intro:

776 views • 26 slides
Using EBS with Auto Scaling Groups How to use the immense power of AWS Auto-Scaling Groups for a

Using EBS with Auto Scaling Groups How to use the immense power of AWS Auto-Scaling Groups for a

Using EBS with Auto Scaling Groups How to use the immense power of AWS Auto-Scaling Groups for a stateful Docker application. Background Background In a service-oriented world where requests can come from anywhere at any time, keeping a

1.04k views • 31 slides
Scaling and Universality in Probability Francesco Caravenna Universit` a degli Studi di

Scaling and Universality in Probability Francesco Caravenna Universit` a degli Studi di

Weak Convergence Brownian Motion A glimpse of SLE Scaling Limits with Disorder Scaling and Universality in Probability Francesco Caravenna Universit` a degli Studi di Milano-Bicocca Luxembourg June 14, 2016 Francesco Caravenna Scaling

1.94k views • 163 slides
Scaling and Universality in Probability Francesco Caravenna Universit` a degli Studi di

Scaling and Universality in Probability Francesco Caravenna Universit` a degli Studi di

Weak Convergence Brownian Motion A glimpse of SLE Scaling Limits with Disorder Scaling and Universality in Probability Francesco Caravenna Universit` a degli Studi di Milano-Bicocca Luxembourg June 14, 2016 Francesco Caravenna Scaling

745 views • 33 slides
Performance Scaling How is my parallel code performing and scaling? Performance metrics

Performance Scaling How is my parallel code performing and scaling? Performance metrics

Performance Scaling How is my parallel code performing and scaling? Performance metrics Measure the execution time T how do we quantify performance improvements? Speed up typically S(N,P) < P Parallel efficiency

377 views • 17 slides
Scaling for Ultrafast, G.FAST, FTTP, 5G and the Cloud Neil J. McRae Chief Architect BT 1

Scaling for Ultrafast, G.FAST, FTTP, 5G and the Cloud Neil J. McRae Chief Architect BT 1

Scaling for Ultrafast, G.FAST, FTTP, 5G and the Cloud Neil J. McRae Chief Architect BT 1 British Telecommunications plc 2018 Agenda BT Core network Traffic growth and trends Scaling the core new router introduction scaling

621 views • 21 slides
Scaling UP PB in County Durham Overview Finding our local PB grass roots! Scaling UP our

Scaling UP PB in County Durham Overview Finding our local PB grass roots! Scaling UP our

Scaling UP PB in County Durham Overview Finding our local PB grass roots! Scaling UP our approach County wide Sharing our experiences and learning From sceptic to supporter! Where to from here? Find out more Our local PB

335 views • 18 slides
Scaling-up SLA Monitoring in Scaling-up SLA Monitoring in Pervasive Environments Pervasive

Scaling-up SLA Monitoring in Scaling-up SLA Monitoring in Pervasive Environments Pervasive

Istituto di Scienza e Tecnologie dell'Informazione A. Faedo Software Engineering Laboratory Scaling-up SLA Monitoring in Scaling-up SLA Monitoring in Pervasive Environments Pervasive Environments Engineering of Software Services for

677 views • 22 slides
CSCI 5417 Information Retrieval Systems Jim Martin Lecture 13 10/6/2011 Text classification

CSCI 5417 Information Retrieval Systems Jim Martin Lecture 13 10/6/2011 Text classification

CSCI 5417 Information Retrieval Systems Jim Martin Lecture 13 10/6/2011 Text classification First Nave Bayes Simple, fast, low training and testing cost Then K Nearest Neighbor classification Simple, can easily

730 views • 27 slides
JET SUBSTRUCTURE AT THE LHC & BEYOND Simone Marzani Universit di Genova & INFN

JET SUBSTRUCTURE AT THE LHC & BEYOND Simone Marzani Universit di Genova & INFN

JET SUBSTRUCTURE AT THE LHC & BEYOND Simone Marzani Universit di Genova & INFN Sezione di Genova Interpreting the LHC Run 2 data and Beyond ICTP Trieste 27 th - 31 st May 2019 1 OUTLINE Jet substructure: where we are

474 views • 29 slides
Assimilation of Geostationary Satellite Land Surface Skin Temperature Observations into the GEOS-5

Assimilation of Geostationary Satellite Land Surface Skin Temperature Observations into the GEOS-5

Assimilation of Geostationary Satellite Land Surface Skin Temperature Observations into the GEOS-5 Global Atmospheric Modeling and Assimilation System Clara Draper 1 , 2 , Rolf Reichle 2 , Gabrielle De Lannoy 1 , 2 , and Qing Liu 3 1. GESTAR,

580 views • 19 slides
A Kernel Density Based Approach for Large Scale Image Retrieval and Its Application to Tattoo

A Kernel Density Based Approach for Large Scale Image Retrieval and Its Application to Tattoo

A Kernel Density Based Approach for Large Scale Image Retrieval and Its Application to Tattoo Identification Wei Tong CMU Outline Why tattoo retrieval Used in forensic to identify Victims, criminals Content-based image

802 views • 68 slides
UP UP AND OUT: SCALING SOFTWARE WITH AKKA Jonas Bonr CTO Typesafe @jboner Scaling software

UP UP AND OUT: SCALING SOFTWARE WITH AKKA Jonas Bonr CTO Typesafe @jboner Scaling software

UP UP AND OUT: SCALING SOFTWARE WITH AKKA Jonas Bonr CTO Typesafe @jboner Scaling software with Jonas Bonr CTO Typesafe @jboner Scaling Scaling software with software with Scaling Scaling software with software with Akka

1.98k views • 174 slides
Designing your SaaS Database for Scale with Postgres Lukas

Designing your SaaS Database for Scale with Postgres Lukas

Designing your SaaS Database for Scale with Postgres Lukas Fi9l Ozgun Erdogan What is Citus? Citus extends PostgreSQL (not a fork) to provide

706 views • 39 slides
disordered field theories Ofer Aharony Weizmann Institute of Science CRM-PCTS workshop, October

disordered field theories Ofer Aharony Weizmann Institute of Science CRM-PCTS workshop, October

Renormalization group flows in disordered field theories Ofer Aharony Weizmann Institute of Science CRM-PCTS workshop, October 4, 2018 Based on the papers 1803.08529,1803.08534 with Vladimir Narovlansky Disorder In QFT we like to assume

483 views • 34 slides
Scaled Machine Learning at Matroid Reza Zadeh @Reza_Zadeh | http://reza-zadeh.com Machine Learning

Scaled Machine Learning at Matroid Reza Zadeh @Reza_Zadeh | http://reza-zadeh.com Machine Learning

Scaled Machine Learning at Matroid Reza Zadeh @Reza_Zadeh | http://reza-zadeh.com Machine Learning Pipeline Learning Replicate Algorithm model Data Trained Serve Model Model Repeat entire pipeline Scaling Machine Learning Datasets and

890 views • 21 slides

More recommend