deploying large payloads at scale
play

Deploying large payloads at scale Ramon van Alteren Wednesday, - PowerPoint PPT Presentation

Aug 07, 2023 •39 likes •489 views

Deploying large payloads at scale Ramon van Alteren Wednesday, November 9, 2011 Hyves 9,7M dutch members (16,7M population) ~7M unique visitors / month (Comscore 09/2011) ~2.3M unique visitors / day 800.000 photo uploads /

  1. Deploying large payloads at scale Ramon van Alteren Wednesday, November 9, 2011

  2. Hyves • 9,7M dutch members (16,7M population) • ~7M unique visitors / month (Comscore 09/2011) • ~2.3M unique visitors / day • 800.000 photo uploads / day • 7M chat messages / day Wednesday, November 9, 2011

  3. Hyves - Operational environment • 3500 node serverpark in 3 datacenters • 6Gbps daily outgoing traffic • System Engineering team: 12 • Development team: 33 Wednesday, November 9, 2011

  4. Weekend project Wednesday, November 9, 2011

  5. Weekend project Result: 4.5 x speed/throughput increase Wednesday, November 9, 2011

  6. Weekend -> Company project Wednesday, November 9, 2011

  7. A Few Minor problems Wednesday, November 9, 2011

  8. A Few Minor problems • compilation took ~40-60 minutes Wednesday, November 9, 2011

  9. A Few Minor problems • compilation took ~40-60 minutes • resulting binary was 750MB Wednesday, November 9, 2011

  10. A Few Minor problems • compilation took ~40-60 minutes • resulting binary was 750MB • Code issues Wednesday, November 9, 2011

  11. A Few Minor problems • compilation took ~40-60 minutes • resulting binary was 750MB • Code issues • gcc 4.5.2 required + added deps Wednesday, November 9, 2011

  12. Part I - Solving the build problem Jenkins to the rescue Wednesday, November 9, 2011

  13. Part I - Solving the build problem Jenkins to the rescue Add distCC to speed up compilation times Wednesday, November 9, 2011

  14. Part I - Solving the build problem Add some serious hardware Wednesday, November 9, 2011

  15. Part I - Solving the build problem Add some serious hardware Compile / build in < 6 mins Wednesday, November 9, 2011

  16. Part II: Deploying Sequential ? Wednesday, November 9, 2011

  17. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds Wednesday, November 9, 2011

  18. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds 450 servers * 8 seconds = 3600 seconds == 1 hour 450 servers * 20 seconds = 9000 seconds == 2.5 hour Wednesday, November 9, 2011

  19. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds 450 servers * 8 seconds = 3600 seconds == 1 hour 450 servers * 20 seconds = 9000 seconds == 2.5 hour Diffs ? Wednesday, November 9, 2011

  20. Part II: Deploying Sequential ? 500MB @ 1Gb/s = 4 seconds 500MB @ 500Mb/s = 8 seconds 500MB @ 200Mb/s = 20 seconds 450 servers * 8 seconds = 3600 seconds == 1 hour 450 servers * 20 seconds = 9000 seconds == 2.5 hour Diffs ? binary diff would be between 10KB - 400MB Even on consecutive runs without Wednesday, November 9, 2011

  21. Part II: Deploying - Bittorrent Wednesday, November 9, 2011

  22. Bittorrent - Previous experiences Naive run using bittorrent to transport 300MB throughout our serverpark • Near-complete network outage due to bandwidth starvation • Several crucial subsystems delayed or unreachable due to network bandwidth shortage Wednesday, November 9, 2011

  23. Bittorrent - Previous experiences Wednesday, November 9, 2011

  24. Bittorrent - The Problem Every server has 1Gb/s link to every other server Wednesday, November 9, 2011

  25. Bittorrent - The Problem Every server has 1Gb/s link to every other server they don’t Wednesday, November 9, 2011

  26. Bittorrent - Actual bandwidth available Core 1-4Gb/s Network Wednesday, November 9, 2011

  27. Bittorrent - Actual bandwidth available Production traffic Core Network Administration traffic Wednesday, November 9, 2011

  28. Murder - Why not ? Murder uses two tricks: • Clients (including the seeder) capped to 1 upload peer • Every client receives every peer from the tracker • No download bandwidth cap (easy to add though) Wednesday, November 9, 2011

  29. Murder - Why not ? Murder uses two tricks: • Clients (including the seeder) capped to 1 upload peer • Every client receives every peer from the tracker • No download bandwidth cap (easy to add though) Peers will still connect all over the place Wednesday, November 9, 2011

  30. Murder - Why not ? Murder uses two tricks: • Clients (including the seeder) capped to 1 upload peer • Every client receives every peer from the tracker • No download bandwidth cap (easy to add though) Peers will still connect all over the place It’s slow, timing run over 25 peers took 7 mins Wednesday, November 9, 2011

  31. DIY: Location aware tracker Wednesday, November 9, 2011

  32. SMDB - Location metadata We have bandwidth information available in our server management database Build two-tier bittorrent swarms: • 1 swarm with 2 peers / rack(uplink) • 1 additional swarm per rack (uplink) • cap every client @ 96mbit/s Wednesday, November 9, 2011

  33. DIY: 2-tier swarms Wednesday, November 9, 2011

  34. DIY - Tracker Tracker in python + Flask: • 1100 lines of code (1900 with tests) • Stores transfer metadata in redis • Connects to our SMDB using REST • Exposes REST interface Wednesday, November 9, 2011

  35. DIY - Client We use a slightly modified rtorrent client Same things twitter modified: • Remove features related to operating on the big bad internet. • Make various timeouts more aggresive • No DHT, UPNP etc. Nice bonus: RPC remote API Wednesday, November 9, 2011

  36. Results - single deploy ~300 hosts 2000 1500 1000 500 0 Without build With build Classic deploy Bittorrent deploy Wednesday, November 9, 2011

  37. Graphs - full webfarm cluster Wednesday, November 9, 2011

  38. Graphs - single node Wednesday, November 9, 2011

  39. Graphs - single rack (24 nodes) Wednesday, November 9, 2011

  40. Bitorrent - Statistics == release: 101482 expected: 287 actual: 107 seeders: 0 progress: 0.00% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 267 seeders: 0 progress: 0.00% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 286 seeders: 0 progress: 0.00% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 1 progress: 0.35% start: 12:04:20 last_completed: none failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 2 progress: 42.11% start: 12:04:20 last_completed: 12:06:01 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 5 progress: 44.48% start: 12:04:20 last_completed: 12:06:05 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 34 progress: 45.80% start: 12:04:20 last_completed: 12:06:09 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 95 progress: 48.17% start: 12:04:20 last_completed: 12:06:10 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 95 progress: 48.17% start: 12:04:20 last_completed: 12:06:10 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 97 progress: 48.46% start: 12:04:20 last_completed: 12:06:15 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 101 progress: 49.23% start: 12:04:20 last_completed: 12:06:21 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 108 progress: 50.72% start: 12:04:20 last_completed: 12:06:24 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 118 progress: 52.95% start: 12:04:20 last_completed: 12:06:27 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 131 progress: 55.88% start: 12:04:20 last_completed: 12:06:30 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 172 progress: 67.77% start: 12:04:20 last_completed: 12:06:34 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 213 progress: 80.53% start: 12:04:20 last_completed: 12:06:37 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 239 progress: 87.36% start: 12:04:20 last_completed: 12:06:40 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 246 progress: 91.24% start: 12:04:20 last_completed: 12:06:43 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 265 progress: 97.55% start: 12:04:20 last_completed: 12:06:46 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 274 progress: 98.79% start: 12:04:20 last_completed: 12:06:49 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 280 progress: 99.07% start: 12:04:20 last_completed: 12:06:52 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 283 progress: 99.26% start: 12:04:20 last_completed: 12:06:55 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 284 progress: 99.36% start: 12:04:20 last_completed: 12:06:56 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 284 progress: 99.36% start: 12:04:20 last_completed: 12:06:56 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 285 progress: 99.52% start: 12:04:20 last_completed: 12:07:04 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 286 progress: 99.85% start: 12:04:20 last_completed: 12:07:06 failed: 0 == == release: 101482 expected: 287 actual: 287 seeders: 287 progress: 100.00% start: 12:04:20 last_completed: 12:07:37 failed: 0 == Wednesday, November 9, 2011

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Deploying Large Scale AVB/TSN Networks Jeff Koftinoff, Meyer Sound Laboratories, Inc. June 19,

Deploying Large Scale AVB/TSN Networks Jeff Koftinoff, Meyer Sound Laboratories, Inc. June 19,

Deploying Large Scale AVB/TSN Networks Jeff Koftinoff, Meyer Sound Laboratories, Inc. June 19, 2015 Deploying Large Scale AVB Networks ACT 1 Deploying Large Scale AVB Networks What does putting audio/video on a

872 views • 63 slides
Data Science Until now Abstractions for writing and deploying large-scale web applications

Data Science Until now Abstractions for writing and deploying large-scale web applications

Data Science Until now Abstractions for writing and deploying large-scale web applications Managing infrastructure (PaaS, IaaS, Infrastructure-as- Code, FaaS, etc.) Constructing applications (ML APIs, Backend-as-a- Service) Portland

1.02k views • 58 slides
Challenges in Delivering and Deploying Software at Scale in Large Clusters Douglas Thain and Kyle

Challenges in Delivering and Deploying Software at Scale in Large Clusters Douglas Thain and Kyle

Challenges in Delivering and Deploying Software at Scale in Large Clusters Douglas Thain and Kyle Sweeney University of Notre Dame {dthain|ksweene3}@nd.edu Software Deployment on HPC Classic Approach Single process MPI app created by end

1.06k views • 50 slides
Deploying large-scale virtual infrastructures with Kadeploy3 Luc Sarzyniec, S ebastien Badia,

Deploying large-scale virtual infrastructures with Kadeploy3 Luc Sarzyniec, S ebastien Badia,

Deploying large-scale virtual infrastructures with Kadeploy3 Luc Sarzyniec, S ebastien Badia, Emmanuel Jeanvoine, Lucas Nussbaum Grid5000 Plan 1 Introduction Kadeploy3 Kabootstrap Our experiment 2 Scalability challenges 3 Experiment 4

205 views • 18 slides
Deploying*Large*Scale*AVB/TSN* Networks*:* Handout * Jeff+Koftinoff+&lt;jeffk@meyersound.com&gt;+

Deploying*Large*Scale*AVB/TSN* Networks*:* Handout * Jeff+Koftinoff+&lt;jeffk@meyersound.com&gt;+

1 ! ! Deploying*Large*Scale*AVB/TSN* Networks*:* Handout * Jeff+Koftinoff+&lt;jeffk@meyersound.com&gt;+ ( Infocomm(2015((Orlando,(Florida( June(19(2015( ! !

216 views • 10 slides
Deploying Large Scale Webapps Thierry Sans Users respond to speed Amazon found every 100ms of

Deploying Large Scale Webapps Thierry Sans Users respond to speed Amazon found every 100ms of

Deploying Large Scale Webapps Thierry Sans Users respond to speed Amazon found every 100ms of latency cost them 1% in sales Google found an extra . 5 seconds in search page generation time dropped traffic by 20% A broker could

559 views • 20 slides
An open, shared platform for developing, deploying, and accessing planetary-scale applications

An open, shared platform for developing, deploying, and accessing planetary-scale applications

An open, shared platform for developing, deploying, and accessing planetary-scale applications Timothy Roscoe Intel Research at Berkeley Thursday, July 01, 2004 APAN, Cairns, 1 July 2004 PlanetLab is Large collection of machines

533 views • 40 slides
FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large

FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large

FINANCING LARGE SCALE SOLAR Large Scale Solar Conference - Sydney Gloria Chan Director, Large Scale Solar Lead April 2017 CONTENTS 1. Introduction to CEFC 2. Investment trends 3. The future of large scale solar 4. Pathway to sustainable

664 views • 21 slides
INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO

INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO

INCORPORATING LARGE-SCALE CITIZEN INCORPORATING LARGE-SCALE CITIZEN DELIBERATION INTO ENVIRONMENTAL CONFLICT RESOLUTION America Speaks presentation to the 2008 ECR Conference Table Introductions Please form groups of 4-5 and give: Your name

848 views • 41 slides
Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008

Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008

Large-Scale Survey Interviewing Following Large Scale Survey Interviewing Following the 2008 WenChuan Earthquake Zhang Huafeng and Jon Pedersen International Blaise Users Conference (IBUC) Baltimore, USA , 1 Introduction Two household

711 views • 22 slides
Large-scale production of graphene: Introduction Large-scale production of graphene: what is

Large-scale production of graphene: Introduction Large-scale production of graphene: what is

Large-scale production of graphene: Introduction Large-scale production of graphene: what is graphene? Graphene is a truly 2D system made of Carbon atoms arranged in an honeycomb hexagonal lattice Zero-gap semiconductor with a low-energy linear

268 views • 6 slides
Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA

Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA

Ethics in Techniques for large-scale data Graham J.L. Kemp TECHNIQUES FOR LARGE-SCALE DATA Masters level course: Chalmers MPALG and MPSOF programmes (DAT345) GU Applied Data Science programme (DIT871) Learning outcomes include:

305 views • 18 slides
Deploying, At An Unusual Scale Andrew Godwin @andrewgodwin

Deploying, At An Unusual Scale Andrew Godwin @andrewgodwin

Deploying, At An Unusual Scale Andrew Godwin @andrewgodwin http://www.flickr.com/photos/whiskeytango/1431343034/ Hi, I'm Andrew. Serial Python developer Django core committer Co-founder of ep.io Hi, I'm Andrew. Serial Python developer

1.03k views • 43 slides
An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of

An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of

An Email Contact Protocol Experiment in a Large-Scale Survey of U.S. in a Large Scale Survey of U.S. Government Employees DC-AAPOR Summer Conference Preview/Review Bureau of Labor Statistics Conference Center Washington DC Washington, DC

1.17k views • 25 slides
Challenges in Deploying and Managing Large Terminologies: NCI Thesaurus For Protg Workshop June

Challenges in Deploying and Managing Large Terminologies: NCI Thesaurus For Protg Workshop June

Challenges in Deploying and Managing Large Terminologies: NCI Thesaurus For Protg Workshop June 22, 2009 Amsterdam Gilberto Fragoso, Sherri de Coronado Challenge Topics Background EVS, NCI Thesaurus, Vocabulary Development, Distribution

521 views • 33 slides
Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E.

Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E.

Efficient Large-Scale Graph Processing on Hybrid CPU and GPU Systems A. Gharaibeh, E. Santos-Neto, L. Costa, M. Ripeanu. IEEE TPC, 2014 Sami (sa894) - R244: Large-scale data processing and optimization Efficient Large-Scale Graph Processing on

381 views • 25 slides
ALICE Erik Edelmann erik.edelmann@csc.fi NeIC / CSC NeIC/NDGF All hands meeting February 2013

ALICE Erik Edelmann erik.edelmann@csc.fi NeIC / CSC NeIC/NDGF All hands meeting February 2013

ALICE Erik Edelmann erik.edelmann@csc.fi NeIC / CSC NeIC/NDGF All hands meeting February 2013 p. 1/ Pledges: CPU 180 Fin 160 Swe Den 140 Nor NDGF 120 % of pledge 100 80 60 40 20 0 May12 Jul12 Sep12 Nov12 Jan13

361 views • 22 slides
Content may be King, but (Peering) Location Matters: A Progress Report on the Evolution of

Content may be King, but (Peering) Location Matters: A Progress Report on the Evolution of

Content may be King, but (Peering) Location Matters: A Progress Report on the Evolution of Content Delivery in the Internet Volker Stocker William Lehr University of Freiburg, Germany MIT Georgios Smaragdakis Steven Bauer TU Berlin/MIT

836 views • 14 slides
P2P Live Streaming: successes and limitations Yong Liu ECE, Polytechnic U. 04/27/2007 joint

P2P Live Streaming: successes and limitations Yong Liu ECE, Polytechnic U. 04/27/2007 joint

P2P Live Streaming: successes and limitations Yong Liu ECE, Polytechnic U. 04/27/2007 joint work with Keith Ross, Xiaojun Hei, Rakesh Kumar, Chao Liang, Jian Liang 1 Next Disruptive Application? Broadband Residential Access

441 views • 23 slides
hi, im missy @missmissylaney SUNDANCE FILM FESTIVAL 2008-2015 Premium All Rights

hi, im missy @missmissylaney SUNDANCE FILM FESTIVAL 2008-2015 Premium All Rights

hi, im missy @missmissylaney SUNDANCE FILM FESTIVAL 2008-2015 Premium All Rights Distribution Deals 17% Films in Purgatory 45% Sub-Par Distribution 18% 21% Service Deal / Split Rights Not that long ago, the choice was

276 views • 16 slides
The Networking Lab in the School of Computing Dr. Jim Martin Associate Professor I want to be a

The Networking Lab in the School of Computing Dr. Jim Martin Associate Professor I want to be a

The Networking Lab in the School of Computing Dr. Jim Martin Associate Professor I want to be a network School of Computing engineer!!!!! WooHoo..I like trains! Clemson University jim.martin@cs.clemson.edu http://www.cs.clemson.edu/~jmarty

280 views • 6 slides
Kitchen Bath Design Presentation Drawing Plans Digital Rendering Nkba Professional Resource

Kitchen Bath Design Presentation Drawing Plans Digital Rendering Nkba Professional Resource

Kitchen Bath Design Presentation Drawing Plans Digital Rendering Nkba Professional Resource Library.pdf Kitchen Bath Design Presentation Drawing Plans Digital Rendering Nkba Professional Resource Library Kitchen Bath Design Presentation Drawing

249 views • 9 slides
TorScan: Tracing Long-lived Connections and Differential Scanning Attacks A. Biryukov, I.

TorScan: Tracing Long-lived Connections and Differential Scanning Attacks A. Biryukov, I.

TorScan: Tracing Long-lived Connections and Differential Scanning Attacks A. Biryukov, I. Pustogarov, R.P. Weinmann University of Luxembourg ivan.pustogarov@uni.lu September 5, 2012 A. Biryukov, I. Pustogarov, R.P. Weinmann (UNI LU) Torscan

376 views • 23 slides
A QoS-Aware Middleware for Fault Tolerant Web Services Zibin Zheng and Michael R. Lyu Department

A QoS-Aware Middleware for Fault Tolerant Web Services Zibin Zheng and Michael R. Lyu Department

A QoS-Aware Middleware for Fault Tolerant Web Services Zibin Zheng and Michael R. Lyu Department of Computer Science &amp; Engineering The Chinese University of Hong Kong Hong Kong, China ISSRE 2008, Seattle, USA, 11-14 November, 2008

596 views • 25 slides

More recommend