scaling distributes systems
play

Scaling Distributes Systems Natalia Chechina and RELEASE Team June - PowerPoint PPT Presentation

Mar 27, 2024 •112 likes •727 views

Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Scaling Distributes Systems Natalia Chechina and RELEASE Team June 11, 2015 N. Chechina, RELEASE team Scaling Distributes Systems Distributed Erlang

  1. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Scaling Distributes Systems Natalia Chechina and RELEASE Team June 11, 2015 N. Chechina, RELEASE team Scaling Distributes Systems

  2. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Who am I? 2011: Received PhD degree in Computer Science from Heriot-Watt University, UK 2011-2015: WP3 lead in the EU RELEASE Project at Glasgow University, UK March 2015: Research Fellow at Glasgow University, UK Main research interest: Scaling distributed computations on commodity hardware N. Chechina, RELEASE team Scaling Distributes Systems

  3. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Sources Research findings Experience from the RELEASE project Funded by EU FP7 Framework 5 academic & 3 industrial partners Aim: To scale the radical actor (concurrency-oriented) paradigm to build reliable general-purpose software, such as server-based systems, on massively parallel machines (10 5 cores) Erlang programming language Experience of other researches and developers N. Chechina, RELEASE team Scaling Distributes Systems

  4. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Scaling a Sysem Scaling ALL aspects of computation Application Language Virtual Machine In-memory data structures Persistent data structures Tools (debugging, monitoring, etc) N. Chechina, RELEASE team Scaling Distributes Systems

  5. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Scaling a Sysem Scaling ALL aspects of computation Application Language Virtual Machine In-memory data structures Persistent data structures Tools (debugging, monitoring, etc) N. Chechina, RELEASE team Scaling Distributes Systems

  6. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Scaling on language level Actor model Functional programming N. Chechina, RELEASE team Scaling Distributes Systems

  7. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Language – Actor Model Built-in concurrency Actors have own states and don’t share them Communication between actor happens only via message passing Actors can spawn new actors N. Chechina, RELEASE team Scaling Distributes Systems

  8. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Language – Functional programming Fundamental operation – application of functions to arguments Higher-order functions – well-structured software Modules – independent, reusable Lazy evaluations Variables given values only once N. Chechina, RELEASE team Scaling Distributes Systems

  9. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Fault Tolerance 10 5 cores – approx. failure of 1 core per hour Non-defensive approach – Supervision & ”Let it crash” N. Chechina, RELEASE team Scaling Distributes Systems

  10. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Philosophy Principles Ideas Core values N. Chechina, RELEASE team Scaling Distributes Systems

  11. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources RELEASE Aim To scale the radical actor (concurrency-oriented) paradigm to build reliable general-purpose software, such as server-based systems, on massively parallel machines (10 5 cores). N. Chechina, RELEASE team Scaling Distributes Systems

  12. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources RELEASE Aim To scale the radical actor (concurrency-oriented) paradigm to build reliable general-purpose software, such as server-based systems, on massively parallel machines (10 5 cores). Erlang N. Chechina, RELEASE team Scaling Distributes Systems

  13. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources RELEASE Aim To scale the radical actor (concurrency-oriented) paradigm to build reliable general-purpose software, such as server-based systems, on massively parallel machines (10 5 cores). Erlang VM aspects , e.g. synchronisation on internal data structures Language aspects , e.g. maintaining a fully connected network of nodes, explicit process placement Tool support N. Chechina, RELEASE team Scaling Distributes Systems

  14. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources RELEASE Aim To scale the radical actor (concurrency-oriented) paradigm to build reliable general-purpose software, such as server-based systems, on massively parallel machines (10 5 cores). Erlang VM aspects , e.g. synchronisation on internal data structures Language aspects , e.g. maintaining a fully connected network of nodes, explicit process placement Tool support N. Chechina, RELEASE team Scaling Distributes Systems

  15. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Typical Target Architecture - 10 5 cores Commodity hardware Non-uniform communication (Level0 – same host, Level1 – same cluster, etc) N. Chechina, RELEASE team Scaling Distributes Systems

  16. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Erlang Overview Erlang is a functional general purpose concurrent programming language developed in 1986 at Ericsson is dynamically typed was designed for distributed, fault-tolerant, massively concurrent, and soft-real time systems follows let it crash and share nothing philosophy The language primitives are processes. Erlang concurrency is handled by the language and not by the operating system [Arm10]. N. Chechina, RELEASE team Scaling Distributes Systems

  17. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Distributed Erlang N. Chechina, RELEASE team Scaling Distributes Systems

  18. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Transitive connections N. Chechina, RELEASE team Scaling Distributes Systems

  19. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Transitive connections Explicit Placement, i.e. spawn(Node, Module, Function, Args) → pid() N. Chechina, RELEASE team Scaling Distributes Systems

  20. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Scalability Limitations Global operations Global operations, i.e. registering names using global module N. Chechina, RELEASE team Scaling Distributes Systems

  21. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Scalability Limitations Global operations Global operations, i.e. registering names using global module Other global operations, e.g. using rpc:call to call multiple nodes N. Chechina, RELEASE team Scaling Distributes Systems

  22. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Scalability Limitations Global operations Global operations, i.e. registering names using global module Other global operations, e.g. using rpc:call to call multiple nodes All-to-all transitive connections N. Chechina, RELEASE team Scaling Distributes Systems

  23. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Scalability Limitations Global operations Global operations, i.e. registering names using global module Other global operations, e.g. using rpc:call to call multiple nodes All-to-all transitive connections But... aren’t global operations and transitivity are optional in distributed Erlang? Why use them if they are a bottleneck? N. Chechina, RELEASE team Scaling Distributes Systems

  24. Distributed Erlang Scalable Distributed (SD) Erlang Operational Semantics Plans Sources Distributed Erlang Scalability Limitations Global operations Global operations, i.e. registering names using global module Other global operations, e.g. using rpc:call to call multiple nodes All-to-all transitive connections But... aren’t global operations and transitivity are optional in distributed Erlang? Why use them if they are a bottleneck? Reliability and fault tolerance – when a process or a node fail, the remaining nodes know about that. The same holds for the recovery It’s already there – no extra effort to connect nodes and distribute information Easy to scale – a new node knows about running nodes, and vice versa N. Chechina, RELEASE team Scaling Distributes Systems

  25. Distributed Erlang SD Erlang Scalable Distributed (SD) Erlang Network Scalability Operational Semantics Validation Plans Semi-Explicit Placement Sources Scalable Distributed (SD) Erlang SD Erlang is a small conservative extension of Distributed Erlang Network Scalability All-to-all connections are not scalable onto 1000s of nodes Aim: Reduce connectivity Semi-explicit Placement Becomes not feasible for a programmer to be aware of all nodes Aim: Automatic process placement in groups of nodes N. Chechina, RELEASE team Scaling Distributes Systems

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Fiber Cement Sheet About : Arrow Systems Arrow Systems, Inc. manufactures and distributes a

Fiber Cement Sheet About : Arrow Systems Arrow Systems, Inc. manufactures and distributes a

Fiber Cement Sheet About : Arrow Systems Arrow Systems, Inc. manufactures and distributes a wide range of materials and equipment for the digital printing and cutting markets. We serve a variety of segments within the indoor/outdoor

265 views • 24 slides
CHAPTER 1: OVERVIEW OF DISTRIBUTED SYSTEMS Dr. Tr n H i Anh Outline 2 Introduction 1.

CHAPTER 1: OVERVIEW OF DISTRIBUTED SYSTEMS Dr. Tr n H i Anh Outline 2 Introduction 1.

CHAPTER 1: OVERVIEW OF DISTRIBUTED SYSTEMS Dr. Tr n H i Anh Outline 2 Introduction 1. Characteristics of Distributes Systems 2. Components of Distributes Systems 3. Topics in Distributed Systems 4. 1. Introduction 3 1.1. Brief

955 views • 30 slides
Headquartered in Tulsa, Oklahoma, TESCORP distributes, fabricates, and services its line of

Headquartered in Tulsa, Oklahoma, TESCORP distributes, fabricates, and services its line of

Headquartered in Tulsa, Oklahoma, TESCORP distributes, fabricates, and services its line of standardized and specialty compression systems stateside and internationally. The heart of the company is our innovative engineered products and service

489 views • 17 slides
Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large

Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large

Scaling Outline Scaling Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Scaling-at-large Principles of Complex Systems Allometry Allometry Definitions Definitions Course 300, Fall, 2008 Examples Examples

568 views • 27 slides
Autoscaling Effects in Speed Scaling Systems Carey Williamson Department of Computer Science

Autoscaling Effects in Speed Scaling Systems Carey Williamson Department of Computer Science

CORS 2016 Autoscaling Effects in Speed Scaling Systems Carey Williamson Department of Computer Science (joint work with Maryam Elahi and Philipp Woelfel) Introduction Dynamic CPU speed scaling systems Service rate adjusted based on

441 views • 14 slides
Polynomial Chaos and Scaling Limits of Disordered Systems Francesco Caravenna Universit` a

Polynomial Chaos and Scaling Limits of Disordered Systems Francesco Caravenna Universit` a

Disordered Systems Partition Function CDPM Further Developments Polynomial Chaos and Scaling Limits of Disordered Systems Francesco Caravenna Universit` a degli Studi di Milano-Bicocca Nantes June 6, 2014 Francesco Caravenna Scaling

1.31k views • 92 slides
Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of

Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of

Scaling Scaling Outline Scalinga Plenitude of Power Laws Scaling-at-large Scaling-at-large Principles of Complex Systems Allometry Allometry CSYS/MATH 300, Fall, 2010 Definitions Definitions Examples Examples History: Metabolism

250 views • 20 slides
Scalinga Plenitude of Power Laws Scaling-at-large Principles of Complex Systems Allometry

Scalinga Plenitude of Power Laws Scaling-at-large Principles of Complex Systems Allometry

Scaling Scalinga Plenitude of Power Laws Scaling-at-large Principles of Complex Systems Allometry CSYS/MATH 300, Fall, 2011 Examples A focus: Metabolism Measuring exponents History: River networks Earlier theories Geometric argument

1.33k views • 114 slides
Communication Avoiding Power Scaling Power Scaling Derivatives of Algorithmic Communication

Communication Avoiding Power Scaling Power Scaling Derivatives of Algorithmic Communication

Communication Avoiding Power Scaling Power Scaling Derivatives of Algorithmic Communication Complexity John D. Leidel, Yong Chen Parallel Programming Models & Systems for High End Computing (P2S2 2015) Sept 1, 2015 1 Overview Intro:

776 views • 26 slides
Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms

Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms

Rafael Oliveira University of Toronto Analysis of Scaling Algorithms for Matrix & Operator Scaling Contents Scaling Algorithms Three Step Analysis Generalization One More Application of Scaling Non-Negative Matrices &

604 views • 15 slides
Scalinga Plenitude of Power Laws Scaling-at-large Principles of Complex Systems Allometry

Scalinga Plenitude of Power Laws Scaling-at-large Principles of Complex Systems Allometry

Scaling Scalinga Plenitude of Power Laws Scaling-at-large Principles of Complex Systems Allometry Course CSYS/MATH 300, Fall, 2009 Definitions Examples History: Metabolism Measuring exponents History: River networks Prof. Peter Dodds

1.46k views • 107 slides
Scaling Pseudonymous Authentication for Large Mobile Systems ACM WiSec19, May 17, 2019

Scaling Pseudonymous Authentication for Large Mobile Systems ACM WiSec19, May 17, 2019

Scaling Pseudonymous Authentication for Large Mobile Systems ACM WiSec19, May 17, 2019 Mohammad Khodaei, Hamid Noroozi, and Panos Papadimitratos Networked Systems Security Group www.eecs.kth.se/nss 1 / 29 Secure Vehicular Communication

597 views • 32 slides
Scaling performance In PowerLimited HPC Systems Prof. Dr. Luca Benini ERC Multitherman Lab

Scaling performance In PowerLimited HPC Systems Prof. Dr. Luca Benini ERC Multitherman Lab

Scaling performance In PowerLimited HPC Systems Prof. Dr. Luca Benini ERC Multitherman Lab DITET, Chair of Digital Dircuits University of Bologna Italy & Systems Switzerland Outline Power and Thermal Walls in HPC

562 views • 37 slides
Scaling Services: Partitioning, Hashing, Key-Value Storage CS 240: Computing Systems and

Scaling Services: Partitioning, Hashing, Key-Value Storage CS 240: Computing Systems and

Scaling Services: Partitioning, Hashing, Key-Value Storage CS 240: Computing Systems and Concurrency Lecture 14 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Selected content adapted from B.

642 views • 48 slides
Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent

Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent

Scaling From simple models to rich strategies PPPLab Day, November 30th Scaling: recent publications Insight Series Tool Animation The rationale for scaling As the SDGs require transformational change, scaling can provide: Reaching more

609 views • 37 slides
Polynomial Chaos and Scaling Limits of Disordered Systems 3. Marginally relevant models

Polynomial Chaos and Scaling Limits of Disordered Systems 3. Marginally relevant models

Polynomial Chaos and Scaling Limits of Disordered Systems 3. Marginally relevant models Francesco Caravenna Universit` a degli Studi di Milano-Bicocca YEP XIII, Eurandom March 7-11, 2016 Francesco Caravenna Scaling Limits of Disordered

299 views • 17 slides
in a Growth Network Keynote Speech Manila, September 12, 2017 ASEAN at 50: Building Partnerships

in a Growth Network Keynote Speech Manila, September 12, 2017 ASEAN at 50: Building Partnerships

ASEAN at 50 : Building Partnerships in a Growth Network Keynote Speech Manila, September 12, 2017 ASEAN at 50: Building Partnerships in a Growth Network Keynote Speech 1. Globalization (as you new it) is dead, Long Live Globalnection (a

251 views • 22 slides
Characterizing Ext xtragalactic Pre-Main- Sequence Stars wit ith Machine and Deep Learnin ing

Characterizing Ext xtragalactic Pre-Main- Sequence Stars wit ith Machine and Deep Learnin ing

Characterizing Ext xtragalactic Pre-Main- Sequence Stars wit ith Machine and Deep Learnin ing Techniques Victor Francisco Ksoll Understanding the Nearby Star-forming Universe with JWST 27.08.2019 Star Forming Regions in the Large Magellanic

333 views • 18 slides
CHAPTER VIII VIII CHAPTER Data Clustering and Data Clustering and Self- -Organizing Feature

CHAPTER VIII VIII CHAPTER Data Clustering and Data Clustering and Self- -Organizing Feature

Ugur HALICI - METU EEE - ANKARA 11/18/2004 CHAPTER VIII VIII CHAPTER Data Clustering and Data Clustering and Self- -Organizing Feature Maps Organizing Feature Maps Self CHAPTER VI : VI : Data Clustering CHAPTER Data Clustering &

337 views • 16 slides
Forward-Lookin Looking g Statemen ements ts This presentation contains forward-looking

Forward-Lookin Looking g Statemen ements ts This presentation contains forward-looking

J.P. P. Mo Morgan an 4 th th An Annua nual l Infrastruct rastructure ure / ML / MLP P 1x1 x1 Forum rum May 14, 2015 Strong. ong. Inn nnovativ tive. . Growing. g. 1 Forward-Lookin Looking g Statemen ements ts This

211 views • 19 slides
Probability Density Function Estimation Based Over-Sampling for Imbalanced Two-Class Problems Ming

Probability Density Function Estimation Based Over-Sampling for Imbalanced Two-Class Problems Ming

Introduction PDF Estimation Based Over-sampling Experiments Conclusions Probability Density Function Estimation Based Over-Sampling for Imbalanced Two-Class Problems Ming Gao a , Xia Hong a , Sheng Chen b , c , Chris J. Harris b a School of

335 views • 29 slides
Sparse Kernel Density Estimation Technique Based on Zero-Norm Constraint Xia Hong 1 , Sheng Chen 2

Sparse Kernel Density Estimation Technique Based on Zero-Norm Constraint Xia Hong 1 , Sheng Chen 2

Motivations Proposed Sparse Kernel Density Estimator Numerical Examples Conclusions Sparse Kernel Density Estimation Technique Based on Zero-Norm Constraint Xia Hong 1 , Sheng Chen 2 , Chris J. Harris 2 1 School of Systems Engineering

596 views • 23 slides
Kernel Density Adaptive Random Testing Matthew Patrick and Yue Jia 13 April 2015 Outline

Kernel Density Adaptive Random Testing Matthew Patrick and Yue Jia 13 April 2015 Outline

Kernel Density Adaptive Random Testing Matthew Patrick and Yue Jia 13 April 2015 Outline Diversification and Intensification Kernel Density Adaptive Random Testing (KD-ART) Is KD-ART more effective than ART? Finding the optimal

503 views • 26 slides
Exploring the Limits of Classification Accuracy Carolyn Kim 1 Lester Mackey 2 1 Computer Science

Exploring the Limits of Classification Accuracy Carolyn Kim 1 Lester Mackey 2 1 Computer Science

Exploring the Limits of Classification Accuracy Carolyn Kim 1 Lester Mackey 2 1 Computer Science Department, Stanford University 2 Statistics Department, Stanford University December 7, 2015 Carolyn Kim , Lester Mackey (Stanford) Exploring the

577 views • 16 slides

More recommend