CS 744: Big Data Systems Shivaram Venkataraman Fall 2018 - PowerPoint PPT Presentation

Dec 01, 2022 •644 likes •822 views

CS 744: Big Data Systems Shivaram Venkataraman Fall 2018 ADMINISTRIVIA - Assignment 1: Due Oct 1 - Sign up for Project meetings - Group updates MapReduce GFS BigTable BORG: WORKLOAD Long-running services (should never go down) Batch

CS 744: Big Data Systems Shivaram Venkataraman Fall 2018
ADMINISTRIVIA - Assignment 1: Due Oct 1 - Sign up for Project meetings - Group updates
MapReduce GFS BigTable
BORG: WORKLOAD Long-running services (should “never” go down) Batch jobs: few seconds to a few days
BORG CONCEPTS Users submit jobs Each job is one or more tasks All tasks that run the same program (binary) Each job runs in one Borg cell
JOB DESCRIPTION
JOB PROPERTIES Name Constraints Properties - Resource requirements - No slots! - Static Binaries
JOB LIFECYLE
QUOTAS, PRIORITIES, BNS Priority High priority can preempt lower priority Quotas Used for admission control Infinite quota at priority zero Service Discovery using BNS
ARCHITECTURE
MASTER, Borglet BorgMaster Single Leader, five-ways replicated Paxos group – using Chubby locks Borglet Daemon on each machine Borgmaster pulls updates from Borglets Health checks used to detect failures
SCHEDULER - Feasibility checking pass, Scoring pass - Task cache (static binaries) - Scalability - Split master into multiple processes - Use replicas for communication - Randomize machines used for scoring …
UTILIZATION: CELL COMPACTION
REQUEST SIZE: NO SWEET SPOT
RECLAMATION
LESSONS, DISCUSSION - Jobs are restrictive, Allocs are useful - IP address per container - Kernel of distributed operating system
QUESTIONS / DISCUSSION ?

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

BSD Data Systems Report to the School Board, April 2019 Data-Driven ESSA Best Practices Data

BSD Data Systems Report to the School Board, April 2019 Data-Driven ESSA Best Practices Data Systems Coordinator Data Systems Coordinator Role Data that is accurate, relevant, actionable, accessible. District systems and central office

512 views • 17 slides

Today Data Acquisition Systems Many embedded systems measure quantities from Data

Today Data Acquisition Systems Many embedded systems measure quantities from Data acquisition the environment and turn them into bits Digital filters and signal processing These are data acquisition systems (DAS) Filter

522 views • 9 slides

DLS VERIFICATION TESTING FIELD TESTS TO CHECK DLS OPERATIONS DATA LOGGER SYSTEMS DATA LOGGER

DLS VERIFICATION TESTING FIELD TESTS TO CHECK DLS OPERATIONS DATA LOGGER SYSTEMS DATA LOGGER SYSTEMS Data LOGGER SYSTEMS (DLS) ARE ELECTRONIC SYSTEMS USED TO: 1. MONITOR MATERIAL APPLICATION CONDITIONS i.e. ambient temperatures 2. PROVIDE REAL

492 views • 23 slides

Big Data, Big Problem Data-intensive systems are highly complex Manipulate large-scale

Towards Big Data Provenance Daniel Deutch Blavatnik School of Computer Science, Raymond and Beverly Sackler Faculty of Exact Sciences Tel Aviv University cartoon by T. Gregorius Big Data, Big Problem Data-intensive systems are highly

358 views • 6 slides

Data Centers & Co-designed Distributed Systems A Data Center Inside a Data Center Data

Data Centers & Co-designed Distributed Systems A Data Center Inside a Data Center Data center 10k - 100k servers: 250k 10M cores 1-100PB of DRAM 100PB - 10EB storage 1- 10 Pbps bandwidth (>> Internet) 10-100MW power - 1-2% of

911 views • 54 slides

CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu

CS 591: Data Systems Architectures Prof. Manos Athanassoulis mathan@bu.edu http://manos.athanassoulis.net/classes/CS591 Today big data I want you to speak up! [and you can always interrupt me] data-driven world data systems which are the

860 views • 54 slides

geotools: Exporting cartography data from Stata to GIS systems Sergiy Radyakin

Introduction Earlier products Data sources Data transformations Data export Output Notes 2018 Canadian Stata Conference Morris J. Wosk Centre for Dialogue, Vancouver, BC geotools: Exporting cartography data from Stata to GIS systems

507 views • 37 slides

File Systems 1. Organizing data into files and directories and supporting primitives to

CSE 421/521 - Operating Systems File Systems Fall 2011 Provides organized and efficient access to data on secondary storage: Lecture - XVIII File Systems 1. Organizing data into files and directories and supporting primitives to

345 views • 6 slides

Big Data Systems Big Data Parallelism Huge data set crawled documents, web request logs,

Big Data Systems Big Data Parallelism Huge data set crawled documents, web request logs, etc. Natural parallelism: can work on different parts of data independently image processing, grep, indexing, many more Challenges

708 views • 55 slides

ZODIAC DATA SYSTEMS ZODIAC AIRCRAFT SYSTEMS ITU Mission (from Y. Henri presentation - 18/09/14)

ZODIAC DATA SYSTEMS ZODIAC AIRCRAFT SYSTEMS ITU Mission (from Y. Henri presentation - 18/09/14) ZODIAC AIRCRAFT SYSTEMS This document is the property of ZODIAC DATA SYSTEMS. It cannot be duplicated or distributed without expressed written

744 views • 38 slides

Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data

Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data Centric Systems and Networking Emergence of Big Data Shift of Communication Paradigm From end-to-end to data centric Data as

767 views • 17 slides

Systems for Data Science Marco Serafini COMPSCI 532 Lecture 1 Course Structure

Systems for Data Science Marco Serafini COMPSCI 532 Lecture 1 Course Structure Fundamentals you need to know about systems Caching, Virtual memory, concurrency, etc Review of several Big-data systems Learn how they

428 views • 31 slides

Follow the data! Algorithms and systems for responsible data science Julia Stoyanovich Drexel

Follow the data! Algorithms and systems for responsible data science Julia Stoyanovich Drexel University & Princeton CITP NYC Algorithmic Transparency Law 1/11/2018 Int. No. 1696-A: A Local Law in relation to automated decision systems

812 views • 49 slides

CSE 232A Database System Implementation Arun Kumar Topic 8: Data Systems for ML Workloads

CSE 232A Database System Implementation Arun Kumar Topic 8: Data Systems for ML Workloads Book: Data Management in ML Systems by Morgan & Claypool Publishing 1 Big Data Systems Parallel RDBMSs and Cloud-Native RDBMSs

552 views • 21 slides

File Systems: All data structures are cached for better performance Create a new file

File Systems: Consistency Issues What about Multiple Updates? File systems maintain many data structures Several file system operations update multiple data structures Free list/bit vector Directories Examples: File headers and inode

257 views • 3 slides

How Complex Systems Thinking Can Tame Big Data: The Limits of Data-Centric Inference (and

Research School 2014 Predictive modeling: from data to models 29-31 October, Mtopole, Toulouse How Complex Systems Thinking Can Tame Big Data: The Limits of Data-Centric Inference (and Math Analysis) and Usefulness of Agent-Based

708 views • 51 slides

Central philosophy of the work Making Gnutella-like P2P File-sharing is a dominant P2P

Central philosophy of the work Making Gnutella-like P2P File-sharing is a dominant P2P application Systems Scalable DHTs might not be suitable for file-sharing Gnutellas design Presented by: Karthik Lakshminarayanan

393 views • 5 slides

Whats new in HIV-associated tuberculosis? December 14, 2019 Sarah Puryear, MD MPH 1

Whats new in HIV-associated tuberculosis? December 14, 2019 Sarah Puryear, MD MPH 1 Disclosure I have no relevant financial relationships with any companies related to the content of this course. 2 1 | [footer text here] Objectives

681 views • 30 slides

Introduction to Computer Science CSCI 109 China Tianhe-2 Andrew Goodney Fall 2019 Lecture

Introduction to Computer Science CSCI 109 China Tianhe-2 Andrew Goodney Fall 2019 Lecture 8: Operating Systems October 21th, 2019 Operating Systems Working Together 1 Schedule 2 Agenda u

772 views • 41 slides

Will it blend? A comparison of oVirt, OpenStack and kubernetes schedulers Martin Sivk

Will it blend? A comparison of oVirt, OpenStack and kubernetes schedulers Martin Sivk Principal Software Engineer Red Hat Czech 3 th of Feb 2018 This presentation is licensed under a Creative Commons Attribution 4.0 International License

410 views • 27 slides

Intergalacc magnec fields Andrii Neronov University of

Intergalac*c magne*c fields Andrii Neronov University of Geneva Off-the-Beaten-Track Dark Ma9er and Astrophysical Probes of Fundamental Physics, Trieste

1.48k views • 36 slides

Realtime Java Christoph Neijenhuis Agenda 2 RT Problems in Java Solutions: RTSJ and

Realtime Java Christoph Neijenhuis Agenda 2 RT Problems in Java Solutions: RTSJ and others Guiding Principles Memory Management Threads Asynchronous Event Handling Scheduling Time Synchronization

802 views • 20 slides

Chapter 4 Priority-Based Real-Time Scheduling Real-Time Embedded Systems Laboratory Northeastern

Spring 2009 - Real-Time Systems http://www.neu-rtes.org/courses/spring2009/ Chapter 4 Priority-Based Real-Time Scheduling Real-Time Embedded Systems Laboratory Northeastern University Objectives In this chapter, you are supposed to learn:

695 views • 32 slides

REPORT OF THE CHIEF EXECUTIVE: 2018 B UDGET P REVIEW S ECOND P RESENTATION 10 AM 12 PM O CTOBER

REPORT OF THE CHIEF EXECUTIVE: 2018 B UDGET P REVIEW S ECOND P RESENTATION 10 AM 12 PM O CTOBER 23, 2017 IFAA O FFICES , 404 K IVA C OURT , S ANTA F E 2018 IFAA B UDGET L ENS W HAT ARE THE ESSENTIALS ? I F MORE WITH LESS . . . WHAT DOES

596 views • 13 slides