CS 744: MESOS Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - - - PowerPoint PPT Presentation

▶
cs 744 mesos
SMART_READER_LITE
LIVE PREVIEW

CS 744: MESOS Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - - - PowerPoint PPT Presentation

Dec 28, 2022 687 likes •943 views

CS 744: MESOS Shivaram Venkataraman Fall 2020 ADMINISTRIVIA - Assignment 1: How did it go? - Assignment 2 out tonight - Project details - Create project groups - Bid for projects/Propose your own - Work on Introduction COURSE FORMAT

slide-1
SLIDE 1

CS 744: MESOS

Shivaram Venkataraman Fall 2020

slide-2
SLIDE 2

ADMINISTRIVIA

  • Assignment 1: How did it go?
  • Assignment 2 out tonight
  • Project details
  • Create project groups
  • Bid for projects/Propose your own
  • Work on Introduction
slide-3
SLIDE 3

COURSE FORMAT

Paper reviews “Compare, contrast and evaluate research papers” Discussion

slide-4
SLIDE 4

Scalable Storage Systems Datacenter Architecture Resource Management Computational Engines Machine Learning SQL Streaming Graph Applications

slide-5
SLIDE 5

MapReduce GFS Spark

slide-6
SLIDE 6

BACKGROUND: OS SCHEDULING

code, static data heap stack code, static data heap stack code, static data heap stack CPU

How do we share CPU between processes ?

slide-7
SLIDE 7

CLUSTER SCHEDULING

slide-8
SLIDE 8

TARGET ENVIRONMENT

Multiple MapReduce versions Mix of frameworks: MPI, Spark, MR Data sharing across frameworks Avoid per-framework clusters

slide-9
SLIDE 9

DESIGN

slide-10
SLIDE 10

RESOURCE OFFERS

slide-11
SLIDE 11

CONSTRAINTS

Examples of constraints Constraints in Mesos:

slide-12
SLIDE 12

DESIGN DETAILS

Allocation: Guaranteed allocation, revocation Isolation Containers (Docker)

slide-13
SLIDE 13

FAULT TOLERANCE

slide-14
SLIDE 14

PLACEMENT PREFERENCES

What is the problem? How do we do allocations?

slide-15
SLIDE 15

CENTRALIZED VS DECENTRALIZED

slide-16
SLIDE 16

CENTRALIZED VS DECENTRALIZED

Framework complexity Fragmentation, Starvation Inter-dependent framework

slide-17
SLIDE 17

COMPARISON: YARN

Per-job scheduler AM asks for resource RM replies

slide-18
SLIDE 18

COMPARISON: BORG

Single centralized scheduler Requests mem, cpu in cfg Priority per user / service Support for quotas / reservations

slide-19
SLIDE 19

SUMMARY

  • Mesos: Scheduler to share cluster between Spark, MR, etc.
  • Two-level scheduling with app-specific schedulers
  • Provides scalable, decentralized scheduling
  • Pluggable Policy ? Next class!
slide-20
SLIDE 20

DISCUSSION

https://forms.gle/urHSeukfyipCKjue6

slide-21
SLIDE 21

What are some problems that could come up if we scale from 10 frameworks to 1000 frameworks in Mesos?

slide-22
SLIDE 22
slide-23
SLIDE 23

List any one difference between an OS scheduler and Mesos

slide-24
SLIDE 24

NEXT STEPS

Next class: Scheduling Policy Further reading

  • https://www.umbrant.com/2015/05/27/mesos-omega-borg-a-survey/
  • https://queue.acm.org/detail.cfm?id=3173558