cutting mapreduce cost with spot market
play

Cutting MapReduce Cost with Spot Market Huan Liu Accenture - PowerPoint PPT Presentation

Dec 09, 2022 •101 likes •185 views

Cutting MapReduce Cost with Spot Market Huan Liu Accenture Technology Labs Why spot market? 2 Challenge with Spot Market 3 Cloud MapReduce Hadoop Our prior work MapReduce App MapReduce App Cloud MapReduce Hadoop Cloud OS Amazon

  1. Cutting MapReduce Cost with Spot Market Huan Liu Accenture Technology Labs

  2. Why spot market? 2

  3. Challenge with Spot Market 3

  4. Cloud MapReduce � Hadoop � Our prior work MapReduce App MapReduce App Cloud MapReduce Hadoop Cloud OS Amazon 4

  5. Cloud MapReduce architecture Output queue Master reduce queue Reducer SimpleDB ….. Reduce queues Input queue Mapper 5

  6. A map node in Spot CMR Input queue Spot CMR This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd input KV pairs Dfji adjfi adf pjdoaf Akdjiji ad jijifad inform This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf User-defined Map Function Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad temp buffer staging buffer Shutdown This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd request Dfji adjfi adf pjdoaf One KV Previous KV Akdjiji ad jijifad Shutdown output outputs script Hypervisor Flush ….. 6 Reduce queues

  7. Summary � First-ever MapReduce implementation that works in a spot market environment � Uses on-demand checkpointing � Dramatic cost savings 7

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data Analy/c Cloud Instance Op/ons MapReduce Spot Instances Evalua/on Data

Data Analy/c Cloud Instance Op/ons MapReduce Spot Instances Evalua/on Data

Navraj Chohan 1 Claris Cas/llo 2 Mike Spreitzer 2 Malgorzata Steinder 2 Asser Tantawi 2 Chandra Krintz 1 UC Santa Barbara 1 IBM Research 2 Data Analy/c Cloud Instance Op/ons MapReduce Spot Instances Evalua/on Data Public Cloud

197 views • 19 slides
Neodymium and Praseodymium NdPr Biggest Blind Spot in the Global Commodity Market Peak

Neodymium and Praseodymium NdPr Biggest Blind Spot in the Global Commodity Market Peak

Neodymium and Praseodymium NdPr Biggest Blind Spot in the Global Commodity Market Peak Resources - Becoming one of worlds lowest cost fully integrated NdPr producers BEHIND EACH BATTERY IS A MOTOR Over 90% of all new energy vehicles will

190 views • 16 slides
Neodymium and Praseodymium NdPr Biggest Blind Spot in the Global Commodity Market Peak

Neodymium and Praseodymium NdPr Biggest Blind Spot in the Global Commodity Market Peak

Neodymium and Praseodymium NdPr Biggest Blind Spot in the Global Commodity Market Peak Resources - Becoming one of worlds lowest cost fully integrated NdPr producers BEHIND EACH BATTERY IS A MOTOR Over 90% of all new energy vehicles will

206 views • 17 slides
MapReduce Andrew Crotty Alex Galakatos What is MapReduce? MapReduce is a framework for:

MapReduce Andrew Crotty Alex Galakatos What is MapReduce? MapReduce is a framework for:

MapReduce Andrew Crotty Alex Galakatos What is MapReduce? MapReduce is a framework for: parallelizable problems large datasets cluster/grid computing Background Google project Implemented many special-purpose computations

373 views • 26 slides
SPOT RATE THE NORMAL RATE QUOTED IN THE FOREIGN EXCHANGE MARKET IS THE SPOT RATE. THIS

SPOT RATE THE NORMAL RATE QUOTED IN THE FOREIGN EXCHANGE MARKET IS THE SPOT RATE. THIS

Chapter No. Page No. 20 350 International Finance COMMERCIAL RATES OF EXCHANGE SPOT RATE THE NORMAL RATE QUOTED IN THE FOREIGN EXCHANGE MARKET IS THE SPOT RATE. THIS RATE IS QUOTED FOR TRANSACTIONS WHERE THE FOREIGN CURRENCY

436 views • 12 slides
Electric Spot Prices and Wind Forecasts: A dynamic Nordic/Baltic Electricity Market Analysis using

Electric Spot Prices and Wind Forecasts: A dynamic Nordic/Baltic Electricity Market Analysis using

Norwegian University of Science and Technology Electric Spot Prices and Wind Forecasts: A dynamic Nordic/Baltic Electricity Market Analysis using Nonlinear Impulse-Response Methodology by Professor Per B Solibakke 4.2% 1 1 Spot Electricity

533 views • 25 slides
Christchurch 20 th November 2012 Cutting Edge 2012 School of Addiction 2013 Cutting Edge 2013

Christchurch 20 th November 2012 Cutting Edge 2012 School of Addiction 2013 Cutting Edge 2013

Leadership Day Christchurch 20 th November 2012 Cutting Edge 2012 School of Addiction 2013 Cutting Edge 2013 Practitioner Registration Cutting Edge 2012 Relevant: 97.5% Enjoyable: 99% Recommend: 89% Cost reasonable 80%

255 views • 21 slides
Presentation to AEMC Presentation to AEMC Physical Market Cap Trigger Physical Market Cap

Presentation to AEMC Presentation to AEMC Physical Market Cap Trigger Physical Market Cap

Presentation to AEMC Presentation to AEMC Physical Market Cap Trigger Physical Market Cap Trigger & Spot Market Insurance Fund & Spot Market Insurance Fund 20 th January 2009 Presentation Outline Defining the Problem; Our experience

686 views • 15 slides
MapReduce 320302 Databases & Web Services (P. Baumann) 1 Why MapReduce? Motivation: Large

MapReduce 320302 Databases & Web Services (P. Baumann) 1 Why MapReduce? Motivation: Large

MapReduce 320302 Databases & Web Services (P. Baumann) 1 Why MapReduce? Motivation: Large Scale Data Processing MapReduce Idea: simple, highly scalable, generic parallelization model Want to process lots of data ( > 1 TB)

780 views • 49 slides
FSD3 Grass Cutting Cost Analysis January 2015 Purpose: The purpose of this study is to analyze

FSD3 Grass Cutting Cost Analysis January 2015 Purpose: The purpose of this study is to analyze

FSD3 Grass Cutting Cost Analysis January 2015 Purpose: The purpose of this study is to analyze the benefits & costs of using contracted services for grass cutting vs using in-district employees & equipment. Method: In-District Grass

249 views • 3 slides
Lecture 16: Overview of MapReduce MapReduce is a parallel, distributed programming model and

Lecture 16: Overview of MapReduce MapReduce is a parallel, distributed programming model and

Lecture 16: Overview of MapReduce MapReduce is a parallel, distributed programming model and implementation used to process and generate large data sets. The map component of a MapReduce job typically parses input data and distills it down to

532 views • 5 slides
Lecture 36: MapReduce Frameworks [Adapted from slides by John DeNero and MapReduce is a

Lecture 36: MapReduce Frameworks [Adapted from slides by John DeNero and MapReduce is a

Lecture 36: MapReduce Frameworks [Adapted from slides by John DeNero and MapReduce is a framework for batch processing of Big Data: http://research.google.com/archive/mapreduce-osdi04-slides] Framework: A system used by programmers to build

186 views • 3 slides
Tony DiLodovico President Tony D Environmental Permitting, LLC cutting red tape with cost -

Tony DiLodovico President Tony D Environmental Permitting, LLC cutting red tape with cost -

Tony DiLodovico President Tony D Environmental Permitting, LLC cutting red tape with cost - effective environmentally protective solutions TonyDEP.llc@g-mail.com 732-740-5725 Avoid disturbance

805 views • 33 slides
Mrs: MapReduce for Scientific Computing in Python Andrew McNabb, Jeff Lund , and Kevin Seppi

Mrs: MapReduce for Scientific Computing in Python Andrew McNabb, Jeff Lund , and Kevin Seppi

MapReduce MapReduce in Scientific Computing Mrs Features Performance and Case Studies Mrs: MapReduce for Scientific Computing in Python Andrew McNabb, Jeff Lund , and Kevin Seppi Brigham Young University November 16, 2012 MapReduce MapReduce

432 views • 29 slides
cutting deployment risk, time & cost 1 Introduction Todays EW systems must meet

cutting deployment risk, time & cost 1 Introduction Todays EW systems must meet

cutting deployment risk, time & cost 1 Introduction Todays EW systems must meet threat environment that are diverse, deceptive & agile. To confront these challenges the systems must offer: Ultimate performance for

326 views • 18 slides
COMP9313: Big Data Management MapReduce Data Structure in MapReduce Key-value pairs are the

COMP9313: Big Data Management MapReduce Data Structure in MapReduce Key-value pairs are the

COMP9313: Big Data Management MapReduce Data Structure in MapReduce Key-value pairs are the basic data structure in MapReduce Keys and values can be: integers, float, strings, raw bytes They can also be arbitrary data structures

1.77k views • 65 slides
DTrace Topics: /* print main headers */ OPT_dump ? printf("%s %s %s % DTraceToolkit

DTrace Topics: /* print main headers */ OPT_dump ? printf("%s %s %s % DTraceToolkit

#pragma D option quiet #pragma D option switchrate=10hz /* * Print header */ dtrace:::BEGIN { /* print optional headers */ OPT_time ? printf("%-14s ", " OPT_timestr ? printf("%-20s ", " OPT_zone ?

451 views • 31 slides
CORS & WEBSOCKETS Dealin g wit h Sam e -Origi n Polic y application lives on llama.com

CORS & WEBSOCKETS Dealin g wit h Sam e -Origi n Polic y application lives on llama.com

CS 498RK SPRING 2020 CORS & WEBSOCKETS Dealin g wit h Sam e -Origi n Polic y application lives on llama.com and you want to pull data from www.alpaca.com var xhr = new XMLHttpRequest(); xhr.open('GET',

773 views • 41 slides
Cross-Site Scripting XSS Many sites allow users to upload information Blogs, photo

Cross-Site Scripting XSS Many sites allow users to upload information Blogs, photo

Cross-Site Scripting XSS Many sites allow users to upload information Blogs, photo sharing, Facebook, etc. Which gets permanently stored And displayed Attack based on uploading a script Other users inadvertently download

428 views • 14 slides
!"##"$%&'(&)*'+,-&#.*",-/,0 ' @;#/?*. ' A' !"#$%&

!"##"$%&'(&)*'+,-&#.*",-/,0 ' @;#/?*. ' A' !"#$%&

!"##"$%&'(&)*'+,-&#.*",-/,0 ' @;#/?*. ' A' !"#$%& '/.'"'.*&#&4*=?/;"5'.&BC&,;&'4D'&%&,*.'"..4;/"*&-'

583 views • 9 slides
Shell Scripting Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your

Shell Scripting Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your

CSCI 2132: Software Development Norbert Zeh Faculty of Computer Science Shell Scripting Dalhousie University Winter 2019 Reading Glass and Ables, Chapter 8: bash Your Shell vs Your File Manager File manager Easy and intuitive to use

267 views • 25 slides
NLU lecture 6: Compositional character representations Adam Lopez alopez@inf.ed.ac.uk Credits:

NLU lecture 6: Compositional character representations Adam Lopez alopez@inf.ed.ac.uk Credits:

NLU lecture 6: Compositional character representations Adam Lopez alopez@inf.ed.ac.uk Credits: Clara Vania 2 Feb 2018 Lets revisit an assumption in language modeling (& word2vec) When does this assumption make sense for language

801 views • 66 slides
STAT 113: FINAL EXAM PRACTICE PROBLEMS COLIN REIMER DAWSON, FALL 2015 Research Design /

STAT 113: FINAL EXAM PRACTICE PROBLEMS COLIN REIMER DAWSON, FALL 2015 Research Design /

STAT 113: FINAL EXAM PRACTICE PROBLEMS COLIN REIMER DAWSON, FALL 2015 Research Design / Describing Samples. (1) The following measures can be used to describe distributions (either population or sample distributions). For each one describe

239 views • 8 slides
using Amazon Key Management Service Jan Lindstrm, Principal Engineer, MariaDB Corporation

using Amazon Key Management Service Jan Lindstrm, Principal Engineer, MariaDB Corporation

Transparent tablespace and log encryption on MariaDB 10.1 using Amazon Key Management Service Jan Lindstrm, Principal Engineer, MariaDB Corporation Amsterdam, Netherlands | October 5, 2016 Agenda 1. Introduction 2. Concepts 3.

795 views • 32 slides

More recommend