Cutting MapReduce Cost with Spot Market Huan Liu Accenture - - PowerPoint PPT Presentation

▶

Dec 09, 2022 101 likes •189 views

Cutting MapReduce Cost with Spot Market Huan Liu Accenture Technology Labs Why spot market? 2 Challenge with Spot Market 3 Cloud MapReduce Hadoop Our prior work MapReduce App MapReduce App Cloud MapReduce Hadoop Cloud OS Amazon

SLIDE 1

Cutting MapReduce Cost with Spot Market

Huan Liu Accenture Technology Labs

SLIDE 2

Why spot market?

SLIDE 3

Challenge with Spot Market

SLIDE 4

Cloud MapReduce

Hadoop

MapReduce App Cloud OS MapReduce App Hadoop Cloud MapReduce

Our prior work

Amazon

SLIDE 5

Cloud MapReduce architecture

Mapper Reducer Input queue Master reduce queue Output queue

….. Reduce queues

SimpleDB

SLIDE 6

A map node in Spot CMR

Input queue

Hypervisor Shutdown request

…..

Reduce queues One KV

utput

temp buffer Previous KV

utputs

staging buffer Spot CMR User-defined Map Function input KV pairs inform Flush Shutdown script

This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad This is a script Asfjialsjdf Sfjiasdfji asjfdif Sdji sdi alkjdi fd Dfji adjfi adf pjdoaf Akdjiji ad jijifad

SLIDE 7

Summary

First-ever MapReduce implementation that works in a

spot market environment

Uses on-demand checkpointing

Dramatic cost savings