CS 744: MAPREDUCE Shivaram Venkataraman Fall 2019 ANNOUNCEMENTS - PowerPoint PPT Presentation

CS 744: MAPREDUCE Shivaram Venkataraman Fall 2019

ANNOUNCEMENTS • Assignment 1 out • CloudLab notes on Piazza • No teams yet?

Applications Machine Learning SQL Streaming Graph Computational Engines Scalable Storage Systems Resource Management Datacenter Architecture

BACKGROUND: PTHREADS void *myThreadFun(void *vargp) { sleep(1); printf(“Hello World\n"); return NULL; } int main() { pthread_t thread_id_1, thread_id_2; pthread_create(&thread_id_1, NULL, myThreadFun, NULL); pthread_create(&thread_id_2, NULL, myThreadFun, NULL); pthread_join(thread_id_1, NULL); pthread_join(thread_id_2, NULL); exit(0); }

BACKGROUND: MPI mpirun -n 4 -f host_file ./mpi_hello_world int main(int argc, char** argv) { MPI_Init(NULL, NULL); // Get the number of processes int world_size; MPI_Comm_size(MPI_COMM_WORLD, &world_size); // Get the rank of the process int world_rank; MPI_Comm_rank(MPI_COMM_WORLD, &world_rank); // Print off a hello world message printf("Hello world from rank %d out of %d processors\n", world_rank, world_size); // Finalize the MPI environment. MPI_Finalize(); }

MOTIVATION Build Google Web Search - Crawl documents, build inverted indexes etc. Need for - automatic parallelization - network, disk optimization - handling of machine failures

OUTLINE - Programming Model - Execution Overview - Fault Tolerance - Optimizations

PROGRAMMING MODEL Data type: Each record is (key, value) Map function: (K in , V in ) à list(K inter , V inter ) Reduce function: (K inter , list(V inter )) à list(K out , V out )

Example: Word Count def def mapper(line): for for word in in line.split(): output(word, 1) def def reducer(key, values): output(key, sum(values))

Word Count Execution Input Map Shuffle & Sort Reduce Output the quick Map brown fox Reduce the fox ate Map the mouse Reduce how now Map brown cow

Word Count Execution Input Map Shuffle & Sort Reduce Output the, 1 brown, 1 the quick brown, 2 Map fox, 1 brown fox fox, 2 Reduce how, 1 how, 1 now, 1 now, 1 brown, 1 the, 1 the, 3 the fox ate Map fox, 1 the mouse the, 1 ate, 1 quick, 1 cow, 1 Reduce how now ate, 1 mouse, 1 Map brown mouse, 1 quick, 1 cow cow, 1

ASSUMPTIONS

ASSUMPTIONS 1. Commodity networking, less bisection bandwidth 2. Failures are common 3. Local storage is cheap 4. Replicated FS

Word Count Execution Submit a Job JobTracker Schedule tasks Automatically with locality split work Map Map Map how now the quick the fox ate brown brown fox the mouse cow

Fault Recovery If a task crashes: – Retry on another node – If the same task repeatedly fails, end the job Map Map Map how now the quick the fox ate brown brown fox the mouse cow

Fault Recovery If a node crashes: – Relaunch its current tasks on other nodes What about task inputs ? File system replication Map Map Map how now the quick the fox ate brown brown fox the mouse cow

Fault Recovery If a task is going slowly (straggler): – Launch second copy of task on another node – Take the output of whichever finishes first Map Map Map how now the quick the fox ate the quick brown brown fox the mouse brown fox cow

MORE DESIGN Master failure Locality Task Granularity

REFINEMENTS - Combiner functions - Counters - Skipping bad records

Jeff Dean, LADIS 2009

DISCUSSION https://forms.gle/hK8wFDxBDfS6chD28

DISCUSSION Indexing pipeline where you start with HTML documents. You want to index the documents after removing the most commonly occurring words. 1. Compute most common words. 2. Remove them and build the index. What are the main shortcomings of using MapReduce?

DISCUSSION

NEXT STEPS • Next lecture: Spark • Assignment 1: Use Piazza! • Project topics: End of this week