multi pass sorted neighborhood blocking with mapreduce
play

Multi-pass Sorted Neighborhood Blocking with MapReduce Lars Kolb, - PowerPoint PPT Presentation

Apr 05, 2024 •246 likes •658 views

Multi-pass Sorted Neighborhood Blocking with MapReduce Lars Kolb, Andreas Thor, Erhard Rahm Jens Hildebrandt, Jakob Zwiener Agenda 2 1. Sorted Neighborhood Method with Map Reduce with Entity Replication 2. Multipass Sorted

  1. Multi-pass Sorted Neighborhood Blocking with MapReduce Lars Kolb, Andreas Thor, Erhard Rahm Jens Hildebrandt, Jakob Zwiener

  2. Agenda 2 1. Sorted Neighborhood Method ■ with Map Reduce ■ with Entity Replication 2. Multipass Sorted Neighborhood Method 3. Load Balancing 4. Benchmarks Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  3. Sorted Neighborhood Method 3 1. Calculate Sorting Key sorting key artist_name disc_title Genre tracks • Genre + tracks Sonny Terry The Blues Blues 18 Fats Waller Portrait Jazz 17 Blind Blake Best Of Blues 18 I'M Fats Domino Walking Blues 18 Stony Chris Rea Blues 17 Road Jazz Jazz Jazz 20 Acustica Acustica Blues 19 Various The Blues Blues 17 Kelis Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  4. Sorted Neighborhood Method 4 1. Calculate Sorting Key sorting key artist_name disc_title Genre tracks • Genre + tracks Blues18 Sonny Terry The Blues Blues 18 2. Sort Jazz17 Fats Waller Portrait Jazz 17 Blues18 Blind Blake Best Of Blues 18 I'M Blues18 Fats Domino Walking Blues 18 Stony Blues17 Chris Rea Blues 17 Road Jazz20 Jazz Jazz Jazz 20 Blues19 Acustica Acustica Blues 19 Blues17 Various The Blues Blues 17 R+B17 Kelis Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  5. Sorted Neighborhood Method 5 1. Calculate Sorting Key sorting key artist_name disc_title Genre tracks • Genre + tracks Stony Blues17 Chris Rea Blues 17 2. Sort Road 3. Move a window over Blues17 Various The Blues Blues 17 the data Window size w = 3 • Blues18 Sonny Terry The Blues Blues 18 • Row count n = 9 Blues18 Blind Blake Best Of Blues 18 I'M Comparisons: ? Blues18 Fats Domino Walking Blues 18 Comparisons: O(n*w) Blues19 Acustica Acustica Blues 19 Jazz17 Fats Waller Portrait Jazz 17 Jazz20 Jazz Jazz Jazz 20 R+B17 Kelis Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  6. Sorted Neighborhood with Map Reduce - Algorithm 6 disc_title Genre tracks The Blues Blues 18 Portrait Jazz 17 Best Of Blues 18 disc_title Genre tracks I'M Blues 18 Walking Stony Road Blues 17 Jazz Jazz 20 disc_title Genre tracks Acustica Blues 19 The Blues Blues 17 Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  7. Sorted Neighborhood with Map Reduce - Algorithm 7 disc_title Genre tracks sort disc_title ... map 1 The Blues Blues 18 The Blues ... Portrait ... Portrait Jazz 17 Best Of ... Best Of Blues 18 disc_title Genre tracks sort disc_title ... I'M map 2 Blues 18 Walking I'M Walking ... Stony Road Blues 17 Stony Road ... Jazz ... Jazz Jazz 20 disc_title Genre tracks sort disc_title ... Acustica Blues 19 map 3 Acustica ... The Blues Blues 17 The Blues ... Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  8. Sorted Neighborhood with Map Reduce - Algorithm 8 Map: disc_title Genre tracks sort disc_title ... 1. Calculate map 1 The Blues Blues 18 Blues18 The Blues ... SortingKey: Jazz17 Portrait ... Portrait Jazz 17 Genre+tracks Blues18 Best Of ... Best Of Blues 18 disc_title Genre tracks sort disc_title ... I'M map 2 Blues 18 Walking Blues18 I'M Walking ... Stony Road Blues 17 Blues17 Stony Road ... Jazz20 Jazz ... Jazz Jazz 20 disc_title Genre tracks sort disc_title ... Acustica Blues 19 map 3 Blues19 Acustica ... The Blues Blues 17 Blues17 The Blues ... R+B17 Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  9. Sorted Neighborhood with Map Reduce - Algorithm 9 Map: disc_title Genre tracks part.sort disc_title ... 1. Calculate map 1 The Blues Blues 18 1.Blues18 The Blues ... SortingKey: Jazz17 Portrait ... Portrait Jazz 17 Genre+tracks 1.Blues18 Best Of ... Best Of Blues 18 2. Calculate Partition: disc_title Genre tracks sorting key partition part.sort disc_title ... I'M B… 1 map 2 Blues 18 Walking 1.Blues18 I'M Walking ... Stony Road Blues 17 1.Blues17 Stony Road ... Jazz20 Jazz ... Jazz Jazz 20 disc_title Genre tracks part.sort disc_title ... Acustica Blues 19 map 3 1.Blues19 Acustica ... The Blues Blues 17 1.Blues17 The Blues ... R+B17 Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  10. Sorted Neighborhood with Map Reduce - Algorithm 10 Map: disc_title Genre tracks part.sort disc_title ... 1. Calculate map 1 The Blues Blues 18 1.Blues18 The Blues ... SortingKey: 2.Jazz17 Portrait ... Portrait Jazz 17 Genre+tracks 1.Blues18 Best Of ... Best Of Blues 18 2. Calculate Partition: disc_title Genre tracks sorting key partition part.sort disc_title ... I'M B… 1 map 2 Blues 18 Walking 1.Blues18 I'M Walking ... J… 2 Stony Road Blues 17 1.Blues17 Stony Road ... 2.Jazz20 Jazz ... Jazz Jazz 20 disc_title Genre tracks part.sort disc_title ... Acustica Blues 19 map 3 1.Blues19 Acustica ... The Blues Blues 17 1.Blues17 The Blues ... R+B17 Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  11. Sorted Neighborhood with Map Reduce - Algorithm 11 Map: disc_title Genre tracks part.sort disc_title ... 1. Calculate map 1 The Blues Blues 18 1.Blues18 The Blues ... SortingKey: 2.Jazz17 Portrait ... Portrait Jazz 17 Genre+tracks 1.Blues18 Best Of ... Best Of Blues 18 2. Calculate Partition: disc_title Genre tracks sorting key partition part.sort disc_title ... I'M B… 1 map 2 Blues 18 Walking 1.Blues18 I'M Walking ... J… 2 R… 2 Stony Road Blues 17 1.Blues17 Stony Road ... 2.Jazz20 Jazz ... Jazz Jazz 20 disc_title Genre tracks part.sort disc_title ... Acustica Blues 19 map 3 1.Blues19 Acustica ... The Blues Blues 17 1.Blues17 The Blues ... 2.R+B17 Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  12. Sorted Neighborhood with Map Reduce - Algorithm 12 disc_title Genre tracks part.sort disc_title ... map 1 The Blues Blues 18 1.Blues18 The Blues ... 2.Jazz17 Portrait ... Portrait Jazz 17 1.Blues18 Best Of ... Best Of Blues 18 Partitioning disc_title Genre tracks part.sort disc_title ... I'M map 2 Blues 18 Walking 1.Blues18 I'M Walking ... Stony Road Blues 17 1.Blues17 Stony Road ... 2.Jazz20 Jazz ... Jazz Jazz 20 disc_title Genre tracks part.sort disc_title ... Acustica Blues 19 map 3 1.Blues19 Acustica ... The Blues Blues 17 1.Blues17 The Blues ... 2.R+B17 Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  13. Sorted Neighborhood with Map Reduce - Algorithm 13 disc_title Genre tracks part.sort disc_title ... part.sort disc_title ... map 1 The Blues Blues 18 1.Blues18 The Blues ... 1.Blues17 The Blues ... 2.Jazz17 Portrait ... Portrait Jazz 17 1.Blues17 Stony Road ... 1.Blues18 Best Of ... 1.Blues18 I'M Walking ... Best Of Blues 18 1.Blues18 Best Of ... 1.Blues18 The Blues ... Partitioning disc_title Genre tracks part.sort disc_title ... 1.Blues19 Acustica ... I'M map 2 Blues 18 Walking 1.Blues18 I'M Walking ... Stony Road Blues 17 1.Blues17 Stony Road ... 2.Jazz20 Jazz ... Jazz Jazz 20 part.sort disc_title ... 2.Jazz17 Portrait ... disc_title Genre tracks part.sort disc_title ... 2.Jazz20 Jazz ... Acustica Blues 19 map 3 1.Blues19 Acustica ... 2.R+B17 Tasty ... The Blues Blues 17 1.Blues17 The Blues ... 2.R+B17 Tasty ... Tasty R+B 17 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  14. Sorted Neighborhood with Map Reduce - Limitations 14 reduce 1 • Neighboring sorting keys must be on the same reducer part.sort disc_title ...  own partition function 1.Blues17 The Blues ... 1.Blues17 Stony Road ... • Self defined partitioning + sorting 1.Blues18 I'M Walking ... • Internal load balancing does not work 1.Blues18 Best Of ... anymore 1.Blues18 The Blues ... 1.Blues19 Acustica ... • Boundary entities reduce 2 • Sliding window cannot compare entities that are assigned to different reduce nodes part.sort disc_title ... • Solution: data replication 2.Jazz17 Portrait ... 2.Jazz20 Jazz ... 2.R+B17 Tasty ... Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  15. Sorted Neighborhood with Entity Replication 15 map 1 reduce 1 map 2 reduce 2 map 3 Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

  16. Sorted Neighborhood with Entity Replication 16 sort disc_title ... Blues18 The Blues ... map 1 Jazz17 Portrait ... reduce 1 Blues18 Best Of ... sort disc_title ... Blues18 I'M Walking ... map 2 Blues17 Stony Road ... Jazz20 Jazz ... reduce 2 sort disc_title ... Blues19 Acustica ... map 3 Blues17 The Blues ... R+B17 Tasty ... Sorted Neighborhood with MapReduce | Jens Hildebrandt, Jakob Zwiener | 6. Mai 2013

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

MapReduce Andrew Crotty Alex Galakatos What is MapReduce? MapReduce is a framework for:

MapReduce Andrew Crotty Alex Galakatos What is MapReduce? MapReduce is a framework for:

MapReduce Andrew Crotty Alex Galakatos What is MapReduce? MapReduce is a framework for: parallelizable problems large datasets cluster/grid computing Background Google project Implemented many special-purpose computations

373 views • 26 slides
MIP Relaxation and Large Neighborhood Search for a Multi-Mode Resource-Constrained Multi-Project

MIP Relaxation and Large Neighborhood Search for a Multi-Mode Resource-Constrained Multi-Project

MIP Relaxation and Large Neighborhood Search for a Multi-Mode Resource-Constrained Multi-Project Scheduling Problem Christian artigues Emmanuel H ebrard LAAS-CNRS, Univ of Toulouse, France August 29, MISTA 2013, Ghent, Belgium C. Artigues,

160 views • 13 slides
Graphs, Polymorphisms, and Multi-Sorted Structures Ross Willard University of Waterloo NSAC 2013

Graphs, Polymorphisms, and Multi-Sorted Structures Ross Willard University of Waterloo NSAC 2013

Graphs, Polymorphisms, and Multi-Sorted Structures Ross Willard University of Waterloo NSAC 2013 University of Novi Sad June 6, 2013 Ross Willard (Waterloo) Graphs, Polymorphisms, Multi-Sorted Struc NSAC 2013 1 / 26 Background Structure :

512 views • 26 slides
Autosubst 2: Towards Reasoning with Multi-Sorted de Bruijn Terms and Vector Substitutions Jonas

Autosubst 2: Towards Reasoning with Multi-Sorted de Bruijn Terms and Vector Substitutions Jonas

Introduction Towards Autosubst 2 Interpreting HOAS Vector Substitutions Case Study Future Work Autosubst 2: Towards Reasoning with Multi-Sorted de Bruijn Terms and Vector Substitutions Jonas Kaiser, Steven Schfer, Kathrin Stark saarland

555 views • 22 slides
Amit Chakrabarti Dartmouth College WAPMDS, IIT Kanpur, Dec 2009 Amit Chakrabarti 1 Multi-Pass

Amit Chakrabarti Dartmouth College WAPMDS, IIT Kanpur, Dec 2009 Amit Chakrabarti 1 Multi-Pass

Multi-Pass Lower Bounds Dec 20, 2009 Multi-pass Data Stream Lower Bounds via Round Elimination Amit Chakrabarti Dartmouth College WAPMDS, IIT Kanpur, Dec 2009 Amit Chakrabarti 1 Multi-Pass Lower Bounds Dec 20, 2009 Lower Bounds Paradigms

846 views • 70 slides
Dimension Independent Matrix Square Introduction using MapReduce The Problem Why Bother

Dimension Independent Matrix Square Introduction using MapReduce The Problem Why Bother

Dimension Independent Matrix Square Reza Zadeh Dimension Independent Matrix Square Introduction using MapReduce The Problem Why Bother MapReduce First Pass Naive Reza Bosagh Zadeh Analysis DIMSUM Algorithm Shuffle Size Correctness

546 views • 36 slides
ARGYLE HOUSING SITE CONSTRAINTS Long facades with 400' frontage Blocking sun from

ARGYLE HOUSING SITE CONSTRAINTS Long facades with 400' frontage Blocking sun from

NEIGHBORHOOD MEETING ARGYLE HOUSING SITE CONSTRAINTS Long facades with 400' frontage Blocking sun from outdoor spaces Disconnected parking in back LIGHT CONNECTIONS 3 connected courtyards Pull public into courtyards and

441 views • 9 slides
MapReduce 320302 Databases & Web Services (P. Baumann) 1 Why MapReduce? Motivation: Large

MapReduce 320302 Databases & Web Services (P. Baumann) 1 Why MapReduce? Motivation: Large

MapReduce 320302 Databases & Web Services (P. Baumann) 1 Why MapReduce? Motivation: Large Scale Data Processing MapReduce Idea: simple, highly scalable, generic parallelization model Want to process lots of data ( > 1 TB)

780 views • 49 slides
Data Blocking Jon K. Nilsen Department of Physics and Scientific Computing Group University of

Data Blocking Jon K. Nilsen Department of Physics and Scientific Computing Group University of

Data Blocking Jon K. Nilsen Department of Physics and Scientific Computing Group University of Oslo, N-0316 Oslo, Norway Spring 2008 Computational Physics II FYS4410 Outline Data Blocking Why blocking? What is blocking? Blocking in

626 views • 14 slides
Lecture 16: Overview of MapReduce MapReduce is a parallel, distributed programming model and

Lecture 16: Overview of MapReduce MapReduce is a parallel, distributed programming model and

Lecture 16: Overview of MapReduce MapReduce is a parallel, distributed programming model and implementation used to process and generate large data sets. The map component of a MapReduce job typically parses input data and distills it down to

532 views • 5 slides
Lecture 36: MapReduce Frameworks [Adapted from slides by John DeNero and MapReduce is a

Lecture 36: MapReduce Frameworks [Adapted from slides by John DeNero and MapReduce is a

Lecture 36: MapReduce Frameworks [Adapted from slides by John DeNero and MapReduce is a framework for batch processing of Big Data: http://research.google.com/archive/mapreduce-osdi04-slides] Framework: A system used by programmers to build

186 views • 3 slides
Multi-marginal optimal transportation and hedonic pricing Brendan Pass University of Alberta

Multi-marginal optimal transportation and hedonic pricing Brendan Pass University of Alberta

Multi-marginal optimal transportation and hedonic pricing Brendan Pass University of Alberta June 4, 2012 Brendan Pass Multi-marginal optimal transportation and hedonic pricing Introduction Probability measures i on X i R n , i = 1 , 2

451 views • 8 slides
Mrs: MapReduce for Scientific Computing in Python Andrew McNabb, Jeff Lund , and Kevin Seppi

Mrs: MapReduce for Scientific Computing in Python Andrew McNabb, Jeff Lund , and Kevin Seppi

MapReduce MapReduce in Scientific Computing Mrs Features Performance and Case Studies Mrs: MapReduce for Scientific Computing in Python Andrew McNabb, Jeff Lund , and Kevin Seppi Brigham Young University November 16, 2012 MapReduce MapReduce

432 views • 29 slides
Blocking in the 2 k Design Blocking may be required because: we cannot perform all required runs

Blocking in the 2 k Design Blocking may be required because: we cannot perform all required runs

ST 516 Experimental Statistics for Engineers II Blocking in the 2 k Design Blocking may be required because: we cannot perform all required runs under homogeneous conditions; e.g. raw material comes in limited batches; we want to carry out runs

487 views • 23 slides
COMP9313: Big Data Management MapReduce Data Structure in MapReduce Key-value pairs are the

COMP9313: Big Data Management MapReduce Data Structure in MapReduce Key-value pairs are the

COMP9313: Big Data Management MapReduce Data Structure in MapReduce Key-value pairs are the basic data structure in MapReduce Keys and values can be: integers, float, strings, raw bytes They can also be arbitrary data structures

1.77k views • 65 slides
Quines Conjecture on Many-Sorted Logic Thomas Barrett and Hans Halvorson Dominik Ehrenfels St

Quines Conjecture on Many-Sorted Logic Thomas Barrett and Hans Halvorson Dominik Ehrenfels St

Many-sorted logic Quines Conjecture on Many-Sorted Logic Thomas Barrett and Hans Halvorson Dominik Ehrenfels St Cross College March 2, 2019 1 / 41 Many-sorted logic Many-sorted logic - Syntax Non-logical vocabulary: 1. non-empty set of

901 views • 41 slides
HLZ - counting and sorting machines for UBCs About us Company founded 1945 Location of

HLZ - counting and sorting machines for UBCs About us Company founded 1945 Location of

HLZ - counting and sorting machines for UBCs About us Company founded 1945 Location of company lholm, Denmark Number of employees about 30 Supplier of brewery equipment since 1945 About us (2) 2003 - delivery of the

617 views • 47 slides
Preliminary results from the operation of the fjrst WEEE sorting centers in Greece K.

Preliminary results from the operation of the fjrst WEEE sorting centers in Greece K.

Preliminary results from the operation of the fjrst WEEE sorting centers in Greece K. Lasaridi*, C. Chroni, C. Abeliotis, G. Protopapas, P . Kirkitsos, A. Dalamaga, R. Charitopoulou, K. Kalafata, C. Angelakopoulos *klasaridi@hua.gr

462 views • 20 slides
Team 10: Automatic Lego Sorting Machine BY: AUSTIN SHORR ERIC PISCIOTTA TRISTIAN VIGUERIA

Team 10: Automatic Lego Sorting Machine BY: AUSTIN SHORR ERIC PISCIOTTA TRISTIAN VIGUERIA

Team 10: Automatic Lego Sorting Machine BY: AUSTIN SHORR ERIC PISCIOTTA TRISTIAN VIGUERIA Project description Design and construct an apparatus capable of automatically sorting Lego pieces by block type Sponsored by David Willy in

405 views • 12 slides
Technical Challenges Emery | Engineering | May 16th, 2018 AEMT Scottish Meeting May 2018

Technical Challenges Emery | Engineering | May 16th, 2018 AEMT Scottish Meeting May 2018

HV Test and Overspeed Facility Technical Challenges Emery | Engineering | May 16th, 2018 AEMT Scottish Meeting May 2018 Agenda 1. Introduction 2. Motor Testing 3. High Speed Rotor Testing AEMT Scottish Meeting May 2018 Introduction

917 views • 29 slides
Graph-Based Remerging of Genealogical Databases D. Randall Wilson fonix Corporation Draper,

Graph-Based Remerging of Genealogical Databases D. Randall Wilson fonix Corporation Draper,

Graph-Based Remerging of Genealogical Databases D. Randall Wilson fonix Corporation Draper, Utah, USA e-mail: WilsonR@fonix.com or randy@axon.cs.byu.edu Workshop on Technology for Family History and Genealogical Research Brigham Young

159 views • 12 slides
PhUSE 2016 Paper CC08 Perish the Sort: Using Indexes and Hash Objects for Efficient Programming

PhUSE 2016 Paper CC08 Perish the Sort: Using Indexes and Hash Objects for Efficient Programming

PhUSE 2016 Paper CC08 Perish the Sort: Using Indexes and Hash Objects for Efficient Programming Jason A Smith Argo Analytics Ltd Contents Indexes to combine or classify data What is an index? Simple index Composite

284 views • 27 slides
Future direction for using FOSS4G CONTENTS Country Background Actions by the National Government

Future direction for using FOSS4G CONTENTS Country Background Actions by the National Government

Case of Managing Early Warning Systems for Monitoring Natural Hazards in El Salvador Future direction for using FOSS4G CONTENTS Country Background Actions by the National Government Natural Threats - Integrated Monitoring Center Tools

505 views • 47 slides
Combating Drivers License Fraud By Owen McShane Director of Investigations New York State

Combating Drivers License Fraud By Owen McShane Director of Investigations New York State

Combating Drivers License Fraud By Owen McShane Director of Investigations New York State Department of Motor Vehicles Combating Drivers License Fraud Facial Recognition Name Matching Software Document Scanning Workstations

518 views • 23 slides

More recommend