Programming MapReduce in Mathematica Paul-Jean Letourneau Data - - PowerPoint PPT Presentation

▶
programming mapreduce in mathematica
SMART_READER_LITE
LIVE PREVIEW

Programming MapReduce in Mathematica Paul-Jean Letourneau Data - - PowerPoint PPT Presentation

Sep 12, 2023 118 likes •609 views

Programming MapReduce in Mathematica Paul-Jean Letourneau Data Scientist, Wolfram Research Commercial Users of Functional Programming Sept 22, 2013 2 cufp-2013-talk-slides.nb personal analytics cufp-2013-talk-slides.nb 3 4

slide-1
SLIDE 1

Programming MapReduce in Mathematica

Paul-Jean Letourneau Data Scientist, Wolfram Research

Commercial Users of Functional Programming Sept 22, 2013

slide-2
SLIDE 2

personal analytics

2 cufp-2013-talk-slides.nb

slide-3
SLIDE 3

cufp-2013-talk-slides.nb 3

slide-4
SLIDE 4

experimental computation

4 cufp-2013-talk-slides.nb

slide-5
SLIDE 5

cufp-2013-talk-slides.nb 5

slide-6
SLIDE 6

bioinformatics

6 cufp-2013-talk-slides.nb

slide-7
SLIDE 7

genomics

cufp-2013-talk-slides.nb 7

slide-8
SLIDE 8

distributed computation

8 cufp-2013-talk-slides.nb

slide-9
SLIDE 9
  • verview

core principles of Mathematica examples programming MapReduce with Mathematica

cufp-2013-talk-slides.nb 9

slide-10
SLIDE 10

the fundamental principles

  • 1. everything is an expression
  • 2. expressions are transformed until they stop changing
  • 3. transformation rules are patterns

10 cufp-2013-talk-slides.nb

slide-11
SLIDE 11
  • 1. everything is an expression

expressions are data structures

Mathematica expression:

head [ arg1, arg2, ...]

LISP expr:

(head arg1 arg2 ...)

cufp-2013-talk-slides.nb 11

slide-12
SLIDE 12
  • 1. everything is an expression

FullForm

1 + 1 2 FullForm@Unevaluated@1 + 1DD Unevaluated@Plus@1, 1DD FullForm@Unevaluated@1 + 1 - 3 aDD Unevaluated@Plus@1, 1, Times@-1, Times@3, aDDDD

12 cufp-2013-talk-slides.nb

slide-13
SLIDE 13
  • 1. everything is an expression

... with lots of syntactic sugar

Ò + 1 & êü Range@10D 82, 3, 4, 5, 6, 7, 8, 9, 10, 11< FullForm@Unevaluated@Ò + 1 & êü Range@10DDD Unevaluated@Map@Function@Plus@Slot@1D, 1DD, Range@10DDD

cufp-2013-talk-slides.nb 13

slide-14
SLIDE 14
  • 2. expressions are transformed until they stop changing

definitions are rules

Clear@aD; a = 1; a 1

14 cufp-2013-talk-slides.nb

slide-15
SLIDE 15
  • 2. expressions are transformed until they stop changing

rules transform expressions: infinite evaluation

OwnValues@aD 8HoldPattern@aD ß 1< a êê Trace 8a, 1< Clear@bD; a = 1; a + b + 1 êê Trace 88a, 1<, 1 + b + 1, 2 + b< b = 2; a + b + 1 êê Trace 88a, 1<, 8b, 2<, 1 + 2 + 1, 4<

cufp-2013-talk-slides.nb 15

slide-16
SLIDE 16
  • 3. rules are patterns

rules have patterns

a = 1; OwnValues@aD 8HoldPattern@aD ß 1<

16 cufp-2013-talk-slides.nb

slide-17
SLIDE 17
  • 3. rules are patterns

functions are rules

Clear@f, g, a, bD; f@x_IntegerD := x + 1 DownValues@fD êê Column HoldPattern@f@x_IntegerDD ß x + 1 Head@1D Integer f@1D 2 f@"a"D f@aD Head@"a"D String

cufp-2013-talk-slides.nb 17

slide-18
SLIDE 18
  • 3. rules are patterns
  • rdering of rules

f@1D := 1000 DownValues@fD êê Column HoldPattern@f@1DD ß 1000 HoldPattern@f@x_IntegerDD ß x + 1 f êü 80, 1, 2, 3, 4, 5< 81, 1000, 3, 4, 5, 6<

18 cufp-2013-talk-slides.nb

slide-19
SLIDE 19

program as data

expressions are immutable

10 = 1

Set::setraw : Cannot assign to raw object 10. à

1 Plus@1, 1D = 3

Set::write : Tag Plus in 1 + 1 is Protected. à

3 a = 10 10 a = 1 1

cufp-2013-talk-slides.nb 19

slide-20
SLIDE 20

program as data

homoiconicity: expressions ARE the data structure

Clear@aD; TreeForm@Unevaluated@1 + 1 - 3 aDD

Plus 2 Times

  • 3

a

20 cufp-2013-talk-slides.nb

slide-21
SLIDE 21

examples

Fibonacci sequence

fib@n_D := fib@nD = fib@n - 2D + fib@n - 1D; fib@1D = 1; fib@2D = 1; Table@fib@nD, 8n, 1, 10<D 81, 1, 2, 3, 5, 8, 13, 21, 34, 55< ListLogLogPlot@Table@fib@nD, 8n, 1, 100<DD

2 5 10 20 50 100 104 108 1012 1016 1020

cufp-2013-talk-slides.nb 21

slide-22
SLIDE 22

examples

scrape a web page

GridüPartition@Show@ImportüÒ, ImageSize Ø 50D & êü Unionü FlattenüTable@Cases@Import@"http:êêcufp.orgêconferenceêsessionsê2013?page=" <> IntegerStringün, "XMLObject"D, s_String ê; StringMatchQ@s, RegularExpression@".*\\.jpg"DD, InfinityD, 8n, 0, 3<D, 5, 5, 1, 8<D

22 cufp-2013-talk-slides.nb

slide-23
SLIDE 23

examples

“everything is a one-liner in Mathematica ... for a sufficiently long line.” (Theo Gray)

Show@ImageAssemble@ Round@Rescale@ImageData@i = Nest@Darker, ImageResize@ExampleData@8"TestImage", "Elaine"<D, 50D, 3DDD 9D ê. n_Integer ß Nest@Lighter, i, nDD, ImageSize Ø 400D

cufp-2013-talk-slides.nb 23

slide-24
SLIDE 24

gateway drug ...

... to declaritive programming

y = 0; For@i = 1, i § 10, i++, y += i^2 D; y 385 Fold@Ò1 + Ò2^2 &, 0, Range@10DD 385

24 cufp-2013-talk-slides.nb

slide-25
SLIDE 25

advanced topics

scoping evaluation control MathLink protocol

cufp-2013-talk-slides.nb 25

slide-26
SLIDE 26

MapReduce

MapReduce in a nutshell

26 cufp-2013-talk-slides.nb

slide-27
SLIDE 27

HadoopLink

WordCount

textRaw = Import@"http:êêwww.gutenberg.orgêcacheêepubê1342êpg1342.txt"D; StringTake@textRaw, 200D The Project Gutenberg EBook of Pride and Prejudice, by Jane Austen This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away o ReverseüSortBy@Tally@StringSplit@textRaw, RegularExpression@"@\\W_D+"DDD, LastD êê Short 88the, 4218<, 8to, 4187<, 8of, 3705<, á7101à, 810, 1<, 8000, 1<<

cufp-2013-talk-slides.nb 27

slide-28
SLIDE 28

HadoopLink

create key-value pairs

paras = StringSplit@textRaw, RegularExpression@"\n82,<"DD; paraPairs = Transpose@8paras, Table@1, 8Lengthüparas<D<D; Grid@8Ò<, Frame Ø All, Background Ø 88LightGreen, LightRed<<D & êü paraPairs@@1 ;; 4DD êê Column The Project Gutenberg EBook of Pride and Prejudice, by Jane Austen 1 This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org 1 Title: Pride and Prejudice 1 Author: Jane Austen 1

28 cufp-2013-talk-slides.nb

slide-29
SLIDE 29

HadoopLink

export to the Hadoop filesystem

<< HadoopLink $$link = OpenHadoopLink@ "fs.default.name" Ø "hdfs:êêhadoopheadlx.wolfram.com:8020", "mapred.job.tracker" Ø "hadoopheadlx.wolfram.com:8021" D; inputfile@"pap"D = "êuserêpaul-jeanêhadooplinkêpap-paras.seq"; DFSExport@$$link, inputfile@"pap"D, paraPairs, "SequenceFile"D êuserêpaul-jeanêhadooplinkêpap-paras.seq Grid@Partition@Names@"HadoopLink`*"D, 4D, Alignment Ø Left, BaseStyle Ø 8FontSize Ø 14<D

DFSAbsoluteFileName DFSCloseSequenceStream DFSCopyDirectory DFSCopyFile DFSCopyFromLocal DFSCopyToLocal DFSCreateDirectory DFSDeleteDirectory DFSDeleteFile DFSDirectoryQ DFSExport DFSFileByteCount DFSFileDate DFSFileExistsQ DFSFileNames DFSFileQ DFSFileType DFSImport DFSOpenSequenceStream DFSReadList DFSRenameDirectory DFSRenameFile DFSSequenceStream HadoopLink HadoopMapReduceJob IncrementCounter OpenHadoopLink Yield

cufp-2013-talk-slides.nb 29

slide-30
SLIDE 30

HadoopLink

mapper

WordCountMapper = Function@8k, v<, With@8 words = ToLowerCase êü StringSplit@k, RegularExpression@"@\\W_D+"DD<, Yield@Ò, 1D & êü words D D;

30 cufp-2013-talk-slides.nb

slide-31
SLIDE 31

HadoopLink

reducer

SumReducer = Function@8k, vs<, Module@ 8sum = 0<, While@vsühasNext@D, sum += vsünext@D D; Yield@k, sumD D D;

cufp-2013-talk-slides.nb 31

slide-32
SLIDE 32

HadoopLink

run the job

inputfile@"pap"D = "êuserêpaul-jeanêhadooplinkêpap-paras.seq";

  • utputdir@"pap"D = "êuserêpaul-jeanêhadooplinkêpap-wordcount";

HadoopMapReduceJob@ $$link, "pap wordcount", inputfile@"pap"D,

  • utputdir@"pap"D,

WordCountMapper, SumReducer D

32 cufp-2013-talk-slides.nb

slide-33
SLIDE 33

HadoopLink

control flow

cufp-2013-talk-slides.nb 33

slide-34
SLIDE 34

genome search engine

prep data

mtseq = GenomeData@8"Mitochondrion", 81, -1<<D; StringTake@mtseq, 30D GATCACAGGTCTATCACCCTATTAACCACT querybases = "GCACACACACA"; StringPosition@mtseq, querybasesD 88515, 525<<

34 cufp-2013-talk-slides.nb

slide-35
SLIDE 35

genome search engine

create key-value pairs

mtchars = Characters@mtseqD; mtbases = Transpose@8mtchars, RangeüLengthümtchars<D; Grid@8Ò<, Frame Ø All, Background Ø 88LightGreen, LightRed<<D & êü mtbases@@1 ;; 20DD 9 G 1 , A 2 , T 3 , C 4 , A 5 , C 6 , A 7 , G 8 , G 9 , T 10 , C 11 , T 12 , A 13 , T 14 , C 15 , A 16 , C 17 , C 18 , C 19 , T 20 =

cufp-2013-talk-slides.nb 35

slide-36
SLIDE 36

genome search engine

mapper

querybases = "GCACACACACA"; GenomeSearchMapper@qchunks : 8__String<D := Function@8base, genomepos<, Module@8pos, querypositions<, querypositions = FlattenüPosition@qchunks, baseD; With@ 8querypos = Ò<, Yield@genomepos - Hquerypos - 1L, queryposD D & êü querypositions D D

36 cufp-2013-talk-slides.nb

slide-37
SLIDE 37

genome search engine

mapper

507 C 1 G 508 C 2 C 509 T 1 G 3 A 510 A 2 C 4 C 511 C 1 G 3 A 5 A 512 C 2 C 4 C 6 C 513 C 1 G 3 A 5 A 7 A 514 A 2 C 4 C 6 C 8 C 515 G 1 G 3 A 5 A 7 A 9 A 516 C 2 C 4 C 6 C 8 C 10 C 517 A 3 A 5 A 7 A 9 A 11 A 518 C 4 C 6 C 8 C 10 C 519 A 5 A 7 A 9 A 11 A 520 C 6 C 8 C 10 C 521 A 7 A 9 A 11 A 522 C 8 C 10 C 523 A 9 A 11 A 524 C 10 C 525 A 11 A 526 C 527 C

cufp-2013-talk-slides.nb 37

slide-38
SLIDE 38

genome search engine

mapper

507 C 1 G 508 C 2 C 509 T 1 G 3 A 510 A 2 C 4 C 511 C 1 G 3 A 5 A 512 C 2 C 4 C 6 C 513 C 1 G 3 A 5 A 7 A 514 A 2 C 4 C 6 C 8 C 515 G 1 G 3 A 5 A 7 A 9 A 516 C 2 C 4 C 6 C 8 C 10 C 517 A 3 A 5 A 7 A 9 A 11 A 518 C 4 C 6 C 8 C 10 C 519 A 5 A 7 A 9 A 11 A 520 C 6 C 8 C 10 C 521 A 7 A 9 A 11 A 522 C 8 C 10 C 523 A 9 A 11 A 524 C 10 C 525 A 11 A 526 C 527 C

38 cufp-2013-talk-slides.nb

slide-39
SLIDE 39

genome search engine

reducer

GenomeSearchReducer@qchunks : 8__String<D := Function@8matchposition, chunkoffsets<, Module@8numchunks, sumoffsets, goalsum<, numchunks = Lengthüqchunks; sumoffsets = 0; goalsum = numchunks * Hnumchunks + 1L ê 2; While@chunkoffsetsühasNext@D, sumoffsets += chunkoffsetsünext@D; D; If@sumoffsets ã goalsum, Yield@StringJoinüqchunks, matchpositionD D D D

cufp-2013-talk-slides.nb 39

slide-40
SLIDE 40

genome search engine

run the job

querybases = "GCACACACACA"; input = DFSFileNames@$$link, "mt-bases.index", "hadooplink"D;

  • ut = "êuserêpaul-jeanêhadooplinkêmt-search-GCACACACACA";

HadoopMapReduceJob@ $$link, "mt search GCACACACACA", input,

  • ut,

GenomeSearchMapper@querybasesD, GenomeSearchReducer@querybasesD D

40 cufp-2013-talk-slides.nb

slide-41
SLIDE 41

genome search engine

import the results

files = DFSFileNames@$$link, "part-*", "êuserêpaul-jeanêhadooplinkêmt-search-GCACACACACA-bases.out"D Join üü HDFSImport@$$link, Ò, "SequenceFile"D & êü filesL 88GCACACACACA, 515<< First êü StringPosition@mtseq, querybasesD 8515<

cufp-2013-talk-slides.nb 41

slide-42
SLIDE 42

challenges

memory consumption

42 cufp-2013-talk-slides.nb

slide-43
SLIDE 43

challenges

memory consumption

cufp-2013-talk-slides.nb 43

slide-44
SLIDE 44

challenges

HadoopLink architecture

44 cufp-2013-talk-slides.nb

slide-45
SLIDE 45

challenges

job-level configurations

HadoopMapReduceJob@ $$link, "hs search GCACACACACA", input,

  • utput,

GenomeSearchMapper@querybasesD, GenomeSearchReducer@querybasesD, "mapred.child.java.opts" -> "-Xmx512m" D

cufp-2013-talk-slides.nb 45

slide-46
SLIDE 46

conclusions

core principles of Mathematica

everything is an expression expressions are transformed until they stop changing transformation rules are patterns

examples

Fibonacci sequence, web scraping, recursive image

MapReduce with Mathematica

mapper and reducer functions running MapReduce jobs using HadoopLink challenges: constrain memory consumption, job-level configurations

46 cufp-2013-talk-slides.nb

slide-47
SLIDE 47

the end

@rule146

rl = MapThread@Rule, 8Tuples@81, 0<, 3D, IntegerDigits@146, 2, 8D<D; ar = NestList@Partition@Ò, 3, 1, 2D ê. rl &, RandomInteger@1, 200D, 150D; gr = ArrayPlot@ar, PixelConstrained Ø 2D

cufp-2013-talk-slides.nb 47