miniMap The team at 2am in the morning Jamie Song - - - PowerPoint PPT Presentation

▶
minimap the team at 2am in the morning
SMART_READER_LITE
LIVE PREVIEW

miniMap The team at 2am in the morning Jamie Song - - - PowerPoint PPT Presentation

Feb 03, 2023 260 likes •525 views

miniMap The team at 2am in the morning Jamie Song - js4390@columbia.edu Olesya Medvedeva - oam2113@columbia.edu Ryan DeCosmo - rd2680@columbia.edu Charis Lam - cl3257@columbia.edu Concept: MapReduce 1. Large input data set. (ex. a book)

slide-1
SLIDE 1

miniMap

slide-2
SLIDE 2

The team… at 2am in the morning

Jamie Song - js4390@columbia.edu Olesya Medvedeva - oam2113@columbia.edu Ryan DeCosmo - rd2680@columbia.edu Charis Lam - cl3257@columbia.edu

slide-3
SLIDE 3

Concept: MapReduce

  • 1. Large input data set. (ex. a book)
  • 2. Data set gets split into chunks. (ex. small text files)
  • 3. A function is applied to each chunk

(ex. return the frequency of the word ‘hitchhiker’)

  • 3. Aggregate all the results into one unit. (ex. 42)
slide-4
SLIDE 4

Inspiration: Apache Hadoop

slide-5
SLIDE 5

Expectations:

  • > BIIIIG DATA
  • > Multi-threaded on graphics

card

  • > GPU-accelerated,
  • > In-memory
  • > Map-reduce replacement

for single workstation users

slide-6
SLIDE 6

reality...

Text processing language <- Small-to-Medium Data <- Sorta.. multi-threaded! <- Lower overhead than the hadoop ecosystem <- *Ideal? For projects / researchers

miniMap:

slide-7
SLIDE 7

so how should it work?

miniMap()

slide-8
SLIDE 8

works like MapReduce

miniMap(File* inputFile, void* splitter(), void* mapper(), File* context, void* reducer())

slide-9
SLIDE 9

the pieces:

  • File* inputFile: an input text file
  • void* splitter(): function pointer to a function that splits the input file
  • mapper(): function pointer to a user defined function
  • File* context: an intermediate step that outsources RAM to disk
  • reducer(): function pointer to a user defined function
slide-10
SLIDE 10

Function headers

File** split_by_size(int x) File** split_by_quant(int x) File** split_by_regex(File*, String) void mapper(File*, File*) void reducer(File*)

void miniMap(input, splitter, mapper, context, reducer)

slide-11
SLIDE 11

so how does it work?

Splitter Function Input File

slide-12
SLIDE 12

Splitter Function

Disk

so how does it work?

slide-13
SLIDE 13

Disk MiniMap

Threads

so how does it work?

slide-14
SLIDE 14

Multiple threads

so how does it work?

slide-15
SLIDE 15

Map Function

so how does it work?

slide-16
SLIDE 16

Architecture

Applied using threads

slide-17
SLIDE 17

Each file chunk has the map function applied to it

so how does it work?

slide-18
SLIDE 18

Reducer combines data from mapper threads

Reducer

so how does it work?

slide-19
SLIDE 19

Result:

File of clean, useful Data

slide-20
SLIDE 20

Built-in Types

  • ints
  • bool
  • float
  • String
  • void
  • File
  • Array
  • Array pointer
slide-21
SLIDE 21

Built-in functions.. links to C standard library!

Prints: print(), printb(), printbig(), printstring() Splitters: split_by_size(), split_by_quant(), split_by_regex() File:

  • pen(), readFile(), isFileEnd(), close()

String: strstr()

slide-22
SLIDE 22

demo!

slide-23
SLIDE 23

Our process:

  • Weekly meetings
  • Internal implementation goals
  • Iterative cycle of concept and coding!

concept implement errors

slide-24
SLIDE 24

possible directions that Minimap could take:

GPU acceleration using Nvidia CUDA Multi-Node Support (multiple multi-core PCs) Optimize File I/O - Sequential Offset (like Kafka)

slide-25
SLIDE 25