R.A.I.D.F.S Randomized Aggregation Independent Distributed File - - PowerPoint PPT Presentation
R.A.I.D.F.S Randomized Aggregation Independent Distributed File System P2P Distributed File System with an API for Map-Reduce Integration Sven Reber, Jrmy Gotteland, David Froelicher, Alban Marguet, Pascal Cudr, Valrian Pittet Context
Randomized Aggregation Independent Distributed File System
P2P Distributed File System with an API for Map-Reduce Integration Sven Reber, Jérémy Gotteland, David Froelicher, Alban Marguet, Pascal Cudré, Valérian Pittet
user machine
Peer to peer DFS that is
○ chunking by line blocks ○ text files
○ simply connect to the network and run your jobs
GlobalChunkField <= 3 (arbitrary) is an unstable state
Look at its neighbors chunkfields
Randomly gets one of the insufficiently replicated chunk
Do not download chunk if it finds enough replicas
File is “stable” when there is enough replicas
New file : “put” command
publish an index update, then neighbors discover every 20s
neighbors try to stabilize file (same process as before)
neighbors get missing chunks randomly to complete their GCF
commands available
○ Each chunk mapped to its result files (can be created in advance) -> One folder for each mapped chunk ○ One key chunk for each key discovered in the original chunk
during map
initiator
ReduceFile on DFS
create a ReduceFile chunk download the needed keyChunks
a reduce is finished when ReduceFile is stable on DFS
the MRp2p paper)