[PPT] - HDFS Hadoop Distributed File System Motivation File Management PowerPoint Presentation

SLIDE 1

HDFS

Hadoop Distributed File System

Motivation File Management Streaming Data Fault Tolerance

1

SLIDE 2

Labs

Run 2–27 October (four weeks) at these times: Monday 9am Monday 10am Tuesday 2pm Wednesday 10am Wednesday 2pm Thursday 9am Thursday 11am Friday 11am Friday 2pm Lab groups will be chosen online: student.inf.ed.ac.uk.

Motivation File Management Streaming Data Fault Tolerance

2

SLIDE 3

Distributed Map-Reduce

Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9

Mapper 1 Mapper 2 Mapper 4 Mapper 3

Motivation File Management Streaming Data Fault Tolerance

3

SLIDE 4

Large Data Sets

file sizes going up to petabytes

Motivation File Management Streaming Data Fault Tolerance

4

SLIDE 5

How to get Data to Mappers?

Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9

Mapper 1 Mapper 2 Mapper 4 Mapper 3

?

Motivation File Management Streaming Data Fault Tolerance

5

SLIDE 6

How to get Data to Mappers?

Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9

Mapper 1 Mapper 2 Mapper 4 Mapper 3

!

Motivation File Management Streaming Data Fault Tolerance

6

SLIDE 7

Bring Mappers to Data!

Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9

Mapper 1 Mapper 2 Mapper 4 Mapper 3

Motivation File Management Streaming Data Fault Tolerance

7

SLIDE 8

But disk access latency is so high!

Motivation File Management Streaming Data Fault Tolerance

8

SLIDE 9

But disk access latency is so high! Yes, but throughput is acceptable.

Motivation File Management Streaming Data Fault Tolerance

9

SLIDE 10

Distributed File System

Motivation File Management Streaming Data Fault Tolerance

10

SLIDE 11

Distributed File System

HDFS is a GFS (Google File System) clone

Motivation File Management Streaming Data Fault Tolerance

11

SLIDE 12

HDFS Design Choices

1 Support handling of large files across multiple nodes Motivation File Management Streaming Data Fault Tolerance

12

SLIDE 13

HDFS Design Choices

1 Support handling of large files across multiple nodes 2 Optimise for streaming access Motivation File Management Streaming Data Fault Tolerance

13

SLIDE 14

HDFS Design Choices

1 Support handling of large files across multiple nodes 2 Optimise for streaming access 3 Run on commodity hardware (e.g. high fault tolerance) Motivation File Management Streaming Data Fault Tolerance

14

SLIDE 15

Large Files

128 MB Block Size

Motivation File Management Streaming Data Fault Tolerance

15

SLIDE 16

Why so large Blocks?

Motivation File Management Streaming Data Fault Tolerance

16

SLIDE 17

HDFS datanode Linux file system

Motivation File Management Streaming Data Fault Tolerance

17

SLIDE 18

HDFS datanode Linux file system Demo

Motivation File Management Streaming Data Fault Tolerance

18

SLIDE 19

HDFS datanode Linux file system HDFS datanode Linux file system

HDFS namenode

File namespace /foo/bar

block 3df2 instructions to datanode datanode state

Motivation File Management Streaming Data Fault Tolerance

19

SLIDE 20

Optimised for Streaming

Append Write Successive Read

write once read many

Motivation File Management Streaming Data Fault Tolerance

20

SLIDE 21

HDFS datanode Linux file system HDFS datanode Linux file system

HDFS namenode

File namespace /foo/bar

block 3df2 instructions to datanode datanode state

Application HDFS Client

(file name, block id) (block id, block location) block data (block id, byte range) ctrl flow data flow

Motivation File Management Streaming Data Fault Tolerance

21

SLIDE 22

HDFS datanode Linux file system HDFS datanode Linux file system

HDFS namenode

File namespace /foo/bar

block 3df2 instructions to datanode datanode state

Application HDFS Client

(file name, block id) (block id, block location) block data (block id, byte range) ctrl flow data flow

Why so large blocks?

Motivation File Management Streaming Data Fault Tolerance

22

SLIDE 23

HDFS datanode Linux file system HDFS datanode Linux file system

HDFS namenode

File namespace /foo/bar

block 3df2 instructions to datanode datanode state

Application HDFS Client

(file name, block id) (block id, block location) block data (block id, byte range) ctrl flow data flow

Why so large blocks?

1 less communication between master and workers Motivation File Management Streaming Data Fault Tolerance

23

SLIDE 24

HDFS datanode Linux file system HDFS datanode Linux file system

HDFS namenode

File namespace /foo/bar

block 3df2 instructions to datanode datanode state

Application HDFS Client

(file name, block id) (block id, block location) block data (block id, byte range) ctrl flow data flow

Why so large blocks?

1 less communication between master and workers 2 reduced communication between client and datanodes Motivation File Management Streaming Data Fault Tolerance

24

SLIDE 25

HDFS datanode Linux file system HDFS datanode Linux file system

HDFS namenode

File namespace /foo/bar

block 3df2 instructions to datanode datanode state

Application HDFS Client

(file name, block id) (block id, block location) block data (block id, byte range) ctrl flow data flow

Why so large blocks?

1 less communication between master and workers 2 reduced communication between client and datanodes 3 less meta data to be saved in namenode Motivation File Management Streaming Data Fault Tolerance

25

SLIDE 26

Which block location is best for the client? Application HDFS Client

block data

?

Motivation File Management Streaming Data Fault Tolerance

26

SLIDE 27

Which block location is best for the client? Application HDFS Client

block data

?

The closest one!

Motivation File Management Streaming Data Fault Tolerance

27

SLIDE 28

Network is represented as a tree. Distance between two nodes is the sum of their distance to their closest common ancestor.

Motivation File Management Streaming Data Fault Tolerance

28

SLIDE 29

Fault Tolerance

Faults are the norm, not the exception.

Motivation File Management Streaming Data Fault Tolerance

29

SLIDE 30

Hadoop keeps three versions by default.

Motivation File Management Streaming Data Fault Tolerance

30

SLIDE 31

How to spread over across the cluster?

Motivation File Management Streaming Data Fault Tolerance

31

SLIDE 32

How to spread over across the cluster?

Demo

Motivation File Management Streaming Data Fault Tolerance

32

SLIDE 33

Anatomy of a Write

Motivation File Management Streaming Data Fault Tolerance

33

SLIDE 34

Summary

1 HDFS handles large files across the cluster 2 HDFS is optimised for streaming access to files 3 HDFS runs on commodity hardware and needs to be fault

tolerant

Motivation File Management Streaming Data Fault Tolerance

34