HP-Mapper: A High Performance Storage Driver for Docker Containers - - PowerPoint PPT Presentation

HP-Mapper: A High Performance Storage Driver for Docker Containers

Fan Guo1, Yongkun Li1, Min Lv1, Yinlong Xu1, John C.S. Lui2

1University of Science and Technology of China 2The Chinese University of Hong Kong

Outline

2

Background & Motivation

1

HP-Mapper Design

2

Evaluation

3

Conclusion

4

n Container

ü Process-level virtualization

l

Share host kernel

l

Namespaces, Cgroup

ü Better performance

l

Fast deployment

l

Low resources usage

l

Near bare-mental performance

n Docker

ü Most popular container engine ü Extensively used in production

3

Container and Docker

VM Container

n Container images

ü Store all requirements for running the containers ü Hierarchical, read-only, sharable

n Storage drivers

ü Support cross-level lookup and copy-on-write (COW)

Storage Management of Containers

4

Docker storage drivers Container Debian (Base Image) nginx (Image) emacs (Image) Apache (Image) writeable layer writeable layer Container …

Cross-level lookup Copy-on-write

Unified view

Container

n Block-based Drivers

ü Block-level COW (≥64KB) ü Cannot share cached data ü DeviceMapper, ZFS, BtrFS

Layer1 (read-only) Layer2 (read-only) Layer3 (writable)

n File-based Drivers

ü File-level COW ü Share cached data ü Overlay2, AUFS

5

Docker Storage Drivers

file4 file5 blk1 blk2 blk5 blk3 blk4

vblk1 vblk2 vblk3 vblk4 vblk5

File1 File2

blk3 File1 File2 File2 File1 blk5 blk4

File2

File-based Drivers (path→file) Block-based Drivers (vblk→blk)

blk3 File1

n File-based storage drivers incur large COW

latency (especially for large files)

ü Incur large write overhead ü Degrade write performance

6

High COW Latency

File Size 4KB 64KB 1MB 16MB DeviceMapper 0.12 0.74 0.96 1.39 BtrFS 0.09 0.09 0.09 0.10 Overlay2 1.99 2.49 7.14 61.7 Copy-on-write latency (ms)

Block-based File-based

n Block-based drivers introduce many redundant

I/Os when reading data from a shared file

ü Degrade I/O performance ü Waste I/O bandwidth

7

Disk I/O Redundancy

Total amount of reading data during the startup

• f 64 containers

Block-based File-based

n Both kinds of storage drivers generate a lot of

redundant cached data

ü Block-based: Read multiple copies of the data (as

cache can not be shared)

ü File-based: Unchanged data in the file are also

copied when performing copy-on-write

8

Cache Redundancy

n Limitations of current storage drivers

ü Tradeoff between write and read performance ü Low cache efficiency

n Our goal: Develop a new storage driver for

docker containers

ü Low COW overhead (or high write performance) ü High read I/O performance ü High cache efficiency

9

Motivation

Outline

10

Background & Motivation

1

HP-Mapper Design

2

Evaluation

3

Conclusion

4

n Key idea: HP-Mapper works at the block level and

manages physical blocks

ü Why block level: Low COW overhead ü Why physical block: Able to detect redundant I/Os &

redundant cached data

n Three modules

ü Address mapper ü I/O interceptor ü Cache manager

11

HP-Mapper: A High Performance Storage Driver

n Tradeoff exists in block-based management

ü Large blocks: high COW latency ü Small blocks: high lookup and storage overhead due to

large metadata size

n HP-Mapper uses a two-level mapping tree

ü Support two different block sizes ü Differentiate different requests w/ on-demand allocation

l

New write: large sequential I/O (large block size)

l

COW: depending on req size

12

Address Mapper

n Two-level mapping tree design

13

Address Mapper

key

…

key

… …

key

… ... …

key

…

1xx

(PBN)

… … … … … …

0xx

(root)

… … …

key

…

key

… …

key

…

PBN

… … ... … … … …

key … PBN … …

… … … …

Root of Level-1 Root of Level-2

index1

• ffset

VBN1 index1

• ffset

index2 VBN2

Storage efficiency Separated block placement + defragmentation

Head entry1

u64 PBN (key) u16 copies_num u64 Super_Block u64 Inode_ID void *next void *tail

Head entry2 ... Tail entry1 Tail entry2 001 002 … … … …

...

n How to detect redundant I/O

ü Indexing with physical block number (PBN)

n PBN-indexing hash table

ü Entries are linked in a two-dimensional list (LRU)

l

Head entry (latest copy) + Tail entry (other copies)

I/O Interceptor – Metadata Management

14

n Detect redundant I/O w/ PBN-index

15

I/O Interceptor - Workflow

Avoid unnecessary check (write I/Os to writable layer)

n How to remove redundant copies in cache

ü Periodically scan the hash table to locate cached pages ü Maintain the hotness of each page (multiple LRU)

n Page eviction: cache hit ratio vs. cache usage

ü Limit # of copies: utilization-aware adjustment ü Hotness-aware eviction

16

Cache Manager

Page1 (cold) Page2 (hot) Page3 (hot) Page4 (hot) Page5 (cold) Page6 (hot) Page7 (hot) Page8 (hot) Page9 (hot)

Cached copies of PBN1 Cached copies of PBN2 Locate & monitor Scan next block Evict (copies_limit = 3)

Page1 (cold) Page2 (hot) Page3 (hot) Page4 (hot) Page5 (cold) Page6 (hot) Page7 (hot) Page8 (hot) Page9 (hot) Hash Table Head entry1 (PBN1) Tail entry Tail entry ... Tail entry Head entry2 (PBN2) Tail entry Tail entry ... Tail entry

Outline

17

Background & Motivation

1

HP-Mapper Design

2

Evaluation

3

Conclusion

4

n Prototype

ü Act as a plugin module in Linux kernel 3.10.0 ü Backing file system: Ext4

n Workloads

ü Container images: Tomcat, Nextcloud, Vault

n Overhead of HP-Mapper

18

Experiment Setup

Overhead of HP-Mapper (MT represents Mapping Tree, HT represents Hash Table)

n Copy-on-write(COW) latency

19

Reduction of COW Latency

File Size 4KB 64KB 1MB 16MB DeviceMapper 0.13 0.74 0.96 1.39 BtrFS 0.09 0.09 0.09 0.10 Overlay2 1.99 2.49 7.14 61.7 HP-Mapper 0.07 0.12 0.55 0.57 Copy-on-write latency (ms) HP-Mapper reduces up to more than 90% COW latency comparing with DeviceMapper and Overlay2

n Total amount of reading/writing data when

launching 64 containers from a single image

20

Reduction of Redundant I/Os

HP-Mapper efficiently removes redundant read I/Os, and also reduces more than 50% writing data on average

n Cache usage when starting 64 containers from a

single image

21

Improvement of Cache Efficiency

HP-Mapper reduces more than 65% cache usage on average

n Total startup time when launching 64 containers

from a single image on SSD/HDD

22

Improvement of Startup Time

HP-Mapper achieves up to 2.0× - 7.2× faster startup speed than the other three storage drivers

n Total startup time when launching 64 containers

in memory-scarce systems

23

Improvement of Startup Time

HP-Mapper achieves larger improvement as memory size decreases

50 100 150 200 12GB 8GB 4GB Total Startup time (s) Total available memory

DM BtrFS Overlay2 HP-Mapper

10 20 30 40 6GB 4GB 2GB Total Startup time (s) Total available memory

DM BtrFS Overlay2 HP-Mapper

Launch 64 Nextcloud containers Launch 64 Vault containers

Outline

24

Background & Motivation

1

HP-Mapper Design

2

Evaluation

3

Conclusion

4

n Tradeoffs exist for Docker storage drivers

ü File-based: High COW overhead ü Block-based: Low cache efficiency & redundant I/O

n We develop HP-Mapper which achieves

ü Low COW overhead by following block-based

design with differentiated block sizes

ü High I/O efficiency by intercepting redundant I/Os ü High cache efficiency by enabling cache sharing

and hotness-aware management

25

Conclusion

Q&A Yongkun Li ykli@ustc.edu.cn http://staff.ustc.edu.cn/~ykli

26

Thanks!