Using Eager Strategies to Improve NFS I/O Performance Stephen Rago, - - PowerPoint PPT Presentation

▶
using eager strategies to improve nfs i o performance
SMART_READER_LITE
LIVE PREVIEW

Using Eager Strategies to Improve NFS I/O Performance Stephen Rago, - - PowerPoint PPT Presentation

Oct 24, 2023 278 likes •920 views

Using Eager Strategies to Improve NFS I/O Performance Stephen Rago, Aniruddha Bohra, Cristian Ungureanu NEC Laboratories America Tuesday, July 26, 2011 Introduction Background Backup appliance development NFS Version 3

slide-1
SLIDE 1

Using Eager Strategies to Improve NFS I/O Performance

Stephen Rago, Aniruddha Bohra, Cristian Ungureanu NEC Laboratories America

Tuesday, July 26, 2011

slide-2
SLIDE 2

Introduction

  • Background
  • Backup appliance development
  • NFS

Version 3

  • Backup over NFS was slower than expected
  • With storage system capable of 400 MB/s, couldn’t saturate a

1Gb Ethernet

  • With 10Gb Ethernet, can’t approach throughput of storage

subsystem

  • Built server testbed with conventional storage subsystem:

ext3 on top of striped, 15K RPM disks

  • Server capable of 300 MB/s throughput to storage subsystem

2

Tuesday, July 26, 2011

slide-3
SLIDE 3

NFS Performance Problems

  • Streaming write performance erratic
  • Tuning the system to cache more data caused write

throughput to vary from 40 MB/s to 200 MB/s on our test systems for the same set of tunable values

  • Slow performance results from:
  • Multiple contexts writing generate out-of-order requests
  • Memory pressure leads to small, synchronous writes
  • Memory pressure also increases commits
  • Streaming read performance lower than expected
  • Less than 100 MB/s on 10Gb Ethernet
  • Out-of-order requests defeat kernel read-ahead logic

3

Tuesday, July 26, 2011

slide-4
SLIDE 4

Concurrency = Out-of-Order NFS Requests

4

Reads Writes Client

Read-ahead Multiple writers (background flushing, pageout, and application) plus asynchronous writes

Server

Multiple NFS threads Multiple NFS threads

Tuesday, July 26, 2011

slide-5
SLIDE 5

Problem 1: Synchronous Operations -

Base System, Slow Run vs. Fast Run for Same Amount of Data Written

5

1 10 100 1000 10000 100000 1000000 Asynchronous Writes Synchronous Writes Commits

34 32855 148 207061 56145

Request Count

Slow (40 MB/s) Fast (200 MB/s)

Tuesday, July 26, 2011

slide-6
SLIDE 6

Problem 2: Small Record Sizes -

Idealized NFS Write Throughput

6

100 200 300 400 500 600 700 800 900 1 KB 2 KB 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB 512 KB 1 MB Throughput (MB/s) Record Size

Tuesday, July 26, 2011

slide-7
SLIDE 7

Problem 3: NFS Write Offset Ordering

(Writing a 32 GB File)

0.75 1.50 2.25 3.00 500 1000 1500 2000 2500 3000 Write Offset (GB) Request Number 7

Tuesday, July 26, 2011

slide-8
SLIDE 8

Problem 4: NFS Read Offset Ordering

(Reading a 32 GB File)

20 40 60 80 100 120 140 500 1000 1500 2000 2500 3000 3500 4000 Read Offset (MB) Request Number 8

Tuesday, July 26, 2011

slide-9
SLIDE 9

Solutions

  • Three general techniques
  • Eager Writeback
  • Reduces concurrency on client and maintains sequentiality
  • Eager Page Laundering
  • Reduces client memory pressure
  • Request Ordering
  • Prevents out-of-order operations on a single file
  • Implemented on Linux 2.6.36
  • Techniques applicable to other operating systems

9

Tuesday, July 26, 2011

slide-10
SLIDE 10

Technique 1: Eager Writeback

  • Client-side mechanism
  • Prevents application from creating dirty pages quickly
  • Pages written eagerly to server
  • Client waits for outstanding requests to complete before

continuing

  • Advantages
  • Starts sending dirty pages earlier -- better server utilization
  • Only one thread writes a file’s pages to the server
  • Better flow control
  • Disadvantages
  • Starts sending dirty pages earlier -- limited page reuse for
  • verwriting patterns

10

Tuesday, July 26, 2011

slide-11
SLIDE 11

Simplified Page State Diagram

11

Free Active

Inactive Clean Inactive Dirty

Unstable Deallocate Allocate Old/Modified Writeback Commit Old/ Unmodified Writeback

Tuesday, July 26, 2011

slide-12
SLIDE 12

Technique 2: Eager Page Laundering

  • Client & Server mechanism
  • Dirty pages on server eventually become clean
  • Communicate largest stable offset from server to client
  • Piggybacked in NFS write response (takes half of verifier)
  • Negotiated at mount time
  • Client reclaims (“launders”) pages eagerly
  • Advantages
  • Reduces memory pressure on client
  • No commits or synchronous writes needed
  • Disadvantages
  • Small protocol change

12

Tuesday, July 26, 2011

slide-13
SLIDE 13

Largest Stable Offset (LSO)

13

Unmodified File Contents

Dirty Dirty Dirty

LSO

Tuesday, July 26, 2011

slide-14
SLIDE 14

Largest Stable Offset (LSO)

13

Unmodified File Contents

Clean Dirty Dirty

LSO

Tuesday, July 26, 2011

slide-15
SLIDE 15

Largest Stable Offset (LSO)

13

Unmodified File Contents

Clean Dirty Clean

LSO

Tuesday, July 26, 2011

slide-16
SLIDE 16

Largest Stable Offset (LSO)

13

Unmodified File Contents

Clean Clean Clean

LSO

Tuesday, July 26, 2011

slide-17
SLIDE 17

Base Client Page Counts

14

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-18
SLIDE 18

Base Client Page Counts

14

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-19
SLIDE 19

Base Client Page Counts

14

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-20
SLIDE 20

Base Client Page Counts

14

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-21
SLIDE 21

Base Client Page Counts

14

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

178 MB/s

Tuesday, July 26, 2011

slide-22
SLIDE 22

Client Page Counts - Eager Writeback Only

15

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-23
SLIDE 23

Client Page Counts - Eager Writeback Only

15

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-24
SLIDE 24

Client Page Counts - Eager Writeback Only

15

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-25
SLIDE 25

Client Page Counts - Eager Writeback Only

15

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-26
SLIDE 26

Client Page Counts - Eager Writeback Only

15

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

230 MB/s

Tuesday, July 26, 2011

slide-27
SLIDE 27

Client Page Counts - Eager Writeback & Eager Page Laundering

16

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-28
SLIDE 28

Client Page Counts - Eager Writeback & Eager Page Laundering

16

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-29
SLIDE 29

Client Page Counts - Eager Writeback & Eager Page Laundering

16

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-30
SLIDE 30

Client Page Counts - Eager Writeback & Eager Page Laundering

16

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

Tuesday, July 26, 2011

slide-31
SLIDE 31

Client Page Counts - Eager Writeback & Eager Page Laundering

16

200 400 600 800 Number of Pages x 1000 Time (s)

Dirty Pages Unstable Pages Total

244 MB/s

Tuesday, July 26, 2011

slide-32
SLIDE 32

Technique 3: Request Ordering

  • Server sorts requests based on RPC transmission ID
  • Server-side mechanism
  • Prevents out-of-order completion of requests from

competing threads

  • Advantages
  • Improves sequential read performance
  • When used during writes, can further improve read

performance (depending on file system implementation)

  • Disadvantages
  • Adds a small delay (50 ns) on reads to facilitate sorting, but
  • nly for sequential reads on files where the queue is empty

17

Tuesday, July 26, 2011

slide-33
SLIDE 33

Sorting Request on the Server

18

Head

  • f

Queue

Tuesday, July 26, 2011

slide-34
SLIDE 34

Sorting Request on the Server

18

2

Head

  • f

Queue

Tuesday, July 26, 2011

slide-35
SLIDE 35

Sorting Request on the Server

18

2 4

Head

  • f

Queue

Tuesday, July 26, 2011

slide-36
SLIDE 36

Sorting Request on the Server

18

1 2 4

Head

  • f

Queue

Tuesday, July 26, 2011

slide-37
SLIDE 37

Sorting Request on the Server

18

1 2 4 6

Head

  • f

Queue

Tuesday, July 26, 2011

slide-38
SLIDE 38

Sorting Request on the Server

18

1 2 4 5 6

Head

  • f

Queue

Tuesday, July 26, 2011

slide-39
SLIDE 39

Sorting Request on the Server

18

1 2 3 4 5 6

Head

  • f

Queue

Tuesday, July 26, 2011

slide-40
SLIDE 40

NFS Write Offset Ordering

19 Base Eager

Tuesday, July 26, 2011

slide-41
SLIDE 41

NFS Write Offset Ordering

500 1000 1500 2000 2500 3000 0.75 1.50 2.25 3.00 Write Offset (GB) Request Number 19 Base Eager

Tuesday, July 26, 2011

slide-42
SLIDE 42

NFS Write Offset Ordering

500 1000 1500 2000 2500 3000 0.75 1.50 2.25 3.00 Write Offset (GB) Request Number 19 Base Eager

Tuesday, July 26, 2011

slide-43
SLIDE 43

NFS Read Offset Ordering

20 Base Eager

Tuesday, July 26, 2011

slide-44
SLIDE 44

NFS Read Offset Ordering

500 1000 1500 2000 2500 3000 3500 4000 20 40 60 80 100 120 140 Read Offset (MB) Request Number 20 Base Eager

Tuesday, July 26, 2011

slide-45
SLIDE 45

NFS Read Offset Ordering

500 1000 1500 2000 2500 3000 3500 4000 20 40 60 80 100 120 140 Read Offset (MB) Request Number 20 Base Eager

Tuesday, July 26, 2011

slide-46
SLIDE 46

Performance Comparisons

  • Micro benchmarks
  • Streaming I/O
  • Random Writes
  • Non-sequential Writes
  • Adversarial Page Reuse
  • Macro benchmarks
  • Filebench Fileserver
  • Filebench

Videoserver

21

Tuesday, July 26, 2011

slide-47
SLIDE 47

Streaming I/O Performance

22

Read Write

Tuesday, July 26, 2011

slide-48
SLIDE 48

Streaming I/O Performance

22

Local NFS Base NFS Eager 100 200 300 400

279 326

Throughput (MB/s)

Read Write

Tuesday, July 26, 2011

slide-49
SLIDE 49

Streaming I/O Performance

22

Local NFS Base NFS Eager 100 200 300 400

279 326 178 96

Throughput (MB/s)

Read Write

Tuesday, July 26, 2011

slide-50
SLIDE 50

Streaming I/O Performance

22

Local NFS Base NFS Eager 100 200 300 400

279 326 178 96 244 321

Throughput (MB/s)

Read Write

Tuesday, July 26, 2011

slide-51
SLIDE 51

Random Write Performance

23

75 150 225 300 4 KB 8 KB 16 KB 32 KB 64 KB 128 KB 256 KB 512 KB 1 MB 2 MB 4 MB 8 MB 16 MB Throughput (MB/s) Record Size

Base Eager

Tuesday, July 26, 2011

slide-52
SLIDE 52

Nonsequential Write Performance

24

75 150 225 300 32 KB Alternating 256 KB Alternating 1MB/11MB Strided

208 235 123 177 194 123

Throughput (MB/s)

Base Eager

Tuesday, July 26, 2011

slide-53
SLIDE 53

Adversarial Example

25

1.5 GB -- 1.4 GB -- dirty_background_ratio 0 GB --

Tuesday, July 26, 2011

slide-54
SLIDE 54

Adversarial Example

25

1.5 GB -- 1.4 GB -- dirty_background_ratio 0 GB --

Footprint Base Eager

Tuesday, July 26, 2011

slide-55
SLIDE 55

Adversarial Example

25

1.5 GB -- 1.4 GB -- dirty_background_ratio 0 GB --

Footprint Base Eager

Tuesday, July 26, 2011

slide-56
SLIDE 56

Adversarial Example

25

1.5 GB -- 1.4 GB -- dirty_background_ratio 0 GB --

Footprint Base Eager

Tuesday, July 26, 2011

slide-57
SLIDE 57

Adversarial Example

25

1.5 GB -- 1.4 GB -- dirty_background_ratio 0 GB --

Footprint Base Eager

1.4 GB

(32 GB total)

1093 MB/s

(1.4 GB to disk)

513 MB/s

(18 GB to disk)

Tuesday, July 26, 2011

slide-58
SLIDE 58

Adversarial Example

25

1.5 GB -- 1.4 GB -- dirty_background_ratio 0 GB --

Footprint Base Eager

1.4 GB

(32 GB total)

1093 MB/s

(1.4 GB to disk)

513 MB/s

(18 GB to disk)

1.5 GB

(32 GB total)

469 MB/s

(18 GB to disk)

253 MB/s

(32 GB to disk)

Tuesday, July 26, 2011

slide-59
SLIDE 59

Filebench Fileserver Workload

26

200 400 600 800 1 2 3 4 Throughput (Operations/sec) Number of Clients

Base Eager

Tuesday, July 26, 2011

slide-60
SLIDE 60

Filebench Videoserver Workload

27

150 300 450 600 1 2 3 4 Throughput (Operations/sec) Number of Clients

Base Eager

Tuesday, July 26, 2011

slide-61
SLIDE 61

Implementation

28

Technique Scope Lines of Code Eager Writeback Eager Page Laundering Request Ordering NFS Client 100 NFS Client & Server 150 NFS Server 120

Tuesday, July 26, 2011

slide-62
SLIDE 62

Related Work

  • Lee, et al. 2000

Eager Writeback - A Technique for Improving Bandwidth Utilization (33rd ACM/IEEE Symposium on Microarchitecture)

  • Ellard & Seltzer 2003

NFS Tips and Benchmarking Traps (USENIX ATC)

  • Ellard, et al. 2003

Passive NFS Tracing of Email and Research Workloads (FAST ’03)

  • Batsakis, et al. 2009

CA-NFS: a Congestion-Aware Network File System (FAST ’09)

29

Tuesday, July 26, 2011

slide-63
SLIDE 63

Summary

  • For writes, memory pressure leads to performance

problems

  • For reads, out-of-order requests disable read-ahead
  • Eager writeback, eager page laundering, and request
  • rdering improve sequential throughput
  • No harm for many nonsequential workloads
  • May even improve throughput when clients experience

memory pressure

30

Tuesday, July 26, 2011