OpenSolaris NFS TM Performance Alain van Hoof February 3, 2010 - - PowerPoint PPT Presentation
OpenSolaris NFS TM Performance Alain van Hoof February 3, 2010 Research Question \begin{quotation} How can the performance bottlenecks be monitored and identified on an OpenSolaris OS NFS server. What are realistic load simulations and
Alain van Hoof February 3, 2010
TM
\begin{quotation}
How can the performance bottlenecks be monitored and identified on an OpenSolaris OS NFS server. What are realistic load simulations and create a base-line.
\end{quotation}
2
Server OpenSolaris Client Linux Physical Devices ZFS FileSystem NFS Server TCP/IP TCP/IP NFS Client ZIL L2ARC VFS Layer
3
Server OpenSolaris Client Linux Physical Devices ZFS FileSystem NFS Server TCP/IP TCP/IP NFS Client ZIL L2ARC VFS Layer nfsstat -c netstat tcpdump/wireshark netstat nfsstat -s zilstat arcstat zpool iostat iostat Dtrace Probes mib:ip:ip_ nfsv3:nfssrv:rfs3_ fbt:zfs:arc_ fbt:zfs:zil_lwb_write io:genunix::start fsstat fsinfo:genunix:fop_
4
Physical Disk ZFS FileSystem NFSv3 Server TCP/IP ZIL ARC VFS Layer
nfsv3:nfssrv:rfs3_read:op-read-start nfsv3:nfssrv:rfs3_write:op-write-start mib:ip:ip_input:ipIfStatsHCInReceives mib:ip:ip_xmit:ipIfStatsHCOutTransmits fsinfo:genunix:fop_write:write fsinfo:genunix:fop_read:read fbt:zfs:zfs_write:entry fbt:zfs:zfs_read:entry fbt:zfs:arc_write:entry fbt:zfs:arc_read:entry fbt:zfs:zil_lwb_write_start:entry io:genunix::start
Write Read
Ard Awr Brd Bwr Crd Cwr Drd Dwr E Frd Fwr G
5
Marcro benchmarks real-world workload Trace benchmarks replay recorded real-world scenario Micro benchmarks
6
Macro benchmark Supported by SUN fileserver: SPECsfs workload
Micro benchmark A benchmark UvA-IC is familiar with Random Read/Write
TM 7
Automate the running of benchmarks Avoid user errors (typo’s) Automatic mount and unmount filesystems Calculate confidence level and run more test to be more confident
8
TestClient VMware 01 LoadClient 01 NFS Server Server Metrics Log LoadClient 02 auto-pilot IOzone VMware 02 LoadClient 03 LoadClient 04 filebench Test Results autopilot filebench autopilot filebench autopilot filebench autopilot filebench
9
Run benchmark on TestClient while load is generated on LoadClients Real-world benchmark is also real-world load generator
10
12 Dtrace probes are logged during testing Output: Every 10 seconds Probes triggered/second
11
A simple test to learn about behavior of the Dtrace probe “Accidentally” Identified NFS bottle-neck Changed number of NFS daemons Positive result
12
200 400 600 800 1000 Time Awr/32 Bwr Cwr Dwr E G/10
Before After Dtrace write probes
13
Filebench and IOzone show performance loss when load increases Filebench more close to “user experience” IOzone tests are not within the confidence interval (95%) after 30 runs
14
5 10 15 20 25 30 35 40 0 1 2 3 4 8 16 MB/s Load filebench
15
5 10 15 20 25 30 35 40 0 1 2 3 4 8 16 MB/s Load IOzone Random Write
16
200 400 600 800 1000 1200 1400 Probes / Second Time Awr/32 Bwr Cwr Dwr E G/10
ZIL write probe
17
18
ZIL write probe
ZFS ZIL Synchronous Write Asynchronous Write
All NFS writes are Synchronous and pass the ZIL
19
Place ZIL on fast device Solid State Disk (SSD) Well known practice
20
Filebench and IOzone show performance increase When the load increases performance decreases IOzone tests are not within the confidence interval (95%) after 30 runs
21
10 20 30 40 50 60 0 1 2 3 4 8 16 MB/s Load filebench ZIL on SSD
filebench normal ZIL
22
200 400 600 800 1000 1200 1400 Probes / Second Time Awr/32 Bwr Cwr Dwr E G/10
ZIL write probe Dtrace write Probes
23
Using Dtrace probes, performance bottlenecks
can be created. A realistic load can be generated using the filebench - fileserver personality Filebench test created a confident baseline (unlike IOzone)
24
Thanks to: Jeroen Roodhart Auke Folkerts
25