SSD Fail ilures in in Datacenters: What? When? And Why? Iyswarya - - PowerPoint PPT Presentation
SSD Fail ilures in in Datacenters: What? When? And Why? Iyswarya Narayanan, Di Wang, Myeongjae Jeon, Bikash Sharma, Laura Caulfield, Anand Sivasubramaniam, Ben Cutler, Jie Liu, Badriddine Khessib, Kushagra Vaid The 9 th ACM Systems And Storage
Iyswarya Narayanan, Di Wang, Myeongjae Jeon, Bikash Sharma, Laura Caulfield, Anand Sivasubramaniam, Ben Cutler, Jie Liu, Badriddine Khessib, Kushagra Vaid
1
The 9th ACM Systems And Storage Conference (SYSTOR 2016)
SSDs’ popularity
2
*Source: IDC, Dec 2015
46.5% annual growth*
Limited field data Datacenter decision support Data reliability
01001100 01001101 11010010 01000000 10011100 10111111 10101111 11000101
SSDs’ popularity
3
*Source: IDC, Dec 2015
46.5% annual growth*
Limited field data Datacenter decision support Data reliability
01001100 01001101 11010010 01000000 10011100 10111111 10101111 11000101
Large scale Field data
4
Flash failures
FTL Mechanisms
refresh, etc.
5
Flash failures
FTL Mechanisms
refresh, etc.
6
Flash failures
FTL Mechanisms
refresh, etc.
7
Flash failures
FTL Mechanisms
refresh, etc.
8
Flash failures
FTL Mechanisms
refresh, etc.
9
Flash failures
FTL Mechanisms
refresh, etc.
Fail-stop failures
10
0.2 0.4 0.6 0.8 1 1.2 1-A 1-B 1-C 1-D 2-A Annualized Failure Rate % SSD Model AFR=0.61 AFR=0.73
Consumer Enterprise
11
0.2 0.4 0.6 0.8 1 1.2 1-A 1-B 1-C 1-D 2-A Annualized Failure Rate % SSD Model AFR=0.61 AFR=0.73
12
0.2 0.4 0.6 0.8 1 1.2 1-A 1-B 1-C 1-D 2-A Annualized Failure Rate % SSD Model AFR=0.61 AFR=0.73
5 large datacenters
13
0.2 0.4 0.6 0.8 1 1.2 1-A 1-B 1-C 1-D 2-A Annualized Failure Rate % SSD Model AFR=0.61 AFR=0.73
4 major workloads
14
0.2 0.4 0.6 0.8 1 1.2 1-A 1-B 1-C 1-D 2-A Annualized Failure Rate % SSD Model AFR=0.61 AFR=0.73
6 different rack SKUs
15
0.2 0.4 0.6 0.8 1 1.2 1-A 1-B 1-C 1-D 2-A Annualized Failure Rate % SSD Model AFR=0.61 AFR=0.73 Various factors in production environment could affect SSD failure trends very differently from lab test conditions Can we understand SSD failures in the presence of various factors ?
16
SSD Reactive Proactive
17
Fever Unexpected weight loss Low blood pressure Data errors 011001?00101? Reallocated sectors SATA downshift
SSD
Program and erase failure
18
Reallocated Sector Count Program and Erase Fail Count CRC and Uncorrectable Error Count SATA Downshift Count 0.5 1 1.5 2 2.5 3 3.5
Reallocated Sector Count Program and Erase Failure Count CRC and Uncorrectable Error Count SATA Downshift Count
AFR % w Symptom w/o Symptom 3.95X 2.76X 18X 3.91X
Insufficiency of symptom only diagnosis
19
10 20 30 40 50 60 70
Reallocations Program and Erase Fail Data Errors SATA Downshift Any
% of devices Failed Healthy Symptoms seen
failed devices
20
Lifestyle Genetics Environmental agents Production environment Workload Design decisions
SSD
21
Average write rate of a device Average read rate of a device Total read and/or write usage Write Amplification Read Write Ratio 0.5 1 1.5 2 2.5 10 15 20 25 30 35 40 45 50 >50 AFR %
More results in the paper Increasing failure trend at higher write rates
22
SSD space utilization Disk space utilization Memory utilization Processor utilization
0.2 0.4 0.6 0.8 1 1.2 10 20 30 40 50 60 70 AFR%
More results in the paper Decreasing failure trend at high disk space usage
Rack SKU Datacenter Facility
23
0.1 0.2 0.3 0.4 0.5 0.6 1-D 2-A 1-D 2-A S1-3a S1-3b AFR % SKU and SSD model
More results in the paper
Same model different behavior
24
SSD
Symptoms Factors Symptoms Factors
MULTI FEATURE ANALYSIS
25
SSD
Symptoms Factors Symptoms Factors Random forest based binary classification Permutation feature ranking
26
are the important factors ? is their order of importance ? are the important combinations?
27
0.2 0.4 0.6 0.8 1 DataErrors ReallocSectors TotalNANDWrites HostWrites TotalReads+Writes AvgMemory AvgSSDSpace UsagePerDay TotalReads ReadsPerDay
Feature Importance
SYMPTOMS
28
0.2 0.4 0.6 0.8 1 DataErrors ReallocSectors TotalNANDWrites HostWrites TotalReads+Writes AvgMemory AvgSSDSpace UsagePerDay TotalReads ReadsPerDay
Feature Importance
DEVICE WORKLOAD
29
0.2 0.4 0.6 0.8 1 DataErrors ReallocSectors TotalNANDWrites HostWrites TotalReads+Writes AvgMemory AvgSSDSpace UsagePerDay TotalReads ReadsPerDay
Feature Importance
SERVER WORKLOAD
30
Condition Class Data Errors <=1 & Reallocated Sectors<=5 H Data Errors<=1& WAF<=1 H Media Wear-out=100 & WAF<=1 H
F
Combinations of top 8 important features Frequent Combinations
SYMPTOMS
31
Condition Class Data Errors <=1 & Reallocated Sectors<=5 H Data Errors<=1& WAF<=1 H Media Wear-out=100 & WAF<=1 H
F
Combinations of top 8 important features Frequent Combinations
SYMPTOMS + WORKLOAD
32
Condition Class Data Errors <=1 & Reallocated Sectors<=5 H Data Errors<=1& WAF<=1 H Media Wear-out=100 & WAF<=1 H
F
Combinations of top 8 important features Frequent Combinations
WORKLOAD
33
is the duration between detection and failure? signatures characterize SSD survivability?
34
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 2 4 6 8 10 12 CDF(x) Time To Fail (months)
50% of failures > 4 months Sufficient time to intervene
35
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 2 4 6 8 10 12 CDF(x) Time To Fail (months)
50% of failures > 4 months Early failures (< 1 month): Rules include symptoms and their thresholds Late failures: Rules contains only workload factors
Understanding SSD Failures – An analogy
36
SSD
Symptoms Factors Symptoms Factors Observation based causal estimate Probabilistic causal models and Pearl’s do-calculus
37
factors impact SSD reliability? is their magnitude of impact?
38
SSD model and symptoms have direct impact Workload impacts failures through media wearout
39