1

Be SMART, Save I/O: Probabilistic Approach to�Avoid Uncorrectable Errors in Storage Systems

Md Arifuzzaman, Masudul Bhuiyan, Mehmet Gumus, and Engin Arslan

University of Nevada, Reno, CISPA Helmholtz Center for Information Security

​

Big Data of Science

3

Error is Inevitable

3

• Hard drives use Error Correcting Code (ECC) to detect and recover from errors.

​

• Yes, a non-negligible portion of errors skips ECC, referred to as Uncorrectable Errors (UE)

​

• Uncorrectable errors take place once in every 10¹² (125GB) to 10¹⁵ (125TB) bits of I/O operation.

​

• 20-57% of disks in Google and Facebook data centers experienced one UE in their lifetime.

Failure is Costly

4

• Uncorrectable errors can lead to data loss, disk failure, or even service interruption.

​

• According to Gartner, server downtime costs $5,600 per minute on average.

​

• The average cost of a data center outage is $740,357.

Problem Statement

5

Can we predict uncorrectable errors before they happen?

Almost!

SMART Values

• SMART (Self-Monitoring, Analysis and Reporting Technology) is a monitoring system
• It is supported by most disk drives.
• It provides various indicators of drive health.
• It reports various types of disk errors such as Seek Error Rate, Read Error Rate, and Uncorrectable Error.
• It also reports operational data, such as drive temperature, and power on hours of the drive.

​

• SMART logs are typically populated once a day for each disk.

6

Dataset

• February 2014 to September 2021 (93 months)

​

• 270M SMART logs.

​

• 200K unique disks in total but 30K-120K of them were active at any given day.

​

• 9.25K disks had at least one uncorrectable error (error percentage 8.7%)

​

• 5% of disks had 89% of all errors and less than 10% of disks had 92.4% of all errors.

​

​

7

SMART Attributes

8

9

Performance Comparison Of Machine Learning Models

Neural network models

Previous proposed Model

Application Scenarios to Utilize the Predictability of Uncorrectable Errors

10

11

Sender

Receiver

​

End-to-End File Transfer

12

Sender

Receiver

​

End-to-End Integrity Verification​​​

2367213558

2367213558

Additional I/O Size for End-to-End Integrity Verification​​​

13

Impact of Integrity Verification on Transfer Time

14

15

Probabilistic Integrity Verification

16

17

Probabilistic Integrity Verification

18

Probabilistic Integrity Verification

p=0.05

p=0.02

p=0.6

p=0.9

19

Probabilistic Integrity Verification

p=0.05

p=0.02

p=0.6

p=0.9

Problem Definition

20

21

X

X

X

X

• Captures all errors
• Incurs high I/O overhead
• Slow transfer speed

​

• No I/O overhead
• Fast transfers
• Fails to capture errors

• Low I/O overhead
• Fast transfers
• Captures most errors

X

X

Probabilistic Integrity Verification aims to select disks to run integrity verification such that most errors captured by checking fewer disks possible

Problem Definition

22

Brute Force Solution

23

.

.

.

No disk

1,2

2,3

.

.

.

1,2,…,N

Greedy Solution

24

.

.

.

.

.

.

.

.

.

25

Evaluation Metrics

p=0.05

p=0.02

p=0.6

p=0.9

Comparison of Brute Force and Greedy algorithms

26

Improvement Ratio

27

• Large improvement ratio leads to an increase in the coverage ratio for all disk types.

​

• Consequently I/O save ratio decreases.

Coverage Ratio Comparison

28

Batch Workload

Streaming Workload

I/O Overhead comparison

29

Batch Workload

Streaming Workload

Overhead Analysis on Production File Systems

30

Impact of probabilistic write verification

31

Darwin

Expanse

Darwin

Expanse

Conclusion

• If not handled properly, uncorrectable errors can lead to complete data loss or even system failure.

​

• We analyzed 270M SMART logs from 200K disks over a period of 93 months and developed prediction models for uncorrectable errors.

​

• Our models achieve up-to 98% accuracy in uncorrectable bit error prediction while keeping false positive rates less than 3%.

​

32

Conclusion

• We incorporated the prediction model into two application scenarios.

​

• Our probabilistic integrity verification method achieves up to 97% reduction in I/O overhead caused by the integrity verification process of wide area file transfers.

​

​

​

​

33

Performance Analysis for Different Improvement Factors

34

• Large improvement ratio leads to an increase in the coverage ratio for all disk types.

​

• Consequently I/O save ratio decreases.

​

• For 90% improvement ratio, we can achieve 85-95% coverage ratio while saving 40-75% I/O.

​

​

Impact of Probabilistic Integrity Verification on Transfer Time

35

Transfer times is reduced by 53% for 90% I/O save

Avoiding Uncorrectable Errors through I/O Redirection

36

37

I/O Redirection

p=0.1

p=0.2

p=0.6

p=0.9

Problem Definition

38

Problem Definition

Let

• 𝛾 : increased error ratio for increasing I/O load

39

Then error probability will be

I/O Redirection

40

.

.

.

.

.

.

p=0.6

p=0.9

p=0.1

p=0.8

41

Evaluation Metrics

p=0.05

p=0.02

p=0.6

p=0.9

Performance Analysis of I/O Redirection

42

• We can achieve more than 50% coverage ratio while keeping overhead less than 5%.

​

• For disk model ST12000NM0007, we can even achieve 85-90% coverage ratio for less than 5% overhead.

Scale is Increasing and So Does Failure

43

• 90% of the current datacenters are established after 2016.

​

• Google uses more than 2.5 million custom servers that are equipped with multiple hard drives.

​

• Facebook’s new data center has 500 racks that each hold 2 petabytes of data.

Disk Models Used in This Work

44

Evaluation Metrics

45

Model Evaluation

46

Random Forest achieves 97% TRP for less than 3% FPR

Performance Evaluation for Random Forest Classifier

• Random Forest achieves more than 95% TPR while keeping FPR less than 5% for ST12000NM007 and ST4000DM000.

​

47

• For ST80000NM0055 and ST80000DM002, 90% TPR causes less than or around 15% FPR.

48

Probabilistic Integrity Verification

p=0.1

p=0.2

p=0.6

p=0.9

Era of Big Data

49

Problem Definition

50

Let

• 𝐸(𝑇) : Error probability without running integrity verification.

51

Probabilistic Integrity Verification

p=0.05

p=0.02

p=0.6

p=0.9

Additional I/O Size for End-to-End Integrity Verification​​​

52

53

Evalution Metrics

54

Evalution Metrics

55

56

Comparison of Brute Force and Greedy algorithms

57

Deployment Challenges and Opportunities

• We developed our model using SMART metrics which is supported by all disk manufacturers.
• Storage footprint to store daily SMART logs is very negligible(32.5MB for 120k disks)
• Transfer learning is challenging but our model can be used as a good starting point.
• Both Raid arrays and Parallel File Systems have options to deactivate disks.

58

Feature Selection

59

Checksum Calculation Speedup Ratio

60

Dataset Charactaristics

• 14.5% of disks raised only one error.

​

• 21.7% of disks raised more than 20 uncorrectable errors.

​

• 6% disks had more than 100 errors.

​

61

Handling Data imbalance

62

Problem Definition

63

Simulation Default Settings

• We simulated our algorithm on BackBlaze SMART logs.
• RAID level ->0(no parity or mirroring)
• Number of disks in RAID array -> 8
• Stripe Size -> 128KB
• Stripe Count -> 1
• the number of disks used for each file transfer -> 8
• Improvement ratio, k -> 90% (10 times improvement)

64

Dataset Characteristics

65

Performance Analysis for different Stripe Count

66

I/O Redirection

67

.

.

.

.

.

.

p=0.6

p=0.9

p=0.1

p=0.8