Wednesday, August 19th 2020

Backblaze Releases Q2 2020 Hard Drive Stats for 142,630 Spinning HDDs

As of June 30, 2020, Backblaze had 142,630 spinning hard drives in our cloud storage ecosystem spread across four data centers. Of that number, there were 2,271 boot drives and 140,059 data drives. This review looks at the Q2 2020 and lifetime hard drive failure rates of the data drive models currently in operation in our data centers and provides a handful of insights and observations along the way. As always, we look forward to your comments.

Quarterly Hard Drive Failure Stats for Q2 2020
At the end of Q2 2020, Backblaze was using 140,059 hard drives to store customer data. For our evaluation we remove from consideration those drive models for which we did not have at least 60 drives (see why below). This leaves us with 139,867 hard drives in our review. The table below covers what happened in Q2 2020.
Notes and Observations
The Annualized Failure Rate (AFR) for Q2 2020 was 0.81% versus Q1 2020 which was 1.07%. The Q2 AFR number is the lowest AFR for any quarter since we started keeping track in 2013. In addition, this is the first time the quarterly AFR has been under 1%. One year ago (Q2 2019), the quarterly AFR was 1.8%.

During this quarter, three drive models had 0 (zero) drive failures: the Toshiba 4 TB (model: MD04ABA400V), the Seagate 6 TB (model: ST6000DX000) and the HGST 8 TB (model: HUH728080ALE600). While the Toshiba 4 TB drives recorded less than 10,000 drive days, we have not had a drive failure for that model since Q4 2018, or 54,054 drive days. In comparing drive days with the Toshiba drive, the Seagate 6 TB and HGST 8 TB drives are just as impressive, having no failures in the quarter yet recording 80,626 and 91,000 drive days respectively in Q2 2020.

There were 192 drives (140,059 minus 139,867) that were not included in the list above because we did not have at least 60 drives of a given model. For example, we have: 20 Toshiba 16 TB drives (model: MG08ACA16TA) we are putting through our certification process. On the other end of the spectrum, we still have 25 HGST 4 TB drives (model: HDS5C4040ALE630), putting in time in Storage Pods. Observant readers might note the model number of those HGST drives and realize they were the last of the drives produced with Hitachi model numbers.

Reminiscing aside, when we report quarterly, yearly, or lifetime drive statistics, those models with less than 60 drives are not included in the calculations or graphs. We use 60 drives as a minimum as there are 60 drives in all newly deployed Storage Pods. Note: The Seagate 16 TB drive (model: ST16000NM001G) does show 59 drives and is listed in the report because the one failed drive had not been replaced at the time the data for this report was collected.

That said, all the data from all of the drive models, including boot drives, is included in the files which can be accessed and downloaded on our Hard Drive Test Data webpage.

What We Deployed in Q2
We deployed 12,063 new drives and removed 1,960 drives via replacements and migration in Q2, giving us a net of 10,103 added drives. Below is a table of the drive models we deployed.
Quarterly Trends by Manufacturer
Quarterly data is just that, data for only that quarter. At the beginning of each quarter we wipe out all the previous data and we start compiling new information. At the end of the quarter, we bundle that data up into a unit (collection, bag, file, whatever), and name it; Q2 2020, for example. This is the type of data you were looking at when you reviewed the Quarterly Chart for Q2 2020 shown earlier in this report. We can also compare the results for a given quarter to other quarters, each their own unique bundle of data. This type of comparison can reveal trends that can help us identify something that needs further attention.

The chart below shows the AFR by manufacturer using quarterly data over the last three years. Following the chart is two tables. The first is the data used to create the chart. The second is the count of the number of hard drives corresponding to each quarter for each manufacturer.
Notes
  • The data for each manufacturer consists of all the drive models in service which were used to store customer data. There were no boot drives or test drives included.
  • The 0.00% values for the Toshiba drives from Q3 2017 through Q3 2018 are correct. There were no Toshiba drive failures during that period. Note, there were no more than 231 drives in service at any one time during that same period. While zero failures over five quarters is notable, the number of drives is not high enough to reach any conclusions.
  • The "n/a" values for the WDC drives from Q2 2019 onward indicate there were zero WDC drives being used for customer data in our system during that period. This does not consider the newer HGST drive models branded as WDC as we do not currently have any of those models in operation.
Observations
  • WDC: The WDC data demonstrate how having too few data points (i.e. hard drives) can lead to a wide variance in quarter to quarter comparisons.
  • Toshiba: Just like the WDC data, the number of Toshiba hard drives for most of the period is too low to reach any decent conclusions, but beginning in Q4 2019, that changes and the data from then on is more reliable.
  • Seagate: After a steady rise in AFR, the last two quarters have been kind to Seagate, with the most recent quarter (AFR = 0.90%) being the best we have ever seen from Seagate since we started keeping stats back in 2013. Good news and worthy of a deeper look over the coming months.
  • HGST: With the AFR fluctuating between 0.36% and 0.61%, HGST drives win the prize for predictability. Boring, yes, but a good kind of boring.
Cumulative Trends by Manufacturer
As opposed to quarterly data, cumulative data starts collecting data at a given point and new data is added until you stop collecting. While quarterly data reflects the events that took place during a given quarter, cumulative data is everything about our collection of hard drives over time. Using cumulative data, we can see longer term trends over the period, as in the chart below, with the data table following.
Down and to the Right
For all manufacturers, you can see a downward trend in AFR over time. While this is a positive occurrence, we do want to understand why and incorporate those learnings into our overall understanding of our environment—just like drive failure, drive "non-failure" matters too. As we consider these findings, if you have any thoughts on the subject, let us know in the comments. Maybe you think hard drives are getting better, or is it more likely that we've added so many new drives in the last three years that they dominate the statistics, or is it something else? Let us know.

Lifetime Hard Drive Failure Rates
The table below shows the lifetime AFR for the hard drive models we had in service as of June 30, 2020. The reporting period is from April 2013 through June 30, 2020. All of the drives listed were installed during this timeframe.
Notes and Observations
The lifetime AFR was 1.64%, the lowest since we started keeping track in 2013. In addition, the lifetime AFR has fallen from 1.86% in Q2 2018 to the current value, even as we've passed milestones like an exabyte of storage under management, opening a data center in Amsterdam, and nearly doubling the size of the company. A busy two years.

All of the Seagate 12 TB drives (model: ST12000NM001G) were installed in Q2, so while we have a reasonable amount of data, as a group these drives are still early in their lifecycle. While not all models follow the bathtub curve as they age, we should wait another couple of quarters to see how they are performing in our environment.

The Seagate 4 TB drives (model: ST4000DM000) keep rambling along. With an average age of nearly five years, they are long past their warranty period (one or two years depending on when they were purchased). Speaking of age, the drive model with the highest average age on the chart is the Seagate 6 TB drive at over 64 months. That same model had zero failures in Q2 2020, so they seem to be aging well.
Add your own comment

17 Comments on Backblaze Releases Q2 2020 Hard Drive Stats for 142,630 Spinning HDDs

#1
Frick
Fishfaced Nincompoop
"Spinning HDD's" must mean the disks themselves are spinning. Rotating racks?
Posted on Reply
#2
Raevenlord
News Editor
Frick
"Spinning HDD's" must mean the disks themselves are spinning. Rotating racks?
In fact, we're all spinning.
Posted on Reply
#3
Verpal
Raevenlord
In fact, we're all spinning.
I think it is appropriate to ask this final question here.....

Is the Universe spinning?
Posted on Reply
#4
windwhirl
Verpal
I think it is appropriate to ask this final question here.....

Is the Universe spinning?
As in the insides or the whole thing?

For that matter, does the multiverse spin too? :D
Posted on Reply
#5
Raevenlord
News Editor
Ah TPU, asking "the realest" existential questions inside the most random comment thread :roll::lovetpu:
Posted on Reply
#6
altcapwn
Raevenlord
Ah TPU, asking "the realest" existential questions inside the most random comment thread :roll::lovetpu:
Everyone know the earth is flat and the sun is spinning around it :laugh: (I'm joking guys please do not attack me).

But did I miss something or I just don't see any data for "Western Digital"?
Yeah I did miss something and I'm blind. WDC isn't part of the data gathering now since Q2 2019.
Posted on Reply
#7
SingularitE
Raevenlord
For all manufacturers, you can see a downward trend in AFR over time. While this is a positive occurrence, we do want to understand why and incorporate those learnings into our overall understanding of our environment—just like drive failure, drive "non-failure" matters too. As we consider these findings, if you have any thoughts on the subject, let us know in the comments. Maybe you think hard drives are getting better, or is it more likely that we've added so many new drives in the last three years that they dominate the statistics, or is it something else? Let us know.
Maybe there are better operating conditions like less start-stop cycles for standby/shutdown(if any), less folder/file fragmentation (fewer random head flying) or constant/lower operating temperatures.
IMO hard drives aren't getting better and, with new models being mostly SMR/hybrid SMR with lots of overhead (just like new QLC/TLC SSDs), they are past the peak of best reliability/durability that was back in the Barracuda 7200.7 days.
Posted on Reply
#8
djisas
SingularitE
Maybe there are better operating conditions like less start-stop cycles for standby/shutdown(if any), less folder/file fragmentation (fewer random head flying) or constant/lower operating temperatures.
IMO hard drives aren't getting better and, with new models being mostly SMR/hybrid SMR with lots of overhead (just like new QLC/TLC SSDs), they are past the peak of best reliability/durability that was back in the Barracuda 7200.7 days.
Barracuda 7200.11 was a disaster, I still bough a coulple 7200.12 drives or more, but i sold the last one a little while ago and I don't want more barracudas...
Posted on Reply
#9
mtcn77
djisas
Barracuda 7200.11 was a disaster, I still bough a coulple 7200.12 drives or more, but i sold the last one a little while ago and I don't want more barracudas...
See, this is why I hate backblaze with a passion. They pot the harddrives most selected with the least and make up such conventions as "no harddrive least of 60" benchmarks to validate their awful predicament.
What in the world has their benchmark which foregoes basic regimens such as keeping the drive temperature in check with ample cooling has anything to do with hard drive reliability lest their own malconduct? They literally abuse the charts for the most used drives to see how far they can tarnish a reputation in disguise of benchmarking sensation. They are the crash squad of unverifiable market share claims.
Posted on Reply
#10
newtekie1
Semi-Retired Folder
Yay, more useless data for consumers!
Posted on Reply
#11
djisas
mtcn77
See, this is why I hate backblaze with a passion. They pot the harddrives most selected with the least and make up such conventions as "no harddrive least of 60" benchmarks to validate their awful predicament.
What in the world has their benchmark which foregoes basic regimens such as keeping the drive temperature in check with ample cooling has anything to do with hard drive reliability lest their own malconduct? They literally abuse the charts for the most used drives to see how far they can tarnish a reputation in disguise of benchmarking sensation. They are the crash squad of unverifiable market share claims.
Had a few drives die and they where all seagate ones, it used to be my brand of choice, the last seagate I bough blew up in smoke swapped with a samsung F3...
Posted on Reply
#12
mtcn77
djisas
Had a few drives die and they where all seagate ones, it used to be my brand of choice, the last seagate I bough blew up in smoke swapped with a samsung F3...
If you notice this is the sort of thing I'm complaining about. Having your basket full of a single brand single lineup, then complaining about it is putting all your eggs in one basket. It is not market wide survey, it is market share survey. You could have 1, or a thousand die on your behalf - it still wouldn't make the least bit of difference in your benchmark credibility.
Posted on Reply
#13
John Naylor
Backblaze again releases totally irrelevant data describing what happens when bargain basement company installs drives in conflict with manufacturer's recommendations. The survey should be entitled "How to Accelerate Product Failure by Installing Consumer Drives in a Server Environment and in Conflict with Manufacturers Installation Requirements.". Luckily I bookmarked the response from the last time these irrelevant statistics were posted. The very features that make a consumer drive a good consumer drive accelerate failure in a server environment.

www.techpowerup.com/forums/threads/most-reliable-hard-drives-nowadays.259783/page-2#post-4170954
Posted on Reply
#14
djisas
mtcn77
If you notice this is the sort of thing I'm complaining about. Having your basket full of a single brand single lineup, then complaining about it is putting all your eggs in one basket. It is not market wide survey, it is market share survey. You could have 1, or a thousand die on your behalf - it still wouldn't make the least bit of difference in your benchmark credibility.
Well to be fair, I had maxtor drives, Hitachi drives, Seagate, WD and samsung in this order in about 18 drives I have owned 1 seagate 7200.11 320 GB died and 2 7200.11 500GB also died, all where drives made in china while drives made in Thailand where fine, I also had 2 7200.12 500GB drives survive several years with bad sectors, my 3 WD's outlived every other drive until sold although one is just about 3 yo. My pc always had 4 or more hdd at the same time with good airflow...

Experience will vary with users, mine has been bad with seagate...
Posted on Reply
#15
mtcn77
djisas
Experience will vary with users, mine has been bad with seagate...
Mine clicked its head once in 15 years. How do I sign up for the bandwagon?
What I recon is, they should get reliability points however much they are tested. If they want to promote others, first rigorous benchmarking is due. Not free points, but I don't know. Maybe, it is Seagate at fault being the choice brand they tested on?
Posted on Reply
#16
Jism
I dont know what is up with seagate, but the faillure of their drives is just immense high for what the company's reputation is worth. Ive seen nummerous disks of seagate years ago, back when i used todo data recovery. WD was 2nd on the list. If you rely on data to be stored for years, dont buy seagate. That is just a fact.
Posted on Reply
Add your own comment