Storage Vaults form the core of Backblaze’s cloud services. Backblaze Vaults are not only incredibly durable, scalable, and performant, but they dramatically improve availability and operability, while still being incredibly cost-efficient at storing data. Back in 2009, we shared the design of the original Storage Pod hardware we developed; here we’ll share the architecture and approach of the cloud storage software that makes up a Backblaze Vault.

Backblaze Vault Architecture for Cloud Storage

The Vault design follows the overriding design principle that Backblaze has always followed: keep it simple. As with the Storage Pods themselves, the new Vault storage software relies on tried and true technologies used in a straightforward way to build a simple, reliable, and inexpensive system.

A Backblaze Vault is the combination of the Backblaze Vault cloud storage software and the Backblaze Storage Pod hardware.

Putting The Intelligence in the Software

Another design principle for Backblaze is to anticipate that all hardware will fail and build intelligence into our cloud storage management software so that customer data is protected from hardware failure. The original Storage Pod systems provided good protection for data and Vaults continue that tradition while adding another layer of protection. In addition to leveraging our low-cost Storage Pods, Vaults take advantage of the cost advantage of consumer-grade hard drives and cleanly handle their common failure modes.

Distributing Data Across 20 Storage Pods

A Backblaze Vault is comprised of 20 Storage Pods, with the data evenly spread across all 20 pods. Each Storage Pod in a given vault has the same number of drives, and the drives are all the same size.

Drives in the same drive position in each of the 20 Storage Pods are grouped together into a storage unit we call a tome. Each file is stored in one tome and is spread out across the tome for reliability and availability.

Every file uploaded to a Vault is divided into pieces before being stored. Each of those pieces is called a shard. Parity shards are computed to add redundancy, so that a file can be fetched from a vault even if some of the pieces are not available.

Each file is stored as 20 shards: 17 data shards and three parity shards. Because those shards are distributed across 20 Storage Pods, the Vault is resilient to the failure of a Storage Pod.

Files can be written to the Vault when one pod is down and still have two parity shards to protect the data. Even in the extreme and unlikely case where three Storage Pods in a Vault lose power, the files in the vault are still available because they can be reconstructed from any of the 17 pods that are available.

Storing Shards

Each of the drives in a Vault has a standard Linux file system, ext4, on it. This is where the shards are stored. There are fancier file systems out there, but we don’t need them for Vaults. All that is needed is a way to write files to disk and read them back. Ext4 is good at handling power failure on a single drive cleanly without losing any files. It’s also good at storing lots of files on a single drive and providing efficient access to them.

Compared to a conventional RAID, we have swapped the layers here by putting the file systems under the replication. Usually, RAID puts the file system on top of the replication, which means that a file system corruption can lose data. With the file system below the replication, a Vault can recover from a file system corruption because a single corrupt file system can lose at most one shard of each file.

Creating Flexible and Optimized Reed-Solomon Erasure Coding

Just like RAID implementations, the Vault software uses Reed-Solomon erasure coding to create the parity shards. But, unlike Linux software RAID, which offers just one or two parity blocks, our Vault software allows for an arbitrary mix of data and parity. We are currently using 17 data shards plus three parity shards, but this could be changed on new vaults in the future with a simple configuration update.

For Backblaze Vaults, we threw out the Linux RAID software we had been using and wrote a Reed-Solomon implementation from scratch, which we wrote about in “Backblaze Open-sources Reed-Solomon Erasure Coding Source Code.” It was exciting to be able to use our group theory and matrix algebra from college.

The beauty of Reed-Solomon is that we can then re-create the original file from any 17 of the shards. If one of the original data shards is unavailable, it can be re-computed from the other 16 original shards, plus one of the parity shards. Even if three of the original data shards are not available, they can be re-created from the other 17 data and parity shards. Matrix algebra is awesome!

Handling Drive Failures

The reason for distributing the data across multiple Storage Pods and using erasure coding to compute parity is to keep the data safe and available. How are different failures handled?

If a disk drive just up and dies, refusing to read or write any data, the Vault will continue to work. Data can be written to the other 19 drives in the tome, because the policy setting allows files to be written as long as there are two parity shards. All of the files that were on the dead drive are still available and can be read from the other 19 drives in the tome.

When a dead drive is replaced, the Vault software will automatically populate the new drive with the shards that should be there; they can be recomputed from the contents of the other 19 drives.

A Vault can lose up to three drives in the same tome at the same moment without losing any data, and the contents of the drives will be re-created when the drives are replaced.

Handling Data Corruption

Disk drives try hard to correctly return the data stored on them, but once in a while they return the wrong data, or are just unable to read a given sector.

Every shard stored in a Vault has a checksum, so that the software can tell if it has been corrupted. When that happens, the bad shard is recomputed from the other shards and then re-written to disk. Similarly, if a shard just can’t be read from a drive, it is recomputed and re-written.

Conventional RAID can reconstruct a drive that dies, but does not deal well with corrupted data because it doesn’t checksum the data.

Scaling Horizontally

Each vault is assigned a number. We carefully designed the numbering scheme to allow for a lot of vaults to be deployed, and designed the management software to handle scaling up to that level in the Backblaze data centers.

The overall design scales very well because file uploads (and downloads) go straight to a vault, without having to go through a central point that could become a bottleneck.

There is an authority server that assigns incoming files to specific Vaults. Once that assignment has been made, the client then uploads data directly to the Vault. As the data center scales out and adds more Vaults, the capacity to handle incoming traffic keeps going up. This is horizontal scaling at its best.

We could deploy a new data center with 10,000 Vaults holding 16TB drives and it could accept uploads fast enough to reach its full capacity of 160 exabytes in about two months!

Backblaze Vault Benefits

The Backblaze Vault architecture has six benefits:

1. Extremely Durable

2. Infinitely Scalable

3. Always Available

4. Highly Performant

5. Operationally Easier

6. Astoundingly Cost Efficient

Summary

Backblaze’s cloud storage Vaults calculated at 99.999999% (eight nines) annual durability (now 11 nines — Editor), horizontal scalability, and 20 Gbps of per-Vault performance, while being operationally efficient and extremely cost effective. Driven from the same mindset that we brought to the storage market with Backblaze Storage Pods, Backblaze Vaults continue our singular focus of building the most cost-efficient cloud storage available anywhere.

•  •  •

Note: This post was updated from the original version posted on March 11, 2015.

The post Backblaze Vaults: Zettabyte-Scale Cloud Storage Architecture appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

At Backblaze we have built an extremely cost-effective storage system that enables us to offer a great price on our online backup service. Along the path to building our storage system, we have used time-tested technologies off the shelf, but we have also built in-house technologies ourselves when things weren’t available, or when the price was too high.

We have taken advantage of many open-source projects, and want to do our part in contributing back to the community. Our first foray into open-source was our original Storage Pod design, back in September of 2009.

Today, we are releasing our latest open-source project: Backblaze Reed-Solomon, a Java library for erasure coding.

An erasure code takes a “message,” such as a data file, and makes a longer message in a way that the original can be reconstructed from the longer message even if parts of the longer message have been lost. Reed-Solomon is an erasure code with exactly the properties we needed for file storage, and it is simple and straightforward to implement. It ensures that an entire data element can be recovered even when part or parts of the original stored data element are lost or unavailable.

Erasure Codes and Storage

Erasure coding is standard practice for systems that store data reliably, and many of them use Reed-Solomon coding.

The RAID system built into Linux uses Reed-Solomon. It has a carefully tuned Reed-Solomon implementation in C that is part of the RAID module. Microsoft Azure uses a similar, but different, erasure coding strategy. We’re not sure exactly what Amazon S3 and Google Cloud Storage use because they haven’t said, but it’s bound to be Reed-Solomon or something similar. Facebook’s new cold storage system also uses Reed-Solomon.

If you want reliable storage that can recover from the loss of parts of the data, then Reed-Solomon is a well-proven technique.

Backblaze Vaults Utilize Erasure Coding

Earlier this year, I wrote about Backblaze Vaults, our new software architecture that allows a file to be stored across multiple Storage Pods, so that the file can be available for download even when some Storage Pods are shut down for maintenance.

To make Backblaze Vaults work, we needed an erasure coding library to compute “parity” and then use it to reconstruct files. When a file is stored in a Vault, it is broken into 17 pieces, all the same size. Then three additional pieces are created that hold parity, resulting in a total of 20 pieces. The original file can then be reconstructed from any 17 of the 20 pieces.

We needed a simple, reliable, and efficient Java library to do Reed-Solomon coding, but didn’t find any. So we built our own. And now we are releasing that code for you to use in your own projects.

Performance

Backblaze Vaults store a vast amount of data and need to be able to ingest it quickly. This means that the Reed-Solomon coding must be fast. When we started designing Vaults, we assumed that we would need to code in C to make things fast. It turned out, though, that modern Java virtual machines are really good, and the just-in-time compiler produces code that runs fast.

Our Java library for Reed-Solomon is as fast as a C implementation, and is much easier to integrate with a software stack written in Java.

A Vault splits data into 17 shards, and has to calculate three parity shards from that, so that’s the configuration we use for performance measurements. Running in a single thread on Storage Pod hardware, our library can process incoming data at 149 megabytes per second. (This test was run on a single processor core, on a Pod with an Intel Xeon E5-1620 v2, clocked at 3.70GHz, on data not already in cache memory.)

Reed-Solomon Encoding Matrix Example

Feel free to skip this section if you aren’t into the math.

We are fortunate that mathematicians have been working on matrix algebra, group theory, and information theory for centuries. Reed and Solomon used this body of knowledge to create a coding system that seems like magic. It can take a message, break it into n pieces, add k “parity” pieces, and then reconstruct the original from n of the (n+k) pieces.

The examples below use a “4+2” coding system, where the original file is broken into four pieces, and then two parity pieces are added. In Backblaze Vaults, we use 17+3 (17 data plus three parity). The math—and the code—works with any numbers as long as you have at least one data shard and don’t have more than 256 shards total. To use Reed-Solomon, you put your data into a matrix. For computer files, each element of the matrix is one byte from the file. The bytes are laid out in a grid to form a matrix. If your data file has “ABCDEFGHIJKLMNOP” in it, you can lay it out like this:

The Original Data

In this example, the four pieces of the file are each four bytes long. Each piece is one row of the matrix. The first one is “ABCD.” The second one is “EFGH.” And so on.
The Reed-Solomon algorithm creates a coding matrix that you multiply with your data matrix to create the coded data. The matrix is set up so that the first four rows of the result are the same as the first four rows of the input. That means that the data is left intact, and all it’s really doing is computing the parity.

Applying the Coding Matrix

The result is a matrix with two more rows than the original. Those two rows are the parity pieces.

Each row of the coding matrix produces one row of the result. So each row of the coding matrix makes one of the resulting pieces of the file. Because the rows are independent, you can cross out two of the rows and the equation still holds.

Data Loss: Two of the Six Rows Are “Lost”

With those rows completely gone, it looks like this:

Data Loss: The Matrix Without the Two “Lost” Rows

Because of all the work that mathematicians have done over the years, we know the coding matrix, the matrix on the left, is invertible. There is an inverse matrix that, when multiplied by the coding matrix, produces the identity matrix. As in basic algebra, in matrix algebra you can multiply both sides of an equation by the same thing. In this case, we’ll multiply on the left by the identity matrix:

Multiplying Each Side of the Equation by the Inverse Matrix
The Inverse Matrix and the Coding Matrix Cancel Out

This leaves the equation for reconstructing the original data from the pieces that are available:

Reconstructing the Original Data

So to make a decoding matrix, the process is to take the original coding matrix, cross out the rows for the missing pieces, and then find the inverse matrix. You can then multiply the inverse matrix and the pieces that are available to reconstruct the original data.

Summary

That was a quick overview of the math. Once you understand the steps, it’s not super complicated. The Java code goes through the same steps outlined above.

There is one small part of the code that does the actual matrix multiplications that has been carefully optimized for speed. The rest of the code does not need to be fast, so we aimed more for simple and clear.

If you need to store or transmit data, and be able to recover it if some is lost, you might want to look at Reed-Solomon coding. Using our code is an easy way to get started.

The post Backblaze Open-sources Reed-Solomon Erasure Coding Source Code appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Stats geeks: Now it’s your turn.

Backblaze has released the raw data collected from the more than 41,000 disk drives in our data center. To the best of our knowledge, this is the largest data set on disk drive performance ever to be made available publicly.

Over the past 16 months, I have been posting information about hard drive reliability based on the raw data that we collect in the Backblaze data center. I have been crunching those numbers to correlate drive failures with drive model numbers, with SMART statistics, and other variables.

There are lots of smart people out there who like working with data, and you may be one of them. Now it’s your turn to pore over the data and find hidden treasures of insight. All we ask is that if you find something interesting, that you post it publicly for the benefit of the computing community as a whole.

What’s in the Data?

The data that we have released is in two files, one containing the 2013 data and one containing the 2014 data. We’ll add data for 2015 and so on in a similar fashion.

Every day, the software that runs the Backblaze data center takes a snapshot of the state of every drive in the data center, including the drive’s serial number, model number, and all of its SMART data. The SMART data includes the number of hours the drive has been running, the temperature of the drive, whether sectors have gone bad, and many more things. (I wrote a blog post correlating SMART data with drive failures a few months ago.)

Each day, all of the drive “snapshots” are processed and written to a new daily stats file. Each daily stats file has one row for every drive operational in the data center that day. For example, there are 365 daily stats files in the 2014 data package with each file containing a “snapshot” for each drive operational on any given day.

What Does It Look Like?

Each daily stats file is in CSV (comma-separated value) format. The first line lists the names of the columns, and then each following line has all of the values for those columns. Here are the columns:

• Date: The date of the file in yyyy-mm-dd format.
• Serial Number: The manufacturer-assigned serial number of the drive.
• Model: The manufacturer-assigned model number of the drive.
• Capacity: The drive capacity in bytes.
• Failure: Contains a “0” if the drive is OK. Contains a “1” if this is the last day the drive was operational before failing.
• SMART Stats: 80 columns of data that are the raw and normalized values for 40 different SMART stats as reported by the given drive. Each value is the number reported by the drive.

The Wikipedia page on SMART (https://en.wikipedia.org/wiki/S.M.A.R.T.) has a good description of all of the data, and what the raw and scaled values are. The short version is that the raw value is the data directly from the drive. For example, the “Power On Hours” attribute reports the number of hours in the raw value. The normalized value is designed to tell you when the drive is OK. It starts at 100 and goes down to zero as the drive gets sick. (Some drives count down from 200.)

How to Compute Failure Rates

One of my statistics professors once said, “It’s all about counting.” And that’s certainly true in this case.

A failure rate says what fraction of drives have failed over a given time span. Let’s start by calculating a daily failure rate, which will tell us what fraction of drives fail each day. We’ll start by counting “drive days” and “failures.”

To count drive days, we’ll take a look every day and see how many drives are running. Here’s a week in the life of a (small) data center:

Each of the blue dots represents a drive running on a given day. On Sunday and Monday, there are 15 drives running. Then one goes away, and from Tuesday through Saturday there are 14 drives each day. Adding them up we get 15 + 15 + 14 + 14 + 14 + 14 + 14 = 100. That’s 100 drive days.

Now, let’s look at drive failures. One drive failed on Monday and was not replaced. Then, one died on Wednesday and was promptly replaced. The red dots indicate the drive failures:

So we have two drive failures in 100 drive days of operation. To get the daily failure rate, you simply divide. Two divided by 100 is 0.02, or 2%. The daily failure rate is 2%.

The annual failure rate is the daily failure rate multiplied by 365. If we had a full year made of weeks like the one above, the annual failure rate would be 730%.

Annual failures rates can be higher than 100%. Let’s think this through. Say we keep 100 drives running in our data center at all times, replacing drives immediately when they fail. At a daily failure rate of 2%, that means two drives fail each day, and after a year 730 drives will have died. We can have an annual failure rate above 100% if drives last less than a year on average.

Computing failure rates from the data that Backblaze has released is a matter of counting drive days and counting failures. Each row in each daily drive stats file is one drive day. Each failure is marked with a “1” in the failure column. Once a drive has failed, it is removed from subsequent daily drive stats files.

To get the daily failure rate of drives in the Backblaze data center, you can take the number of failures counted in a given group of daily stats files, and divide by the number of rows in the same group of daily stats files. That’s it!

Where Is the Data?

You’ll find links to download the data files here. You’ll also find instructions on how to create your own sqlite database for the data, and other information related to the files you can download.

Let Us Know What You Find

That’s about all you need to know about the data to get started. If you work with the data and find something interesting, let us know!

The post Reliability Data Set for 41,000 Hard Drives Now Open-source appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

It was one year ago that I first blogged about the failure rates of specific models of hard drives, so now is a good time for an update.

At Backblaze, as of December 31, 2014, we had 41,213 disk drives spinning in our data center, storing all of the data for our unlimited backup service. That is up from 27,134 at the end of 2013. This year, most of the new drives are 4TB drives, and a few are the new 6TB drives.

Hard Drive Failure Rates for 2014

Let’s get right to the heart of the post. The table below shows the annual failure rate through the year 2014. Only models where we have 45 or more drives are shown. I chose 45 because that’s the number of drives in a Backblaze Storage Pod and it’s usually enough drives to start getting a meaningful failure rate if they’ve been running for a while.

Notes:

1. The total number of drives in this chart is 39,696. As noted, we removed from this chart any model of which we had less than 45 drives in service as of December 31, 2014. We also removed Storage Pod boot drives. When these are added back in we have 41,213 spinning drives.
2. Some of the HGST drives listed were manufactured under their previous brand, Hitachi. We’ve been asked to use the HGST name and we have honored that request.

What Is a Drive Failure for Backblaze?

A drive is recorded as failed when we remove it from a Storage Pod for one or more of the following reasons:

1. The drive will not spin up or connect to the OS.
2. The drive will not sync, or stay synced, in a RAID Array.
3. The SMART stats we use show values above our thresholds.

Sometimes we’ll remove all of the drives in a Storage Pod after the data has been copied to other (usually higher-capacity) drives. This is called a migration. Some of the older Pods with 1.5TB drives have been migrated to 4TB drives. In general, migrated drives don’t count as failures because the drives that were removed are still working fine and were returned to inventory to use as spares.

This past year, there were several Pods where we replaced all the drives because the RAID storage was getting unstable, and we wanted to keep the data safe. After removing the drives, we ran each of them through a third-party drive tester. The tester takes about 20 minutes to check the drive; it doesn’t read or write the entire drive. Drives that failed this test were counted as failed and removed from service.

Takeaways: What Are the Best Hard Drives?

4TB Drives Are Great.

We like every one of the 4TB drives we bought this year. For the price, you get a lot of storage, and the drive failure rates have been really low. The Seagate Desktop HDD.15 has had the best price, and we have a LOT of them. Over 12 thousand of them. The failure rate is a nice low 2.6% per year. Low price and reliability is good for business.

The HGST drives, while priced a little higher, have an even lower failure rate, at 1.4% 1.0% (for all HGST 4TB models). It’s not enough of a difference to be a big factor in our purchasing, but when there’s a good price, we grab some. We have over 12 thousand of these drives.

Where are the WD 4TB drives?

There is only one Storage Pod of Western Digital 4TB drives. Why? The reason is simple: price. We purchase drives through various channel partners for each manufacturer. We’ll put out an RFQ (Request for Quote) for say 2,000 4TB drives, and list the brands and models we have validated for use in our Storage Pods. Over the course of the last year, Western Digital drives were often not quoted and when they were, they were never the lowest price. Generally the WD drives were $15-$20 more per drive. That’s too much of a premium to pay when the Seagate and HGST drives are performing so well.

3TB Drives Are Not So Great.

The HGST Deskstar 5K3000 3TB drives have proven to be very reliable, but expensive relative to other models (including similar 4TB drives by HGST). The Western Digital Red 3TB drives’ annual failure rate of 7.6% is a bit high but acceptable. The Seagate Barracuda 7200.14 3TB drives are another story. We’ll cover how we handled their failure rates in a future blog post.

Confidence in Seagate 4TB Drives

You might ask why we think the 4TB Seagate drives we have now will fare better than the 3TB Seagate drives we bought a couple years ago. We wondered the same thing. When the 3TB drives were new and in their first year of service, their annual failure rate was 9.3%. The 4TB drives, in their first year of service, are showing a failure rate of only 2.6%. I’m quite optimistic that the 4TB drives will continue to do better over time.

6TB Drives and Beyond: Not Sure Yet

We’re beginning the transition from using 4TB to using 6TB drives. Currently we have 270 of the Western Digital Red 6TB drives. The failure rate is 3.1%, but there have been only three failures. The statistics give a 95% confidence that the failure rate is somewhere between 0.1% and 17.1%. We need to run the drives longer, and see more failures, before we can get a better number.

We have just 45 of the Seagate 6TB SATA 3.5 drives, although more are on order. They’ve only been running a few months, and none have failed so far. When we have more drives, and some have failed, we can start to compute failure rates.

Which Hard Drive Should I Buy?

All hard drives will eventually fail, but based on our environment, if you are looking for good drive at a good value, it’s hard to beat the current crop of 4TB drives from HGST and Seagate. As we get more data on the 6TB drives, we’ll let you know.

What About the Hard Drive Reliability Data?

We will publish the data underlying this study in the next couple of weeks. There are over 12 million records covering 2014, which were used to produce the failure data in this blog post. There are over five million records from 2013. Along with the data, I’ll explain step by step how to compute an annual failure rate.

Related Posts

• “How Long Do Disk Drives Last?”
• “Hard Drive Temperature: Does It Matter?”

The post What Is the Best Hard Drive? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

I’ve shared a lot of Backblaze data about hard drive failure statistics. While our system handles a drive failing, we prefer to predict drive failures, and use the hard drives’ built-in SMART metrics to help. The dirty industry secret? SMART stats are inconsistent from hard drive to hard drive.

With nearly 40,000 hard drives and over 100,000,000GB of data stored for customers, we have a lot of hard-won experience. See which five of the SMART stats are good predictors of drive failure below. And see the data we have started to analyze from all of the SMART stats to see which other ones predict failure.

S.M.A.R.T.

Every disk drive includes Self-Monitoring, Analysis, and Reporting Technology (SMART), which reports internal information about the drive. Initially, we collected a handful of stats each day, but at the beginning of 2014 we overhauled our disk drive monitoring to capture a daily snapshot of all of the SMART data for each of the 40,000 hard drives we manage. We used Smartmontools to capture the SMART data.

But, before we dig into the data, we first need to define what counts as a failure.

What Is a Failure?

Backblaze counts a drive as failed when it is removed from a Storage Pod and replaced because it has 1) totally stopped working, or 2) because it has shown evidence of failing soon.

A drive is considered to have stopped working when the drive appears physically dead (e.g. won’t power up), doesn’t respond to console commands or the RAID system tells us that the drive can’t be read or written.

To determine if a drive is going to fail soon we use SMART statistics as evidence to remove a drive before it fails catastrophically or impedes the operation of the Storage Pod volume.

From experience, we have found the following five SMART metrics indicate impending disk drive failure:

• SMART 5: Reallocated_Sector_Count.
• SMART 187: Reported_Uncorrectable_Errors.
• SMART 188: Command_Timeout.
• SMART 197: Current_Pending_Sector_Count.
• SMART 198: Offline_Uncorrectable.

We chose these five stats based on our experience and input from others in the industry because they are consistent across manufacturers and they are good predictors of failure.

The Other SMART Stats

We compiled and placed online our list of all the SMART stats across all hard drives we use. For each stat we display the failure rate charts based on the raw and normalized values we recorded. Remember, this is raw data and since different disk drive manufacturers report SMART stats differently, be careful how you use this.

Choosing the Right Stats to Use

There are over 70 SMART statistics available, but we use only five. To give some insight into the analysis we’ve done, we’ll look at three different SMART statistics here. The first one, SMART 187, we already use to decide when to replace a drive, it’s really a test of the analysis. The other two are SMART stats we don’t use right now, but have potentially interesting correlations with failure

SMART 187: Reported_Uncorrect

Backblaze uses this one.
Number 187 reports the number of reads that could not be corrected using hardware ECC. Drives with zero uncorrectable errors hardly ever fail. This is one of the SMART stats we use to determine hard drive failure; once SMART 187 goes above zero, we schedule the drive for replacement.

This first chart shows the failure rates by number of errors. Because this is one of the attributes we use to decide whether a drive has failed, there has to be a strong correlation:

The next question you might ask is: How many drives fall into each of those ranges? That’s answered by the next chart:

This looks at the full time range of the study, and counts “drive years.” Each day that a drive is in one of the ranges counts as 1/365 of a drive year for that range. Those fractions are all added up to produce the chart above. It shows that most of the daily samples show drives without errors.

For SMART 187, the data appears to be consistently reported by the different manufacturers, the definition is well understood, and the reported results are easy to decipher: zero is good, above zero is bad. For Backblaze this is a very useful SMART stat.

SMART 12: Power_Cycle_Count

Backblaze does not use this one.
The number of times the power was turned off and turned back on correlates with failures:

We’re not sure whether this is because cycling the power is bad for the drive, or because working on the Pods is bad for the drives, or because “new” drives have flaws that are exposed during the first few dozen power cycles and then things settle down.

Most of our drives have very few power cycles. They just happily sit in their Storage Pods holding your data. If one of the drives in a Storage Pod fails, we cycle down the entire Storage Pod to replace the failed drive. This only takes a few minutes and then power is reapplied and everything cycles back up. Occasionally we power cycle a Storage Pod for maintenance and on rare occasions we’ve had power failures, but generally, the drives just stay up.

As a result, the correlation of power cycles to failure is strong, but the power cycles may not be the cause of the failures because of our limited number of power cycles for a drive (less than 100) and also considering the variety of other possible failure causes during that period.

In addition to reporting the raw value, drives also report a “normalized” value in the range from 253 (the best) down to one (the worst). The drive is supposed to know what its design criteria and failure modes are, then interpret the raw value and tell you whether it’s good or bad. Unfortunately, with the Power_Cycle_Count, the drives all say the value is 100, which doesn’t lead to a very useful chart.

As shown, SMART 12 does not produce a useful normalized value, it doesn’t think that power cycling is a problem at all.

You may ask whether there is a correlation of power cycle count with failures because power cycle count correlates with age, and age correlates with failures. The answer is no. The correlation of power cycle count with age is very weak: 0.05. New drives can have higher power cycle counts, and old drives can have low power cycle counts.

Because Backblaze does not power-cycle our drives very often, this SMART stat is not very useful to us in determining the potential failure of a drive. It also does not answer the age-old question that asks whether turning off your computer every night is better or worse for the disk—that mystery remains.

SMART 1: Read_Error_Rate

Backblaze does not use this one.
The Wikipedia entry for this one says “The raw value has different structure for different vendors and is often not meaningful as a decimal number.” So the numeric values here probably don’t count anything directly, but it’s clear from the failure rate chart that they have something to do with drives failing, and that non-zero values are worse. Once the value goes above zero, bigger is not worse, though.

And a lot of the drives have a zero value, presumably meaning “no problem:”

Unlike the Power_Cycle_Count, the scaled value for this the Raw_Read_Error_Rate does what it’s supposed to do: failure rates are higher for lower normalized values, although it’s not a nice smooth progression:

For Backblaze to use this SMART stat we’d like to have a better sense of the values as reported by each vendor. While a value above zero is not good, the inconsistency of the reported values above zero is wildly inconsistent as seen in the charts above using normalized values. Since the manufacturers don’t tell us what their attribute values could be, this SMART stat is not very useful, especially across multiple drive manufacturers.

Tell Us What They Mean

Backblaze uses SMART 5, 187, 188, 197, and 198 for determining the failure or potential failure of a hard drive. We would love to use more—ideally the drive vendors would tell us exactly what the SMART attributes mean. Then we, and the rest of the storage community, could examine the data and figure out what’s going on with the drives.

In the meantime, at Backblaze, we’ll continue gathering data and working to correlate it as best we can. One thing we are looking at is to break down each SMART stat by the drive model, but there are challenges with how drive manufacturers change drive model numbers and how firmware changes occur within a given model. We’ll see if there is anything interesting and let you know.

Remember, you can find charts like the ones above for all of the SMART attributes on our website here. If you see something interesting there and figure out what it means, or know more yourself, be sure to let us know.

The post Hard Drive SMART Stats appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

At Backblaze we now have 34,881 drives and store over 100 petabytes of data. We continually track how our disk drives are doing, which ones are reliable, and which ones need to be replaced.

I did a blog post back in January, called “What Hard Drive Should I Buy?” It covered the reliability of each of the drive models that we use. This month, I’m updating those numbers and sharing some surprising new findings.

Reliability of Hard Drive Brands

Losing a disk drive at Backblaze is not a big deal. Every file we back up is replicated across multiple drives in the data center. When a drive fails, it is promptly replaced, and its data is restored. Even so, we still try to avoid failing drives, because replacing them costs money.

We carefully track which drives are doing well and which are not, to help us when selecting new drives to buy.

The good news is that the chart today looks a lot like the one from January, and that most of the drives are continuing to perform well. It’s nice when things are stable.

The surprising (and bad) news is that Seagate 3TB drives are failing a lot more, with their failure rate jumping from 9% to 15%. The Western Digital 3TB drives have also failed more, with their rate going up from 4% to 7%.

In the chart below, the grey bars are the failure rates up through the end of 2013, and the colored bars are the failure rates including all of the data up through the end of June 2014.

Share this Image On Your Site

<a href=”https://www.backblaze.com/blog/hard-drive-reliability-update-september-2014″><img src=”https://www.backblaze.com/blog/wp-content/uploads/2014/09/blog-fail-drives-manufactureX.jpg” alt=”Hard Drive Failure Rates by Model” width=”560″ border=”0″ /></a>

You can see that all the HGST (formerly Hitachi) drives, the Seagate 1.5TB and 4TB, and Western Digital 1TB drives are all continuing to perform as well as they were before. But the Seagate and Western Digital 3TB drives failure rates are up quite a bit.

What is the likely cause of this?

It may be that those drives are less well-suited to the data center environment. Or it could be that getting them by drive farming and removing them from external USB enclosures caused problems. We’ll continue to monitor and report on how these drives perform in the future.

Should we switch to enterprise drives?

Assuming we continue to see a failure rate of 15% on these drives, would it make sense to switch to “enterprise” drives instead?

There are two answers to this question:

1. Today on Amazon, a Seagate 3TB “enterprise” drive costs $235 while a Seagate 3TB “desktop” drive costs $102. Most of the drives we get have a three year warranty, making failures a non-issue from a cost perspective for that period. However, even if there were no warranty, a 15% annual failure rate on the consumer “desktop” drive and a 0% failure rate on the “enterprise” drive, the breakeven would be 10 years, which is longer than we expect to even run the drives for.
2. The assumption that “enterprise” drives would work better than “consumer” drives has not been true in our tests. I analyzed both of these types of drives in our system and found that their failure rates in our environment were very similar—with the “consumer” drives actually being slightly more reliable.

Detailed Reliability of Hard Drive Models

This table shows the detailed breakdown of how many of which drives we have, how old they are on average, and what the failure rate is. It includes all drive models of which we have at least 200. A couple of models are new to Backblaze and show a failure rate of “n/a” because there isn’t enough data yet for reliable numbers.

We use two different models of Seagate 3TB drives. The Barracuda 7200.14 is having problems, but the Barracuda XT is doing well with less than half the failure rate.

There is a similar pattern with the Seagate 1.5TB drives. The Barracuda 7200.11 is having problems, but the Barracuda LP is doing well.

Summary

While the failure rate of Seagate and Western Digital 3TB hard drives has started to rise, most of the consumer-grade drives in the Backblaze data center are continuing to perform well, and are a cost-effective way to provide unlimited online backup at a good price.

Notes

September 30, 2014: We were nicely asked by the folks at HGST to replace the name Hitachi with the name HGST given that HGST is no longer a Hitachi company. To that end, we have changed Hitachi to HGST in this post and in the graph.

The post Hard Drive Reliability Update: September 2014 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

How much does operating temperature affect the failure rates of disk drives? Not much.

The unlimited online backup service provided by Backblaze requires a lot of storage. In fact, we recently passed the 100PB mark in our data center. This means we use disk drives. A lot of disk drives.

The Backblaze Storage Pod is designed to provide good air flow over the disk drives, so they don’t get too hot. Still, different locations inside a Pod, and different locations within a data center will have different temperatures, and we wondered whether that was a problem for the drives.

What Other People Say

Google and Microsoft have both done studies on disk drive temperature in their data centers. Google found that temperature was not a good predictor of failure, while Microsoft and the University of Virginia found that there was a significant correlation.

Disk drive manufacturers tell Backblaze that in general, it’s a good idea to keep disks cooler so they will last longer.

All Drives: No Correlation

After looking at data on over 34,000 drives, I found that overall there is no correlation between temperature and failure rate.

To check correlations, I used the point-biserial correlation coefficient on drive average temperatures and whether drives failed or not. The result ranges from -1 to 1, with 0 being no correlation, and 1 meaning hot drives always fail.

Correlation of Temperature and Failure: 0.0

Disk Drive Temperature Range

It turns out that different drive models run at different temperatures, and this can throw off the stats when looking at the entire population. If in a given ambient air temperature, drive model A runs warmer than drive B, and drive A fails more, that will make it look like there is a correlation when there isn’t.

This table shows the average temperature, in degrees Celsius, of different drive models:

Each Storage Pod in our data center is initially deployed with one model of drive in all 45 slots. It tends to stay that way over time, too, as drives are replaced. Pods with different models of drives are distributed somewhat randomly around the data center, so on the average, each model runs in an environment that is about the same. The temperatures in the table above are due to differences in the disk drives more than differences in their environment.

The first five drives in the above list are all advertised as “green,” low-power drives. It makes sense that they run cooler because they generate less heat.

The chart below shows the distribution of drive temperatures for our four most popular drives. As you can see, all of the drives are well within the 0° (or 5°) to 60° that the manufacturers specify for the drives. And almost all of the drives are in the nice comfortable range from 15° to 30°.

Correlations Between Temperature and Failure for Different Drives

Now, let’s look at the correlation between temperatures and failures for each drive model. Here’s the same set of models, this time sorted by correlation. The correlations that are statistically significant are in bold:

Seagate Barracuda & Barracuda LP 1.5TB Heat Failure

This is the one drive that does show some correlation between temperature and failure rates. The correlations of 0.17 and 0.11 are weak, but they are statistically significant.

It’s interesting that the correlations are similar for the regular 7200 RPM drive and the low power 5900 RPM drive. The average temperature of the low power drives is 21.9, while the average for the regular drives is 25.7.

Comparing the failure rates of drives that are below the average temperature for the model, and those above the average temperature, there is a clear difference for these drives:

Why is the correlation weak when these numbers look so obvious? It’s because there’s a lot of overlap between the temperatures of the failed drives and the temperatures of the working drives, so you can’t predict for sure which drives will fail, but the low p-value means that there is a meaningful difference in failure rates.

The failure rate of the ST31500541AS does go up at higher temperatures:

This contrasts with most other drives we have, which don’t show that trend. The Hitachi HDS722020ALA330 is another one of our older drives, and it shows a more typical non-pattern:

Seagate Barracuda 3TB and Hitachi Deskstar 7K2000

These are the remaining two drives that have a statistically significant correlation between temperature and failures, but they show very weak correlations and they are in opposite directions. The Seagate drives fail very slightly more when they are warmer, while the Hitachi drives fail very slightly more when they are cooler. The correlations of 0.03 and -0.04 are weak enough that we shouldn’t draw conclusions from them.

Hard Drive Temperature Takeaways

Overall, there is not a correlation between operating temperature and failure rates. The one exception is the Seagate Barracuda 1.5TB drives, which fail slightly more when they run warmer.

As long as you run drives well within their allowed range of operating temperatures, keeping them cooler doesn’t matter.

The post Hard Drive Temperature—Does It Matter? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

My last two blog posts were about expected drive lifetimes and drive reliability. These posts were an outgrowth of the careful work that we’ve done at Backblaze to find the most cost effective disk drives. When we started Backblaze, our goal was to run a truly unlimited online backup service for only $5 per month, meaning our cloud storage needs to be very efficient and we need to quickly figure out which drives work.

Because Backblaze has a history of openness, many readers expected more details in my previous posts. They asked what drive models work best and which last the longest. Given our experience with over 25,000 drives, they asked which ones are good enough that we would buy them again. In this post, I’ll answer those questions.

Drive Population

At the end of 2013, we had 27,134 consumer grade drives spinning in Backblaze Storage Pods. The breakdown by brand looks like this:

As you can see, they are mostly Seagate and Hitachi drives, with a good number of Western Digital thrown in. We don’t have enough Toshiba or Samsung drives for good statistical results.

Why do we have the drives we have? Basically, we buy the least expensive drives that will work. When a new drive comes on the market that looks like it would work, and the price is good, we test a Pod full and see how they perform. The new drives go through initial set up tests, a stress test, and then a couple of weeks in production. (A couple of weeks is enough to fill the Pod with data.) If things still look good, that drive goes on the buy list. When the price is right, we buy it.

We are willing to spend a little bit more on drives that are reliable, because it costs money to replace a drive. We are not willing to spend a lot more, though.

Excluded Drives

Some drives just don’t work in the Backblaze environment. We have not included them in this study. It wouldn’t be fair to call a drive “bad” if it’s just not suited for the environment it’s put into.

We have some of these drives running in Storage Pods, but are in the process of replacing them because they aren’t reliable enough. When one drive goes bad, it takes a lot of work to get the RAID back online if the whole RAID is made up of unreliable drives. It’s just not worth the trouble.

The drives that just don’t work in our environment are Western Digital Green 3TB drives and Seagate LP (low power) 2TB drives. Both of these drives start accumulating errors as soon as they are put into production. We think this is related to vibration. The drives do somewhat better in the new low vibration Backblaze Storage Pod, but still not well enough.

These drives are designed to be energy efficient, and spin down aggressively when not in use. In the Backblaze environment, they spin down frequently, and then spin right back up. We think that this causes a lot of wear on the drive.

Failure Rates

We measure drive reliability by looking at the annual failure rate, which is the average number of failures you can expect running one drive for a year. A failure is when we have to replace a drive in a Pod.

This chart has some more details that don’t show up in the pretty chart, including the number of drives of each model that we have, and how old the drives are:

The following sections focus on different aspects of these results.

1.5TB Seagate Drives

The Backblaze team has been happy with Seagate Barracuda LP 1.5TB drives. We’ve been running them for a long time—their average age is pushing four years. Their overall failure rate isn’t great, but it’s not terrible either.

The non-LP 7200 RPM drives have been consistently unreliable. Their failure rate is high, especially as they’re getting older.

The Seagate Barracuda Green 1.5TB drive, though, has not been doing well. We got them from Seagate as warranty replacements for the older drives, and these new drives are dropping like flies. Their average age shows 0.8 years, but since these are warranty replacements, we believe that they are refurbished drives that were returned by other customers and erased, so they already had some usage when we got them.

Bigger Seagate Drives

The bigger Seagate drives have continued the tradition of the 1.5TB drives: they’re solid workhorses, but there is a constant attrition as they wear out.

The good pricing on Seagate drives along with the consistent, but not great, performance is why we have a lot of them.

Hitachi Drives

If the price were right, we would be buying nothing but Hitachi drives. They have been rock solid, and have had a remarkably low failure rate.

Western Digital Drives

Back at the beginning of Backblaze, we bought Western Digital 1.0TB drives, and that was a really good choice. Even after over four years of use, the ones we still have are going strong.

We wish we had more of the Western Digital Red 3TB drives (WD30EFRX). They’ve also been really good, but they came after we already had a bunch of the Seagate 3TB drives, and when they came out their price was higher.

What About Drives that Don’t Fail Completely?

Another issue when running a big data center is how much personal attention each drive needs. When a drive has a problem, but doesn’t fail completely, it still creates work. Sometimes automated recovery can fix this, but sometimes a RAID array needs that personal touch to get it running again.

Each Storage Pod runs a number of RAID arrays. Each array stores data reliably by spreading data across many drives. If one drive fails, the data can still be obtained from the others. Sometimes, a drive may “pop out” of a RAID array but still seem good, so after checking that its data is intact and it’s working, it gets put back in the RAID to continue operation. Other times a drive may stop responding completely and look like it’s gone, but it can be reset and continue running.

Measuring the time spent in a “trouble” state like this is a measure of how much work a drive creates. Once again, Hitachi wins. Hitachi drives get “four nines” of untroubled operation time, while the other brands just get “two nines.”

Drive Lifetime by Brand

The chart below shows the cumulative survival rate for each brand. Month by month, how many of the drives are still alive?

Hitachi does really well. There is an initial die-off of Western Digital drives, and then they are nice and stable. The Seagate drives start strong, but die off at a consistently higher rate, with a burst of deaths near the 20 month mark.

Having said that, you’ll notice that even after three years, by far most of the drives are still operating.

What Drives Is Backblaze Buying Now?

We are focusing on 4TB drives for new Pods. For these, our current favorite is the Seagate Desktop HDD.15 (ST4000DM000). We’ll have to keep an eye on them, though. Historically, Seagate drives have performed well at first, and then had higher failure rates later.

Our other favorite is the Western Digital 3TB Red (WD30EFRX).

We still have to buy smaller drives as replacements for older Pods where drives fail. The drives we absolutely won’t buy are Western Digital 3TB Green drives and Seagate 2TB LP drives.

A year and a half ago, Western Digital acquired the Hitachi disk drive business. Will Hitachi drives continue their excellent performance? Will Western Digital bring some of the Hitachi reliability into their consumer grade drives?

Correction: Hitachi’s 2.5″ hard drive business went to Western Digital, while the 3.5″ hard drive business went to Toshiba.

At Backblaze, we will continue to monitor and share the performance of a wide variety of disk drive models. What has your experience been?

The post What Hard Drive Should I Buy? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Last month I dug into drive failure rates based on the 25,000+ consumer drives we have and found that consumer drives actually performed quite well. Over 100,000 people read that blog post and one of the most common questions asked was:

“Okay, so the consumer drives don’t fail that often. But aren’t enterprise drives so much more reliable that they would be worth the extra cost?”

Well, I decided to try to find out.

In the Beginning

As many of you know, when Backblaze first started the unlimited online backup service, our founders bootstrapped the company without funding. In this environment, one of our first and most critical design decisions was to build our backup software on the premise of data redundancy. That design decision allowed us to use consumer drives instead of enterprise drives in our early Storage Pods as we used the software, not the hardware, to manage redundancy. Given that enterprise drives were often twice the cost of consumer drives, the choice of consumer drives was also a relief for our founders’ thin wallets.

There were warnings back then that using consumer drives would be dangerous, with people saying:

• “Consumer drives won’t survive in the hostile environment of the data center.”
• “Backblaze Storage Pods allow too much vibration—consumer drives won’t survive.”
• “Consumer drives will drop dead in a year. Or two years. Or…”

As we have seen, consumer drives didn’t die in droves, but what about enterprise ones?

Failure Rates

In my post last month on disk drive life expectancy, I went over what an annual failure rate means. It’s the average number of failures you can expect when you run one disk drive for a year. The computation is simple:

Annual Failure Rate = (Number of Drives that Failed / Number of Drive-Years)

Drive-years is a measure of how many drives have been running for how long. This computation is also simple:

Drive-Years = (Number of Drives x Number of Years)

For example, one drive for one year is one drive-year. Twelve drives for one month is also one drive-year.

Backblaze Storage Pods: Consumer Class Drives

We have detailed day-by-day data about the drives in the Backblaze Storage Pods since mid-April of 2013. With 25,000 drives ranging in age from brand-new to over four years old, that’s enough data to slice the data in different ways and still get accurate failure rates. Next month, I’ll be going into some of those details, but for the comparison with enterprise drives, we’ll just look at the overall failure rates.

We have data that tracks every drive by serial number, which days it was running, and if/when it was replaced because it failed. We have logged:

• 14,719 drive-years on the consumer grade drives in our Storage Pods.
• 613 drives that failed and were replaced.

Commercially Available Servers: Enterprise-Class Drives

We store customer data on Backblaze Storage Pods which are purpose-built to store data very densely and cost-efficiently. However, we use commercially available servers for our central servers that store transactional data such as sales records and administrative activities. These servers provide the flexibility and throughput needed for such tasks. These commercially available servers come from Dell and from EMC.

All of these systems were delivered to us with enterprise-class hard drives. These drives were touted as solid long-lasting drives with extended warranties.

The specific systems we have are:

• Six shelves of enterprise class drives in Dell PowerVault storage systems.
• One EMC storage system with 124 enterprise drives that we just brought up this summer. One of the drives has already failed and been replaced.

We have also been running one Backblaze Storage Pod full of enterprise drives storing users’ backed-up files as an experiment to see how they do. So far, their failure rate has been statistically consistent with drives in the commercial storage systems.

In the two years since we started using these enterprise grade storage systems, they have logged:

• 368 drive-years on the enterprise-grade drives.
• 17 drives that failed and were replaced.

Enterprise vs. Consumer Drives

At first glance, it seems the enterprise drives don’t have that many failures. While true, the failure rate of enterprise drives is actually higher than that of the consumer drives!

It turns out that the consumer drive failure rate does go up after three years, but all three of the first three years are pretty good. We have no data on enterprise drives older than two years, so we don’t know if they will also have an increase in failure rate. It could be that the vaunted reliability of enterprise drives kicks in after two years, but because we haven’t seen any of that reliability in the first two years, I’m skeptical.

You might object to these numbers because the usage of the drives is different. The enterprise drives are used heavily. The consumer drives are in continual use storing users’ updated files and they are up and running all the time, but the usage is lighter. On the other hand, the enterprise drives we have are coddled in well-ventilated low-vibration enclosures, while the consumer drives are in Backblaze Storage Pods, which do have a fair amount of vibration. In fact, the most recent design change to the Pod was to reduce vibration.

Overall, I argue that the enterprise drives we have are treated as well as the consumer drives. And the enterprise drives are failing more.

So, Are Enterprise Drives Worth the Cost?

From a pure reliability perspective, the data we have says the answer is clear: no.

Enterprise drives do have one advantage: longer warranties. That’s a benefit only if the higher price you pay for the longer warranty is less than what you expect to spend on replacing the drive.

This leads to an obvious conclusion: If you’re okay with buying the replacements yourself after the warranty is up, then buy the cheaper consumer drives.

The post Enterprise Drives: Fact or Fiction? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.