How much does operating temperature affect the failure rates of disk drives? Not much.

The unlimited online backup service provided by Backblaze requires a lot of storage. In fact, we recently passed the 100PB mark in our data center. This means we use disk drives. A lot of disk drives.

The Backblaze Storage Pod is designed to provide good air flow over the disk drives, so they don’t get too hot. Still, different locations inside a Pod, and different locations within a data center will have different temperatures, and we wondered whether that was a problem for the drives.

What Other People Say

Google and Microsoft have both done studies on disk drive temperature in their data centers. Google found that temperature was not a good predictor of failure, while Microsoft and the University of Virginia found that there was a significant correlation.

Disk drive manufacturers tell Backblaze that in general, it’s a good idea to keep disks cooler so they will last longer.

All Drives: No Correlation

After looking at data on over 34,000 drives, I found that overall there is no correlation between temperature and failure rate.

To check correlations, I used the point-biserial correlation coefficient on drive average temperatures and whether drives failed or not. The result ranges from -1 to 1, with 0 being no correlation, and 1 meaning hot drives always fail.

Correlation of Temperature and Failure: 0.0

Disk Drive Temperature Range

It turns out that different drive models run at different temperatures, and this can throw off the stats when looking at the entire population. If in a given ambient air temperature, drive model A runs warmer than drive B, and drive A fails more, that will make it look like there is a correlation when there isn’t.

This table shows the average temperature, in degrees Celsius, of different drive models:

Each Storage Pod in our data center is initially deployed with one model of drive in all 45 slots. It tends to stay that way over time, too, as drives are replaced. Pods with different models of drives are distributed somewhat randomly around the data center, so on the average, each model runs in an environment that is about the same. The temperatures in the table above are due to differences in the disk drives more than differences in their environment.

The first five drives in the above list are all advertised as “green,” low-power drives. It makes sense that they run cooler because they generate less heat.

The chart below shows the distribution of drive temperatures for our four most popular drives. As you can see, all of the drives are well within the 0° (or 5°) to 60° that the manufacturers specify for the drives. And almost all of the drives are in the nice comfortable range from 15° to 30°.

Correlations Between Temperature and Failure for Different Drives

Now, let’s look at the correlation between temperatures and failures for each drive model. Here’s the same set of models, this time sorted by correlation. The correlations that are statistically significant are in bold:

Seagate Barracuda & Barracuda LP 1.5TB Heat Failure

This is the one drive that does show some correlation between temperature and failure rates. The correlations of 0.17 and 0.11 are weak, but they are statistically significant.

It’s interesting that the correlations are similar for the regular 7200 RPM drive and the low power 5900 RPM drive. The average temperature of the low power drives is 21.9, while the average for the regular drives is 25.7.

Comparing the failure rates of drives that are below the average temperature for the model, and those above the average temperature, there is a clear difference for these drives:

Why is the correlation weak when these numbers look so obvious? It’s because there’s a lot of overlap between the temperatures of the failed drives and the temperatures of the working drives, so you can’t predict for sure which drives will fail, but the low p-value means that there is a meaningful difference in failure rates.

The failure rate of the ST31500541AS does go up at higher temperatures:

This contrasts with most other drives we have, which don’t show that trend. The Hitachi HDS722020ALA330 is another one of our older drives, and it shows a more typical non-pattern:

Seagate Barracuda 3TB and Hitachi Deskstar 7K2000

These are the remaining two drives that have a statistically significant correlation between temperature and failures, but they show very weak correlations and they are in opposite directions. The Seagate drives fail very slightly more when they are warmer, while the Hitachi drives fail very slightly more when they are cooler. The correlations of 0.03 and -0.04 are weak enough that we shouldn’t draw conclusions from them.

Hard Drive Temperature Takeaways

Overall, there is not a correlation between operating temperature and failure rates. The one exception is the Seagate Barracuda 1.5TB drives, which fail slightly more when they run warmer.

As long as you run drives well within their allowed range of operating temperatures, keeping them cooler doesn’t matter.

The post Hard Drive Temperature—Does It Matter? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

My last two blog posts were about expected drive lifetimes and drive reliability. These posts were an outgrowth of the careful work that we’ve done at Backblaze to find the most cost effective disk drives. When we started Backblaze, our goal was to run a truly unlimited online backup service for only $5 per month, meaning our cloud storage needs to be very efficient and we need to quickly figure out which drives work.

Because Backblaze has a history of openness, many readers expected more details in my previous posts. They asked what drive models work best and which last the longest. Given our experience with over 25,000 drives, they asked which ones are good enough that we would buy them again. In this post, I’ll answer those questions.

Drive Population

At the end of 2013, we had 27,134 consumer grade drives spinning in Backblaze Storage Pods. The breakdown by brand looks like this:

As you can see, they are mostly Seagate and Hitachi drives, with a good number of Western Digital thrown in. We don’t have enough Toshiba or Samsung drives for good statistical results.

Why do we have the drives we have? Basically, we buy the least expensive drives that will work. When a new drive comes on the market that looks like it would work, and the price is good, we test a Pod full and see how they perform. The new drives go through initial set up tests, a stress test, and then a couple of weeks in production. (A couple of weeks is enough to fill the Pod with data.) If things still look good, that drive goes on the buy list. When the price is right, we buy it.

We are willing to spend a little bit more on drives that are reliable, because it costs money to replace a drive. We are not willing to spend a lot more, though.

Excluded Drives

Some drives just don’t work in the Backblaze environment. We have not included them in this study. It wouldn’t be fair to call a drive “bad” if it’s just not suited for the environment it’s put into.

We have some of these drives running in Storage Pods, but are in the process of replacing them because they aren’t reliable enough. When one drive goes bad, it takes a lot of work to get the RAID back online if the whole RAID is made up of unreliable drives. It’s just not worth the trouble.

The drives that just don’t work in our environment are Western Digital Green 3TB drives and Seagate LP (low power) 2TB drives. Both of these drives start accumulating errors as soon as they are put into production. We think this is related to vibration. The drives do somewhat better in the new low vibration Backblaze Storage Pod, but still not well enough.

These drives are designed to be energy efficient, and spin down aggressively when not in use. In the Backblaze environment, they spin down frequently, and then spin right back up. We think that this causes a lot of wear on the drive.

Failure Rates

We measure drive reliability by looking at the annual failure rate, which is the average number of failures you can expect running one drive for a year. A failure is when we have to replace a drive in a Pod.

This chart has some more details that don’t show up in the pretty chart, including the number of drives of each model that we have, and how old the drives are:

The following sections focus on different aspects of these results.

1.5TB Seagate Drives

The Backblaze team has been happy with Seagate Barracuda LP 1.5TB drives. We’ve been running them for a long time—their average age is pushing four years. Their overall failure rate isn’t great, but it’s not terrible either.

The non-LP 7200 RPM drives have been consistently unreliable. Their failure rate is high, especially as they’re getting older.

The Seagate Barracuda Green 1.5TB drive, though, has not been doing well. We got them from Seagate as warranty replacements for the older drives, and these new drives are dropping like flies. Their average age shows 0.8 years, but since these are warranty replacements, we believe that they are refurbished drives that were returned by other customers and erased, so they already had some usage when we got them.

Bigger Seagate Drives

The bigger Seagate drives have continued the tradition of the 1.5TB drives: they’re solid workhorses, but there is a constant attrition as they wear out.

The good pricing on Seagate drives along with the consistent, but not great, performance is why we have a lot of them.

Hitachi Drives

If the price were right, we would be buying nothing but Hitachi drives. They have been rock solid, and have had a remarkably low failure rate.

Western Digital Drives

Back at the beginning of Backblaze, we bought Western Digital 1.0TB drives, and that was a really good choice. Even after over four years of use, the ones we still have are going strong.

We wish we had more of the Western Digital Red 3TB drives (WD30EFRX). They’ve also been really good, but they came after we already had a bunch of the Seagate 3TB drives, and when they came out their price was higher.

What About Drives that Don’t Fail Completely?

Another issue when running a big data center is how much personal attention each drive needs. When a drive has a problem, but doesn’t fail completely, it still creates work. Sometimes automated recovery can fix this, but sometimes a RAID array needs that personal touch to get it running again.

Each Storage Pod runs a number of RAID arrays. Each array stores data reliably by spreading data across many drives. If one drive fails, the data can still be obtained from the others. Sometimes, a drive may “pop out” of a RAID array but still seem good, so after checking that its data is intact and it’s working, it gets put back in the RAID to continue operation. Other times a drive may stop responding completely and look like it’s gone, but it can be reset and continue running.

Measuring the time spent in a “trouble” state like this is a measure of how much work a drive creates. Once again, Hitachi wins. Hitachi drives get “four nines” of untroubled operation time, while the other brands just get “two nines.”

Drive Lifetime by Brand

The chart below shows the cumulative survival rate for each brand. Month by month, how many of the drives are still alive?

Hitachi does really well. There is an initial die-off of Western Digital drives, and then they are nice and stable. The Seagate drives start strong, but die off at a consistently higher rate, with a burst of deaths near the 20 month mark.

Having said that, you’ll notice that even after three years, by far most of the drives are still operating.

What Drives Is Backblaze Buying Now?

We are focusing on 4TB drives for new Pods. For these, our current favorite is the Seagate Desktop HDD.15 (ST4000DM000). We’ll have to keep an eye on them, though. Historically, Seagate drives have performed well at first, and then had higher failure rates later.

Our other favorite is the Western Digital 3TB Red (WD30EFRX).

We still have to buy smaller drives as replacements for older Pods where drives fail. The drives we absolutely won’t buy are Western Digital 3TB Green drives and Seagate 2TB LP drives.

A year and a half ago, Western Digital acquired the Hitachi disk drive business. Will Hitachi drives continue their excellent performance? Will Western Digital bring some of the Hitachi reliability into their consumer grade drives?

Correction: Hitachi’s 2.5″ hard drive business went to Western Digital, while the 3.5″ hard drive business went to Toshiba.

At Backblaze, we will continue to monitor and share the performance of a wide variety of disk drive models. What has your experience been?

The post What Hard Drive Should I Buy? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Last month I dug into drive failure rates based on the 25,000+ consumer drives we have and found that consumer drives actually performed quite well. Over 100,000 people read that blog post and one of the most common questions asked was:

“Okay, so the consumer drives don’t fail that often. But aren’t enterprise drives so much more reliable that they would be worth the extra cost?”

Well, I decided to try to find out.

In the Beginning

As many of you know, when Backblaze first started the unlimited online backup service, our founders bootstrapped the company without funding. In this environment, one of our first and most critical design decisions was to build our backup software on the premise of data redundancy. That design decision allowed us to use consumer drives instead of enterprise drives in our early Storage Pods as we used the software, not the hardware, to manage redundancy. Given that enterprise drives were often twice the cost of consumer drives, the choice of consumer drives was also a relief for our founders’ thin wallets.

There were warnings back then that using consumer drives would be dangerous, with people saying:

• “Consumer drives won’t survive in the hostile environment of the data center.”
• “Backblaze Storage Pods allow too much vibration—consumer drives won’t survive.”
• “Consumer drives will drop dead in a year. Or two years. Or…”

As we have seen, consumer drives didn’t die in droves, but what about enterprise ones?

Failure Rates

In my post last month on disk drive life expectancy, I went over what an annual failure rate means. It’s the average number of failures you can expect when you run one disk drive for a year. The computation is simple:

Annual Failure Rate = (Number of Drives that Failed / Number of Drive-Years)

Drive-years is a measure of how many drives have been running for how long. This computation is also simple:

Drive-Years = (Number of Drives x Number of Years)

For example, one drive for one year is one drive-year. Twelve drives for one month is also one drive-year.

Backblaze Storage Pods: Consumer Class Drives

We have detailed day-by-day data about the drives in the Backblaze Storage Pods since mid-April of 2013. With 25,000 drives ranging in age from brand-new to over four years old, that’s enough data to slice the data in different ways and still get accurate failure rates. Next month, I’ll be going into some of those details, but for the comparison with enterprise drives, we’ll just look at the overall failure rates.

We have data that tracks every drive by serial number, which days it was running, and if/when it was replaced because it failed. We have logged:

• 14,719 drive-years on the consumer grade drives in our Storage Pods.
• 613 drives that failed and were replaced.

Commercially Available Servers: Enterprise-Class Drives

We store customer data on Backblaze Storage Pods which are purpose-built to store data very densely and cost-efficiently. However, we use commercially available servers for our central servers that store transactional data such as sales records and administrative activities. These servers provide the flexibility and throughput needed for such tasks. These commercially available servers come from Dell and from EMC.

All of these systems were delivered to us with enterprise-class hard drives. These drives were touted as solid long-lasting drives with extended warranties.

The specific systems we have are:

• Six shelves of enterprise class drives in Dell PowerVault storage systems.
• One EMC storage system with 124 enterprise drives that we just brought up this summer. One of the drives has already failed and been replaced.

We have also been running one Backblaze Storage Pod full of enterprise drives storing users’ backed-up files as an experiment to see how they do. So far, their failure rate has been statistically consistent with drives in the commercial storage systems.

In the two years since we started using these enterprise grade storage systems, they have logged:

• 368 drive-years on the enterprise-grade drives.
• 17 drives that failed and were replaced.

Enterprise vs. Consumer Drives

At first glance, it seems the enterprise drives don’t have that many failures. While true, the failure rate of enterprise drives is actually higher than that of the consumer drives!

It turns out that the consumer drive failure rate does go up after three years, but all three of the first three years are pretty good. We have no data on enterprise drives older than two years, so we don’t know if they will also have an increase in failure rate. It could be that the vaunted reliability of enterprise drives kicks in after two years, but because we haven’t seen any of that reliability in the first two years, I’m skeptical.

You might object to these numbers because the usage of the drives is different. The enterprise drives are used heavily. The consumer drives are in continual use storing users’ updated files and they are up and running all the time, but the usage is lighter. On the other hand, the enterprise drives we have are coddled in well-ventilated low-vibration enclosures, while the consumer drives are in Backblaze Storage Pods, which do have a fair amount of vibration. In fact, the most recent design change to the Pod was to reduce vibration.

Overall, I argue that the enterprise drives we have are treated as well as the consumer drives. And the enterprise drives are failing more.

So, Are Enterprise Drives Worth the Cost?

From a pure reliability perspective, the data we have says the answer is clear: no.

Enterprise drives do have one advantage: longer warranties. That’s a benefit only if the higher price you pay for the longer warranty is less than what you expect to spend on replacing the drive.

This leads to an obvious conclusion: If you’re okay with buying the replacements yourself after the warranty is up, then buy the cheaper consumer drives.

The post Enterprise Drives: Fact or Fiction? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

New Ultrabook Design Decisions Limit Data Recovery Options

When my girlfriend recently dropped (and killed) her new Ultrabook, I found out the Solid State Drive (SSD) inside of it was not easily removable like the old hard drives always were. This means a dead laptop = completely lost data if you aren’t fully backed up. Now don’t get me wrong, storing my valuable data on precariously spinning platters inside laptop hard drives has always worried me. Even before I co-founded an online backup company, I’d wince when a friend dropped a laptop roughly on a desk or spun the laptop around with a fast nudge so others could see the screen. So as soon as I could, I moved over to using SSDs that contain no moving parts. The new SSDs are more reliable than the old spinning platters overall, but the current designs cause catastrophic failure modes that will result in MORE data loss in many situations.

How I Found This Out

The Asus Zenbook UX31 Ultrabook in the pictures below was dropped on the floor and stopped working. So I took it apart to triage what was wrong, plus maybe pull out the hard drive and recover some data from it.

Recovering Data by Pulling Out a Hard Drive

As a computer professional, I help out friends and family with their computer problems. When a computer won’t boot, one of the first things I try is to recover the valuable data through the time honored step of pulling out the hard drive and putting it in an external USB enclosure, then connecting that enclosure to a working computer. This usually takes about five minutes, and is a great way to not cause any more damage and see if you can extricate any good data that is left. Since it was invented 20 years ago, the connector inside every laptop, desktop, and most servers in corporate data centers are “Serial ATA” (SATA). The fact that SATA is so common makes external USB enclosures cost about $3, and they are even reusable!

Ultrabook SSDs Make This Much Harder

So I was surprised when I opened the broken Asus Zenbook UX31 Ultrabook and couldn’t find the SATA connector I was expecting!

To save space and cost, the SSD drive is just a set of chips on a board mounted on the motherboard by a connector I didn’t immediately recognize. A little Googling and I found a technical page about the SanDisk SSD drive but it wasn’t entirely clear which of the available connectors it had, and exactly how I could connect this drive to a separate working computer. I think this is mSATA (?) but if you know for sure, can you point me at a spec for the ASUS Zenbook that makes that clear?

Recommended Solution: A Hard Drive “Black Box”

In my opinion, an ideal laptop storage system would be a small, incredibly tough, heat resistant, metal, enclosure. I’m picturing the hardened “black boxes” on commercial airplanes which are designed to survive an airplane crash. The idea is if you drive a car over your laptop and shatter it into thousands of pieces, you can sort through the wreckage and find the data enclosure, brush it off, and plug it into a standard port on another computer to recover your data. Honestly, this wouldn’t increase the price of a laptop more than a buck on a $1,000 laptop. The old hard drives with spinning platters were always encased in fairly tough steel packaging, albeit this was probably just a side effect of getting the ridiculous system to work where spinning platters with magnetic rust accurately read and write microscopic 1s and 0s at 75 miles per hour (the speed of the outside perimeter on a 7200 RPM 3.5 inch drive).

Ultrabook Makers, Please Spend the $1 to Help Save Customer Data

I might be jaded based on my current job, but “dead laptop that won’t boot” is the most common outcome for ALL LAPTOPS, followed only by “laptop was stolen.” I suppose I’ve known a few people who upgraded computers because their old computer was too slow, but now that the performance aspects of Moore’s law have collapsed (CPUs are no longer getting faster), that won’t be happening anymore. To make matters so much worse, only 10 short years ago it was hard to lose every family photo you have of your life and your whole record collection at the same time, it only occurred if your house burned down. All it takes now is your leg brushing lightly against the power cord pulling the laptop off your desk. One second every picture of your children from birth through high school graduation is safe, and a moment later they are gone forever. All because ASUS or Apple wanted to make $1 more profit? Is that all your lifetime of memories is worth?

Backups are great, but for the 94% of people who do not back up their computer, I think those people should be offered the choice of a “survivable memory module” in the computer. Heck, I always have at least two backups of any data I care about, and I would STILL choose this option.

The post Data Survivability With Ultrabook SSDs: A Dropped Zenbook and Teardown Findings appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What do the following have in common: Google providing search, Coca-Cola operating its systems to track inventory, and Backblaze backing up your data? The computers that handle all of this live in data centers. And those data centers use power—lots of it.

In the U.S. alone there are over 20,000 data centers—each of which houses thousands or tens of thousands of servers. Combined, these data centers make up 3% of all U.S. energy consumption (not just electricity). That’s more than the entire domestic air fleet.

So, when I went to an event on Wednesday called:
THE TRUTH ABOUT THE FUTURE OF THE DATA CENTER: CLOUD, COLOCATION, & DATA CENTER REAL ESTATE
it should be no surprise that the focus was on power, power, power.

And lest you think this is people getting wrapped up in the green movement or just jumping on a marketing trend—let me dissuade you. Data centers in the U.S. spend $23 billion a year on electricity according to KC Mares of MegaWatt Consulting. In fact, electricity can often cost over 50% of the purchase price of a server over its lifetime. Minor improvements can have massive implications not only on global warming but also company bottom lines.

Mares provided a fascinating overview of innovations and experiments that operators of data centers and the companies building out large server deployments are pursuing. Some examples:

• VFDs: Variable frequency drives to maintain the speed of blower fans that adjust to need rather than spinning at a constant rate.
• Natural cooling: Using outside air and fans rather than air conditioning to keep data centers cool, it turns out most servers are perfectly happy running at temperatures much higher than what data centers attempt to keep them at.
• Shorter cooling regions: Having air flow almost directly around a server in the process of cooling it rather than through the entire building; shorter distances mean less air friction and less energy spent moving it around.
• Eliminating UPS systems: Getting rid of the backup power systems and assuming servers will go down…and having backup servers or data centers instead.
• Using 480 volts: Higher voltage means lower amperage and thus less heat loss and higher efficiency. More of today’s server systems are capable of handling this voltage.
• Higher efficiency power supplies: Switching to 90% efficient power supplies on servers rather than using 70% or 80% ones. These are more expensive upfront but can still pay off fairly quickly.

A number of these items pay for themselves in a couple months and then generate savings ongoing from then on. Mares has a variety of information on his site and blog.

The post The Future of the Data Center Is Green: Takeaways From the WiredRE Data Center Event appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Don Honabach has the honor of being the first person to successfully build his own Backblaze Storage Pod. (At least the first we know about.)

With four servers running at home for media storage, Don was using a fair bit of power (and probably generating a lot of heat and noise and taking up space.) For five years he was working to come up with an “extreme media server” and after reading about the Backblaze Storage Pod, he decided this may be the way to go.

Having expertise in the space, Don customized a variety of items in the Pod including:

• The operating system (switching to Microsoft Windows Server 2008 R2)
• Power supplies
• Motherboard
• and more…

In just a couple weeks Don had completed his “extreme media server.” Combining all four servers into one, Don is saving 500 watts of power, and can run 16 independent movie streams across two monitors from a single Storage Pod.

Don created a blog that describes his experiences building his “extreme media server.”

Congratulations Don, and good luck watching all those movies at the same time!

The post User Builds “Extreme Media Server” Based on a Backblaze Storage Pod appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Last month’s blog post about building our Backblaze Storage Pods generated a ton of interest and many people are building their own Pods! Our post also generated a ton of questions so, below we answer the common ones and provide more details about where to get components.

Detailed Backblaze Storage Pod Parts and Vendor List

*Backblaze now uses the low power drives.

**Protocase is excellent at making custom cases in small quantities and in response to demand have dropped the price of the Backblaze Storage Pod enclosure to $700 each if you order 20 or more. Other case manufacturers can also build these cases from the 3-D model.

***Zippy Technology Corp. provides the power supplies. List price is about $500/unit. For quantities of 100+ (50 pairs), contact Mason Lee at the U.S. West Pacific branch of Zippy: +1-949-366-9525.

For smaller quantities, the Enermax Modu82+ 525W PSUs works well in the Backblaze Storage Pod and since those power supplies are modular it is easier to design the custom wiring harness. For $25 Enermax used to make the custom wiring harness but we have heard reports that they will no longer do this. If this is the case, since the power supply is modular, you could make your own custom wiring harness. The only catch is that you need a motherboard stub to turn the second PSU on/off. This is a $1 part but can be hard to find.

****CFI has now setup a complete, and very reasonable, pricing table for people who wish to purchase these port multiplier backplanes directly. Their pricing is:

200pcs US$40.00
150pcs US$41.00
100pcs US$42.00
50pcs US$43.00
40pcs US$43.50
30pcs US$44.00
20pcs US$44.50
10pcs US$45.00
5pcs US$45.50
1pc US$46.00

Note: If you are using latching SATA cables, make sure to ask for the port multipliers that support a latch-housing on the SATA connector.

*****The Western Digital drive listed has recently been discontinued. This is the closest match:
Western Digital Caviar SE WD800AAJB 80GB 7200 RPM 8MB Cache IDE Ultra ATA100 3.5″ Internal Hard Drive – OEM

However, most drives will work for the boot drive. We prefer PATA drives since this keeps the device name separate from the SATA data drives.

† House of Foam recently went out of business, but similar products can be ordered from other foam product suppliers.

Tips and Tricks for Building a Backblaze Storage Pod

Drives

Many people said, “Oh, I’ll build one of these Pods but switch the drives for 2TB drives or ones from XYZ vendor.” We tried drives from Western Digital, Hitachi, Samsung, and Seagate; 1TB, 1.5TB, and 2TB drives; and consumer and enterprise drives. Some of these drives performed terribly—dropping out of the RAID array on a regular basis. Other performed OK. The Seagate Barracuda 1.5TB low power drives have worked best for us. The 2TB Seagate drives have also worked well but currently cost more per GB than the 1.5TB drives; the Samsung low power drives have extremely high error rates; Hitachi and Samsung 7200 RPM drives performed well but we have done limited testing; and the Western Digital drives have not performed well in our setup. (Note: this is not necessarily to say these drives are good or bad, just how well they performed in this configuration.)

Heat

Tons of commenters worried about Backblaze Storage Pods BBQ’ing the drives. On the contrary, heat has not been an issue at all, six fans are definitely overkill and there are opportunities to reduce the number of fans. However, if you change the case design, type of drives, etc., make sure you monitor what impact your changes have made.

Power

We settled on our power supplies by figuring out the least expensive way to get high-efficiency power. If switching power supplies, ensure you have smooth power and there is plenty to spin up the drives from a cold start without being too close to the margin. In general, you need significant power on five volts (about 40-50 amps) which is what makes finding alternate power supplies a challenge.

Vibration

We worked hard to reduce vibration in the storage pods. Nylon screws, rubber fasteners for the fans, HD anti-vibration sleeves, foam insulation, etc., were all to dampen vibration. There is some data to show that our focus on vibration was important…though in discussions with the drive manufacturers, they claim that vibration is more likely to impact performance, not reliability. Having said that, if you change parts, make sure you are not increasing vibration.

Cables

While it’s tempting to lace up the cables tightly, they should actually be fairly loose so they do not pull on the connectors. And, even more important, ensure you are getting cables with good connectors. The actual cables rarely break, but connectors make a big difference: gold-plated, well-curved springy connectors that are shiny and not pitted if viewed under a microscope are ideal. (The ones we specified above are fine.) Also, ensure you get right angle SATA cable connectors, not left angle ones, so the cables route cleanly under the backplanes.

HD Anti-Vibration Sleeves

Vibration sleeves should be put on the drives at the level of the top grid such that they completely insulate it; the drives should not touch the grid. It takes a bit of practice at first, but quickly becomes easier than the standard process of putting four screws into a drive caddy.

Cold Swap Drives

Technically the system will support hot-swapping drives. However, we never do this. Hot-swapping drives on any system increases the likelihood of something going wrong. Since these Pods are top-loaded and require moving the server to replace drives, it’s one more reason to power down before swapping drives. Here is a picture of me doing this:

Powering Up the Pod

We mentioned this in the original blog post, but it is worth repeating here: we recommend powering up PSU1 first, waiting until the drives are spun-up (and the power draw decreases to a reasonable level), and then powering up PSU2.

Test the Storage Pod

There are a number of tests that should minimally be run before deploying a Storage Pod into production. First, completely sync all the drives into a RAID array. This forces every sector on every drive to be read and some sectors to be written—a process that will take about three days for a Pod full of 1.5TB drives. Second, run a memory test on the system. And finally, run a load test on the Storage Pod. There are various tools out there to help with this process. (At Backblaze, we wrote our own to best simulate a real production environment.)

Hopefully this helps those of you who are working on building your own version of the Backblaze Storage Pod. We still intend to provide blog posts covering vibration, drives, and other areas of the Pod in detail. You should feel free to experiment; try different motherboards, CPUs, cases, fans, etc. Customize the Backblaze Storage Pod for your particular purpose. Do keep in mind that while it may seem that every component should work seamlessly with every other, it is not the case. If you are switching out pieces, expect to need to experiment to get the system working right.

The post Backblaze Storage Pod: Vendors, Tips, and Tricks appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Introduction

I’ve had a lot of success in my 20-year software engineering career with developing cross platform C and C++ code. At Backblaze, we just released the Mac beta version of our online backup service, so I thought it an apt time to discuss my 10 rules for writing cross-platform code. We develop an online backup product where a small desktop component (running on either Windows or Macintosh) encrypts and then transmits users’ files across the internet to our data centers (running Linux). We use the same C and C++ libraries on Windows, Mac, and Linux interchangeably. I estimate it slows down software development by about 5% overall to support all three platforms. However, I run into other developers or software managers who mistakenly think cross platform code is difficult, or might double or triple the development schedules. This misconception is based on their bad experiences with badly run porting efforts. So this article is to quickly outline the 10 simple rules I live by to achieve efficient cross platform code development.

The Target Platforms: Microsoft Windows, Apple Macintosh, and Linux

The concepts listed here apply to all platforms, but the three most popular platforms on Earth right now are Microsoft Windows (“Windows” for short), Apple Macintosh (“Mac” for short), and Linux. We use the same C and C++ libraries on all three platforms interchangeably by following the 10 simple rules below.

I always believe in using the most popular (most highly supported) compiler and environment on each platform, so on Windows that’s Microsoft Visual Studio, on Apple Macintosh that is Xcode, and on Linux it is GCC. It wouldn’t be worth it to write cross platform code if you had to use a non-standard tool on a particular platform.  Luckily you can always use the standard tools and it works flawlessly.

Why Take the Extra Time and Effort to Implement Cross-platform?

Money! At Backblaze we run Linux in our data center because it’s free (among other reasons), and every penny we save in data center costs is a penny earned to Backblaze. At the same time, over 90% of the world’s desktops run Windows and virtually all of the remaining desktops run Apple Macintosh. Supporting the major desktop platforms and free data center platform maximizes our opportunity.

Another reason to implement cross-platform is that it raises the overall quality of the code. The compilers on each platform differ slightly, and can provide excellent warnings and hints on a section of code that “compiled without error” on another platform but would crash in some cases. The debugger and run-times also differ on each platform, so sometimes a problem that is stumping the programmer in Microsoft Visual Studio on Windows will show its base cause quickly in Xcode on the Macintosh, or vice versa. You also benefit from the tools available on all the target platforms, so if gprof helps the programmer debug a performance issue then it is a quick compiler flag away on Linux, but not readily available on Windows or Xcode.

If the above paragraph makes it sound like you have to become massively proficient in all development environments, let me just say we can usually teach a Windows-centric programmer to navigate and build on the Mac or Linux in less than an hour (or vice versa for a Macintosh-centric programmer). There isn’t any horrendous learning curve here, programmers just check out the source tree on the other platforms and select the “Build All” menu item, or type “make” at the top level. Most of the build problems that occur are immediately obvious and fixed in seconds, like an undefined symbol that is CLEARLY platform specific and just needs a quick source code tweak to work on all platforms.

So, onto the 10 rules that make cross platform development this straightforward:

Rule #1: Simultaneously develop—don’t “port” it later, and DO NOT OUTSOURCE the effort!

When an engineer designs a new feature or implements a bug fix, he or she must consider all the target platforms from the beginning, and get it working on all platforms before they consider the feature “done.” I estimate that simultaneously developing C code across our three platforms lengthens the development by less than 5% overall. But if you developed a Windows application for a full year then tried to “port” it to the Mac it might come close to doubling the development time.

To be clear, by “simultaneously” I mean the design takes all target platforms into account before coding starts, but then the C code is composed and compiled on one platform first (pick any one platform, whichever is the programmer’s favorite or has better tools for the task at hand). Then within a few hours of finishing the code on the first platform it is compiled, tested, touched up, and then finished on all the other platforms by the original software engineer.

There are several reasons it is so much more expensive to “port a year later.” First, while a programmer works on a feature he or she remembers all the design criteria and corner cases. By simultaneously getting the feature working on two platforms the design is done ONCE, and the learning curve is done ONCE. If that same programmer came back a year later to “port” the feature the programmer must reacquaint themselves with the source code. Certain issues or corner cases are forgotten about then rediscovered during QA.

But the primary reason it is expensive to “port a year later” is that programmers take short cuts, or simply through ignorance or lack of self control don’t worry about the other platforms. A concrete example is over-use of the Windows (in)famous registry. The Windows registry is essentially an API to store name-value pairs in a well known location on the file system. It’s a perfectly fine system, but it’s approximately the same as saving name-value pairs in an XML file to a well known location on the file system. If the original programmer does not care about cross-platform and makes the arbitrary decision to write values into the registry, then a year later the “port” must re-implement and test that code from scratch as XML files on the other platforms that do not have a registry. However, if the original programmer thinks about all target platforms from the beginning and chooses to write an XML file, it will work on all platforms without any changes.

Finally, the WORST thing you can possibly do is outsource the port to another company or organization. The most efficient person to work on any section of code is the original programmer, and the most efficient way to handle any (small) cross-platform issues is right inline in the code. By outsourcing the port you must deal with communication issues, code merges a year later and the resulting code destabilization, misaligned organizational goals, misaligned schedules (the original team is charging ahead while the port team is trying to stabilize their port) and other such issues. Outsourcing any coding task is almost always a mistake, outsourcing a platform port is a guaranteed disaster.

Rule #2: Factor out the GUI into non-reusable code—then develop a cross-platform library for the underlying logic.

Some engineers think “cross-platform” means “least common denominator programs” or possibly “bad port that doesn’t embrace the beauty of my favorite platform.” Not true! You should NEVER sacrifice a single bit of quality or platform specific beauty! What we’re shooting for is the maximum re-use of code WITHOUT sacrificing any of the end user experience. Towards that end, the least re-useable code in most software programs is the GUI. Specifically the buttons, menus, pop-up dialogs or slide down panes, etc. On Windows the GUI is probably in a “.rc” Windows specific resource file laid out by the Visual Studio dialog editor. On the Mac the GUI is typically stored in an Apple specific “.xib” laid out by Apple’s “Interface Builder.” It’s important to embrace the local tools for editing the GUI and just admit these are going to be re-implemented completely from scratch, possibly even with some layout changes. Luckily, these are also the EASIEST part of most applications and done by dragging and dropping buttons in a GUI builder.

But other than the GUI drawing and layout step that does not share any code, much of the underlying logic CAN be shared. Take Backblaze as a concrete example. Both the Mac and PC have a button that says “Pause Backup” which is intended to temporarily pause any backup that is occurring. On the Mac the “Pause Backup” button lives in an Apple “Preference Pane” under System Preferences. On Windows the button lives in a dialog launched from the Microsoft System Tray. But on BOTH PLATFORMS they call the same line of code: BzUiUtil::PauseBackup(); The BzUiUtil class and all of the functions in it are shared between the two implementations because those can be cross-platform; both platforms want to stop all the same processes and functions and pause the backup.

Furthermore, if at all possible you should factor the GUI code all the way out into it’s own stand-alone process. In Backblaze’s case, the GUI process (called “bzbui”) reads and writes a few XML files out to the filesystem instructing the rest of the system how to behave, for example which directories to exclude from backup. There is a completely separate process called “bztransmit” that has no GUI (and therefore can be much more cross-platform) which reads the XML configuration files left by the “bzbui” GUI process and knows not to transmit the directories excluded from backup. It turns out having a process with no GUI is a really good feature for a low-level service such as online backup, because it allows the backups to continue when the user is logged out (and therefore no GUI is authorized to run).

Finally, notice that this design is really EASY (and solid, and has additional benefits) but only if you think about cross platform from the very beginning of a project. It is much more difficult if we ignore the cross platform design issues at first and allow the code to be peppered randomly with platform specific GUI code.

Rule #3: Use standard ‘C’ types, not platform specific types.

This seems painfully obvious, but it’s one of the most common mistakes that lead to more and more code that is hard to fix later. Let’s take a concrete example: Windows offers an additional type not specified in the C language called DWORD which is defined by Microsoft as “typedef unsigned long DWORD.” It seems really obvious that using the original C type of “unsigned long” is superior in every way AND it is cross platform, but it’s a common mistake for programmers embedded in the Microsoft world to use this platform specific type instead of the standard type.

The reason a programmer might make this mistake is that the return values from a particular operating system call might be in a platform specific type, and the programmer then goes about doing further calculations in general purpose source code using this type. Going further, the programmer might even declare a few more variables of this platform specific type to be consistent. But instead, if the programmer immediately switches over to platform neutral, standard C variables as soon as possible then the code easily stays cross platform.

The most important place to apply this rule is in cross platform library code. You can’t even call into a function that takes a DWORD argument on a Macintosh, so the argument should be passed in as an unsigned long.

Rule #4: Use only built in #ifdef compiler flags, do not invent your own.

If you do need to implement something platform specific, wrap it in the 100% STANDARD #ifdefs. Do not invent your own and then additionally turn them off and on in your own Makefiles or build scripts.

One good example is that Visual Studio has an #ifdef compiler flag called “_WIN32” that is absolutely, 100% present and defined all the time in all C code compiled on Windows. There is NO VALID REASON to have your own version of this!! As long as you use the syntax as follows:

#ifdef _WIN32
// Microsoft Windows Specific Calls here
#endif

Then the build will always be “correct,” regardless of which Makefiles or Visual Studio you use, and regardless of if an engineer copies this section of code to another source file, etc. Again, this rule seems obvious, but many (most?) free libraries available on the internet insist you set many, MANY compiler flags correctly, and if you are just borrowing some of their code it can cause porting difficulties.

Rule #5: Develop a simple set of re-useable, cross-platform “base” libraries to hide per-platform code.

Let’s consider a concrete example at Backblaze, the “BzFile::FileSizeInBytes(const char *fileName)” call which returns how many bytes are contained in a file. On a Windows system this is implemented with a Windows specific GetFileAttributesEx(), and on Linux this is implemented as a lstat64(), and on the Macintosh it uses the Mac specific call getattrlist(). So INSIDE that particular function is a big #ifdef where the implementations don’t share any code. But now callers all over the Backblaze system can call this one function and know it will be very fast, efficient, and accurate, and that it is sure to work flawlessly on all the platforms.

In practice, it takes a very short amount of time to wrap common calls and then they are used HUNDREDS of times throughout the cross-platform code for great advantage. So you must buy-off on building up this small, simple set of functionality yourself, and this will slow down initial development just a bit. Trust me, you’ll see it’s worth it later.

Rule #6: Use Unicode (specifically UTF-8) for all APIs.

I don’t want this to become a tutorial on Unicode, so I’ll just sum it up for you: use Unicode. It is absolutely 100% supported on all computers and all applications on Earth now, and specifically the encoding of Unicode called UTF-8 is “the right answer.” Windows XP, Windows Vista, Macintosh OS X, Linux, Java, C#, all major web browsers, all major email programs like Microsoft Outlook, Outlook Express, or Gmail, everything, everywhere, all the time support UTF-8. There’s no debate. This web page that you are reading is written in UTF-8, and your web browser is displaying it perfectly, isn’t it?

To give Microsoft credit, they went Unicode before most other OS manufacturers did when Windows NT was released in 1993 (so Microsoft has been Unicode for more than 15 years now). However, Microsoft (being early) chose for their C APIs the non-fatal but unfortunate path of using UTF-16. They have corrected that mistake in their Java and C# APIs and use UTF-8, but since this article is all about C it deserves a quick mention here. For those not that acquainted with Unicode, just understand UTF-8 and UTF-16 are two different encodings of the same identical underlying string, and you can translate any string encoded in one form into a string encoded to the other form in one line of code very quickly and efficiently with no outside information; it’s a simple algorithmic translation that loses no data.

So, let’s take our example from “Rule #5″ above: BzFile::FileSizeInBytes(const char *fileName).” The “fileName” is actually UTF-8 so that we can support file names in all languages such as Japanese (example: C:\tmp\子犬.txt). One of the nice properties of UTF-8 is that it is backward compatible with old US ascii, while fully supporting all international languages such as Japanese. The Macintosh file system API calls and the Linux file system API calls already take UTF-8, so their implementation is trivial. But so that the rest of our system can speak UTF-8 and so we can write cross-platform code calling BzFile::FileSizeInBytes(const char *fileName),” on Windows the implementation must do a conversion step as follows:

int BzFile::FileSizeInBytes(const char *fileName)
{
#ifdef _WIN32
wchar_t utf16fileNameForMicrosoft[1024]; // wchar_t is UTF-16
ConvertUtf8toUtf16(fileName, utf16fileNameForMicrosoft); // convert
GetFileAttributesEx(utf16fileNameForMicrosoft,
GetFileExInfoStandard, &fileAttr);
return (win32fileInfo.nFileSizeLow);
#endif
}

The above code is just approximate, and suffers from buffer over-run potentials, and doesn’t handle files larger than 2GB in size, but you get the idea. The most important thing to realize here is that as long as you take UTF-8 into account BEFORE STARTING YOUR ENTIRE PROJECT then supporting multi-platform (and also international characters) will be trivial. However, if you first write the code assuming Microsoft’s unfortunate UTF-16, if you wait a year then try to port to the Macintosh and Linux with your code assuming UTF-16 or (heaven forbid) US-ASCII it will be a nightmare.

Rule #7: Don’t use third party “Application Frameworks” or “Runtime Environments.”

Third party libraries are great—for NEW FUNCTIONALITY you need. For example, I can’t recommend OpenSSL highly enough. It’s free, re-distributable, implements incredibly secure, fast encryption, and we could not have done better at Backblaze. But this is different than STARTING your project with some library that doesn’t add any functionality other than it is supposed to make your programming efforts “cross-platform.” The whole premise of the article you are reading is that C and C++ are THEMSELVES cross-platform. You do not need a library or “Application Framework” to make C cross-platform. This goes double for the GUI layer (see “Rule #2” above).  If you use a so called “cross-platform” GUI layer you will probably end up with an ugly and barely functioning GUI.

There are many of these Application Frameworks that will claim to save you time, but in the end they will just limit your application. The learning curve of these will eclipse any benefit they give you. Here are just a few examples: Qt (by TrollTech), ZooLib, GLUI/GLUT, CPLAT, GTK+, JAPI, etc.

In the worst examples, these application frameworks will bloat your application with enormous extra libraries and “runtime environments” and actually cause additional compatibility problems such as if their runtimes do not install or update correctly then YOUR application will no longer install or run correctly on one of the platforms.

The astute reader might notice this almost conflicts with “Rule #5” but the key is that you really should develop your OWN cross-platform base set of libraries, not try to use somebody else’s. While this might be against the concept of code re-use, the fact is it take just a SMALL amount of time to wrap the few calls you will actually need yourself. And for this small penalty, it gives you full control over extending your own set of cross-platform base libraries. It’s worth doing just to de-mystify the whole thing. It will show you just how easy it really is.

Rule #8: Build the raw source directly on all platforms—do not use a “Script” to transmogrify it to compile.

The important concept here is that the same exact foo.cpp and foo.h file can be checked out and always built on Windows, Macintosh, and Linux. I am mystified why this isn’t universally understood and embraced, but if you compile OpenSSL there is a complex dance where you run a PERL script on the source code, THEN you build it. Don’t get me wrong, I really appreciate that OpenSSL has amazing functionality, is free, and can be built on all these platforms with a moderate amount of effort. I just don’t understand why the OpenSSL authors don’t PRE-RUN the PERL script on the source code BEFORE they check it into the tree so that it can be compiled directly!

The whole point of this article is how to write software in a cross-platform manner. I believe that the above PERL scripts allow the coders to make platform specific mistakes and have the PERL script hide those mistakes.  Just write the source correctly from the beginning and you do not need the PERL script.

Rule #9: Require all programmers to compile on all platforms.

It only takes 10 minutes to teach an entry-level programmer how to checkout and build on a new platform they have never used before. You can write the four or five steps of instructions on the back of a business card and tape it to their monitor. If this entry-level programmer writes their new cross-platform code according to the rules in this article, it will build flawlessly (or be fixable in a few minutes with very basic changes). THIS IS NOT SOME HUGE BURDEN. And ALL programmers must be responsible for keeping their code building on ALL platforms, or the whole cross-platform effort won’t go smoothly.

With a small programming team (maybe less than 20 programmers), it is just fine to use the source code control system (like Subversion or CVS) as the synchronization point between the multi-platform builds. By this I mean that once a new feature is implemented and tested on one platform (let’s say Windows), the programmer checks it into Subversion, then IMMEDIATELY checks it out on both Macintosh and Linux and compiles it. This means that the build can be broken for one or two minutes while any small oversights are corrected. You might jump to the conclusion that the other programmers will commonly notice this one or two minute window once a day, but you would be dead wrong. In the last 15 years of using this system in teams ranging from 10 programmers to 50 programmers, I can only remember two incidents where I happened to “catch” the tree in a broken state due to this “cross-platform issue window” and it only cost me five minutes of productivity to figure out I was just unlucky in timing. During that same 15 years there were probably HUNDREDS of broken builds I discovered that had nothing to do with cross-platform issues at all, or had specifically to do with a stubborn or incompetent programmer that refused to obey this “Rule #9” and compile the code themselves on all platforms. This last point leads me to our final rule below.

Rule #10: Fire the lazy, incompetent, or bad-attitude programmers who can’t follow these rules.

Sometimes in a cross-platform world the build can be broken on one platform, and that’s okay as long as it doesn’t happen often. Your very best programmer can get distracted at the critical moment they were checking in code and forget to test the compile on the other platforms. But when one of your programmers is CONSISTENTLY breaking the build, day after day, and always on the same platform, it’s time to fire that programmer. Once an organization has stated their goals to develop and deliver cross-platform, it is simply unprofessional to monkey up the system by ignoring these rules.

Either the offending programmer is patently incompetent or criminally lazy, and either way it is more than just “grounds for termination,” I claim it is a moral obligation to fire that programmer. That programmer is giving decent, hard-working programmers a bad name, and it reflects badly on our honorable profession to allow them to continue to produce bad code.

The post Brian’s 10 Rules for How to Write Cross-Platform Code appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Goal: Security Done Right

Protecting the privacy of our users’ data is a top priority for us here at Backblaze and that means encryption. But providing a service that is extremely easy to use is also a key part of our vision and far too often encryption makes a product hard to use. This trade-off was unacceptable to us so we set out to build a system that delivers military grade encryption without compromise! Here is the full list of our requirements:

• 1. Protect data with military grade encryption
• 2. Implement encryption transparently so users don’t have to deal with it
• 3. Allow users to change their password without re-encrypting their data
• 4. In business environments, allow IT access to data without the user’s password

The Solution: Military Grade Encryption Made Easy

To accomplish the ambitious goals above we used a mix of public/private and symmetric key algorithms. The math behind this cryptography is hard but the idea is simple… Public/private keys allow you to encrypt data with one key and decrypt it with another one. Typically data is encrypted with the public key and decrypted with a private key that is kept secret but the reverse also works. This is very useful because it allows us to encrypt data in the background without requiring the user to type in their password.

Unfortunately, public/private key algorithms are slow and can’t be used to encrypt a large amount of data. Symmetric key algorithms use the same key to encrypt and decrypt data and are very fast on large amounts of data. But since the same key is used to decrypt the data, the data is only secure if the symmetric key is secure.

Combining these algorithms, here’s how our system works.


We generate a new 2048-bit RSA public/private key pair when our client is installed, store the public key on the local disk and transmit the private key to our data center via https. Then, for each backup session, we generate a new random 128-bit Advanced Encryption Standard (AES) symmetric key which we use to encrypt the user’s data. We secure the 128-bit AES key by encrypting it with the user’s public key and transmit the encrypted file along with the encrypted key to our data center over https. We destroy the unencrypted 128-bit AES key at the end of each backup session and never write it to disk. To decrypt a file, the user’s private key is used to decrypt the 128-bit AES which is then used to decrypt the file.

The user’s private key which is stored safely in our data center is protected by a password that is highly guarded. But for some users this is not good enough and we allow the user to secure this file with their own password. When this is done it is impossible to access the data without the user’s password. Unfortunately, this also means we can’t help the user if they ever forget this password so we don’t recommend it for most users.

The real beauty of this scheme becomes clear when you look back at our goals above. AES is the encryption standard adopted by the US government to protect classified information. #1 solved. Using the user’s public key we can safely run transparently in the background without compromising security. #2, check. Since a password is used to secure the private key rather than to encrypt the data directly, the password can be changed by re-encrypting only the private key with the new password. #3 accomplished. And last but not least, you can make several copies of the user’s private key and encrypt each copy with a different password to provide IT access to data without the need to share passwords. #4 done!

The post How to Make Strong Encryption Easy to Use appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.