This week, we’re celebrating the one year anniversary of the launch of Backblaze B2 Cloud Storage. Today’s post is focused on giving you a peek behind the curtain about the costs of providing cloud storage. Why? Over the last 10 years, the most common question we get is still “How do you do it?” In this multi-billion dollar global industry exhibiting exponential growth, none of the other major players seem to be willing to discuss the underlying costs. By exposing a chunk of the Backblaze financials, we hope to provide a better understanding of what it costs to run “the cloud,” and continue our tradition of sharing information for the betterment of the larger community.

Context

Backblaze built one of the industry’s largest cloud storage systems and we’re proud of that accomplishment. We bootstrapped the business and funded our growth through a combination of our own business operations and just $5.3 million in equity financing ($2.8 million of which was invested into the business—the other $2.5 million was a tender offer to shareholders). To do this, we had to build our storage system efficiently and run as a real, self-sustaining business. After over a decade in the data storage business, we have developed a deep understanding of cloud storage economics.

Definitions

I promise we’ll get into the costs of cloud storage soon, but some quick definitions first:

Revenue:

• Money we collect from customers.

Cost of Goods Sold (COGS):

• The costs associated with providing the service.

Operating Expenses (OpEx):

• The costs associated with developing and selling the service.

Income/Loss:

What is left after subtracting the COGS and OpEx from revenue.

I’m going to focus today’s discussion on the COGS: What goes into it, how it breaks down, and what percent of revenue it makes up. Backblaze is a roughly break-even business with COGS accounting for 47% of our revenue and the remaining 53% spent on our OpEx like developing new features, marketing, sales, office rent, and other administrative costs that are required for us to be a functional company.

This post’s focus on COGS should let us answer the commonly asked question of “How do you provide cloud storage for such a low cost?”

Breaking Down Cloud COGS

Providing a cloud storage service requires the following components (COGS and OpEx—below we break out COGS):

Hardware: 23% of Revenue

Backblaze stores data on hard drives. Those hard drives are “wrapped” with servers so they can connect to the public and store data. We’ve discussed our approach to how this works with our Vaults and Storage Pods. Our infrastructure is purpose-built for data storage. That is, we thought about how data storage ought to work, and then built it from the ground up. Other companies may use different storage media like Flash, SSD, or even tape. But it all serves the same function of being the thing that data actually is stored on. For today, we’ll think of all this as “hardware.”

We buy storage hardware that, on average, will last five years (60 months) before needing to be replaced. To account for hardware costs in a way that can be compared to our monthly expenses, we amortize them and recognize 1/60th of the purchase price each month.

Storage Pods and hard drives are not the only hardware in our environment. We also have to buy the cabinets and rails that hold the servers, core servers that manage accounts/billing/etc., switches, routers, power strips, cables, and more. (Our post on bringing up a data center goes into some of this detail.) However, Storage Pods and the drives inside them make up about 90% of all the hardware cost.

Data Center (Space & Power): 8% of Revenue

“The cloud” is a great marketing term and one that has caught on for our industry. That said, all “clouds” store data on something physical like hard drives. Those hard drives (and servers) are actual, tangible things that take up actual space on earth, not in the clouds.

At Backblaze, we lease space in colocation facilities which offer a secure, temperature controlled, reliable home for our equipment. Other companies build their own data centers. It’s the classic rent vs. buy decision; but it always ends with hardware in racks in a data center.

Hardware also needs power to function. Not everyone realizes it, but electricity is a significant cost of running cloud storage. In fact, some data center space is billed simply as a function of an electricity bill.

Every hard drive storing data adds incremental space and power need. This is a cost that scales with storage growth.

I also want to make a comment on taxes. We pay sales and property tax on hardware, and it is amortized as part of the hardware section above. However, it’s valuable to think about taxes when considering the data center since the location of the hardware actually drives the amount of taxes on the hardware that gets placed inside of it.

People: 7% of Revenue

Running a data center requires humans to make sure things go smoothly. The more data we store, the more human hands we need in the data center. All drives will fail eventually. When they fail, “stuff” needs to happen to get a replacement drive physically mounted inside the data center and filled with the customer data (all customer data is redundantly stored across multiple drives). The individuals that are associated specifically with managing the data center operations are included in COGS since, as you deploy more hard drives and servers, you need more of these people.

Customer support is the other group of people that are part of COGS. As customers use our services, questions invariably arise. To service our customers and get questions answered expediently, we staff customer support from our headquarters in San Mateo, CA. They do an amazing job! Staffing models, internally, are a function of the number of customers and the rate of acquiring new customers.

Bandwidth: 3% of Revenue

We have over 350PB of customer data being stored across our data centers. The bulk of that has been uploaded by customers over the internet (the other option, our Backblaze Fireball service, is six months old and is seeing great adoption). Uploading data over the internet requires bandwidth—basically, an internet connection similar to the one running to your home or office. But, for a data center, instead of contracting with Time Warner or Comcast, we go “upstream.” Effectively, we’re buying wholesale.

Understanding how that dynamic plays out with your customer base is a significant driver of how a cloud provider sets its pricing. Being in business for a decade has explicit advantages here. Because we understand our customer behavior, and have reached a certain scale, we are able to buy bandwidth in sufficient bulk to offer the industry’s best download pricing at $0.02/GB (compared to $0.05 from Amazon, Google, and Microsoft). (See notice of download price change from $0.02 to $0.01 per gigabyte at top of post—Editor.)

Why does optimizing download bandwidth charges matter for customers of a data storage business? Because it has a direct relationship to you being able to retrieve and use your data, which is important.

Other Fees: 6% of Revenue

We have grouped the remaining costs inside of “Other Fees.” This includes fees we pay to our payment processor as well as the costs of running our Restore, Return, Refund program.

A payment processor is required for businesses like ours that need to accept credit cards securely over the internet. The bulk of the money we pay to the payment processor is actually passed through to pay the credit card companies like AmEx, Visa, and Mastercard.

The Restore, Return, Refund program is a unique program for our Backblaze Computer Backup and Backblaze Business Backup products. Customers can download any of their files directly from our website. We also offer customers the ability to order a hard drive with some or all of their data on it, we then FedEx to the customer wherever in the world the customer might be. Any customer can opt to return the drive to us for a full refund. Customers love the program, but it does cost Backblaze money. We choose to subsidize the cost associated with this service in an effort to provide the best customer experience we can.

The Big Picture

At the beginning of the post, I mentioned that Backblaze is, effectively, a break-even business. The reality is that our products drive a profitable business but those profits are invested back into the business to fund product development and growth. That means growing our team as the size and complexity of the business expands; it also means being fortunate enough to have the cash on hand to fund “reserves” of extra hardware, bandwidth, data center space, etc. In our first few years as a bootstrapped business, having sufficient buffer was a challenge. Having weathered that storm, we are particularly proud of being in a financial place where we can afford to make things a bit more predictable.

All this adds up to answer the question of how Backblaze has managed to carve out its slice of the cloud market—a market that is a key focus for some of the largest companies of our time. We have innovated a novel, purpose-built storage infrastructure with our Vaults and Pods. That infrastructure allows us to keep costs very, very low. Low costs enable us to offer the most affordable, reliable cloud storage.

Does reliable, affordable storage matter? For a company like Vintage Aerial, it enables them to digitize 50 years’ worth of aerial photography of rural America and share that national treasure with the world. Having the best download pricing in the storage industry means Austin City Limits, a PBS show out of Austin, can digitize and preserve over 550 concerts.

We think offering purpose-built, affordable storage is important. It empowers our customers to monetize existing assets, make sure data is backed up (and not lost), and focus on their core business because we can handle their data storage needs.

The post The Cost of Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Dear Customer,

Over the last 72 hours, our security team has noticed an increase in automated attempts to log into our users’ accounts using credentials stolen from other websites. To protect your account, we recommend that you:

• Change your password.
• Add two-factor authentication for additional security.

NOTE: The Backblaze login database has not been compromised—the credentials were stolen from other sources.

Regrettably, we live in an era where companies have been breached and their customers’ credentials have been leaked—Dropbox, Adobe, and LinkedIn are just a few, high profile examples. What happens in these attacks is that the attacker acquires “the Dropbox list” and simply tries those usernames and passwords on another site. If your credentials were leaked in one of those hacks and you used the same username/password combination to sign up for other services (such as ours), you are vulnerable.

While we have a number of methods in place to thwart nefarious attacks, there is a limit to what we can do to prevent someone from signing in to an account with a valid username and password. We are sending this message to you today because we know that some of our users’ credentials are in these stolen lists.

Changing your password now ensures you’re not using a password that was previously leaked. Adding two-factor authentication provides an extra layer of security and protection if credentials end up on one of these lists in the future.

Thank you,

Tim
Chief Cloud Officer
Backblaze

The post Protecting Your Account appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

A couple of months ago, we gave you a sneak peek as we designed and prototyped our next generation Storage Pod. We’re done, and Storage Pod 5.0 is here and it is faster, beefier, easier to assemble, easier to maintain, and less expensive than Storage Pod 4.5. It is also the hardware that enables us to offer our newly announced Backblaze B2 Cloud Storage for just $0.005/GB/month. As always, the Storage Pod design is open-source and we’ve included the instructions and diagrams you’ll need to build your very own Storage Pod 5.0.

Building for the Future

Storage Pods have always been designed and built to provide Backblaze with low-cost data storage. When we decided that we were going to offer B2 Cloud Storage as the lowest cost cloud storage in the world, we doubled down on making our storage infrastructure more durable and more performant without raising the cost of storing data. Step one was designing and deploying the Backblaze Vault architecture to improve durability. Step two was designing and building Storage Pods that were more performant and lower cost to use in the Vaults. That’s why we built Storage Pod 5.0.

What’s New in Storage Pod 5.0

When you look at Storage Pod 5.0, nearly everything, inside and out, looks different than previous versions. Here’s Tim, our Chief Cloud Officer (CCO), to walk you through the changes…

We’ve grouped the changes as follows: performance, drive support, backplane tray, power, chassis, and fans. Let’s take a look at each.

Performance: We upgraded the motherboard, CPU, and SATA cards, and increased the memory to 32GB. Let’s compare:

With the increased cost of making Storage Pod 5.0 more performant to support Backblaze Vaults, the task was to reduce the costs of the other components. The remaining changes were implemented to reduce the combination of manufacturing, assembly, and maintenance costs.

The drawings/specifications for each of the following Storage Pod components are available for download as a 42MB ZIP file. Depending on the component there are 2D, 3D and/or STEP files available.

Drive Supports: The drive grids and 15-drive-wide drive lids have been replaced by a combination of drive support beams, plastic drive guides, and shorter drive lids which cover five drives instead of 15. These components work together to hold each hard drive firmly in place during operation and dampen vibrations while enabling easier installation and removal. Slits have also been cut in the drive lids so that the drive serial numbers can be read without removing the lid.

Drive Supports in ChassisDrive Lids (Top View)

There are two plastic drive guides for each hard drive in the Storage Pod, 90 total. A guide attaches to the side of each drive into the mounting screw holes. Once attached, the drive slides into drive supports to guide the drive directly into the socket on the backplane.

Drive GuidesDrive Supports (Top View)

The color of a given drive guide is not important. As shown above, the orange drive guide was 3D printed and the black drive guide was made by a third party plastics manufacturer.

Backplane Tray: New to Storage Pod 5.0 is the Backplane Tray. The tray holds the power and SATA cable wiring and the backplanes are attached to the tray, creating a modular component which can be preassembled then installed and removed as a unit. This reduces assembly time as well as downtime caused by the replacement of a bad backplane or wire.

Backplane Tray (Front View)Backplane Tray With Backplane

As part of the preassembly process, the SATA and power cables are run in a specific pattern. Here are the cable routing diagrams for both the Power and SATA cables, as well as which SATA cables connect to which SATA cards on the motherboard.

• Cable routing: Power cables.
• Cable routing: SATA cables.
• SATA cable connections to SATA cards on motherboard.

Power: In Storage Pod 5.0 we qualified and now use the EVGA SuperNOVA 750 G1 Power Supply. This reduces the cost of the power supply component by nearly 50%. There are two power supplies, each with their own wiring harness. We modularized each wiring harness into two assemblies. The first assembly, the pigtail, goes from the power supply to a 20- or 24-pin Molex connector and the second assembly goes from the Molex connector to the individual backplanes. To use an alternate power supply, you replace the pigtail assembly so it has the correct connections for the given power supply. The specifications for these cables are listed later in this post.

Power Supplies (Back View)Power Supplies (Top View)

Chassis: The chassis is still a red 4U box, but it looks different. First, we changed the design to support recommendations for minimizing electromagnetic emissions. Second, we changed the faceplate to improve air flow-through with our cool laser-cut logo. Third, we made the two removable covers tool-less, creating covers that slide and latch into place. This removed the need to remove/insert screws to access the drive bay or motherboard bay, saving time during assembly and during maintenance (replacing a failed drive, for example).

Tool-less Cover – Latch	Tool-less Cover – Tab

Fans: About a year ago, we took a group of Storage Pods and removed the three fans at the end, leaving just three middle fans to cool the unit. We placed these Pods into production and monitored the temperature of the hard drives utilizing the SMART stats we take each day. Nothing changed, as the drives stayed cool and didn’t fail at higher rates. In Storage Pod 5.0 we only use three fans which are now thinner (25mm vs. 38mm) and lower in power than before. We also created a “fan module” to hold all three fans in one bracket. After removing the covers, it takes just two screws to remove the new fan module. This speeds up assembly and reduces downtime during replacement of a failed fan. It also lowers the overall cost of the chassis.

Rails: Or should we say, tool-less rails. We redesigned the rails to be tool-less as you can see below. They are easier to install and remove, and are less expensive to manufacture then the previous type of rails. Here are the specifications if you’d like to make yourself a pair.

Rails – Old Design	Rails – Tool-less Design

What We Didn’t Change

To the delight/dismay of the reddit community, we did not change the on-off switch. Call us sentimental fools, but we continue to shell out the big bucks for our special blue-light switch. Oh, we looked at a few others; we even tried a couple out, but in the end we stuck with the one we had. OK, it actually has a lot to do with wiring and what type of power is needed versus what’s available. But let’s just say we like the blue light and leave it at that.

We also did not change the five-port backplanes. The SATA 3 (6 Gbit/s) backplane speed provides the throughput needed at the right price, so there was no need to change or upgrade this item. If there is a place for improvement, it is the PCIe bus. Our current PCIe x2 SATA cards utilize one PCIe lane for two SATA ports. We are looking at different strategies to remove this last potential bottleneck.

The Cost

There are actually three different prices for a Storage Pod. Here are those three costs of an 180TB Storage Pod 5.0 system, including 45 Seagate (4TB) ST4000DM000 drives, for the scenarios noted:

1. $7,974.44<./strong> The cost for Backblaze, given that we purchase 300+ Storage Pods per year.
2. $8,493.11. The cost for you to build one Storage Pod by buying the parts and assembling it yourself.
3. $10,439.10. The cost to you to purchase one already assembled Storage Pod from a third-party supplier, and then purchase and install the hard drives yourself.

These prices do not include packaging, shipping, taxes, VAT, etc.

Below are the Backblaze costs for building the different Storage Pod versions.

As you can see in the table above, Storage Pod 5.0 is less expensive per GB to build and fill with hard drives than all its predecessors. The entire system, fully populated with 180TB worth of hard drives, costs Backblaze just over $0.044 per GB, and that includes assembly costs (labor). In addition, with the price of larger capacity hard drives starting to move down the pricing curve, the cost per GB for these hard drives should continue to decrease and drive down the cost of each Storage Pod we build.

Why Did the Cost Go Down?

Before we celebrate the drop in the Storage Pod cost, let’s take a look at why it decreased. Was it market forces at play driving down the per GB cost of the hard drives? Or did the cost of the Storage Pod chassis and components drop, meaning our design efforts using Agile and Scrum actually made a difference? Here’s the breakdown of the costs (labor not included) comparing Storage Pod 4.5 with Storage Pod 5.0. Remember, these are Backblaze costs based on our purchasing 300+ Storage Pods per year.

Given that Storage Pod 5.0 is faster and beefier than 4.5, and given the expected decrease in assembly and maintenance costs, the time spent designing Storage Pod 5.0 using Agile seems to have been worth the effort. Based on our calculations, we estimate the time to recover our investment in the process to be 8-10 months. That includes the startup costs and learning curve associated with Agile and Scrum.

Buying a Storage Pod

Backblaze does not sell Storage Pods or parts, but below you’ll find a source for Storage Pod hardware.

45drives.com:

They are a subsidiary of Protocase and make Backblaze inspired Storage Pods based on the 3.0 and 4.0 Storage Pod designs. 45drives.com currently builds Storage Pods based on their direct-wire design. They offer multiple versions of their Storinator line starting at $3,350 for a base 45-drive system, with a Storage Pod 5.0 comparable system costing $4,600–$5,000 (not including hard drives).

Building Your Own Storage Pod

If you are so inclined, you can build your own Storage Pod.

1. 1. Start by reviewing the drawings/specifications for the various components. We’ve created a ZIP file (42MB) containing the following drawings and specifications to help you out:
      • Chassis_edrawings_file_P5.EASM
      • Pod5_Chassis-2D.PDF
      • Pod5_Chassis_STEP_file.STEP
      • PSU1_power_to_backplane_P5.pdf
      • PSU1_pigtail_P5.pdf
      • PSU2_power_to_backplane.pdf
      • PSU2_pigtail_P5.pdf
      • Power_cable_routing_P5.pdf
      • PSU1_all_connections_P5.pdf
      • PSU2_all_connections_P5.pdf
      • SATA_cable_routingP5.pdf
      • sata_card_motherboard_locations_P5.pdf
      • Drive_guide_2D_P5.PDF
      • Drive_guide_Solidworks_drawing_file_p5.SLDDRW
      • Drive_guide_Solidworks_partsfile.SLDPRT
      • Drive_guide_Solidworks_exchange.STL
      • Rail_left_2D_P5.PDF
      • Rail_right_2D_P5.PDF
      • Rail_left_STEP_file_p5.STEP
      • Rail_right_STEP_file_p5.STEP

If you are still interested, here’s where you can download the ZIP file containing these files.

1. Next, you’ll want to download the Storage Pod 5.0 Build Book. This guide will walk you through the assembly process for Storage Pod 5.0.
2. Finally, you can gather up all the parts you’ll need to build your Storage Pod. You can use the Parts List in Appendix A as a shopping list to buy the components needed, although you may find it easier to buy kits (assembled or as parts) from the vendors noted above.

As a reminder, Backblaze does not sell Storage Pods, and the design is open-source, so we don’t provide support or warranty for people who choose to build their own Storage Pod. That said, if you do build your own, we’d like to hear from you. In the meantime, you can find fellow Pod builders at www.openstoragepod.org and www.45drives.com/community.

What’s Next?

As part of using the Agile design methodology, we are continually working on improving the Storage Pod design. So even though Storage Pod 5.0 is here, we’re already working on improvements. Here are a few of the items we are working on:

• Using only one power supply.
• Upgrading the motherboard to the Supermicro X11 line.
• Upgrading the CPU.
• Extending the chassis to support 60 drives.
• Adding a hot spare data drive.
• And (dare we say), replacing the power switch…

As new ideas make it into production, we’ll let you know.

Appendix A: Storage Pod 5.0 Parts List

Below is the list of parts you’ll need to build your own Storage Pod 5.0. The prices listed are “street” prices, meaning you should be able to find these online or from manufacturer in quantities sufficient to build one Storage Pod. Good luck and happy building.

The post The Hardware Inside Backblaze B2 Cloud Storage: Storage Pod 5.0 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Do you like big data? Big red boxes? Are you passionate about technology?

Backblaze has over 30 petabytes of spinning disks and is adding about three petabytes every month! We like people who have the creativity to come up with new ways of building bigger and cheaper cloud storage—yet are meticulous about data center operations. Sound like you? Our Operations team is staffing up and we are hiring folks at all levels so… Backblaze Team: Assemble!

All three positions require:

• Good attitude and willingness to do whatever it takes to get the job done.
• Strong desire to work for a small fast paced company.
• Desire to learn and adapt to rapidly changing technologies.
• Understanding of the importance of following best practices.
• Relentless attention to detail.
• Excellent interpersonal skills and good oral/written communication.
• Excellent troubleshooting and problem solving skills.

Director of Web Services

Responsibilities:

• Lead efforts to maintain and scale Backblaze’s web services, including the static/dynamic web servers and monitoring systems
• Be hands on in deploying systems, configuring new services and setting up our network infrastructure
• Implement security measures required by corporate security policies
• Maintain monitoring systems to measure system availability and detect issues
• Help create, maintain, and improve operational bash shell scripts and manual procedures
• Help hire and train IT staff as needed, including those in remote (global) locations
• Participate in the pager rotation and respond to alerts
• Handle daily operational tasks including racking new servers, replacing drives, reviewing logs, etc.

Requirements:

• Professional experience in large scale, high-availability, and high-load environments
• Expert knowledge of Linux system administration; Debian experience preferred
• Experience supporting Apache, Tomcat, and Java services
• Bash scripting skills desired
• Ability to lift/move 50-75 lbs and work down near the floor as needed

Systems Administrator

Responsibilities:

• Deploy and configure new systems, networks, and services including firewalls, DNS servers, and monitoring systems
• Research new services and recommend specific packages and configurations
• Install new software patches, releases, and system upgrades
• Troubleshoot and resolve operational problems with minimal supervision
• Help maintain monitoring systems to measure system availability and detect issues
• Participate in the pager rotation and respond to alerts as needed
• Handle daily operational tasks including racking new servers, replacing drives, reviewing logs, etc.

Requirements:

• Good understanding of Linux system administration; Debian experience preferred
• Experience supporting Apache, Tomcat, and Java services a plus
• Bash scripting skills desired
• Ability to lift/move 50-75 lbs and work down near the floor on a weekly basis

Datacenter Technican

Responsibilities:

• Rack, cable, and label servers and network hardware
• Perform basic system setups and configurations
• Manage equipment delivery schedules and work with vendors to ensure timely receipt of critical equipment
• Manage spare parts inventory
• Help maintain infrastructure including replacing hard drive and other system components as needed
• Help troubleshoot and resolve operational issues with minimal supervision
• Help manage facilities including scouting new locations
• Follow and improve data center best practices documentation
• Have on-call responsibilities that will include occasional trips to data center to resolve issues that can’t be handled remotely

Requirements:

• Working knowledge of Linux and/or ability to learn quickly
• Excellent time management skills
• Ability to lift/move 50-75 lbs and work down near the floor on a daily basis

Interested? Check out this video on our data center Operations team.

Want to join our team? Follow these three steps:

Send an email to jobscontact@backblaze.com[/embed] with one of the positions listed above in the subject line, include your resume, include your answers to two of the following three questions:

• What about working at Backblaze excites you the most?
• Provide three adjectives that best describe your personal workspace.
• How would you manage boot images and system configurations on 1,000+ servers (i.e. Backblaze Storage Pods)?

Note: All positions are based in the San Francisco Bay Area, California.

The post Cloud Computing Superheroes Wanted appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

It’s been over a year since Backblaze revealed the designs of our first generation (67TB) Storage Pod. During that time, we’ve remained focused on our mission to provide an unlimited online backup service for $5 per month. To maintain profitability, we continue to avoid overpriced commercial solutions, and we now build the Backblaze Storage Pod 2.0: a 135TB, 4U server for $7,384. It’s double the storage and twice the performance—at a lower cost than the original.

In this post, we’ll share how to make a 2.0 Storage Pod, and you’re welcome to use the design. We’ll also share some of our secrets from the last three years of deploying more than 16PB worth of Backblaze Storage Pods. As before, our hope is that others can benefit from this information and help us refine the Pods. (Some of the enhancements are contributions from helpful kindred Pod builders, so if you do improve your Backblaze Pod farm, please balance the karma and send us your suggestions!)

Quick Review: What Makes a Backblaze Storage Pod?

A Backblaze Storage Pod is a self-contained unit that puts storage online. It’s made up of a custom metal case with commodity hardware inside. You can find a parts list in Appendix A. You can also link to a power wiring diagram, see an exploded diagram of parts, and check out a half-assembled Pod. The two most noteworthy factors are that the cost of the hard drives dominates the price of the overall Pod and that the system is made entirely of commodity parts. For more background, read the original blog post. Now let’s talk about the changes.

Density Matters: Double the Storage in the Same Enclosure

We upgraded the hard drives inside the 4U sheet metal Pod enclosure to store twice as much data in the same space. After the cost of filling a rack with Pods, one data center rack containing 10 Pods costs Backblaze about $2,100 per month to operate, roughly divided equally into thirds for physical space rental, bandwidth, and electricity. Doubling the density saves us half of the money spent on both physical space and electricity. The picture below is from our data center, showing 15PB racked in a single row of cabinets. The newest cabinets squeeze 1PB into three-quarters of a single cabinet for $56,696.

Our online backup cloud storage is our largest cost, and we are obsessed with providing a service that remains secure, reliable, and above all, inexpensive. We’ve seen competitors unable to react to these demands who were forced to exit the market, like Iron Mountain, or raise prices, like Mozy and Carbonite. Controlling the hardware design has allowed us to keep prices low.

We are constantly looking at new hard drives and evaluating them for reliability and power consumption. The Hitachi 3TB drive (Hitachi Deskstar 5K3000 HDS5C3030ALA630) is our current favorite for both its low power demand and astounding reliability. The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise hard drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully.

Twice as Fast

We’ve made several improvements to the design that have doubled the performance of the Storage Pod. Most of the improvements were straightforward and helped by Moore’s Law. We bumped the CPU up from the Intel dual core CPU to the Intel i3 540 and upgraded the motherboard from one gigabit ethernet port to a Supermicro motherboard with two gigabit ethernet ports. RAM dropped in price, so we doubled it to 8GB in the new Pod. More RAM enables our custom Backblaze software layer to create larger disk caches that can really speed up certain types of disk I/O.

In the first generation Storage Pod, we ran out of the faster PCIe slots and had to use one slower PCI slot, creating a bottleneck. Justin Stottlemyer from Shutterfly found a better PCIe SATA card, which enabled us to reduce the SATA cards from four to three. Our upgraded motherboard has three PCIe slots, completely eliminating the slower PCI bottleneck from the system. The updated SATA wiring diagram is seen below. Hint: The Pod will work if you connect every port multiplier backplane to a random SATA connection, but if you wire it up as shown below, the 45 drives will appear named in sequential order.

We upgraded the Linux 64-bit OS from Debian 4 to Debian 5, but we no longer use JFS as the file system. We selected JFS years ago for its ability to accommodate large volumes and low CPU usage, and it worked well. However, ext4 has since matured in both reliability and performance, and we realized that with a little additional effort we could get all the benefits and live within the unfortunate 16TB volume limitation of ext4. One of the required changes to work around ext4’s constraints was to add LVM (Logical Volume Manager) above the RAID 6 but below the file system. In our particular application (which features more writes than reads), ext4’s performance was a clear winner over ext3, JFS, and XFS.

With these performance improvements, we see the new Storage Pods in our data center accepting customer data more than twice as fast as the older generation Pods. It takes approximately 25 days to fill a new Pod with 135TB of data. The chart below shows the measured fill rates of an old Pod versus a new Pod, both under real-world maximum load in our data center.

Please note: The above graph is not the benchmarked write performance of a Pod; we have easily saturated the gigabit pipes copying data from one Pod to another internally. This graph shows Pods running in production, accepting data from thousands of simultaneous and independent desktop machines running Windows and Mac OS, where each desktop is forming HTTPS connections to the Tomcat web server and pushing data to the Pod. At the same time, as customers are preparing restores that read data off those drives, there are system cleanup processes running, occasional RAID repairs, etc. In this end-to-end measurement, the new Pods are twice as fast in our environment.

Lessons Learned: Three Years, 16PB and Counting

Backblaze is employee owned (with no VC funding or other deep pockets), so we have two choices: 1) stay profitable by keeping costs low or 2) go out of business. Staying profitable is not just about upfront hardware costs; there are ongoing expenses to consider.

One of the hidden costs to a data center is the headcount (salary) for the employees who deploy Pods, maintain them, replace bad drives with good, and generally manage the facility. Backblaze has 16PB and growing, and we employ one guy (Sean) whose full time job is to maintain our fleet of 201 Pods, which hold 9,045 drives. Typically, once every two weeks, Sean deploys six Pods during an eight-hour work day. (He gets a little help from one of us to lift each Pod into place because they each weigh 143 pounds.)

Our philosophy is to plan for equipment failure and build a system that operates in spite of it. We have a lot of redundancy, ensuring that if a drive fails, immediate replacement isn’t critical. So at his leisure, Sean also spends one day each week replacing drives that have gone bad. As of this week, Backblaze has more than 9,000 hard drives spinning in the data center, the oldest of which we purchased four years ago. We see fairly high infant mortality on the hard drives deployed in brand new Pods, so we like to burn the Pods in for a few days before storing any customer data. We have yet to see any drives die because of old age, which will be fascinating to monitor in the next few years. All told, Sean replaces approximately 10 drives per week, indicating a 5% per year drive failure rate across the entire fleet, which includes infant mortality and also the higher failure rates of previous drives. (We are currently seeing failures in less than 1% of the Hitachi Deskstar 5K3000 HDS5C3030ALA630 drives that we’re installing in Pod 2.0.)

We monitor the temperature of every drive in our data center through the standard SMART interface, and we’ve observed in the past three years that: 1) hard drives in Pods in the top of racks run three degrees warmer on average than Pods in the lower shelves; 2) drives in the center of the Pod run five degrees warmer than those on the perimeter; 3) Pods do not need all six fans—the drives maintain the recommended operating temperature with as few as two fans; and 4) heat doesn’t correlate with drive failure (at least in the ranges seen in Storage Pods).

One important note: Because all of the parts (including drives) in the Backblaze Storage Pod come with a three-year warranty, we rarely pay for a replacement part. The drive manufacturers take back failed drives with “no questions asked” and send free replacements. If you figure that storage resellers, such as NetApp and EMC, tack on a three-year support fee, a petabyte of Backblaze storage costs less than their support contract alone. A chart below takes all of our experience into account and shows what it costs to own and maintain a petabyte of storage for three years:

In the chart above, the economies of scale only kick in if you really do need to store a full petabyte or more. For a small amount of data (a few terabytes), Amazon S3 could easily save money, but the Amazon option is clearly a dubious financial choice for a company with large, multi-petabyte storage needs.

Final Thoughts

The Backblaze Storage Pod is just one building block in making a cloud storage service. If all you need is cheap storage, this may suffice. If you need to build a reliable, redundant, monitored storage system, you’ve got more work ahead of you. At Backblaze we’ve developed software that manages and monitors the cloud service, proprietary technology that we’ve developed over the years.

We offer our Storage Pod design free of any licensing or any future claims of ownership. Anybody is allowed to use and improve upon it. You may build your own cloud system and use the Backblaze Storage Pod as part of your solution. The steps to assemble a Storage Pod, including diagrams, can be found on our original blog post, and an updated list of parts is provided below in Appendix A. We don’t sell the design, so we don’t provide support or a warranty for people who build their own. To all of those builders who take up the challenge, we’d love to hear from you and welcome any insights you provide about the experience. And please send us a photo of your new 135TB Pod.

Appendix A: Price List:

Custom wiring harnesses for PSU1 and PSU2 (the Zippy power supplies):
See detailed wiring harness diagrams.

SATA Chipsets
SiI3726 on each port multiplier backplane to attach five drives to one SATA port.
SiI3124 on three PCIe SATA cards. Each PCIe card has four SATA ports on it, although we only use three of the four ports.

The post Petabytes on a Budget 2.0: Revealing More Secrets appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Last month’s blog post about building our Backblaze Storage Pods generated a ton of interest and many people are building their own Pods! Our post also generated a ton of questions so, below we answer the common ones and provide more details about where to get components.

Detailed Backblaze Storage Pod Parts and Vendor List

*Backblaze now uses the low power drives.

**Protocase is excellent at making custom cases in small quantities and in response to demand have dropped the price of the Backblaze Storage Pod enclosure to $700 each if you order 20 or more. Other case manufacturers can also build these cases from the 3-D model.

***Zippy Technology Corp. provides the power supplies. List price is about $500/unit. For quantities of 100+ (50 pairs), contact Mason Lee at the U.S. West Pacific branch of Zippy: +1-949-366-9525.

For smaller quantities, the Enermax Modu82+ 525W PSUs works well in the Backblaze Storage Pod and since those power supplies are modular it is easier to design the custom wiring harness. For $25 Enermax used to make the custom wiring harness but we have heard reports that they will no longer do this. If this is the case, since the power supply is modular, you could make your own custom wiring harness. The only catch is that you need a motherboard stub to turn the second PSU on/off. This is a $1 part but can be hard to find.

****CFI has now setup a complete, and very reasonable, pricing table for people who wish to purchase these port multiplier backplanes directly. Their pricing is:

200pcs US$40.00
150pcs US$41.00
100pcs US$42.00
50pcs US$43.00
40pcs US$43.50
30pcs US$44.00
20pcs US$44.50
10pcs US$45.00
5pcs US$45.50
1pc US$46.00

Note: If you are using latching SATA cables, make sure to ask for the port multipliers that support a latch-housing on the SATA connector.

*****The Western Digital drive listed has recently been discontinued. This is the closest match:
Western Digital Caviar SE WD800AAJB 80GB 7200 RPM 8MB Cache IDE Ultra ATA100 3.5″ Internal Hard Drive – OEM

However, most drives will work for the boot drive. We prefer PATA drives since this keeps the device name separate from the SATA data drives.

† House of Foam recently went out of business, but similar products can be ordered from other foam product suppliers.

Tips and Tricks for Building a Backblaze Storage Pod

Drives

Many people said, “Oh, I’ll build one of these Pods but switch the drives for 2TB drives or ones from XYZ vendor.” We tried drives from Western Digital, Hitachi, Samsung, and Seagate; 1TB, 1.5TB, and 2TB drives; and consumer and enterprise drives. Some of these drives performed terribly—dropping out of the RAID array on a regular basis. Other performed OK. The Seagate Barracuda 1.5TB low power drives have worked best for us. The 2TB Seagate drives have also worked well but currently cost more per GB than the 1.5TB drives; the Samsung low power drives have extremely high error rates; Hitachi and Samsung 7200 RPM drives performed well but we have done limited testing; and the Western Digital drives have not performed well in our setup. (Note: this is not necessarily to say these drives are good or bad, just how well they performed in this configuration.)

Heat

Tons of commenters worried about Backblaze Storage Pods BBQ’ing the drives. On the contrary, heat has not been an issue at all, six fans are definitely overkill and there are opportunities to reduce the number of fans. However, if you change the case design, type of drives, etc., make sure you monitor what impact your changes have made.

Power

We settled on our power supplies by figuring out the least expensive way to get high-efficiency power. If switching power supplies, ensure you have smooth power and there is plenty to spin up the drives from a cold start without being too close to the margin. In general, you need significant power on five volts (about 40-50 amps) which is what makes finding alternate power supplies a challenge.

Vibration

We worked hard to reduce vibration in the storage pods. Nylon screws, rubber fasteners for the fans, HD anti-vibration sleeves, foam insulation, etc., were all to dampen vibration. There is some data to show that our focus on vibration was important…though in discussions with the drive manufacturers, they claim that vibration is more likely to impact performance, not reliability. Having said that, if you change parts, make sure you are not increasing vibration.

Cables

While it’s tempting to lace up the cables tightly, they should actually be fairly loose so they do not pull on the connectors. And, even more important, ensure you are getting cables with good connectors. The actual cables rarely break, but connectors make a big difference: gold-plated, well-curved springy connectors that are shiny and not pitted if viewed under a microscope are ideal. (The ones we specified above are fine.) Also, ensure you get right angle SATA cable connectors, not left angle ones, so the cables route cleanly under the backplanes.

HD Anti-Vibration Sleeves

Vibration sleeves should be put on the drives at the level of the top grid such that they completely insulate it; the drives should not touch the grid. It takes a bit of practice at first, but quickly becomes easier than the standard process of putting four screws into a drive caddy.

Cold Swap Drives

Technically the system will support hot-swapping drives. However, we never do this. Hot-swapping drives on any system increases the likelihood of something going wrong. Since these Pods are top-loaded and require moving the server to replace drives, it’s one more reason to power down before swapping drives. Here is a picture of me doing this:

Powering Up the Pod

We mentioned this in the original blog post, but it is worth repeating here: we recommend powering up PSU1 first, waiting until the drives are spun-up (and the power draw decreases to a reasonable level), and then powering up PSU2.

Test the Storage Pod

There are a number of tests that should minimally be run before deploying a Storage Pod into production. First, completely sync all the drives into a RAID array. This forces every sector on every drive to be read and some sectors to be written—a process that will take about three days for a Pod full of 1.5TB drives. Second, run a memory test on the system. And finally, run a load test on the Storage Pod. There are various tools out there to help with this process. (At Backblaze, we wrote our own to best simulate a real production environment.)

Hopefully this helps those of you who are working on building your own version of the Backblaze Storage Pod. We still intend to provide blog posts covering vibration, drives, and other areas of the Pod in detail. You should feel free to experiment; try different motherboards, CPUs, cases, fans, etc. Customize the Backblaze Storage Pod for your particular purpose. Do keep in mind that while it may seem that every component should work seamlessly with every other, it is not the case. If you are switching out pieces, expect to need to experiment to get the system working right.

The post Backblaze Storage Pod: Vendors, Tips, and Tricks appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

When we started Backblaze, our goal was to provide unlimited computer backup for $5 a month, so we had to figure out how to store hundreds of petabytes of customer data in a reliable, scalable way—and keep our costs low. After looking at several overpriced commercial solutions, we decided to build our own custom Backblaze Storage Pods: 67 terabyte 4U servers for $7,867.

What we actually provide for our customers is online backup for home and business online backup for work. However, in this post, we’ll share how to make one of these Storage Pods, and you’re welcome to use this design. Our hope is that by sharing, others can benefit and ultimately, refine this concept and send improvements back to us. Evolving and lowering costs is critical to our continuing success at Backblaze.

Below is a video that shows a 3-D model of the Backblaze Storage Pod. Continue reading to learn the exact details of the design.

You can download the full 3-D model of the Backblaze Storage Pod here.

Backblaze Needs Plenty of Reliable, Cheap Storage

To say that Backblaze needs lots of storage is an understatement. We’re a backup service, so our data center contains a complete copy of all of our customers’ data, plus multiple versions of files that change. In rough terms, every time one of our customers buys a hard drive, Backblaze needs another hard drive. A long time ago we stopped measuring storage in our data center in gigabytes or terabytes and started measuring in petabytes.

To get a sense of what this looks like, here is a shot of me deploying new Pods in our data center. The small stack of six Pods in the rack I’m working on contains just under half a petabyte of storage.

To offer our service at a reasonable price, we need affordable storage at a multi-petabyte scale.

No One Sells Cheap Storage, So We Designed It

Before realizing that we had to solve this storage problem ourselves, we considered Amazon S3, Dell or Sun Servers, NetApp Filers, EMC SAN, etc. As we investigated these traditional off-the-shelf solutions, we became increasingly disillusioned by the expense. When you strip away the marketing terms and fancy logos from any storage solution, data ends up on a hard drive. But when we priced various off-the-shelf solutions, the cost was 10 times as much (or more) than the raw hard drives. Here’s a comparison chart of the price for one petabyte from various vendors:

Based on the expense, we decided to build our own Backblaze Storage Pods. We had two primary goals: keep upfront costs low by using consumer-grade drives and readily available commodity components and be as power and space efficient as possible by using green components and squeezing a lot of storage into a small box.

The result is a 4U rack-mounted Linux-based server that contains 67TB at a material cost of $7,867, the bulk of which goes to purchase the drives themselves. This translates to just three-tenths of one penny per gigabyte per month over the course of three years. Even including the surrounding costs—such as electricity, bandwidth, space rental, and IT administrators’ salaries—Backblaze spends one-tenth of the price in comparison to using Amazon S3, Dell Servers, NetApp Filers, or an EMC SAN.

What Makes a Backblaze Storage Pod?

A Backblaze Storage Pod is a self-contained unit that puts storage online. It’s made up of a custom metal case with commodity hardware inside. Specifically, one Pod contains one Intel Motherboard with four SATA cards plugged into it. The nine SATA cables run from the cards to nine port multiplier backplanes that each have five hard drives plugged directly into them (45 hard drives in total).

Above is an exploded diagram, and you can see a detailed parts list in Appendix A at the bottom of this post. The two most important factors to note are that the cost of the hard drives dominates the price of the overall Pod and that the rest of the system is made entirely of commodity parts.

Wiring It Up: How to Assemble a Backblaze Storage Pod

The power wiring diagram of a Backblaze Storage Pod is seen below. Power supply units (PSUs) provide most of their power on two different voltages: 5V and 12V. We use two power supplies in the Pod because 45 drives draw a lot of 5V power, yet high wattage ATX PSUs provide most of their power on 12V. This is not an accident: 1,500 watt and larger ATX power supplies are designed for powerful 3-D graphics cards that need the extra power on the 12V rail. We could have switched to a power supply designed for servers, but two ATX PSUs are cheaper.

PSU1 powers the front three fans and port multiplier backplanes one, two, three, four, and seven. PSU2 powers everything else. (See Appendix A for a detailed list of the custom connectors on each PSU.) To power the port multiplier backplanes, the power cables run from the PSUs through four holes in the divider metal plate that holds the fans at the center of the case (near the base of the fans) and then continue to the underside of the nine backplanes. Each port multiplier backplane has two molex male connectors on the underside. Hard drives draw the most power during initial spin-up, so if you power up both PSUs at the same time, it can draw a large (14 amp) spike of 120V power from the socket. We recommend powering up PSU1 first, waiting until the drives are spun-up (and the power draw decreases to a reasonable level), and then powering up PSU2. Fully booted, the entire Pod will draw approximately 4.8 amps idle and up to 5.6 amps under heavy load.

Below is a picture of a partially assembled Backblaze Storage Pod (click on the photo for a larger image). The metal case has screws mounted on the bottom, facing upward, where we attach nylon standoffs (the small white pieces in the picture below). Nylon helps dampen vibration, and this dampening is a critical aspect of server design. The circuit boards shown on top of the nylon standoffs are a few of the nine SATA port multiplier backplanes that take a single SATA connection on their underside and allow five hard drives to be mounted vertically and plugged into the topside of the board. All the power and SATA cables run underneath the port multiplier backplanes. One of the backplanes in the picture below is fully populated with hard drives to show the positioning.

A note about drive vibration: the drives vibrate too much if you leave them sitting as shown in the picture above, so we add an “anti-vibration sleeve” (essentially a rubber band) around the hard drive in between the red metal grid and the drives. This seats the drives tightly in the rubber. We also lay a large (16″ x 17″ x 1/8″) piece of foam along top of the hard drives after all 45 are in the case. The lid then screws down on top of the foam to hold the drives securely. In the future, we will dedicate an entire blog post to vibration.

The SATA wiring diagram is seen below.

The Intel Motherboard has four SATA cards plugged into it: three SYBA two-port SATA cards and one Addonics four-port card. The nine SATA cables connect to the top of the SATA cards and run in tandem with the power cables. All nine SATA cables measure 36 inches and use locking 90-degree connectors on the backplane end and non-locking straight connectors into the SATA cards.

A note about SATA chipsets: each of the port multiplier backplanes has a Silicon Image SiI3726 chip so that five drives can be attached to one SATA port. Each of the SYBA two-port PCIe SATA cards has a Silicon Image SiI3132, and the four-port PCI Addonics card has a Silicon Image SiI3124 chip. We use only three of the four available ports on the Addonics card because we have only nine backplanes. We don’t use the SATA ports on the motherboard because, despite Intel’s claims of port multiplier support in their ICH10 south bridge, we noticed strange results in our performance tests. Silicon Image pioneered port multiplier technology, and their chips work best together.

A Backblaze Storage Pod Runs Free Software

A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The Pods boot 64-bit Debian 4 Linux and the JFS file system, and they are self-contained appliances, where all access to and from the Pods is through HTTPS. Below is a layer cake diagram.

Starting at the bottom, there are 45 hard drives exposed through the SATA controllers. We then use the fdisk tool on Linux to create one partition per drive. On top of that, we cluster 15 hard drives into a single RAID6 volume with two parity drives (out of the 15). The RAID6 is created with the mdadm utility. On top of that is the JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5. After taking all this into account, the formatted (useable) space is 87% of the raw hard drive totals. One of the most important concepts here is that to store or retrieve data with a Backblaze Storage Pod, it is always through HTTPS. There is no iSCSI, no NFS, no SQL, no Fibre Channel. None of those technologies scales as cheaply, reliably, goes as big, nor can be managed as easily as stand-alone Pods with their own IP address waiting for requests on HTTPS.

A Backblaze Storage Pod Is a Building Block

We have been extremely happy with the reliability and excellent performance of the Pods, and a Backblaze Storage Pod is a fully contained storage server. But the intelligence of where to store data and how to encrypt it, deduplicate it, and index it is all at a higher level (outside the scope of this blog post). When you run a data center with thousands of hard drives, CPUs, motherboards, and power supplies, you are going to have hardware failures—it’s irrefutable. Backblaze Storage Pods are building blocks upon which a larger system can be organized that doesn’t allow for a single point of failure. Each Pod in itself is just a big chunk of raw storage for an inexpensive price; it is not a “solution” in itself.

Cloud Storage: The Next Step

The first step to building cheap cloud storage is to already have cheap storage, and above we demonstrate how to create your own. If all you need is cheap storage, this may suffice. If you need to build a cloud, you’ve got more work ahead of you.

Building a cloud includes not only deploying a large quantity of hardware but, critically, deploying software to manage it. At Backblaze we have developed software that de-duplicates and chops data into blocks; encrypts and transfers it for backup; reassembles, decrypts, re-duplicates, and packages the data for recovery; and monitors and manages the entire cloud storage system. This process is proprietary technology that we have developed over the years.

You may have your own system for this process and incorporate the Backblaze Storage Pod design, or you may simply seek inexpensive storage that won’t be deployed as part of a cloud. In either case, you’re free to use the Storage Pod design above. If you do, we would appreciate credit at Backblaze and welcome any insights, though this isn’t a requirement. Please note that because we’re not selling the design or the Storage Pods themselves, we provide no support nor warranties.

Coming next: In the next few weeks, we’ll talk about iPhone vibration sensors, Swiss cheese Pod designs, why electricity costs more than bandwidth, and more about the design of big cloud storage.

Credits and Standing on the Shoulders of Giants

The Backblaze Storage Pod design would not have been possible without an enormous amount of help, usually requested with little notice, from some amazingly smart and generous people who answered our questions, worked with us, and provided key insights at critical moments. First, we thank Chris Robertson for the inspiration to build our own storage and his early work on prototypes; Kurt Schaefer for advice on metal work and the concept of “furniture” for circuit boards; Dominic Giampaolo from Apple Computer for his advice on hard drives, vibration, and certifications; Stuart Cheshire from Apple Computer and Nick Tingle from Alcatel-Lucent for low-level network advice; Aaron Emigh (EVP & GM, Core Technology) at Six Apart for his help on initial design work; Gary Orenstein for insight into drive reliability and the storage industry in general; Jonathan Beck for invaluable advice on vibration, fans, cooling, and case design; Steve Smith (Senior Design Manager), Imran Pasha (Director of Software Engineering), and Alex Chervet (Director of Strategic Marketing) at Silicon Image who helped us debug SATA protocol problems and loaned us 10 different SATA cards to test against; James Lee at Chyang Fun Industries in Taiwan for customizing SATA boards to simplify our design; Wes Slimick, Richard Crockett, Don Shields, and Robert Knowles from Western Digital for their help debugging Western Digital drive logs; Christa Carey, Jennifer Hurd, and Shirley Evely at Protocase for putting up with hundreds of small 3-D case design tweaks; Chester Yeung at Central Computer for coming through quickly and repeatedly with locally supplied parts when it really mattered; Mason Lee at Zippy for power supply advice and custom wiring harnesses; and Angela Lai for knowing just the right people and providing gracious introductions.

Finally, we thank the thousands of engineers who slaved away for millions of hours to bring us the Pod components that are either inexpensive or totally free, such as the Intel Processor, Gigabit Ethernet, ridiculously dense hard drives, Linux, Tomcat, JFS, etc. We realize we’re standing on the shoulders of giants.

Appendix A: Detailed Backblaze Storage Pod Parts List

Custom wiring harnesses for PSU1 (1st Zippy power supply):

• 5x 4-pin 90-degree molex HD connectors with two connectors each. Length should be 36″ to the farthest connector, 32.5″ to the closest (3.5″ apart)
• 3x 4-pin 12V fan connectors that should be 32″ in length with extender and RPM signal that can attach to motherboard

Custom wiring harnesses for PSU2 (2nd Zippy power supply):

• 1x 24-pin motherboard connector, 8″
• 1x 4-pin ATX12V for CPU, 8″
• 4x 4-pin 90-degree molex HD connectors with two connectors each. Length should be 36″ to the farthest connector, 32.5″ to the closest (3.5″ apart)
• 1x 4-pin 90-degree molex connector, 12″ long
• 3x 3-pin, 12V fan connectors, 12″ long, with extender for RPM signal that can attach to motherboard

SATA Chipsets

• SiI3726 on each port multiplier backplane to attach five drives to one SATA port.
• SiI3132 on each of the three PCIe SATA cards to attach two backplanes each (six ports total)
• SiI3124 on the one PCI SATA card to attach up to four port multiplier backplanes (we only use three of the four ports)

The post Petabytes on a Budget: How to Build Cheap Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Goal: Security Done Right

Protecting the privacy of our users’ data is a top priority for us here at Backblaze and that means encryption. But providing a service that is extremely easy to use is also a key part of our vision and far too often encryption makes a product hard to use. This trade-off was unacceptable to us so we set out to build a system that delivers military grade encryption without compromise! Here is the full list of our requirements:

• 1. Protect data with military grade encryption
• 2. Implement encryption transparently so users don’t have to deal with it
• 3. Allow users to change their password without re-encrypting their data
• 4. In business environments, allow IT access to data without the user’s password

The Solution: Military Grade Encryption Made Easy

To accomplish the ambitious goals above we used a mix of public/private and symmetric key algorithms. The math behind this cryptography is hard but the idea is simple… Public/private keys allow you to encrypt data with one key and decrypt it with another one. Typically data is encrypted with the public key and decrypted with a private key that is kept secret but the reverse also works. This is very useful because it allows us to encrypt data in the background without requiring the user to type in their password.

Unfortunately, public/private key algorithms are slow and can’t be used to encrypt a large amount of data. Symmetric key algorithms use the same key to encrypt and decrypt data and are very fast on large amounts of data. But since the same key is used to decrypt the data, the data is only secure if the symmetric key is secure.

Combining these algorithms, here’s how our system works.


We generate a new 2048-bit RSA public/private key pair when our client is installed, store the public key on the local disk and transmit the private key to our data center via https. Then, for each backup session, we generate a new random 128-bit Advanced Encryption Standard (AES) symmetric key which we use to encrypt the user’s data. We secure the 128-bit AES key by encrypting it with the user’s public key and transmit the encrypted file along with the encrypted key to our data center over https. We destroy the unencrypted 128-bit AES key at the end of each backup session and never write it to disk. To decrypt a file, the user’s private key is used to decrypt the 128-bit AES which is then used to decrypt the file.

The user’s private key which is stored safely in our data center is protected by a password that is highly guarded. But for some users this is not good enough and we allow the user to secure this file with their own password. When this is done it is impossible to access the data without the user’s password. Unfortunately, this also means we can’t help the user if they ever forget this password so we don’t recommend it for most users.

The real beauty of this scheme becomes clear when you look back at our goals above. AES is the encryption standard adopted by the US government to protect classified information. #1 solved. Using the user’s public key we can safely run transparently in the background without compromising security. #2, check. Since a password is used to secure the private key rather than to encrypt the data directly, the password can be changed by re-encrypting only the private key with the new password. #3 accomplished. And last but not least, you can make several copies of the user’s private key and encrypt each copy with a different password to provide IT access to data without the need to share passwords. #4 done!

The post How to Make Strong Encryption Easy to Use appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.