A Behind the Scenes Look at Our US East Data Center

In the last couple of years, Backblaze has taken residence in three new data center facilities. In the West, we added the CyrusOne facility in Chandler, Arizona and the Nautilus Data Center in Stockton, California. In the East, we just added the CoreSite facility in Reston, Virginia as our anchor to our new U.S. East data region. Each of these data centers will house over an exabyte of customer data. A number like an exabyte of storage is nice, and we’ll share a bunch more numbers as we go. But what I really wanted to share are the stories behind the scenes as we moved into the CoreSite facility in Reston, which we call IAD 1.

The process of turning a big empty room into a data center location is complex and requires intense coordination, an adaptable project plan, and folks who can think on their feet. Let’s face it, no project is perfect, and building out a data center is no different. But, once in a while, it’s fun to peek behind the curtain to see how the actual work gets done. Let’s take a look.

Big Boxes, Tight Spaces, and Construction

There were over 700 boxes of various shapes and sizes to be received at the IAD site. This included server cabinets, storage servers, support servers, networking equipment, and other odds and ends. The cabinet and storage server boxes were particularly large, and only four of these boxes at a time could fit into the service elevator used to move everything from the loading dock to the Backblaze facility on the fourth floor.

Cabinets and servers are normally received almost daily, but the IAD data center was undergoing construction which limited where and how trucks could unload their cargo. One poor fellow spent the better part of three hours trying to manipulate his semi into the loading dock. The construction also limited the amount of cargo that could stay in the dock area to basically zero. When a truck arrived and started unloading Backblaze goods, our process was: load four boxes, go up four floors, unload four boxes, go down four floors, and repeat until the truck was empty.

Each time a truck arrived, the race was on, and there were a lot of trucks as everything was scheduled to arrive within a 30-day window. Why 30 days? We wanted to install the cabinets all at once, so we could run the networking once and not have to piecemeal it in. To help out, we enlisted the help of the CoreSite staff to assemble and install the cabinets, while we ran the networking and installed the servers and other gear.

Taking the Long Way Home

The Backblaze presence is spread across two buildings. One building houses the data center itself and includes the maintenance area. The Backblaze office is located in another building. It’s not ideal, but sometimes things work out that way. But, what it means is this: If you don’t know where you are going you’re either not supposed to be there or you are new. Data centers are not known for having a lot of “you are here” signs, nor are there a lot of folks around to ask if you are lost.

Being new can turn a five minute walk from the office to the data center floor into a 30 minute expletive laden stroll through unmarked halls and deadend corridors complete with visions of serial killers being behind every door. Lucky for us, CoreSite does background checks on all their employees.

Let’s Talk Boxes

While Backblaze was responsible for getting the boxes to the new space and unboxing the contents, CoreSite helped out by providing us with some temporary storage space as we built out our facility. Given our aggressive schedule, things occasionally got messy (as seen below).

Shortly after this photo was taken, we learned the site had “the crusher”, which takes boxes or garbage or whatever and, well, crushes such things into dumpster-suitable or recycling-suitable packages. While not as fun as a Megabot, the crusher ensured that we didn’t lose any employees under an avalanche of boxes.

How many boxes? Well, there were 126 boxes containing cabinets, one per box. Each cabinet was assembled and installed by the CoreSite folks. There were hundreds of smaller boxes containing networking servers, conduits, networking cables, and, of course, thousands of various types of cable ties used by the Backblaze cabling ninjas as we see below.

The cabinets are 52U tall, and 120 of them will be used to house 1,440 storage servers which will make up 72 Backblaze Vaults. Each vault consists of twenty storage servers. Each storage server has 60 16TB drives, which totals 960TB of raw storage per server. Doing the math, the IAD data center will have over 1.3EB of raw storage. Subtracting formatting and parity, the capacity is still over 1EB. Of course, over time we expect to use larger hard drives in all of our data centers as the cost per gigabyte for hard drives continues to decrease.

My Kingdom for a Storage Pod

Not to be a villain here, but there are no Backblaze Storage Pods in the IAD 1 data center. All 100 of the storage servers used for the initial build out of the IAD data center are the Supermicro models we detailed in the recent Storage Pod Story blog post. You can see from the photo below each of the five vaults are racked and waiting for their hard drives to be installed. Maybe one day Supermicro will make us some pretty red bezels.

An Old Friend to the Rescue

The 52U Enconnex cabinets are 94.49 inches (2400 mm) tall. The 4U Supermicro storage servers will eventually be stacked 12 high in the cabinet, leaving 4U at the top of each for 1U core servers and IPMI (Intelligent Platform Management Interface) switches. Lifting a 4U 150lb (68kg) storage server is difficult, but so is lifting a 1U core server to nearly eight feet high. We needed some muscle, and there’s no one better than Guido, our first and most experienced server lift. He was flown in from the Phoenix data center to help the IAD staff get set up, and if he likes the gig he can stay. After all, he earned the right to a choice after nearly 12 years of heavy lifting for Backblaze.

Power

The data center provides fully redundant power to each cabinet. Each separate power source connects to a PDU in each cabinet. In each cabinet, there is a red PDU and a blue PDU with each color representing a power source. Since most of the servers we use in the data center support redundant power, a given server connects to each PDU (red and blue) in their cabinet as shown below.

The PDUs that we used were recommended by the data center and not a brand we had used before. The PDU manufacturer does not make the power cables, but they do recommend a couple of brands. We happened to like our red and blue cables and used them instead. We were surprised to discover they were a bad fit and kept falling out of the PDUs—so much for standards. Amazingly, it just so happens that a company makes PDU plug locks to keep the plugs from falling out. The plug locks also help when someone accidentally bumps into a power plug connected to the PDU while working on some equipment, so there’s that.

Security

As with all data center facilities, security is a prime concern. At the CoreSite facility, Backblaze personnel must pass through a minimum of four checkpoints to get from the parking lot to the Backblaze data center facility or the Backblaze office. Along the way, both badge access and biometric scans are employed—sometimes separately and sometimes together. In addition, Backblaze personnel are limited in where they can go. For example, they are not allowed on the second and third floors of the data center building, only the fourth floor, and then they can only enter our facility. Getting lost while going from the office to the datacenter floor should make a little more sense now.

Within the Backblaze facility there are cameras that monitor everything inside. There are also cameras used by the CoreSite staff to monitor the common areas such as hallways, the loading dock, and the parking lot. Before you can enter or leave the parking lot, an access badge and visual confirmation are required by the CoreSite staff. This led to a very interesting dinner one evening for Backblaze and CoreSite personnel…

Huevos Rancheros

Several of the Backblaze staff were temporarily deployed to Reston to set up the IAD data center. One of their favorite places to eat was Ted’s Bulletin, located in Reston near the data center. They serve breakfast all day until they close at 10 p.m. or so. Working into the evening is typical for data center set ups, and the gang decided to order from Ted’s and have it delivered via DoorDash so they could keep working.

The Dasher arrived with their order at the back gate of the compound. That’s not a public entrance and the Dasher was told to go around to the public gate. “This is where it says to go,” said the Dasher. He wasn’t even sure where he was; he just followed the GPS. Jack, who placed the DoorDash order, got a call from the Dasher. He was going to leave if someone didn’t meet him at the gate. Not wanting to see his huevos rancheros go to waste, Jack found his way to the back gate talking to the Dasher all the way so he wouldn’t leave. Jack showed his credentials to the security camera, but they would not open the gate. Why? The Dasher was a visitor at a non-visitor gate, and Jack was not a vehicle that needed to exit. The compromise; the Dasher was allowed to hand Jack the containers of food through a narrow opening in the gate. Jack showed his huevos rancheros and the other delights to security as he passed through the various checkpoints to get to a Backblaze office, and breakfast for dinner was had by all.

Three days later they wanted Ted’s again. They drove.

Epilogue

The Backblaze U.S. East data region is ready to go with five Backblaze Vaults online and accepting data. That’s 100 servers and thousands of connections open for business on day one with more vaults waiting in the wings to be deployed by the end of the year. Many thanks to Jack, Jessie, Zachary, Brent, Rich, Mark, and the supporting cast back at Backblaze HQ in San Mateo for getting IAD 1 up and running. Jessie and Zachary are part of the permanent crew at IAD 1 with more folks joining them over the coming months.

One last shoutout to the IAD crew for having the courage and sense of humor to share their stories with me. Having a Dasher squeezing your huevos through a gate in the dark while security folks watch on a live feed is not something I could ever make up. Thanks again.