There’s almost no way to quantify the impacts COVID-19 has had on the world. Personally, communally, and economically—there isn’t a part of our lives it hasn’t touched in some way. We’ve discussed how it’s affected our operations and our culture, but at the end of the day, the central focus at Backblaze is providing the best cloud storage and backup in the world—a mission that’s especially important in a time when cloud storage and data security has become more vital to day-to-day life than ever.

At the most basic level, our services and products rely on a singular building block: the hard drive. And today, we’re going to discuss how our team has ensured that, as more businesses and individuals turn to cloud storage to solve their rapidly evolving data storage and management needs, we’ve had what we need to care for the petabytes of inbound data.

We’re no strangers to navigating an external threat to business as usual. In 2011, flooding in Thailand impacted nearly 50% of the world’s hard drive manufacturing capability, limiting supply and dramatically raising hard drive prices. At the time, Backblaze was only about four years into providing its computer backup service, and we needed to find a way to keep up with storage demand without going broke. We came up with a hack that became internally known as “drive farming.”

What does it mean to farm hard drives? Well, everyone on our staff, and many of our friends and family members went out and bought every hard drive we could get our hands on, at every retail outlet nearby. It was a bit unconventional, but it worked to maintain our storage demand. We wrote the whole story about how we weathered that crisis without compromising our services in this blog post.

This year, most of us thought the eruption of the volcano Taal in the Philippines was going to be the biggest threat to the hard drive supply chain. We were wrong. Instead, we’ve been called to apply some of the resourcefulness we learned during the Thailand drive crisis to deal with the disruptions to production, manufacturing, and supply chains that COVID-19 has caused.

No, this isn’t “Drive Farming: Part II, the Drivening!” Rather, faced with an uncertain and rapidly shifting business environment, we turned to someone on our team who knew, even before 2020 began, that a global pandemic was a much more likely challenge to our operations than any volcano: our Senior Director of Supply Chain, Ariel Ellis.

Recently, Ahin (our VP of Marketing) sat down with Ariel to discuss how he has been managing our supply chain efforts within the context of these extraordinary times. The Q&A that follows has been edited for brevity (give a marketer a microphone…). It covers a wide range of topics: how business has changed since the emergence of COVID; how our supply chain strategy adjusted; and what it’s like for Ariel to do all of this while battling COVID himself.

Ahin Thomas: Wow! What a ride. Let’s start by understanding the baseline—what was considered “business as usual” in the supply chain before COVID? Can you give me a sense of our purchasing volumes of hard drives and who makes them?

Ariel Ellis: Pre-COVID we were buying hard drives on a quarterly basis and deploying around 20-30PB of data storage a month. We were doing competitive bidding between Seagate, Toshiba, and Western Digital—the only three hard drive manufacturers in the world.

AT: It doesn’t seem that long ago that 30PB in a year would have been a big deal! But are you saying things were pretty stable pre-COVID?

Ariel: Everything was relatively stable. I joined Backblaze in 2014 and pre-COVID, 2019 has probably been the most consistent in regards to the hard drive supply chain that I have seen during my tenure.

AT: Well that’s because neither of us was here in 2011 when the floods in Thailand disrupted the global hard drive supply chain! How did the industry learn from 2011 and did it help in 2020?

Ariel: The Thailand flooding caught the manufacturers, and the industry, off guard. Since then the manufacturers have become better at foreseeing disruptions and having contingency plans in place. They’ve also become more aware of how much routine cloud storage demand there is, so they are increasingly thoughtful about our kind of businesses and making sure supply is provided accordingly. It’s worth noting that the industry has also really shifted—manufacturers are no longer trying to provide high capacity hard drives for personal computers at places like Costco and Best Buy because consumers now use services like ours, instead.

AT: Interesting. How did we learn from 2011?

Ariel: We now have long term planning in place, directly communicate with the manufacturers, and spend more time thinking about durability and buffers.

I was actually brought in right after the Thailand crisis because Backblaze realized that they needed someone to specialize in building out supply strategies.

The Thailand flooding really changed the way we manage storage buffers. I run close to three to four months of already deployed, forecasted storage as a buffer. We never want to get caught without availability that could jeopardize our durability. In a crisis, this four-month buffer should provide me with enough time to come up with an alternative solution to traditional procurement methods.

Our standard four-month deployed storage buffer is designed to withstand either a sudden rise in demand (increase to our burn rate), or an unexpected shortage of drives—long enough that we can comfortably secure new materials. Lead times for enterprise hard drives are in the 90-day range, while manufacturing for our Pods is in the 120-day range. In the event of a shortage we will immediately accelerate all open orders, but to truly replenish a supply gap it takes about four months to fully catch up. It’s critical that I maintain a safety buffer large enough to ensure we never run out of storage space for both existing and new customers.

As soon as we recognized the potential risks that COVID-19 posed for hard drive manufacturing, we decided to build a cache of hard drives to last an additional six months beyond the deployed buffers. This was a measured risk because we lost some price benefits due to buying stock early, but we decided that having plenty of hard drives “in house” was worth more than any potential cost savings later in the year. This proved to be the correct strategy because manufacturers struggled for several months to meet supply and prices this year have not decreased at the typical 5% per quarter.

AT: So, in a sense, you can sacrifice dollars to help remove risk. But, you probably don’t want to pull that lever too often (or be too late to pull it, either). When did you become aware of COVID and when was it clear to you that it would have a global impact?

Ariel: As a person in charge of supply chains, I had been following COVID since it hit the media in late December. As soon as China started shutting down municipalities I recognized that this was going to have a global impact and there would be scarcities. By January most of us in the industry were starting to ask the question: How is this going to affect us? I wasn’t getting a lot of actionable feedback from any of the manufacturers, so we knew something was coming but it was very hard to figure out what to do.

AT: That’s tough—you can see it coming but can’t tell how far away it is. But seeing it in January—on a relative basis—is early. How did you get to that point?

Ariel: I’m part of the Backblaze COVID preparation team and the Business Continuity Team, which is a standing team of cross-functional leaders that are part of the overall crisis response plan—we don’t want to have to meet, but we know what to do when it happens. I also had COVID. As we were making firm business decisions on how to plan for disruptions I developed a cough, a fever, and had to take naps to make it through the day. It was brutal.

In December of 2019, we realized we had to stay ahead of decision making on sourcing hard drives. We had to be aggressive and we had to be fast. We first discussed doing long term contracts with the manufacturers to cover the next 12 months. Then, as a team, we realized that contracts weren’t an option because if shelter-in-place initiatives were rolled out across the country then we were going to lose access to the legal teams and decision makers needed to make that process work. It was during the second week of March 2020 that we decided to bypass long term contracts and do the most viable thing we could think of, which was to issue firm purchase orders. A purchase order accepted between both companies is the most certain way to stay at the front of the line and ensure hard drive stock.

We immediately committed to purchase orders for the hard drives needed to cover six months out. This was on top of our typical four-month deployment buffer and would ultimately give us about 10 months of capacity. This is a rolling six months, so since then I’ve continued to ensure we have an additional six months of capacity committed.

Issuing these purchase orders required a great deal of effort and coordination across the Business Continuity Team, and in particular with our finance team. I worked side-by-side with our chief financial officer to quickly leverage the resources needed to commit to stock outside of our normal cycles. We ordered around 40,000 hard drives rapidly, which is about 400PB of usable space (meaning after parity), or roughly $10 million worth of capital equipment. Overall, this action has proved to be smart and put us one to two weeks ahead of the curve.

AT: We’re all grateful you made it through. OK, so a couple weeks into a global pandemic, while you’ve contracted COVID-19, we increased our purchasing by an order of magnitude! How are the manufacturers performing? Are we still waiting on drives from the purchase orders we issued?

Ariel: We’ve deployed many of the drives we’ve received, and we have a solid inventory of about 20,000 drives—which equals about a couple hundred petabytes of capacity—but we’ve continued to add to the open orders and are still waiting for around 20,000 drives to finish out the year. The answer to manufacturer performance changes on a constant basis. All three manufacturers have struggled due to mandated factory shutdowns, limited transportation options, and component shortages. We consistently experience small-to-medium delays in shipments, which was somewhat expected and the reason we extended our material buffers.

AT: Is there a sense of “new normal” for the buffer? Will it return to four months?

Ariel: This is going to change my world forever. Quarterly buying and competitive bid based strategies were a calculated risk, and the current crisis has caused me to rethink risk calculation. Moving forward we are going to better distribute our demand across the three manufacturers so I stay front and center if there is ever constrained supply. We will also be assessing quarterly bidding, which while price effective, gives us limited capacity and it is somewhat short-sighted. It might be more advantageous to look at six-month, and maybe even rough, 12-month capacity plans with the manufacturers.

This year has reminded me how tentative the supply of enterprise hard drives is for a company at our scale. We rely on hard drives for our life blood and the manufacturers rely on a handful of cloud storage companies like us as the primary consumers of high-capacity storage. I will continue to develop long term supply strategies with each of the manufacturers as I plan the next few years of growth.

AT: I know we are still very much in the middle of the pandemic, but have things somewhat stabilized for your team?

Ariel: From a direct manufacturing perspective we’re just now starting to see a return to regular manufacturing. In Malaysia, Thailand, and the Philippines, there were government-imposed factory shutdowns. Those restrictions are slowly being lifted and production is returning to full capacity in steps. Assuming there is no pendulum swing back to reinfection in those areas, the factories expect to return to full capacity any day now. It’s going to take a number of months for them to work through their backlog of orders, so I would expect that by October we will see a return to routine manufacturing.

It’s interesting to point out that one of the most notable impacts of COVID to the supply chain was not loss of manufacturing, but loss of transportation. In fact, that was the first challenge we experienced—factories having 12,000 hard drives ready to ship, but they couldn’t get them on an airplane.

AT: This might be a bit apocalyptic, but what was your worst case scenario? What would have happened if you couldn’t secure drives?

Ariel: We would fully embrace our scrappy, creative spirit and I would pursue any number of secondary options. For example, deploying Dell servers, which come with hard drives, or looking for recertified hard drives, drives that were made for one of the tier one hardware manufacturers but were unused and went back to the factory to be retested and recertified. Closer to actionable would have been slowing down our growth rate. While not ideal, we could certainly pump the brakes on accelerating new customer acquisitions and growth, which would extend the existing buffers and give us breathing room.

AT: Historically, the manufacturers have a fairly consistent cycle of increased densities and trying to drive down costs. Are you seeing any trends in drive development driven by this moment or is everyone simply playing catch-up?

Ariel: It’s unclear how much this year has slowed down new technology growth as we have to assume the development labs haven’t been functioning as normal. Repercussions will become clear over the next six months, but as of now I have to assume that the push towards higher capacities, and new esoteric technologies to get to those higher capacities, has been delayed. I expect that companies are going to funnel all of their resources into current platforms, meeting pent up demand, and rebuilding their revenue base.

AT: Obviously, this is not a time for predicting the future—anyone who had been asked about 2020 in February was likely very wrong, after all—but what do you see in the next 12 months?

Ariel: I don’t think we’ve seen all of the repercussions from manufacturing bottlenecks. For example, there could be disruption to the production of the subcomponents required to make hard drives that the manufacturers have yet to experience because they have a cache of them. And whether it’s through lost business or lost potential, the hard drive manufacturers’ revenue streams are going to take a hit. We are deeply vested in seeing the hard drive manufacturers thrive so we hope they are able to continue business as usual and are excited to work with us to grow more business.

I think there will also be a further shift towards hard drive manufacturers relying on their relationship with cloud storage providers. During COVID, cloud service providers either saw no decline in business or an increase in business with people working from home and spending more time online. That is just going to accelerate the shift of hard drive production going exclusively towards large scale infrastructure instead of being dispersed amongst retail products and end-users across the planet.

Even more surprising is that our team hasn’t changed that much. Something Ariel mentioned after we finished our conversation was how, when he had to go offline due to complications from COVID, the rest of our team easily stepped in to cover him in his absence. And a lot of those folks that stepped into the breach were the same people wheeling shopping carts of hard drives out of big box stores in 2011. Sure, we manage more data and employ more people, but when it comes down to it, the same scrappy approach that got us where we are today continues to carry us into the future.

Floods, viruses, volcanoes: We’re going to have more global disruptions to our operations. But we’ve got a team that’s proven for 14 years that they can chart any uncertain waters together.

The post Floods, Viruses, and Volcanoes: Managing Supply Chain in Uncertain Times appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Spoiler alert: At the end of this post, we announce our Cloud to Cloud Migration program—an offer to pay the transfer costs for customers that want to migrate their data from Amazon S3 to Backblaze B2 Cloud Storage. Yup, we’re so confident in our service that we’ll pay for you to save 75% of your cloud storage bill. If you want to stop reading and start saving: Click here.

On May 4^th, we released the beta version of our Backblaze S3 Compatible APIs. It was our most requested feature, so we knew it was something our customers wanted. But what we’ve seen has been simply incredible—thousands of customers uploading petabytes of data to Backblaze B2 Cloud Storage. Today, we’re moving those S3 Compatible APIs out of beta and into general availability (GA), as we continue to remove the barriers that are keeping your data locked within the cloud oligarchy.

Not Just Compatible, VERY Compatible

At Backblaze, we take great pride in our track record of enduring innovation. Whether it’s our Storage Pod, Reed-Solomon erasure coding, or our S3 Compatible APIs, when we release something to the public, our sense of craftsmanship demands that it should just work.

By moving our S3 Compatible APIs into GA, we’re announcing that you can expect a bug free, highly functional, and stable experience.

Our functionality and associated infrastructure is fully ready for global, exabyte scale business. Nick Craig-Wood, the founder of rclone, put it well. Our Backblaze S3 compatible layer is not just compatible—according to him, it’s “Very compatible.”

“Very Compatible” Means It Works with Your Workflows

The testing suites are helpful, but what matters most is the actual experience customers are having. During the beta period, more than 1,000 unique S3-compatible tools were used by customers to interact with our new Backblaze S3 compatible layer. We know this because of something called a user agent—an identifier that tells our servers what tool is being used to upload data. Our monitoring looks at the system as a whole as well as at user agents to make sure things are performing as planned. And everything is going smoothly!

Tools like MSP360 (formerly CloudBerry Lab), MinIO, and Synology are among the third parties that had pre-existing integrations with Backblaze but now also have customers uploading through our S3 Compatible APIs. Perhaps more notably, customers brought a wide variety of tools that we had not previously seen upload data to Backblaze. User agents from Commvault, Cohesity, and Veeam all register now in our internal reporting. As do many AWS SDKs and even AWS Lambda.

The recurring theme? Customers can point their existing workflows and tools at Backblaze B2 and not miss a beat.

What It Means for Our Customers

During our beta period, we’ve seen literally thousands of success stories. One of our favorites comes from a company called CloudSpot—a software as a service platform offering photographers the easiest way to share their work online.

CloudSpot’s storage infrastructure is a critical component of both their product and P&L. But, with his company scaling, CEO Gavin Wade realized that his data (and company) were captive to Amazon S3. As CloudSpot grew, his storage related costs threatened to turn his business upside down.

With over 700TB stored and data transfer fees starting at 9 cents/GB, Gavin felt stuck inside of Amazon. He had to start cutting back valued functionality for his customers simply because the AWS pricing was untenable.

With B2 Cloud Storage—which is one-fourth of the cost of Amazon S3—Gavin has slashed his cloud bill, freeing up cash for critical investments in his business and team. After seeing a seamless transition for his active workflows, he migrated over 700TB from Amazon S3 to B2 Cloud Storage in less than six days. Most importantly, there was no service disruption.

“With Backblaze, we have a system for scaling infinitely. It lowers our breakeven customer volume while increasing our margins, so we can reinvest back into the business. My investors are happy. I’m happy. It feels incredible.”

—Gavin Wade, Founder & CEO, CloudSpot

The CloudSpot story is a good one, but it’s just one of the many from our beta period. Customers are tired of being taxed to use their data and need a platform that will not punish them for scaling. We’re grateful that these customers are migrating to Backblaze and have accelerated the month over month growth rate of B2 Cloud Storage by more than 25%. Today, we’d like to encourage you to join those that have liberated their data while significantly reducing their costs.

Transcend the Cloud Oligarchy: Backblaze Will Pay for You to Move Your Data

People want storage that will empower their business and they want a provider that doesn’t try to hide fine print or surprise fees. With our S3 Compatible APIs now generally available, they can have both, all while knowing that their tools and workflows will seamlessly integrate with Backblaze B2.

But customers still need a solution for the excessive fees the cloud oligarchy charges for migrating to a cloud that is better for the customer’s business.

Today, we’re proud to remove the last obstacle between you and shaving 75% off your cloud storage costs: Cloud to Cloud Migration.

And for customers that don’t want to commit to storing it for 12 months? No problem, you can still use our service to directly transfer your data from Amazon S3 to Backblaze B2 for 4 cents/GB. By month three, your storage savings will have paid for the migration.

And if you’re interested in trying Backblaze first? Creating an account is free, your first 10GB of storage are free, and there’s never been a better time to start.

Thanks to all the people, companies, and partners that helped make the beta period such a success. We are excited about what the future holds and are glad that you are coming with us.

The post How to Leave AWS: Backblaze S3 Compatible APIs & Free Cloud to Cloud Migration appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Choosing a cloud storage solution that fits your use case is challenging in the best business climate. Throw in the unpredictable economy we’re all trying to navigate during the COVID-19 pandemic, and making the right next step can feel next to impossible.

The simple fact is: The majority of cloud storage providers make it hard to plan for the future by making it difficult to understand how much you’re storing and the details of what it’s costing you.

Our approach to pricing is to be transparent, straightforward, and predictable. For Backblaze B2 Cloud Storage, this means that no matter how much data you have, the cost for Backblaze B2 is $0.005/GB per month for data storage and $0.01/GB for download. There are no costs to upload. We also throw in your first 10GB of storage for free when you open your account.

The complexity of other cloud storage providers is so epic that there are now whole businesses devoted to interpreting their billing. Just ask NASA—literal rocket scientists—who signed on to a cloud storage service without accurately projecting the cost to access their data when they need it. While they might need a panel of experts to understand their next invoice, we don’t think you should be in the same boat.

Cloud Storage Price Comparison

To help you plan for the future of your data storage—and to show you how you can save on your next cloud storage bill—we’ve built the Cloud Storage Pricing Calculator, where anyone can enter in their specific use case and get pricing back for B2, AWS S3, Microsoft Azure, and Google Cloud Storage.

B2 Cost Calculator

Backblaze provides this calculator as an estimate.

Sample Storage Scenarios

A Real World Example

Hypotheticals are nice, but how about a real world example? We’re glad you asked.

Last year, Nodecraft, a gaming company providing a server management platform for multiplayer games, decided to move to B2 Cloud Storage in an effort to reduce prices and improve their service. They migrated 23TBs of data from S3 to Backblaze B2 in only seven hours with zero service interruption. The result? The switch from AWS S3 to Backblaze B2 reduced Nodecraft’s monthly storage and egress costs by 85%.

Looking Forward to Working Together

We hope our Cost Calculator helps you cut through the fog (heh) of cloud storage pricing. If not, we have an incredible team of solution engineers who will help talk you through whatever sort of challenge you’re facing (go ahead, drop them a line).

This is a post about pricing, so that’s what we’ve focussed on here. But we don’t like to dwell on having the most affordable cloud storage on the planet—it’s simply the price we can charge based on the technology we developed, and it just so happens it’s a great deal.

We prefer to focus on having the best cloud storage for your use case, and with architecture that calculated at 11 nines of durability, a whole host of customer success stories, and more than an exabyte of data under management, we’d like to think we’re doing well on this front. But don’t take our word for it: Sign up today (you’ll get a free 10GB of storage) and try it out.

The post Transparency in Cloud Storage Costs: Still Elusive appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

At the beginning of any given quarter, conference rooms fill with managers extolling the virtues of the key objectives and tactics for the next 90 days. I’ve always been amused by the little voice inside my head as I stand up in front of my team for these talks—in the best of times, it’s saying “C’mon… The probability that this goes exactly as planned is something like 2%.” But, you shrug it off because a good plan is one that creates a clear end point, proposes a viable way to get there, and has the agility to adapt when you come into contact with reality. 89 days later, you look back, chuckle a little at the twists and turns…and, hopefully, celebrate a mission accomplished.

That’s in the best of times. Today is nowhere near that. Today, it’s hard enough to know where the world will be in 90 days’ time, let alone where our business goals and initiatives will end up. To quote our CEO, Gleb, who was borrowing an idea from “Frozen 2,” the best thing is to focus on doing “the next right thing.”

So what, exactly, is that? Well, for Backblaze, it’s to remember that we’re Backblaze. Besides being the right thing, an advantage to being a transparent company is that it’s an easy thing to do even in uncertain times. Quite literally, it made a lot of sense to write a post that looks at the marketing and communications challenges for our business during a pandemic. Our hope is that in sharing what is happening behind the scenes, we might help those that are facing similar challenges.

Writing this post, I am very aware that some of our readers might be furloughed, laid off, or have their businesses entirely gone. I, and our team, hope that each day gets a little brighter and you get back to stable, soon. At the least, we hope you find something interesting or thought provoking here. At the most, we hope you’ll find inspiration to tackle whatever the next right thing is for you.

What Do We Know Right Now?

Fundamental demand is unaffected: The earliest data points we have for Backblaze’s business are that the top of funnel demand (web visits, account creates) have yet to be materially affected by the crisis. In March, those growth metrics hit the forecasts we created at the end of 2019.

These results are a bit head-scratching given the sheer dominance of pandemic news everywhere. Intuitively, one would assume data and cloud storage content wouldn’t be as interesting to people. And yet looking at the data, for now at least, the market for cloud storage is unchanged (more on that later).

Decision making by customers is affected: We can use our sales team’s pipeline to look a little deeper into customer behavior. What we’re seeing is that customers we were working on longer term projects with have largely paused or changed their decision-making approach.

Let’s say you’ve been planning some cool enhancement to your infrastructure since the beginning of the year and were just about to start executing. Well, migrating the archives off of those LTO tapes isn’t today’s problem (setting aside the fact that you may not even be allowed in your office to do that actual work). So those types of projects are delayed. “Change” or “enhance” takes a back seat to “stabilize.”

This does mean that new projects are springing up very quickly—all of a sudden you have a fully distributed workforce. Are your team members’ computers being backed up? How are they accessing mission critical data? While final decisions on existing projects have been delayed, new needs are being identified with accelerated timelines.

This is where a marketer’s post to other marketers feels particularly awkward. But Backblaze is a business: we have customers that rely on the services we provide, families that rely on the salaries we pay, and a broader community that enjoys our occasional gifs. In other words, we still have a responsibility to ensure the future success of the business. In a world where very little non-COVID related news seems to be relevant, what should our messaging look like?

What Can We Do?

There are three fundamental paths we could take with our outward messaging:

• 1. 1. We could go dark. Trying to run a for-profit enterprise is odd during a pandemic. That said, infrastructure as a service is an increasingly relevant thing to

  help

  with the problems our customers are dealing with.
• 2. We could just blissfully execute against our pre-existing plans. Our business model is fortunate to be resilient against these types of events. But ignoring the elephant in the room is odd at best and, at worst, offensively tone deaf.
• 1. 3. We could attempt to be in conversation with the moment. Backblaze prides itself on being a transparent company. When other disasters have hit—

  like the 2011 flooding in Thailand

  —we’ve had a unique perspective to share. Just like every company, we’re bordering on obsessive consumption of the news on the pandemic. Unlike a lot of companies, we have statisticians capable of running their own models. However, customers come to us for perspectives on data storage topics, not public health—what unique, value added perspective, if any, can we offer?

Option one feels irresponsible. Option two is troubling on a number of levels. Option three is why this post was written (and will be followed up with similar posts covering the challenges faced by our Supply Chain and People Operations teams).

Tactically, we feel that option three means that we adjust our plans, speak to the moment in whatever manner we’re uniquely able to, monitor the market for when storage news becomes interesting again, and try to remain a good corporate citizen along the way.

How We’ve Resolved to Do It:

Backblaze made the decision to move the majority of our workforce to work from home on March 6^th (prior to California’s “shelter-in-place” orders). Over that weekend, as our business continuity plans kicked in, a number of proposals were floated. Two broad themes emerged in those discussions:

Should we notify our customers? Sending an email out felt like the right thing to do. But what were we notifying people of? Our operations and services weren’t changing from a customer perspective. We generally don’t send out emails that say, “Here’s 500 words on something that will not affect your experience with Backblaze.” Again, our approach was to speak when and where we could add value to our customers’ lives. An immediate note didn’t pass the sniff test.

Do we adjust our marketing plans? In the most literal sense, we wanted to tone down some celebratory messaging we knew was coming. We cleared an exabyte of customer data under management, but it felt prudent to cancel the party (particularly with guests coming in from around the globe). We had a 5,300 word profile in Inc. magazine (it’s a fun read)…but promoting that in the middle of everything else felt odd. But in the end, if we’re working on products and releases that we think will help our customers, we should still get them out to market.

As we worked through those big picture questions, we arrived at guiding principles for ourselves:

• We’re Still Running the Business: Our customers rely on us for their data storage. Every day, more people are looking for solutions and tools that can help them face their new reality. If we have a solution to a customer problem, let’s make sure the customer knows about it.
• Plan for an Eventual Recovery: Most would agree that we are already in a recession. And things are likely to get worse before they get better. That fact can make it hard to plan for the future. So this principle underlines two important realities: admit that there’s a disruption today; believe that things will eventually get better. We can’t know when things will get better. But, by assuming they will, we’ll do our part to create that sunnier tomorrow.
  • This does mean that we should keep releasing our feature/functionality development. While our launches might be somewhat muted, if we have something that will help customers solve problems, let’s try to get it in their hands. (To that end, stay tuned for something big next month!)
• We Don’t Traffic in Fear, Uncertainty, or Doubt (FUD): There’s plenty of FUD to go around right now, and a lot of companies will try and take advantage of it. That’s never been our style, and we’re not going to start now. We will never be predatory, either in action or perception.
• Our Reality Isn’t Everyone’s Reality: Our business might be doing well, and much of our staff might be working from home, but that’s not the case for everyone. Our team may be relatively healthy, but many people and their families are fighting a deadly virus. Our communications need to include a statement or a sentiment that communicates, “This is a strange time. We’re staying safe and productive, and are trying to provide you tools/interesting stories.” Also, we need to review our automated messaging. As a backup company, the word “disaster” is often used in the context of “disaster recovery.” Those sentences could read poorly in today’s environment.
• We’re Still Backblaze: We’re known for providing unique insights and information on our blog that is gleaned from how we operate. We should continue to find ways to share knowledge that might help (or uniquely interest others) from our blog perspective.
  • As part of this, remember that we’re a data storage company, not a health organization. There are many companies—like retailers, restaurants, medical providers—that need to communicate their relative status to the community and customers. Our customers trust that we are doing what’s right to keep their data safe and accessible. Always filter for “why does Backblaze have a value-add perspective?”

When Is the Beginning of Whatever Is Next?

As a business, it’s always important to stay in tune with how our markets and public are feeling. So how do we understand where society stands? With some hope and a lot of science, eventually, we think there will be a vaccine. But what are the steps before then? Will evidence that we’ve flattened the curve bring confidence that we’ve done the right thing and bring people back to more mundane topics?

At some point, we believe that there will be some leading indications that the recovery is starting from a psychological perspective. In that sense, we think that will be when customers will feel “stabilized” and begin thinking again about “changes” and “enhancements.” Here are three things we are monitoring.

• The curve gets flattened in major markets. We can measure the reduction in rates of infection and mortality in California and New York. Two large media markets, but in terms of coverage and where the majority of media lives. Also, two of the initial “hotspots” for COVID in the US. As those two states progress, we believe that the media environment will start covering other news (and that there will be increasing numbers of people receptive to non-COVID information). The sad realities of overlooking mid and small markets are not lost on us. This is just our perception of how public sentiment will go.
• Ability to care for those that are sick—as measured by having enough hospital beds for those that need it. As a society, we are facing many challenges in managing this outbreak. But, just like in wartime, the resilience and abilities of humans should never be underestimated. Factories are converting to produce more sanitizer, masks, and ventilators. The Army Corps of Engineers are converting spaces (auditoriums, arenas, etc.) into hospitals. We can track the projected number of beds needed and the capacity in given locales. When supply meets demand, it will be a good thing.
• Consumers are consuming more non-COVID stories—as measured in week over week podcast listening. The good people at Podtrac are providing statistics on podcast consumption during this time. It’s hard to imagine total consumption returning until people start commuting again, but it should begin to rise as consumer sentiment starts having interest in non-pandemic news.

When I became part of the Backblaze family four years ago, we were a third of the size we are today. The growth and expansion of the business is incredibly energizing, but does bring increased costs around keeping the team on the same page. Writing this post started as an internal memo to help organize our team. We thought it was potentially interesting enough to merit sharing with you. In keeping with our principles, we’ll continue to share our thinking as it evolves and whatever insights we believe might be helpful. We hope you and your family stay as safe and healthy as possible. As always, we’re happy to engage with you in the comments below.

The post Business as (Un)Usual appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

With the impact of coronavirus on all of our lives, it’s been a struggle to find pieces of good news to share. But we wanted to take a break from the usual programming and share a milestone we’re excited about, one that’s more than 12 years in the making.

Since the beginning of Backblaze—back in 2007, when our five co-founders were working out of Brian Wilson’s apartment in Palo Alto—watching the business grow has always been profoundly exciting.

Our team has grown. From five, way back in the Palo Alto days, to 145 today. Our customer base has grown. Today, we have customers in over 160 countries… it’s not so long ago that we were excited about having our 160^th customer.

More than anything else, the data we manage for our customers has grown.

In 2008, not long after our launch, we had 750 customers and thought ten terabytes was a lot of data. But things progressed quickly, and just two years later we reached 10 petabytes of customer data stored (1,000x more). (Good thing we designed for zettabyte-scale cloud architecture!)

By 2014, we were storing 100 petabytes—the equivalent of 11,415 years of HD video.

Years passed, our team grew, the number of customers grew, and—especially after we launched B2 Cloud Storage in 2015—the data grew. At some scale it got harder to contextualize what hundreds and hundreds of petabytes really meant. We like to remember that each byte is part of some individual’s beloved family photos or some organization’s critical data that they’ve entrusted us to protect.

That belief is part of every single Backblaze job description. Here’s how we put it in that context:

Our customers use our services so they can pursue dreams like curing cancer (genome mapping is data-intensive), archive the work of some of the greatest artists on the planet (learn more about how Austin City Limits uses B2), or simply sleep well at night (anyone that’s spilled a cup of coffee on a laptop knows the relief that comes with complete, secure backups).”

It’s critically important for us that we achieved this growth by staying the same in the most important ways: being open & transparent, building a sustainable business, and caring about being good to our customers, partners, community, and team. That’s why I’m excited to announce a huge milestone today—our biggest growth number yet.

We’ve reached 1.

Or, by another measurement, we’ve reached 1,000,000,000,000,000,000.

Yes, today, we’re announcing that we are storing 1 exabyte of customer data.

What does it all mean? Well. If you ask our engineers, not much. They’ve already rocketed past this number mentally and are considering how long it will take to get to a zettabyte (1,000,000,000,000,000,000,000 bytes).

But, while it’s great to keep our eyes on the future, it’s also important to celebrate what milestones mean. Yes, crossing an exabyte of data is another validation of our technology and our sustainably independent business model. But I think it really means that we’re providing value and earning the trust of our customers.

Thank you for putting your trust in us by keeping some of your bytes with us. Particularly in times like these, we know that being able to count on your infrastructure is essential. We’re proud to serve you.

As the world grapples with a pandemic, celebrations seem inappropriate. But we did want to take a moment and share this milestone with you, both for those of you who have been with us over the long haul and in the hopes that it provides a welcome distraction. To that end, we’ve been working on a few things that we’d planned to launch in the coming weeks. We’ve made the decision to push forward with those launches in hopes that the tools may be of some use for you (and, if nothing else, to try to do our part to provide a little entertainment). For today, here’s to our biggest 1 yet. And many more to come.

The post Exabyte Unlocked appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

We’re looking for someone who loves to go deep on making marketing technology actually work: process oriented campaign execution, implementing and interpreting actionable analytics, and—ultimately—blending all of the above together to understand and improve the customer journey. Most importantly, we’re looking for someone who is ready to achieve this alongside a team that shows genuine care for each other.

Backblaze is an extraordinary place to work. With a great reputation for service built over the last 13 years, we are entrusted with nearly an exabyte of customer data, placing us among the largest cloud storage providers on the planet. And we’ve built a significantly profitable business with a total of just $3M in outside investment, empowering us to set our own course and live by our own ideals. Our approach is guided by honesty, transparency, and a commitment to doing the right thing for our customers and coworkers. Importantly, we’ve created an environment that makes us both happy and proud enough to recruit our friends to the team.

And it’s working quite nicely! Our business lines have achieved sustainable, double-digit growth—with some approaching triple digits. And we’ve accomplished these results without running any significant paid acquisition programs. Our customers are happy, and so are our coworkers: In the most recent “Great Place to Work” survey, 99% of our team rated Backblaze as “a great place to work.”

But while there is a lot to celebrate in our past, there is almost as much opportunity ahead of us: Our marketing systems and processes are functional, but relatively basic. We seek a Director of Marketing Operations who can help build, operate, and evolve our approach to executing and understanding our marketing efforts.

More About Backblaze

Backblaze provides cloud storage that’s astonishingly easy to use and low cost. Our customers use our services so they can pursue dreams like curing cancer (genome mapping is data-intensive), archive the work of some of the greatest artists on the planet (learn more about how Austin City Limits uses B2), or simply sleep well at night (anyone that’s spilled a cup of coffee on a laptop knows the relief that comes with complete, secure backups).

How Marketing at Backblaze Works

Our business is almost entirely driven by inbound customer interest. We have a blog that will attract more than 3M visits this year, with zero paid audience acquisition. Our team creates compelling, if slightly wonky, content on how data storage actually works. Our secret is writing enlightening, useful stories about customer problems and the solutions they might employ to succeed. The result is a highly engaged readership that yields a significant number of new customers. Approximately half of our customer base is “self-serve” while the other half works with our sales team to help them get started.

We operate across multiple verticals and sell 2 different SaaS products. To do so, our Marketing team organizes around the following team concepts:

• Product Marketing: For any given vertical, we have at least one Product Marketer acting as the General Manager for her business. The Product Marketers identify their customer personas and expected buyer journeys. They also define the go-to-market activities for their verticals.
• Editorial: Helps refine and polish the messaging for internal and external audiences.
• Design: Brings the messages to life.
• Brand: When you have a question about how our audience will react to thing X, we have a good answer. In addition, the “Brand” team owns our social presence.
• Growth: Acts as both our CRO and SEO functions. We use Optimizely and have dedicated resources to help within Front End Engineering.
• Marketing Ops: Owns our “stack,” processes, and analytics. Given content and strategy, enables elegant execution and provides actionable insights on results.

The Role: Director of Marketing Ops

Reporting directly to the VP of Marketing, you will build out our Marketing Ops function.

We think of Marketing Ops as having 5 key areas of ownership:

• Marketing Technology (aka “the Marketing Stack”): Strategy, evaluation, implementation and operation of our marketing tools.
  • Today, the key pieces of our stack are Hubspot, Optimizely, BrightTalk, AdRoll, Google Adwords (brand terms), JustUno, Google Analytics, and Google Tag Manager.
  • Our company uses Tableau for visualization and Salesforce for Sales Ops.
• Data: Create and implement the strategy for what our systems collect and how they are connected.
• Team Alignment: Collaborate with team leads to establish processes and training for long-term success.
• Campaign Management: Enable our various go-to-market activities by creating processes that are both predictable and scalable; establish clear priorities for Front End Engineering to ensure quality launches.
• Insights: Analytics and analysis of all aspects of funnel with the goal of providing actionable intelligence for the other Marketing teams.

The Roadmap for Marketing Ops

As mentioned at the start of this post, our Marketing Operations are evolving. We believe there is a clear roadmap ahead of us, but one that will take a couple years to execute well. Here are some milestones that we’re looking forward to:

• Grow understanding of the existing funnels. In your first 12 months, the majority of the work will be collaborating with the VP of Marketing, the Business Intelligence team, and the Sales Operations team to create a quantifiable understanding of our funnels. We have the right pieces in place and a company willing to invest in making them work together, but we need to solidify our fundamental systems.
• Optimize existing funnels. Once we refine the understanding of our business drivers, we should create and execute against deliberate tests to identify new opportunities for growth.
  • Example: Once we have insights on what types of content are leading to what types of customers, provide that intelligence to the Product Marketers & Editorial team so they can create more of the desired content. They will create the incremental content, you will be responsible for measuring the results of the experiment.
• Identify new growth initiatives (aka: find the rocket fuel). Understand the existing business, prove we understand it by optimizing it, and then start working on new streams of demand generation.
• Tailor systems and tools to the most effective growth initiatives. Today, we have all the pieces for a functional marketing stack. Let’s get everything working, prove out areas for growth, and then invest in more specialized tools/operations for the areas of growth we want to pursue.

The Right Fit for our Director of Marketing Operations

We take a lot of pride in the culture we’ve built and are looking for team members that are a good fit. For a better understanding of who we are, check out this About Us video. Beyond “cultural fit,” for this specific role, we’re looking for an experienced operator that has a customer-journey-focused understanding of marketing. Here are the role-specific characteristics we’ll be looking for:

• Possess the right amount of experience. You’ve probably been doing Marketing Operations in some form or another for 6-10 years. We don’t need someone with a specific degree or certification. We do need someone that will come in and take control of the existing systems and immediately start moving us forward on the envisioned roadmap.
  • Expertise in at least one of the major Marketing Automation systems is required, with a strong preference for Hubspot experience and/or mastery.
• Skilled at figuring how the puzzle pieces fit together. Different tools in different configurations do different things. We need someone that can work with the larger Marketing team to understand what we’re trying to achieve, design a stack and processes for achieving those outcomes, and then implement and execute with precision.
• Ability to work cross-functionally. The role of Marketing Ops engages with a variety of teams across our company. Beyond the Marketing team, our ideal candidate excels in interactions with:
  • Front End Engineering: Determine the prioritization of what gets worked on when as well as be the “business owner” for questions they might have.
  • Business Intelligence: Collaborate with the BI team to get answers from non-marketing systems, facilitate their requests for access into the marketing systems.
  • Sales Operations: Work with our Sales Ops teams to ensure precision in lead management, seamless systemic handoff, and generally find solutions to challenges as they arise.
  • Executive Team: Provide distilled, big picture takeaways on what’s happening because of your granular understanding of the little things.
• Knows Demand Gen Mechanics for SaaS. Our ideal candidate is focused on generating incremental revenue and understands how to activate/optimize the different stages of the customer journey to achieve that goal.

Some of Our More Popular Perks

Backblaze offers an unlimited vacation policy; generous health/dental/vision/childcare benefits; fully stocked kitchens, twice a week catered breakfast and lunch, and superior coffee; and a generous skills training policy to continue your professional development. Our office in San Mateo is easily accessible from CalTrain, 280, and 101.

If This Sounds Interesting To You

We’d love to learn more about you! Please email us at jobscontact@backblaze.com with your resume. All applications are reviewed by our team. If we think there might be a good fit, our recruiting lead, Michele, will schedule time to chat, answer any questions you might have, and outline our process.

The post Wanted: Director of Marketing Operations appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Imagine a globe spinning (or simply look at the top of this blog post). When you start out on a data center search, you could consider almost any corner of the globe. For Backblaze, we knew we wanted to find an anchor location in the European Union. For a variety of reasons, we quickly narrowed in on Amsterdam, Brussels and Dublin as the most likely locations. While we were able to generate a list of 40 qualified locations, narrowed it down to ten for physical visits, and then narrowed it yet again to three finalists, the question remained: How would we choose our ultimate partner? Data center searches have changed a lot since 2012 when we circulated our RFP for a previous expansion.

The good news is we knew our top line requirements would be met. Thinking back to the 2×2 that our Chief Cloud Officer, Tim Nufire, had drawn on the board at the early stages of our search, we felt good that we had weighed the tradeoffs appropriately.

Similarly to hiring an employee, after the screening and the interviews, one runs reference checks. In the case of data centers, that means both validating certain assertions and going into the gory details on certain operational capabilities. For example, in our second post in the EU DC series, we mentioned environmental risks. If one is looking to reduce the probability of catastrophe, making sure that your DC is outside of a flood zone is generally advisable. Of course, the best environmental risk factor reports are much more nuanced and account for changes in the environment.

To help us investigate those sorts of issues, we partnered with PTS Consulting. By engaging with third party experts, we get dispassionate, unbiased, thorough reporting about the locations we are considering. Based on PTS’s reporting, we eliminated one of our finalists. To be clear, there was nothing inherently wrong with the finalist, but it was unlikely that particular location would sustainably meet our long term requirements without significant infrastructure upgrades on their end.

In our prior posts, we mentioned another partner, UpStack. Their platform helped us with the sourcing and narrowing down to a list of finalists. Importantly, their advisory services were crucial in this final stage of diligence. Specifically, UpStack brought in electrical engineering expertise to give us a deep, detailed assessment of the electrical mechanical single line diagrams. For those less versed in the aspects of DC power, that means UpStack was able to go into incredible granularity in looking at the reliability and durability of the power sources of our DCs.

Ultimately, it came down to two finalists:

• DC 3: Interxion Amsterdam
• DC 4: The pre-trip favorite

DC four had a lot of things going for it. The pricing was the most affordable and the facility had more modern features and functionality. The biggest downsides were open issues around sourcing and training what would become our remote hands team.

Which gets us back to our matrix of tradeoffs. While more expensive than DC three, Interxion facility graded out equally well during diligence. Ultimately, the people at Interxion and confidence in the ability to build out a sturdy remote hands team made the choice of Interxion clear.

Looking back at Tim’s 2×2, DC four presented as financially more affordable, but operationally a little more risky (since we had questions about our ability to effectively operate on a day to day basis).

Interxion, while a little more financially expensive, reduced our operational risks. When thinking of our anchor location in Europe, that felt like the right tradeoff to be making.

Ready, Set, More Work!

The site selection only represented part of the journey. In parallel, our sourcing team has had to learn how to get pods and drives into Europe. Our Tech Ops & Engineering teams have worked through any number of issues around latency, performance, and functionality. Finance & Legal has worked through the implications of having a physical international footprint. And that’s just to name a few things.

If you’re in the EU, we’ll be at IBC 2019 in Amsterdam from September 13 to September 17. If you’re interested in making an appointment to chat further, use our form to reserve a time at IBC, or drop by stand 7.D67 at IBC (our friends from Cantemo are hosting us). Or, if you prefer, feel free to leave any questions in the comments below!

The post Making the Data Center Choice (the Work Has Just Begun) appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Ten locations, three countries, three days. Even the hardest working person in show business wouldn’t take on challenges like that. But for our COO, John Tran, and UpStack’s CEO, Chris Trapp, that’s exactly what they decided to do.

In yesterday’s post, we discussed the path to getting 40 bids from vendors that could meet our criteria for our new European data center (DC). This was a remarkable accomplishment in itself, but still only part way to our objective of actually opening a DC. We needed to narrow down the list.

With help from UpStack, we began to filter the list based on some qualitative characteristics: vendor reputation, vendor business focus, etc. Chris managed to get us down to a list of 10. The wonders of technology today, like the UpStack platform, help people get more information and cast wider nets then at any other time in human history. The downside of that is you get a lot of information on paper, but that is a poor substitute to what you can gather in person. If you’re looking for a good, long term partner then understanding things like how they operate and their company DNA is imperative to finding the right match. So, to find our newest partner, we needed to go for a trip.

Chris took the lead on booking appointments. The majority of the shortlist clustered in the Netherlands and Ireland. The others were in Belgium and with the magic of Google Maps, one could begin to envision an efficient trip to all three countries. The feeling was it could all be done with just three days on the ground in Europe. Going in, they knew it would be a compressed schedule and that they would be on the move. As experienced travelers, they brought small bags that easily fit in the overhead and the right power adapters.

Hitting the Road

On July 23rd, 2018, John left San Francisco International Airport (SFO) at 7:40 a.m. on a non-stop to Amsterdam. Taking into account the 5,448 miles between the two cities and the time change, John landed at Amsterdam Airport Schiphol (AMS) hours at 7:35 a.m. on July 24th. He would land back home on July 27th at 6:45 p.m.

Tuesday (Day One)

The first day officially started when John’s redeye touched down in Amsterdam at 7:35 a.m. local. Thankfully, Chris’ flight from New York’s La Guardia was also on time. With both flights on time, they were able to meet at the airport: literally, for they had never met before.

Both adjourned to the airport men’s room to change out of their travel clothes and into their suits — choosing a data center is serious business, after all. While airport bathroom changes are best left for spy novels, John and Chris made short work of it and headed to the rental car area.

That day, they’ll ended up touring four DCs. One of the biggest takeaways of the trip was that it turned out visiting data centers is similar to wine tasting. While some of the differences can be divined from the specs on paper, when trying to figure out the difference between A and B, it’s very helpful to compare side by side. Also similar to wine tasting, there’s a fine line between understanding nuances between multiple things and it all starting to blend together. In both cases, after a full day of doing it, you feel like you probably shouldn’t operate heavy machinery.

On day one, our team saw a wide range of options. The physical plant is itself one area of differentiation. While we have requirements for things like power, bandwidth, and security, there’s still a lot of room for tradeoffs among those DCs that exceed the requirement. And that’s just the physical space. The first phase of successful screening (discussed in our prior post) is being effective at examining non-emotional decision variables — specs, price, reputation — but not the people. Every DC is staffed by human beings and cultural fit is important with any partnership. Throughout the day, one of the biggest differences we noticed was the culture of each specific DC.

The third stop of the day was Interxion Amsterdam. While we didn’t know it at the time, they would end up being our partner of choice. On paper, it was clear that Interxion would be a contender. Its impressive facility meets all our requirements and, by happenstance, happens to have a footprint available that is almost exactly to the spec of what we were looking for. During our visit, the facility was impressive, as expected. But the connection we felt with the team there would prove to be the thing that would ultimately be the difference.

After leaving the last DC tour around 7pm, our team drove from Amsterdam to Brussels. Day 2 would be another morning start and, after arriving in Brussels a little after 9pm, they had earned some rest!

Insider Tip: Earlier in his career, John had spent a good amount of time in Europe and, specifically, Brussels. One of his favorite spots is the Grand Place (Brussels’ Central Market). If in the neighborhood, he recommends you go and enjoy a Belgium beer sitting at one of the restaurants in the market. The smart move is to take the advice. Chris, newer to Brussels, gave John’s tour a favorable TripAdvisor rating.

Wednesday (Day Two)

After getting a well-deserved couple hours of sleep, the day officially started with an 8:30 a.m. meeting for the first DC of the day. Major DC operators generally have multiple locations and DCs five and six are operated by companies that also operate sites visited on day one. It was remarkable, culturally, to compare the teams and operational variability across multiple locations. Even within the same company, teams at different locations have unique personalities and operating styles, which all serves to reinforce the need to physically visit your proposed partners before making a decision.

After two morning DC visits, John and Chris hustled to the Brussels airport to catch their flight to Dublin. At some point during the drive, it was realized that tickets to Dublin hadn’t actually been purchased. Smartphones and connectivity are transformative on road trips like this.

The flight itself was uneventful. When they landed, they got to the rental car area and their car was waiting for them. Oh, by the way, minor detail but the steering wheel was on the wrong side of the car! Chris buckled in tightly and John had flashbacks of driver’s ed having never driven on the right side of the car. Shortly after leaving the airport, it was realized that one also drives on the left side of the road in Ireland. Smartphones and connectivity were not required for this discovery. Thankfully, the drive was uneventful and the hotel was reached without incident. After work and family check ins, another day was put on the books.

Our team checked into their hotel and headed over to the Brazen Head Pub for dinner. Ireland’s oldest pub is worth the visit. It’s here that we come across our it really is a small world nomination for the trip. After starting a conversation with their neighbors at dinner, our team was asked what they were doing in Dublin. John introduced himself as Backblaze’s COO and the conversation seemed to cool a bit. Apparently their neighbor was someone from another large cloud storage provider. Apparently, not all companies like sharing information as much as we do.

Thursday (Day Three)

The day again started with an 8:30 a.m. hotel departure. Bear in mind, during all of this, John and Chris both had their day jobs and families back home to stay in touch with. Today would feature four DC tours. One interesting note about the trip: operating a data center requires a fair amount of infrastructure. In a perfect world, power and bandwidth come in at multiple locations from multiple vendors. This often causes DCs to cluster around infrastructure hubs. Today’s first two DCs were across the street from one another. We’re assuming, but could not verify, a fierce inter-company football rivalry.

While walking across the street was interesting, in the case of the final two DCs, they literally shared the same space; the smaller provider subleasing space from the larger. Here, again, the operating personalities differentiated the companies. It’s not necessarily that one was worse than the other, it is a question of whom you think will be a better partnership match for your own style. In this case, the smaller of the two providers stood out because of the passion and enthusiasm we felt from the team there, and it didn’t hurt that they are long time Hard Drive Stats enthusiasts (flattery will get you everywhere!).

While the trip, and this post, were focused on finding our new DC location, opening up our first physical operations outside of the U.S. had any number of business ramifications. As such, John made sure to swing by the local office of our global accounting firm to take the opportunity to get to know them.

The meeting wrapped up just in time for Chris and John to make it to the Guinness factory by 6:15 p.m. Upon arrival, it was then realized that the last entry into the Guinness factory is 6 p.m. Smartphones and connectivity really can be transformative on road trips like this. All that said, without implicating any of the specific actors, our fearless travelers managed to finagle their way in and could file the report home that they were able to grab a pint or two at St. James’ place.

The team would leave for their respective homes early the next morning. John made it back to California in time for a (late) dinner with his family and a well earned weekend.

After a long, productive trip, we had our list of the three finalists. Tomorrow, we’ll discuss how we narrowed it down from three to one. Until then, slainte (cheers)!

The post The Logistics of Finding the Right Data Center: The Great European (Non) Vacation appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

There’s an old saying, “How do you eat an elephant? One bite at a time.” The best way to tackle big problems is to simplify as much as you can.

In our case, with almost an exabyte of customer data under management and customers in over 160 countries, expanding the geographic footprint of our data centers (DCs) has been a frequently discussed topic. Prior to opening up EU Central, we had three DCs, but all in the western U.S. The topic of opening a DC in Europe is not a new one within Backblaze, but going from idea to storing customer data can be a long journey.

As our team gathered to prioritize the global roadmap, the first question was an obvious one: Why do we want to open a DC in Europe? The answer was simple: Customer demand.

While nearly 15 percent of our existing customer base already resides in Europe, the requests for an EU DC come from citizens around the globe. Why?

• Customers like keeping data in multiple geographies. Doing so is in line with the best practices of backup (long before there was a cloud, there was still 3-2-1).
• Geopolitical/regulatory concerns. For any number of reasons, customers may prefer or be required to store data in certain physical locations.
• Performance concerns. While we enjoy a debate about the effects of latency for most storage use cases, the reality is many customers want a copy of their data as physically close to where it’s being used as possible.

With the need established, the next question was predictably obvious: How are we going to go about this? Our three existing DCs are all in the same timezone as our California headquarters. Logistically, opening and operating a DC that has somewhere around an eight hour time difference from our headquarters felt like a significant undertaking.

Organizing the Search for the Right Data Center

To help get us organized, our co-founder and Chief Cloud Officer, Tim Nufire, drew the following on a whiteboard.

This basic matrix frames the challenge well. If one were willing to accept infinite risk (have customers write to scrolls and “upload” via sealed bottle transported across the ocean), we’d have low financial and effort investments outlays to open the data enter. However, we’re not in the business of accepting infinite risk. So we wanted to achieve a low risk environment for data storage while sustaining our cost advantage for our customers.

But things get much more nuanced once you start digging in.

Risks

There are multiple risk factors to consider when selecting a DC. Some of the leading ones are:

• Environmental: One could choose a DC in the middle of a floodplain, but, with few exceptions, most DCs don’t work well underwater. We needed to find an area to minimize adverse environmental impact.
• Political: DCs are physical places. Physical places are governed by some form of nation state. Some customers want (or need) their data to be stored within certain regulatory or diplomatic parameters. In the case of the requests for opening a DC in Europe, many of our customers want their data to be inside of the European Union (EU). That requirement strikes Switzerland off our list. For similar reasons, another requirement we imposed was operating inside of a country that is a NATO member. Regrettably, that eliminated any location inside of Finland. Our customers want EU, not Europe.
• Financial: By opening a DC in Europe, we will be conducting business with a partner that expects to be paid in euros. As an American company, we primarily operate in dollars. So now the simple timing of when we pay our bills may change the cost (depending on exchange rate fluctuations).

Costs

The other dimension on the board was costs, expressed as Affordable to Expensive. Costs can be thought of both as financial as well as effort:

• Operating Efficiency: Generally speaking, the climate of the geography will have an effect on the heating/cooling costs. We needed to understand climate nuances across a broad geographic area.
• Cost of Inputs: Power costs vary widely, often due to fuel sources having different availability at a local level. For example, nuclear power is generally cheaper than fossil fuel, but may not be available in a given region. Complicating things is that power source X may cost one thing in the first country, but something totally different in the next. Our DC negotiations may be for physical space, but we needed to understand our total cost of ownership.
• Staffing: Some DCs provide remote hands (contract labor) while others expect us to provide our own staffing. We needed to get up to speed on labor laws and talent pools in desired regions.

Trying to Push Forward

We’re fortunate to have a great team of Operations people that have earned expertise in the field. So with the desire to find a DC in the EU, a working group formed to explore our options. A little while later, when the internal memo circulated, the summary in the body of the email jumped out:

“It could take 6-12 months from project kick-off to bring a new EU data center online.”

That’s a significant project for any company. In addition, the time range was sufficiently wide to indicate the number of unknowns in play. We were faced with a difficult decision: How can we move forward on a project with so many unknowns?

While this wouldn’t be our first data center search, prior experience told us we had many more unknowns in front of us. Our most recent facility searches mainly involved coordinating with known vendors to obtain facility reports and pricing for comparison. Even with known vendors, this process involved significant resources from Backblaze to relay requirements to various DC sales reps and to take disparate quotes and create some sort of comparison. All DCs will quote you $/Kilowatt Hour or $/kWh, but there is no standard definition of what is and isn’t included in that. Generally speaking, a DC contract has unit costs that decline as usage goes up. So is the $/kWh in a given quote the blended lifetime cost? Year one? Year five? Adding to this complexity would be all the variables discussed above (and more).

Serendipity Strikes: UpStack

Despite the obstacles in our path, our team committed to finding a location inside the EU that makes sense for both our customers’ needs and our business model. We have an experienced team that has demonstrated the ability to source and vet DCs already. That said, our experienced team were already quite busy with their day jobs. This project looked to come at a significant opportunity cost as it would fully occupy a number of people for an extended period of time.

At the same time as we were trying to work through the internal resource planning, our CEO happened across an interesting article from our friends at Data Center Knowledge; they were covering a startup called UpStack (“Kayak for data center services”). The premise was intriguing — the UpStack platform is designed to gather and normalize quotes from qualified vendors for relevant opportunities. Minimizing friction for bidding DCs and Backblaze would enable both sides to find the right fit. Intrigued, we reached out to their CEO, Chris Trapp.

UpStack is a free, vendor-neutral data center sourcing platform that allows businesses to analyze and compare level-set pricing and specifications in markets around the world. Find them at upstack.com.

We were immediately impressed with how easy the user experience was on our side. Knowing how much effort goes into normalizing the data from various DCs, having a DC shopping experience comparable to that of searching for plane tickets was mind blowing. With a plane ticket, you might search for number of stops and layover airports. With UpStack, we were able to search for connectivity to existing bandwidth providers, compliance certifications, and location before asking for pricing.

Once vendors returned pricing, UpStack’s application made it easy to compare specifications and pricing on an apples-to-apples basis. This price normalization was a huge advantage for us as it saved many hours of work usually spent converting quotes into pricing models simply for comparison sake. We have the expertise to do what UpStack does, but we also know how much time that takes us. Being able to leverage a trusted partner was a tremendous value add for Backblaze.

Narrowing Down The Options

With the benefit of the UpStack platform, we were able to cast a much wider net than would have been viable hopping on phone calls from California.

We specified our load ramp. There’s a finite amount of data that will flow into the new DC on day one, and it only grows from there. So part of the pricing negotiation is agreeing to deploy a minimum amount of racks on day one, a minimum by the end of year one, and so on. In return for the guaranteed revenue, the DCs return pricing based on those deployments. Based on the forecasted storage needs, UpStack’s tool then translates that into estimated power needs so vendors can return bids based on estimated usage. This is an important change from how things are usually done; many quotes otherwise price based on the top estimated usage or a vendor-imposed minimum. By basing quotes off of one common forecast, we could get the pricing that fits our needs.

There are many more efficiencies that UpStack provides us and we’d encourage you to visit their site at https://upstack.com to learn more. The punchline is that we were able to create a shortlist of the DCs that fit our requirements; we received 40 quotes provided by 40 data centers in 10 markets for evaluation. This was a blessing and a curse, as we were able to cast a wider net and learn about more qualified vendors than we thought possible, but a list of 40 needed to be narrowed down.

Based on our cost/risk framework, we narrowed it down to the 10 DCs that we felt gave us our best shot to end up with a low cost, low risk partner. With all the legwork done, it was time to go visit. To learn more about our three country trip to 10 facilities that lasted less than 72 hours, tune in tomorrow. Same bat time, same bat station.

The post Getting Ready to Go to Europe appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.