From Cyclones to Cybercriminals: How One University Protects Data in Tornado Alley

Protecting Five Decades of University Data

As the director of IT infrastructure and security for Pittsburg State University in Kansas, Tim Pearson knows all too well the absolute necessity for universities to invest in comprehensive data protection. However, unlike most IT directors, he has some added insight that comes from the other hat he wears: that of a fire chief.

‍

Pearson has volunteered as the fire chief for Carterville, Missouri for several years. He was on the ground the night an intensely destructive tornado struck Joplin, Missouri in May 2011. “We were up close and personal with the destruction, and being an IT guy, I was also in contact with a lot of my peers in Joplin who had lost their data centers,” Pearson recalled.

‍

While lots of areas of the country face the destructive potential of strong tornadoes, and most areas are at risk of at least some kind of natural disaster, Tornado Alley is uniquely at risk. States in the heart of the region rank in the top 10 list of states with the highest average number of tornadoes per year. That’s where Pitt State sits, right in the southeast corner of Kansas, second on the list. Disaster planning is non-negotiable, and Pearson knows that better than anyone.

‍

As a self-professed “belt and suspenders kind of guy” when it comes to data protection, Pearson takes a no-excuses approach based on years of experience and near misses. “I have been in IT long enough to remember when hard drives were nowhere near as reliable as they are today. When I started my career, we had some real near-disaster scenarios, and I vowed I would do everything I could to prevent that from ever happening again,” he said.

Tornado-Proofing With a Private Cloud

For Pearson, preventing disasters started with deploying private cloud infrastructure via Commvault Distributed Storage that could withstand the threat of a tornado while also aggressively managing costs.

‍

He established a three-node system with two data centers on campus and one across the state at another university within the Kansas Research and Educational Network—a nonprofit that offers an incredibly high-speed, low-latency connection between the state’s six universities. The two on-campus nodes run an active-active split with half of their compute infrastructure running on each node, but both data centers have enough horsepower to run their entire workload in the event one were to go down. The third node hosts a third copy of their data for geographic separation, and enough compute horsepower to run critical services should they need to.

‍

The three nodes sync in real time, so Pitt State has three updated copies of their data as production runs throughout the day. “We didn’t want to depend on the previous night’s backup should anything terrible happen,” Pearson noted. They have real-time coverage throughout the day right up to the moment of failure, and they’d have to lose three different locations, one of which is across the state, to incur any data loss.

‍

Notably, they implemented it all at a fraction of the cost of using a diversified public cloud provider for all of their needs. “We think there needs to be a business case for moving to the cloud,” Pearson explained. “Which is why we deployed a private cloud instead of moving our workload to Amazon. We’re saving over six figures every year by deploying our own private cloud with a similar level of survivability.” Pearson readily admits he has a sign on his office wall that reads, “There is no cloud. It’s just someone else’s computer.”

Responding to Data Integrity Warning Signals With Immutability

With his innovative private cloud in place, Pearson felt disaster recovery was well in hand. Even if a mile-wide tornado took out both on-campus data centers, the third node would allow them to continue functioning. The next challenge Pearson sought to solve was data integrity: What happens if someone accidentally deletes half a million records or the university suffers some kind of data breach?

‍

“The techniques that work well for disaster recovery don’t necessarily work well for protecting the integrity of data,” Pearson explained. He had good reason to be concerned. Universities store highly sensitive, personally identifiable data, and cybercriminals know it. Emsisoft, a cybersecurity company, reported that over 2,000 educational, government, and medical institutions were hit by ransomware attacks in 2021. That figure includes 26 colleges. The attacks also caused operational interruptions and at least 118 data breaches.

‍

For the task of protecting data integrity, “Nothing can beat immutability at the file system,” Pearson said. In short, he wanted to create immutability with an air gap between backups and the active network by storing data on something that was either physically or virtually isolated.

‍

Pearson explored network-attached storage (NAS) solutions which he would then have to physically disconnect from the network to create a physical air gap. It appealed to his do-it-yourself mentality but he knew the manual lift would be cumbersome. Though he avoided cloud-based solutions in the past, he recognized this as an area where the business case was strong. He started looking into cloud storage to strengthen Pitt State’s data integrity.

Object Lock Makes a Case for the Cloud

Pearson didn’t have to look far to find a cloud solution he thought might work. As an amateur videographer, Pearson already used Backblaze to back up his personal files. He knew it to be easy and cost-effective. He reached out to learn more about storing Pitt State’s backup data in Backblaze B2 with Object Lock.

‍

Object Lock is a powerful backup protection tool that prevents a file from being altered or deleted until a given date. It stores data using a Write Once, Read Many (WORM) model, meaning after it’s written, data cannot be modified or deleted for a defined period of time set by the admin at the outset. Any attempts to manipulate, copy, encrypt, change, or delete the file will fail during that time. The files may be accessed, but no one can change them, including the file owner or whoever set the Object Lock.

‍

That level of protection appealed to Pearson. “We always try to look at worst case scenarios,” he said. “The worst scenario I can think of is if all of our other protections fail—multifactor authentication, login groups, firewalls, VPNs—and a bad guy somehow gets my backup admin’s credentials. Even if that happens, with Object Lock, they can’t alter the data.” Convinced it provided the data integrity protection he was looking for, Pearson set up Backblaze and integrated it as a destination for his Commvault backups in less than a day. Now, Commvault de-duplicates the data, then sends a copy to Backblaze nightly where it’s stored in a bucket protected by Object Lock, giving them four times the protection combined with the three nodes on-premises.

Pitt State Gets the Best of the Cloud With Backblaze B2

For Pearson to invest the university’s dollars in a solution, there had better be a bulletproof business case, especially when it comes to moving anything to the cloud. For this specific use case, Backblaze B2 Cloud Storage stood up to Pearson’s measured decision-making process. It effectively and affordably filled the university’s need for two key pieces of the data integrity puzzle—immutability and geographic diversity. “Backblaze B2 had the capability we lacked,” he confirmed. “I bolted it onto our system, so now I have off-site backup that is safe and well-protected from even a regional disaster in Kansas as well as ransomware protection with Object Lock.”

‍

Now, Pearson is free to focus on the next puzzle—expanding the university’s compute footprint to harden up their disaster recovery even more. What’s more, Pearson says he appreciated Backblaze’s responsiveness and overall culture. He cited the Backblaze solution engineers’ enthusiasm, outside-the-box thinking, and plain-spoken approach as traits that made the process a pleasure instead of just another box to be checked off, or worse, an exercise in corporate puffery.

‍

In addition to the obvious and direct benefits of redundant backups and data integrity, Pearson says Backblaze along with the Commvault private cloud is insurance against future shifts in the wind (not to mention actual tornadoes). “There’s a lot of upheaval as higher education tries to figure out what’s going to be the new normal for delivering quality education to students,” he said. “We’re trying to stay agile enough that we’ll be able to take advantage of opportunities as they present themselves, and adding Backblaze was one of those opportunities.”

‍