Second, more importantly it hugely relates to how often you scrub the disks to check for errors. If, for example, you let the disks sit for one year without checking for errors, then one year later there is a high chance yo suddenly find more than 3 disk failures, and lose data. If you scrub the disk in a theoretical impossible once per second there is no chance you find more than one disk failure at a time. The truth is some where in between, depending on how often you scrub. My calculation shows you need to scrub all disks every 2 days to achieve 11 nines. Which is impossible to achieve. If you scrub every quarter. Since each quarter is 90/2 = 45 times longer, the chance of data loss is 45^3 = 91125 times higher than 11 nines, or the data resilience is only 6 nines in reality, not 11 nines.

Bottom line, your 17+3 erasure coding is just not good enough. Your first catastrophic customer data loss probably will likely happen within the next three years, based on the scale of your current customer data. You need a more powerful erasure coding than the Reed-Solomon you implemented in Java currently. Talk to me. I will show you how to spend less over head, scrub no more than once a year, and yet achieve better than 11 nines resilience. You cannot achieve that with the legacy Reed-Solomon codes.

Your model
assumes that you are back to square one with all 20 drives running at
the beginning of each 56 frames in a year, and it is
clearly not true.

The proper way of analyzing such systems is to use Markov Chains, which is available in the literature: https://www.researchgate.net/publication/2680057_MDS_Disk_Array_Reliability
If
you compare the results in this paper with yours, you will notice that
you are overestimating the reliability which is a result of the
implicit assumption I discussed above.

Let me know if you disagree.

But… Do you have any stats on files or objects that have had such data failures? Have you seen any in the wild or do you intentionally create some as part of your code deploy process to see what happens and how the process from recovery failure as well as recovery successes but delayed accessibility during the process, are handled at the various layers up to the UI?

I’ve seen A/C failures – usually more that one A/C unit goes offline for various reasons – and the remaining A/C units can’t maintain the desired temperature. Everything (in the datacenter) continues to work without any indication of issues. The failed A/C units are put back online. And then over the next 5 to 10 days, you’ll see a “rash” of disk drive failures. The failure rate will usually follow a “bell curve” type distribution. During the peak of the bell curve, when many of the failures occur in a relatively short timeframe – you usually have to hustle to replace the failed drives so that you don’t have a cascading type failure – where more that one device will fail in a RAID 5 config. [yes – I understand that you guys are not using RAID 5] My point is, that to maximize the probability of being able to resolve this “A/C Event” without data loss, you have a relatively short window in which to swap out the failed drives. This is generally referred to as MTTR (Mean Time To Repair).

The big issue I see from what you have described is your MTTR number – 1Tb in 24 hours. That is *way* too long – especially considering that you’re moving to 14Tb drives.

What I don’t understand it why that MTTR is excessive (?). Modern disk drives can move *approximately* (insert nit-picking argument here – but I say *approximately*) 200Mb/Sec. So .. again approximately 1Tb in 90 minutes. And I’m assuming a minimum of 10Gbit/Sec for your ethernet interconnections.

IMHO – this is your current challenge. To drive down your MTTR.