In the github example code, the shards contain the file size (a 4 byte integer) followed by the file data, with each shard padded to a word (4 byte int) boundary. The example code assumes the file will fit in a buffer. I assume actual server code would use a larger file size parameter (8 byte integer), and would be able to handle large files that don’t fit in memory.

Java is responsible for 91%* of
security attacks. Backblaze’s code is native to Mac and PC and doesn’t use Java.”

Here are some single-threaded benchmarks of other open-source (BSD licensed) packages for error correction codes at the same small sizes as in the article:

http://github.com/catid/longhair/
Longhair Encoded k=17 data blocks with m=3 recovery blocks in 248.894 usec : 4371.34 MB/s
Longhair Decoded 3 erasures in 396.618 usec : 2743.19 MB/s

http://github.com/catid/cm256/
Encoder: 64000 bytes k = 17 m = 3 : 408.348 usec, 2664.4 MBps
Decoder: 64000 bytes k = 17 m = 3 : 232.032 usec, 4689 MBps

http://github.com/catid/leopard/
Leopard Encoder(1.088 MB in 17 pieces, 3 losses): Input=5258.58 MB/s,
Output=927.985 MB/s
Leopard Decoder(1.088 MB in 17 pieces, 3 losses): Input=1162.52 MB/s,
Output=205.15 MB/s

http://github.com/catid/fecal/
FEC-AL Encoder(1.088 MB in 17 pieces, 3 losses): Input=1528.07 MB/s,
Output=269.659 MB/s, (Encode create: 2022.57 MB/s)
FEC-AL Decoder(1.088 MB in 17 pieces, 3 losses): Input=1557.02 MB/s,
Output=274.769 MB/s, (Overhead = 0 pieces)

And I’ll provide a description of each for your convenience:

Leopard-RS *new*: O(K Log M) FFT MDS Reed-Solomon codec

12x faster than existing MDS approaches to encode, and almost 5x faster to decode. This uses a recent result from 2014 introducing a novel polynomial basis permitting FFT over fast Galois fields. This is an MDS Reed-Solomon similar to Jerasure, Zfec, ISA-L, etc, but much faster. It requires SSSE3 or newer Intel instruction sets for this speed. Otherwise it runs much slower. Requires data is a multiple of 64 bytes. This type of software gets slower as O(K Log M) where K = input count, M = recovery count. It is practical for extremely large data. There is no other software available online for this type of error correction code.

CM256: Traditional O(N^2) Cauchy matrix MDS Reed-Solomon codec

This is an MDS code that uses a Cauchy matrix for structure. Other examples of this type would be most MDS Reed-Solomon codecs online: Jerasure, Zfec, ISA-L, etc. It requires SSSE3 or newer Intel instruction sets for this speed. Otherwise it runs much slower. This type of software gets slower as O(K*M) where K = input count and M = recovery count. It is practical for either small data or small recovery set up to 255 pieces.

It is available for production use under BSD license here: http://github.com/catid/cm256 (Note that the inner loops can be optimized more by applying the GF256 library.)

Longhair: Binary O(N^2) Cauchy matrix MDS Reed-Solomon codec

This is an MDS code that uses a Cauchy matrix for structure. This one only requires XOR operations so it can run fast on low-end processors. Requires data is a multiple of 8 bytes. This type of software gets slower as O(K*M) where K = input count and M = recovery count. It is practical for either small data or small recovery set up to 255 pieces. There is no other optimized software available online for this type of error correction code. There is a slow version available in the Jerasure software library.

It is available for production use under BSD license here: http://github.com/catid/longhair (Note that the inner loops can be optimized more by applying the GF256 library.)

Wirehair: O(N) Hybrid LDPC Erasure Code

Does not get slower as the data size increases. This is not an MDS code. It has about a 3% chance of failing to recover and requiring one extra block of data. It uses mostly XOR so it only gets a little slower on lower-end processors. This type of software gets slower as O(K) where K = input count. This library incorporates some novel ideas that are unpublished. The new ideas are described in the source code. It is practical for data up to 64,000 pieces and can be used as a “fountain” code. There is no other optimized software available online for this type of error correction code. I believe there are some public (slow) implementations of Raptor codes available online for study.

It is available for production use under BSD license here: http://github.com/catid/wirehair

There’s a pre-production version that needs more work here using GF256 for more speed, which is what I used for the benchmark: http://github.com/catid/wh256

FEC-AL *new*: O(N^2/8) XOR Structured Convolutional Matrix Code

This is not an MDS code. It has about a 1% chance of failing to recover and requiring one extra block of data. This library incorporates some novel ideas that are unpublished. The new ideas are described in the README. It uses mostly XOR operations so only gets about 2-4x slower on lower-end processors. It gets slower as O(K*M/8) for larger data, bounded by the speed of XOR. This new approach is ideal for streaming erasure codes; two implementations are offered one for files and another for real-time streaming reliable data. It is practical for data up to about 4,000 pieces and can be used as a “fountain” code. There is no other software available online for this type of error correction code.

It is available for production use under BSD license here: http://github.com/catid/fecal

It can also be used as a convolutional streaming code here for e.g. rUDP: http://github.com/catid/siamese