Large Files

Large files can range from 5 MB to 10 TB which is well beyond the 5 GB limit on normal files. You must upload files individually, and you cannot upload a folder full of files.

Primarily, the 5 GB limit on standard files is there because of how much you can reliably upload with a single HTTP request in a reasonable amount of time. Large files are created by assembling parts; each part can either be uploaded or copied from existing files in any bucket that belongs to the same account as the large file. This means that a large file can be assembled from a mix of uploaded and copied parts.

Parts of a large file can be uploaded and copied in parallel, which can significantly reduce the time it takes to upload terabytes of data. Each part can be anywhere from 5 MB to 5 GB, and you can pick the size that is most convenient for your application. For best upload performance, Backblaze recommends that you use the recommendedPartSize parameter that is returned by the b2_authorize_account operation.

To upload larger files and data sets, you can use the command-line interface (CLI), the Native API, or an integration, such as Cyberduck.

Usage for Large Files

Generally, large files are treated the same as small files. The costs for the API calls are the same.

You are charged for storage for the parts that you uploaded or copied. Usage is counted from the time the part is stored. When you call the b2_finish_large_file operation, the parts are combined into one large file, but the number of bytes that are stored remains the same so it does not affect the charge for the storage.

SHA-1 Checksums

When you upload large files to the Backblaze B2 servers, they are broken up into smaller parts (using the b2 upload_part call). Each part of the file is assigned a secure hash algorithm 1 (SHA-1) checksum. The entire large file itself is not automatically assigned a SHA-1.

Large files do not require SHA-1 checksums on the entire file, but Backblaze recommends that you add one. You should add the SHA-1 in the fileInfo parameter during the call to the b2_start_large_file operation. Specify large_file_sha1 in one of the keys, and specify a 40 byte hex string that represents the SHA-1 as the value.

When you upload a file part, you must provide a SHA-1 checksum. This value is used to validate the data that is sent and is stored internally with that part.

When you download a large file, or a range of a large file, there is no checksum for the entire file, so the string none is returned in the X-Bz-Content-Sha1 header.

Limits

Large files can range in size from 5 MB to 10 TB.

Each large file must consist of at least two parts, and all of the parts except the last one must be at least 5 MB. The last part must contain at least one byte.