Upload Files with the Native API

The Backblaze B2 Cloud Storage Native API requires a separate call to get an upload URL. You can use this URL until it returns a 50X error, at which point you must request another upload URL. In the event of a 50X error, retry the request with the new URL.

Upload Files

When you upload files, use the exact URL that is returned from b2_get_upload_url or b2_get_upload_part_url. The version number in that URL matches the version number of the call that returned the response. The headers of the upload request hold the parameters, and the body holds the file that you upload.

The URL to upload a file looks like the following example.

https://pod-000-1013-02.backblaze.com/b2api/v3/b2_upload_file/8f433564b42f9c515e21021e/c001_v0001013_t0020

When you upload a file, the body of the POST is the file being uploaded and the other information is passed in the HTTP headers. You must first call b2_get_upload_url to retrieve specific data to pass in.

ACCOUNT_ID=... # Application Key from the Backblaze web site
ACCOUNT_AUTH_TOKEN=... # Comes from the b2_authorize_account call
BUCKET_ID=... # ID for the bucket to upload to
API_URL=... # Comes from the b2_authorize_account call

curl \
    -H "Authorization: $ACCOUNT_AUTH_TOKEN" \
    -d "{\"bucketId\": \"$BUCKET_ID\"}" \
    "$API_URL/b2api/v3/b2_get_upload_url"

The previous example returns the following response including the upload authorizationToken and the uploadUrl to use.

{
  "authorizationToken": "4_0028bc628f87f040000000001_019e8f5e_263f9e_upld_Yr3aQphj7BktivNK60zSnoh6UDU=",
  "bucketId": "e1256f0973908bfc71ed0c1z",
  "uploadUrl": "https://pod-000-1143-18.backblaze.com/b2api/v3/b2_upload_file/e1256f0973908bfc71ed0c1z/c002_v0001143_t0002"
}

You can now complete the upload call as shown in the following example.

FILE_TO_UPLOAD=~/Downloads/logo.jpeg # Name and URL of the file to upload
MIME_TYPE=b2/x-auto # Mime type of the file. X-auto can also be leveraged.
SHA1_OF_FILE=$(openssl dgst -sha1 $FILE_TO_UPLOAD | awk '{print $2;}') # SHA1 of the file
UPLOAD_URL=... # From the b2_get_upload_url call
UPLOAD_AUTHORIZATION_TOKEN=... # From the b2_get_upload_url call

curl \
    -H "Authorization: $UPLOAD_AUTHORIZATION_TOKEN" \
    -H "X-Bz-File-Name: $FILE_TO_UPLOAD" \
    -H "Content-Type: $MIME_TYPE" \
    -H "X-Bz-Content-Sha1: $SHA1_OF_FILE" \
    -H "X-Bz-Info-Author: unknown" \
    --data-binary "@$FILE_TO_UPLOAD" \
    $UPLOAD_URL

The previous example returns a JSON response that contains the File ID of the new file.

{
    "fileId" : "4_he1256f0973908bfc71ed0c1z_f000000000000472a_d20140104_m032022_c001_v0000123_t0104",
    "fileName" : "typing_test.txt",
    "accountId" : "12f634bf3cbz",
    "bucketId" : "e1256f0973908bfc71ed0c1z",
    "contentLength" : 46,
    "contentSha1" : "bae5ed658ab3546aee12f23f36392f35dba1ebdd",
    "contentType" : "text/plain",
    "fileInfo" : {
       "author" : "unknown"
    }
}

SHA1 Checksums

You must always include the X-Bz-Content-Sha1 header with your upload request. The value that you provide can be: (1) the 40-character hex checksum of the file, (2) the string hex_digits_at_end, or (3) the string do_not_verify.

Whenever possible, Backblaze recommends the first option including the checksum in the header. A request to upload a 5-byte file containing the string "hello" looks like the following example.

Authorization: <auth_token>
X-Bz-File-Name: hello.txt
Content-Length: 5
Content-Type: text/plain
X-Bz-Content-Sha1: f572d396fae9206628714fb2ce00f72e94f2258f

hello

With the second option, you append the 40-character hex sha1 to the end of the request body immediately after the contents of the file that you are uploading. Note that the content length is the size of the file plus 40.

Authorization: <auth_token>
X-Bz-File-Name: hello.txt
Content-Length: 45
Content-Type: text/plain
X-Bz-Content-Sha1: hex_digits_at_end

hellof572d396fae9206628714fb2ce00f72e94f2258f

Backblaze does not recommend specifying do_not_verify as the checksum and letting Backblaze B2 compute the checksum of the file. In the event of data corruption and the checksum does not match the data that is sent, the first two options give Backblaze B2 the opportunity to verify the checksum and reject the upload without storing anything in Backblaze B2. With this option, the file is stored and you have to delete it yourself if there is a problem with the checksum. The following example shows this third option.

Authorization: <auth_token>
X-Bz-File-Name: hello.txt
Content-Type: text/plain
X-Bz-Content-Sha1: do_not_verify

hello

If you choose the do_not_verify option, the checksum that is returned in the response from uploading, when listing files, and when downloading the file will have "unverified:" prepended to the checksum, like the following example.

X-Bz-Content-Sha1: unverified:f572d396fae9206628714fb2ce00f72e94f2258f

Setting the Content-Length HTTP Header

Upload Single Files

To upload a single file, first you call b2_get_upload_url to get a URL. Then call b2_upload_file using that URL.

The upload URL that is returned is targeted at a single storage pod in the Backblaze data center. This makes uploads efficient because you send the data directly to the place where it will be stored. But, it means that if that storage pod is unable to take your data at the time, you will have to get a new upload URL and try again.

Backblaze recommends that you write your code to try five different upload URLs before you report an error. Two attempts are almost always sufficient, and five failures is a sign that something is wrong with your request or that you are having problems connecting to the Backblaze B2 service.

Some of the errors that are returned from b2_upload_file mean that you should get a new upload URL and try again, but other errors mean that there is a problem with your request and trying again will not help. The following errors indicate that you should get a new upload URL and try again:

• Unable to make an HTTP connection, including connection timeout.
• Status of 401 Unauthorized and an error code of expired_auth_token
• Status of 408 Request Timeout
• Any HTTP status in the 5xx range, including 503 Service Unavailable
• "Broken pipe" sending the contents of the file.
• A timeout waiting for a response (socket timeout).

A broken pipe occurs when you send a file big enough that the buffers in the HTTP connection cannot hold it. HTTP client libraries send the entire request before looking for a response, and if the Backblaze B2 server already replied with an error, you are unable to send the entire file and a "broken pipe" error is returned.

You may receive the following errors while you upload files:

• 400 Bad Request
  • bad_request Various problems, including the file is already being finished.
  • cap_exceeded You have reached the storage cap that you set.
• 401 Unauthorized
  • missing_auth_token There is no Authorization header.
  • bad_auth_token The authorization token is not valid.
• 403 Forbidden
  • cap_exceeded You have reached the storage cap that you set.

Upload in Parallel

The URL and authorization token that you get from b2_get_upload_url can be used by only one thread at a time. If you want multiple threads running, each one needs its own URL and authorization token. It can keep using that URL and authorization token for multiple uploads until it gets a returned status indicating that it should get a new upload URL.

Upload Large Files

The process for uploading the parts of a large file is just like uploading individual files, except that you use b2_get_upload_part_url to get the upload URL and authorization token, and you use b2_upload_part for each of the parts.

As with regular files, each thread that uploads must make its own call to b2_get_upload_part_url.

After you upload the parts, use b2_finish_large_file to combine the uploaded parts into one large file.

It may be that the call to finish a large file succeeds, but you do not know because the request timed out or the connection was broken. In that case, retrying results in a 400 Bad Request response because the file is already finished. If that happens, Backblaze recommends that you call b2_get_file_info to verify whether the file is there; if the file is there, you can count the upload as a success.

For specific steps to create and upload large files, see Create Large Files with the Native API.

Upload Multiple Files

The following Java-like code is an outline that shows you how to upload multiple files. It can be used either in a single-threaded application or as one of the threads in a parallel uploader. It assumes that it has a queue of files to upload and runs forever uploading files from the queue. It gets a new URL and authorization token when it has a file to upload and the old one is invalid.

void uploadFiles(Queue<UploadInfo> queue) {
    // Initially, we don't have an upload URL and authorization token
    UrlAndAuthToken urlAndAuthToken = null;

    // Keep looping and uploading files forever
    while (true) {
        // Get the info on the next file to upload
        UploadInfo uploadInfo = queue.take();

        // Try several times to upload the file.  It's normal
        // for uploads to fail if the target storage pod is
        // too busy.  It's also normal (but infrequent) to get
        // a 429 Too Many Requests if you are uploading a LOT
        // of files.

        boolean succeeded = false;
        for (int i = 0; i < 5 && !succeeded; i++) {

            // Get a new upload URL and auth token, if needed.
            if (urlAndAuthToken == null) {
                B2Request getUrlRequest = makeGetUploadUrlRequest();
                B2Response getUrlResponse = callB2WithBackOff(request);
                int status = response.status;
                if (status != 200 /*OK*/) {
                     reportFailure(uploadInfo, response);
                     return;
                }
                urlAndAuthToken = response.getUrlAndAuthToken();
            }

            // Upload the file.  When calling upload, don't use
            // back-off.  If there's any problem, we want to go
            // around the loop again and get another upload URL.
            B2Request uploadRequest = makeUploadRequest(uploadInfo);
            B2Response response = callHttpService(uploadRequest)
            int status = response.status;
            if (status == 200 /*OK*/) {
                reportSuccess(uploadInfo);
                succeeded = true;
                break;
            }
            else if (response.isFailureToConnect()) {
                // Try connecting somewhere else next time.
                urlAndAuthToken = null;
            }
            else if (response.isBrokenPipe()) {
                // Could not send entire file.  Try connecting somewhere else next time.
                // If upload caps are exceeded, the next call to get an upload URL will
                // respond with a useful error message.
                urlAndAuthToken = null;
            }
            else if (status == 401 /* Unauthorized */ &&
                       (response.status_code.equals("expired_auth_token") || response.status_code.equals("bad_auth_token")) {
                // Upload auth token has expired.  Time for a new one.
                urlAndAuthToken = null;
            }
            else if (status == 408 /* Request Timeout */) {
                // Retry and hope the upload goes faster this time
                exponentialBackOff();
            }
            else if (status == 429 /* Too Many Requests */) {
                // We are making too many requests
                exponentialBackOff();
            }
            else {
                // Something else went wrong.  Give up.
                reportFailure(uploadInfo, response);
                return;
            }
        }

        if (!succeeded) {
            reportFailure(uploadInfo, response);
            return;
        }
    }
}

B2Response callB2WithBackOff(B2Request request) {
    int delaySeconds = 1;
    int maxDelay = 64;
    while (true) {
        B2Response response = callHttpService(request);
        int status = response.status;
        if (status == 429 /*Too Many Requests*/) {
            sleepSeconds(response.getHeader('Retry-After'));
            delaySeconds = 1.0; // reset 503 back-off
        }
        else if (status == 503 /*Service Unavailable*/) {
            if (maxDelay < delaySeconds) {
                // give up -- delay is too long
                return response
            }
            sleepSeconds(delaySeconds);
            delaySeconds = delaySeconds * 2;
        }
        else {
            return response;
        }
    }
}