Cloud Replication

Cloud Replication lets you copy data from a source bucket to a destination bucket using the Backblaze web UI or the Native API. The S3-Compatible API does not include Cloud Replication at this time. However, after you create a replication rule, it applies to all files regardless of whether they are created using the Native API or the S3-Compatible API.

You can use Cloud Replication to automate the following processes:

• Configuring a rule to replicate data between a source bucket and destination bucket.
• Setting automatic retries for any transient errors that are encountered during replication.
• Adding a replication status field to every source file and destination file.

To create a Cloud Replication rule using the Backblaze web UI, click here.

To create a Cloud Replication rule using the Native API, click here.

Replication Times

Replication times can vary based on the number of files you replicate. Initial replication of a large data set usually takes longer than a subsequent replication. For subsequently uploaded files, replication begins when you upload a new file to a replication-enabled bucket. For existing files, replication occurs each day at midnight UTC. If the replication fails with a non-fatal error, the replication retries after midnight UTC the following day.

You can check the status of your replication rule from the source account on a file-by-file basis. For more information, see Check the Status of a Cloud Replication Rule.

Replicating Data to a Different Region

Backblaze B2 Cloud Storage was designed for an account to tie to a single region. Therefore, to replicate data to a different region, you must create an additional account for that different region. Setting up this additional account is quick and easy, and it does not require that you speak with a Sales or Support resource.

Deleting and Pausing Replication Rules

When you create a replication rule, it automatically replicates per the logic of the rule. To stop the replication, you must pause or delete the rule.

On the Cloud Replication page, the options to either “Pause” or “Delete” the rule are displayed for the rule you created.

Pausing the rule stops the replication while retaining the rule that you created. Pausing the rule also allows you to resume replicating files at any time from the point at which you pause the rule without your files requiring re-evaluation. 

Deleting the rule also stops the replication rule and is a good option if you no longer intend to replicate to the destination bucket. If you delete the rule, you must recreate the entire rule again.

Managing Backblaze Accounts

You can create a Group and enable Backblaze B2 for the group. Invite your source account and your destination account to this group. Create a replication rule in the source account to replicate the objects that are in that account’s bucket to a bucket in the destination account. Billing for storage and egress fees for downloaded files are charged to the group. There are no egress fees for data replication. For more information about Groups, see Create and Manage Groups.

Lifecycle Rules for Replicated Files

Lifecycle Rules apply to replica files in the same manner as they apply to source files. When you replicate files between buckets with different lifecycle rules, be aware of the following effects that the different lifecycle rules have on your files.

Example 1:

• You have a source bucket with Lifecycle Rules that hides files after 30 days and deletes them five days later.
• You have a destination bucket with Lifecycle Rules that hides files after 10 days and deletes them one day later.

In this situation, you may have a file in the source bucket that is 15 days old. If you replicate that file to the destination bucket, the file becomes hidden. Specifically, the file becomes hidden the next time the system applies the Lifecycle Rules. Lifecycle Rules are applied once per day.

Example 2:

• You have a source bucket with Lifecycle Rules that hides files after 10 days and deletes them one day later.
• You have a destination bucket with Lifecycle Rules that hides files after 30 days and deletes them five days later.

In this situation, you may have a file in the source bucket that reaches an age of 11 days. Due to the source bucket's Lifecycle Rules, this file becomes hidden after 10 days and deleted on the 11th day. However, if you replicate that file to the destination bucket before it is deleted, the file remains visible in the destination bucket until it is 30 days old even though the original file was deleted from the source bucket.

Existing File Replication

When a replication rule is configured to include existing files, the replica files are created with the same upload timestamp as their source file.

• Backblaze does not guarantee any ordering of individual file version replications. While the replication of existing files is ongoing, it is possible that a filename-based query will have different results between the source and destination buckets. This situation is temporary, and both buckets will provide the same response after the existing files have finished replicating. Consider disabling Lifecycle Rules in the destination bucket if this is an issue.
• Existing files are replicated to the destination bucket regardless of the configuration of the destination bucket's Lifecycle Rules. It is possible that some newly created file replicas will be hidden or deleted when Lifecycle Rules are next applied (typically every 24 hours).
• Replica files are created with the same upload timestamp as their source file. If the destination bucket has a default Object Lock retention period set, the retention period is calculated based on the timestamp that is inherited from the source file.

Restrictions

Cloud Replication is not built to serve every use case. The following list provides cases in which the feature is not recommended or is limited to specific scenarios:

• Setting manual data caps is not recommended.
  This can cause unexpected results. If a data cap is set below the 10 GB limit, authenticating to this account does not return any buckets. In this instance, the replication rule should be deleted and the data cap removed. You will then need to recreate the exact replication rule.
• Replication chaining is not possible.
  All destination files are seen as replicants and cannot be used for new replications.
• Multiple replication rules
  Only two rules can be set per bucket for replication.
• File IDs are not replicated.
  Therefore, Veeam backup replication is not currently supported.
• Size limitations
  Files with metadata greater than 2,048 bytes are not replicated.
• Deleted files
  Deleted files are not replicated.
• SSE-C or encryption managed by the client is not supported.
  Backblaze supports encryption (which you must set up on a bucket at the time of creation). SSE-C, or encryption managed by the client, is not supported.
• Metadata changes are not replicated.
  If you need to change a file’s metadata, but you do not change the body, the metadata change on the replicated file is ignored. 
• Existing vs new data replication
  In the API, you can enable the replication of all existing files in a bucket. If this is not enabled, the bucket is set up with New Data replication, where only the new files that were uploaded after the rule was set up are replicated.
  For rules that you configure in the Backblaze web UI, all rules are set to existing data, meaning the parameter is set to 'true' as part of the automatic settings that are configured.
• The following items are not replicated:
  • Replica Files: Any files that are replicated cannot be set as the source of a new replication.
  • SSE-C files: Encrypted files that are managed by the user are not replicated.
  • Hide Markers: Hide markers are not replicated. This means that the files that are hidden in a source are not hidden in the destination. Hide markers can be applied to the destination after replication completes.

Replication Statuses

Additional Notes

You should keep the following key areas in mind when you use the Cloud Replication feature:

• Backblaze does not charge service or egress fees for replication. Standard Backblaze B2 rates apply to replicated data storage.
• Backblaze data storage costs do not differ from one region to another, and you do not incur any service or egress fees. Standard Backblaze B2 rates apply to replicated data storage.
• Backblaze B2 was designed for an account to tie to a single region. Therefore, to replicate data to a different region, you must create an additional account for that different region. Setting up this additional account is quick and easy, and does not require that you speak with a Sales or Support resource.
• When you configure a replication source, you must provide an app key that has read access to the bucket. Similarly, for destination buckets, you must provide an app key that has write access to the destination bucket.
• If you set the replication configuration for a source bucket that references a destination bucket that does not exist, the configuration attempt will fail.
• Ensure that both the source account and the destination account are configured correctly for billing. If they are not, you will be unable to save the related Cloud Replication rule.
• You cannot use a master key as the app key for Cloud Replication.
• If you set Object Lock on a source bucket, you must also enable Object Lock on the destination bucket or the replication returns a failed status.
• To centrally manage your Backblaze B2 accounts, you can create a Group and enable Backblaze B2 for the group. Invite your source account and your destination account to this group. Create a replication rule in the source account to replicate the objects that are in that account’s bucket to a bucket in the destination account. Billing for storage and egress fees for downloaded files are charged to the group. There are no egress fees for data replication. For more information about Groups, see Create and Manage Groups.
  
  Note

  Groups apply only to consumption-based accounts. The Groups feature is currently not supported for B2 Reserve accounts.
• Replication times can vary based on the number of files you replicate. Initial replication of a large data set usually takes longer than a subsequent replication. For subsequently uploaded files, replication begins when you upload a new file to a replication-enabled bucket. For existing files, replication occurs each day at midnight UTC. If the replication fails with a non-fatal error, the replication retries after midnight UTC the following day.
• Currently, you can apply only two rules for the same bucket. If you create a third rule on the same bucket, the rule fails when you attempt to create the bucket.
• You can check the status of your replication rule only on the source account and only on a file-by-file basis. For more information, see Create and Manage Cloud Replication Rules.

For assistance with Cloud Replication, contact the Support team.