Lifecycle Rules

Lifecycle Rules enable the Backblaze B2 Cloud Storage service to automatically hide and delete older versions of files that are stored in Backblaze B2. This capability is available for the Backblaze web UI, command-line interface (CLI), and API. You can apply Lifecycle Rules to some or all of the files in a Backblaze B2 bucket, and each bucket has its own Lifecycle Rules. For example, you can set up rules to delete old versions of files 30 days after you upload a newer version.

File Name Prefix

You can set up to 100 Lifecycle Rules on one bucket. Each rule has a fileNamePrefix that specifies the files to which the rule applies. Any file name that begins with the prefix is subject to the rule. A prefix of an empty string ("") means that the rule applies to all of the files in the bucket.

You cannot create two rules that both apply to the same files. For example, a rule with a file name prefix of photos/ and a rule with a file name prefix of photos/kittens/ both apply to a file named photos/kittens/fluffy.jpg.

File Versions

By default, Backblaze B2 keeps all of the files that you upload and all of the different versions of the files that you upload. While Backblaze B2 retains the different versions, it “hides” the older versions to keep the file list easier to view as shown in the following example.

There are two indicators that there are multiple versions of the same file.

• The file name has a (2) at the end. This is the file version count and, in this case, it means there are two (2) versions of the file. The date/time stamp that is shown is for the most recent version of the file. When you click on the file version count (the 2), the list expands to show all of the file versions.
• The “missing” file information icon is displayed.

The older version, V2 in the example, is hidden. All of the versions of a given file are available to download, delete, or keep. If the file names are different, they are stored as different files, so myfile-version1.doc is not the same as myfile-version2.doc, even if the content is identical.

There is no limit to the number of versions you can have of a given file. By default, older versions are kept forever, unless you delete them or you stop using the Backblaze B2 service.

Since all of the versions of a given file are kept, you can use Lifecycle Rules to help manage the older versions. 

The file version rules apply to all of the files in the bucket, unless otherwise specified. The four options for File Versions work as follows:

• Keep all version of the file (default)
  This option removes all of the lifecycle rules from the bucket and keeps all of the versions of a given file until you explicitly delete them.
• Keep only the last version of the file
  This rule keeps only the most current version of a file. The previous version of the file is “hidden” for one day and then deleted.

  {
     "daysFromHidingToDeleting": 1,
     "daysFromUploadingToHiding": null,
     "fileNamePrefix": ""
   }

• Keep prior versions for this number of days
  This rule deletes the older versions of a file after the number of days that you can specify. The number of days can be one or greater. The most current version of a file is always kept unless it is explicitly deleted.

  {
     "daysFromHidingToDeleting": 30,
     "daysFromUploadingToHiding": null,
     "fileNamePrefix": ""
  }

• Use custom lifecycle rules
  This rule allows you to construct a rule that applies to one or more files using the following variables:
  • fileNamePrefix
  • daysFromUploadingtoHiding
  • daysFromHidingtoDeleting

You can handle the most common situations using one of the first three options.

Construct a Lifecycle Rule

There are multiple variables to construct a Lifecycle Rule.

You can set any of these values to null so that part of the rule is not applied. You cannot set all of the values to null because the rule would do nothing.

You cannot set daysFromUploadingToHiding or daysFromHidingToDeleting to zero (0). When you set a value for these options, you must specify a positive number.

Every Lifecycle Rule includes an implicit rule: when the oldest version of a file is a hide marker, the marker is deleted, independent of its age.

fileNamePrefix

This variable specifies the files in the Backblaze B2 bucket to which the rules apply. For example, a value of “workinprocess/”  applies to all of the files in the “workinprocess” directory/folder. If you leave the prefix field blank, the rule applies to all of the files that are in the bucket. 

A prefix of * or  "" (empty quotes) results in a lifecycle rule that literally applies to files that have a prefix of * or "". 

daysFromHidingToDeleting

This value is the most commonly used setting, and it causes the specified hidden files, explicitly hidden with b2_hide_file, to be automatically deleted after the number of days that you specify. Remember, files are automatically hidden when they are replaced with a newer version. For example, if you set this variable to 10, then 10 days after you upload a newer version of the file, the older version is deleted. Valid values are “null” or numbers one (1) and greater. Null means that no files are deleted based on this rule.

If you back up your files to Backblaze B2 using the command-line tool or another software package that uploads files when they change, Backblaze B2 keeps old versions of the file. This is very helpful because the old versions are there if the original file is accidentally deleted. On the other hand, keeping them forever can clutter things. For an application like this, you might want a Lifecycle Rule that keeps old versions in the backup/folder for 30 days, and then deletes them as in the following example:

{
   "daysFromHidingToDeleting": 30,
   "daysFromUploadingToHiding": null,
   "fileNamePrefix": "backup/"
}

daysFromUploadingToHiding

This setting is less frequently used. This value causes the specified files to be automatically hidden after the specified number of days. This applies to all of the copies of the file, even the most current version. For example, setting this variable to 30 hides all of the files in the specified directory/folder 30 days after you upload the files – this includes the most current version of the file. 

Use this option only if you want to hide all of the versions of the specified files. Valid values are “null” or numbers one (1) and greater. Null, means that no files are hidden based on this rule.

This setting can be useful for things like server log files that can be deleted after a certain time period. This rule keeps files in the logs/folder for seven days, and then hides them. They are then deleted one day after hiding.

{
   "daysFromHidingToDeleting": 1,
   "daysFromUploadingToHiding": 7,
   "fileNamePrefix": "logs/"
}

Lifecycle Rules API Operations

The following API calls are related to Lifecycle Rules:

• b2_create_bucket This operation creates a new bucket.
• b2_update_bucket This operation changes the settings on a bucket.
• b2_list_buckets This operation lists all of the buckets in your account.

When you create a new bucket, you can specify the Lifecycle Rules. Later, you can update the rules on a bucket.

Each API call has an optional parameter called lifecycleRules that requires three variables in the following JSON format:

{
   “daysFromHidingToDeleting”: 30,
   “daysFromUploadingToHiding”: null,
   “fileNamePrefix”: “backup/”
}

In addition, the b2_delete_bucket operation returns the list of lifecycleRules for the bucket that you want to delete.

Lifecycle Rules in the Command-Line Tool

To use Lifecycle Rules with the Backblaze B2 Cloud Storage command-line interface (CLI), you need to run version 0.7.0 of the Backblaze B2 command-line tool.

Use the following commands to create and manage Lifecycle Rules in the CLI.

b2 create-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]

b2 update-bucket [–bucketInfo <json>] [–lifecycleRules <json>] <bucketName> [allPublic | allPrivate]

b2 get-bucket <bucketName>

The b2 get-bucket command returns all of the variables for the named bucket.

Lifecycle Rules and Replica Files

Lifecycle Rules apply to replica files in the same manner as they apply to source files. When you replicate files between buckets with different Lifecycle Rules, the different rules may affect your files.

Example 1:

• You have a source bucket with Lifecycle Rules that hides files after 30 days and then deletes them five days later.
• You have a destination bucket with Lifecycle Rules that hides files after 10 days and then deletes them one day later.

In this situation, you might have a file in the source bucket that is 15 days old. If you replicate that file to the destination bucket, the file becomes hidden. Specifically, the file becomes hidden the next time the system applies the rule. Lifecycle Rules are applied once per day.

Example 2:

• You have a source bucket with Lifecycle Rules that hides files after 10 days and then deletes them one day later.
• You have a destination bucket with Lifecycle Rules that hides files after 30 days and then deletes them five days later.

In this situation, you might have a file in the source bucket that reaches an age of 11 days. Due to the source bucket's Lifecycle Rules, this file becomes hidden after 10 days and then deleted on the 11th day. However, if you replicate that file to the destination bucket before it is deleted, the file remains visible in the destination bucket until it is 30 days old, even though the original file was already deleted from the source bucket.

Setting Times for Lifecycle Rules

Lifecycle Rules are applied once per day. The fewest days you can set for a rule is one. Rules are applied at the next daily run after that number of days has passed.

As an example, suppose you upload the first version of file.txt on May 2. Then you upload a new version of file.txt on May 9 at 06:00. At this point there are two versions, the May 2 version and the May 9 version. The May 2 version is hidden by the May 9 version, so it became hidden at the time the May 9 version was uploaded.

Suppose there's a Lifecycle Rule to delete files one day after they are hidden. If the Lifecycle Rules get applied at 01:00 on May 10, nothing will happen yet. It has been 19 hours since the May 2 version was hidden. Because it hasn't been a full day, nothing will happen.

The next day, on May 11, when the lifecycle is applied, something will happen. If it runs at 01:00, it's now been 43 hours since the May 2 version was hidden, so the rule now applies and the old version will be deleted.

NOTE: Lifecycle Rules are based on the upload timestamp of the file/object. When migrating from other cloud providers, you can maintain existing Lifecycle Rules by customizing the upload timestamp to set the timestamp in the past. 

For more information about custom upload timestamps, see b2_upload_file and b2_start_large_file. 

To use custom upload timestamps, please contact Backblaze support to have the feature enabled on your account.

Object Lock and Lifecycle Rules

Lifecycle Rules potentially delete specific versions of files. However, if the file version to which the rule applies has Object Lock enabled, the deletion will not occur.

Existing File Replication

When a replication rule is configured to include existing files, the replica files are created with the same upload timestamp as their source file. This has several ramifications:

• Backblaze does not guarantee any ordering of individual file version replications. While replication of existing files is ongoing, it is possible that a filename-based query will have different results between the source and destination buckets. This situation is temporary, and both buckets will provide the same response once the existing files have finished replicating. Consider disabling Lifecycle Rules in the destination bucket if this will be an issue.
• Existing files will be replicated to the destination bucket regardless of the destination bucket's Lifecycle Rule configuration. It is possible that some newly created file replicas will be hidden or deleted when Lifecycle Rules are next applied (typically every 24 hours).
• Replica files are created with the same upload timestamp as their source file. If the destination bucket has a default object lock retention period set, the retention period is calculated based on the timestamp inherited from the source file.

Integrations and Lifecycle Rules

Application developers are encouraged to look at Lifecycle Rules to use in their applications that integrate into Backblaze B2. This is especially true for applications that perform Sync or Backup operations to Backblaze B2. For example, the user interface of the application could allow the user/admin to specify the number of days that older versions of files should be kept in support of data retention rules in an organization.

Lifecycle Rule Examples

If you back up your “log” files to Backblaze B2 one time per day, and these files are stored in a directory/folder named log., then you can set up your Lifecycle Rules to manage the log files based on your data retention policies.

Example 1:

fileNamePrefix: log/
daysFromUploadingtoHiding: 7
daysFromHidingtoDeleting: null

Result:

The files in the log/folder will be hidden seven (7) days after you upload them to Backblaze B2. This means that the current version, as well as any previous versions of every file in the log/folder, will be hidden after seven (7) days. The hidden files are still available for download using their specific file_id and continue to count as part of your Backblaze B2 storage amount.

Example 2:

fileNamePrefix: log/
daysFromUploadingtoHiding: 7
daysFromHidingtoDeleting: 30

Result:

The files in the log/folder will be hidden seven (7) days after you upload them to Backblaze B2. After the files have been hidden for 30 days, they are permanently deleted. Until they are deleted, the files are still available for download and count as part of your Backblaze B2 storage amount.

Example 3:

fileNamePrefix: log/
daysFromUploadingtoHiding: null
daysFromHidingtoDeleting: 180

Result:

The files in the log/folder will be deleted 180 days after they have been either explicitly hidden or they been replaced by a file of the same name (they are an older version of the file – remember older file versions are automatically hidden). Until the files are deleted, all of the versions of a file are available for download and count as part of your Backblaze B2 storage amount.