How to set up Archival

Archival consists of the following elements:

• Configuration: Archival is controlled by the server configuration (i.e. the config/development.yaml file).
• Provider: Location where the data should be archived. Supported providers are S3, GCloud, and the local file system.
• URI: Specifies which provider should be used. The system uses the URI schema and path to make the determination.

Take the following steps to set up Archival:

1. Set up the provider of your choice.
2. Configure Archival.
3. Create a Namespace that uses a valid URI and has Archival enabled.

Providers

Temporal directly supports several providers:

• Local file system: The filestore archiver is used to archive data in the file system of whatever host the Temporal server is running on. This provider is used mainly for local installations and testing and should not be relied on for production environments.
• Google Cloud: The gcloud archiver is used to connect and archive data with Google Cloud.
• S3: The s3store archiver is used to connect and archive data with S3.
• Custom: If you want to use a provider that is not currently supported, you can create your own archiver to support it.

Make sure that you save the provider's storage location URI in a place where you can reference it later, because it is passed as a parameter when you create a Namespace.

Configuration

Archival configuration is defined in the config/development.yaml file. Let's look at an example configuration:

# Cluster level Archival config
archival:
  # Event History configuration
  history:
    # Archival is enabled at the cluster level
    state: "enabled"
    enableRead: true
    # Namespaces can use either the local filestore provider or the Google Cloud provider
    provider:
      filestore:
        fileMode: "0666"
        dirMode: "0766"
      gstorage:
        credentialsPath: "/tmp/gcloud/keyfile.json"

# Default values for a Namespace if none are provided at creation
namespaceDefaults:
  # Archival defaults
  archival:
    # Event History defaults
    history:
      state: "enabled"
      # New Namespaces will default to the local provider
      URI: "file:///tmp/temporal_archival/development"

You can disable Archival by setting archival.history.state and namespaceDefaults.archival.history.state to "disabled".

Example:

archival:
  history:
    state: "disabled"

namespaceDefaults:
  archival:
    history:
      state: "disabled"

The following table showcases acceptable values for each configuration and what purpose they serve.

Config	Acceptable values	Description
archival.history.state	enabled, disabled	Must be enabled to use the Archival feature with any Namespace in the cluster.
archival.history.enableRead	true, false	Must be true to read from the archived Event History.
archival.history.provider	Sub provider configs are filestore, gstorage, s3, or your_custom_provider.	Default config specifies filestore.
archival.history.provider.filestore.fileMode	File permission string	File permissions of the archived files. We recommend using the default value of "0666" to avoid read/write issues.
archival.history.provider.filestore.dirMode	File permission string	Directory permissions of the archive directory. We recommend using the default value of "0766" to avoid read/write issues.
namespaceDefaults.archival.history.state	enabled, disabled	Default state of the Archival feature whenever a new Namespace is created without specifying the Archival state.
namespaceDefaults.archival.history.URI	Valid URI	Must be a URI of the file store location and match a schema that correlates to a provider.

Namespace creation

Although Archival is configured at the cluster level, it operates independently within each Namespace. If an Archival URI is not specified when a Namespace is created, the Namespace uses the value of defaultNamespace.archival.history.URI from the config/development.yaml file. The Archival URI cannot be changed after the Namespace is created. Each Namespace supports only a single Archival URI, but each Namespace can use a different URI. A Namespace can safely switch Archival between enabled and disabled states as long as Archival is enabled at the cluster level.

Archival is supported in Global Namespaces (Namespaces that span multiple clusters). When Archival is running in a Global Namespace, it first runs on the active cluster; later it runs on the standby cluster. Before archiving, a history check is done to see what has been previously archived.

Test setup

To test Archival locally, start by running a Temporal server:

./temporal-server start

Then register a new Namespace with Archival enabled.

./tctl --ns samples-namespace namespace register --gd false --history_archival_state enabled --retention 3

note

If the retention period isn't set, it defaults to 2 days. The minimum retention period is 1 day. The maximum retention period is 30 days.

Setting the retention period to 0 results in the error A valid retention period is not set on request.

Next, run a sample Workflow such as the helloworld temporal sample.

When execution is finished, Archival occurs.

Retrieve archives

You can retrieve archived Event Histories by copying the workflowId and runId of the completed Workflow from the log output and running the following command:

./temporal --ns samples-namespace wf show --wid <workflowId> --rid <runId>