Constructing bucketURL for Different Object Stores.

This page describes how to properly construct the bucketURL flag for various different object stores.

If you're wondering how to correctly configure an object storage bucket, as well as the required set of permissions, check out our documentation on configuring buckets and permissions.

The bucketURL flag is the URL of the object storage bucket that the WarpStream Agent should write to. See the table below for how to configure it for different object store implementations.

Note that the WarpStream Agents will automatically write all of their data to a top-level warpstream prefix in the bucket. In addition, each cluster will write its data to a cluster-specific prefix (derived from the cluster ID) within the warpstream prefix so multiple WarpStream clusters can co-exist within the same object storage bucket without issue.

An S3 bucket with 16 different cluster prefixes under the top-level warpstream prefix.

If you want the WarpStream Agents to store data in a specific prefix in the bucket, you can add the prefix as a query argument to the bucket URL. The prefix must terminate with a "/". Example: s3://my_warpstream_bucket_123?region=us-east-1&prefix=my_prefix/

AWS S3

Format: s3://$BUCKET_NAME?region=$BUCKET_REGION

Example: s3://my_warpstream_bucket_123?region=us-east-1

The WarpStream Agent embeds the official AWS Golang SDK V2 so authentication/authorization with the specified S3 bucket can be handled in any of the expected ways, like using a shared credentials file, environment variables, or simply running the Agents in an environment with an appropriate IAM role with Write/Read/Delete/List permissions on the S3 bucket.

If you want to use an AssumeRole provider to authenticate, you can add the WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_DEFAULT environment variable to your Agent. For example:

WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_DEFAULT=arn:aws:iam::103069001423:role/YourRoleName

GCP GCS

Format: gs://$BUCKET_NAME

Example: gs://my_warpstream_bucket_123

The WarpStream Agent embeds the official GCP Golang SDK so authentication/authorization with the storage bucket can be handled in any of the expected ways.

Azure Blob Storage

Format: azblob://$CONTAINER_NAME

Example: azblob://my_warpstream_container_123

The WarpStream Agent embeds the official Azure Golang SDK which expects the AZURE_STORAGE_ACCOUNT environment variable to be set, along with one of the two following environment variables: AZURE_STORAGE_KEY or AZURE_STORAGE_SAS_TOKEN.

Memory

For testing and local development only. All data will be lost once the Agent shuts down.

Example: mem://my_memory_bucket

File

For testing and local development only. The file store implementation is not robust.

Format: file://$PATH_TO_DIRECTORY

Example: file:///tmp/warpstream_tmp_123

If you're using an "S3 compatible" object store that is not actually S3, like MinIO or Oracle Cloud Object Store, then read this reference on how to configure the bucket URL.