Multi Buckets

Combine several object storage buckets into one logical bucket that replicates writes across them for durability and availability.

A multi bucket combines several object storage buckets into a single logical bucket by replicating every write across them. It is a durability and availability feature: because each file is written to a quorum of buckets, the cluster can lose a full bucket (for example, an entire cloud region's object storage) without losing data or availability.

Multi buckets are configured through the same -bucketURL flag (WARPSTREAM_BUCKET_URL) as a normal single bucket — you just pass a warpstream_multi:// URL instead of a plain s3:// (or other provider) URL.

If your goal is to scale write throughput past a single bucket's request-rate ceiling rather than to add redundancy, see Striped Buckets instead. Multi buckets replicate every object (N× storage and write cost); striped buckets store each object once.

URL format

warpstream_multi://$BUCKET_1_URL<>$BUCKET_2_URL<>...<>$BUCKET_N_URL

The sub-buckets are separated by <>, and each sub-URL is constructed exactly as described on the Object Storage Configuration page. Any combination of providers and regions is allowed, though most deployments use the same provider across regions.

For example, three buckets spread across three AWS regions:

-bucketURL "warpstream_multi://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-west-2<>s3://bucket-c?region=us-east-2"

How it works

• Writes are sent to all sub-buckets and acknowledged as soon as a quorum (a majority) succeed, so a single slow or unavailable bucket does not block writes.
• Reads can be served from any sub-bucket that holds the file, so losing one bucket does not make data unreadable.

The net effect is that a multi bucket tolerates the loss of a minority of its sub-buckets with no data loss and no downtime.

Primary use case: multi-region data planes

The most common reason to use a multi bucket is to spread a cluster's data plane across multiple cloud regions so it can survive a region-wide object storage outage with a Recovery Point Objective of 0. See Multi-Region Clusters for how this pairs with a multi-region control plane.

Requirements

• Every WarpStream Agent must be able to read and write all of the sub-buckets.
• Each sub-bucket needs the same permissions and bucket configuration (no object retention policy, versioning, or soft deletion) as a normal WarpStream bucket.

Migrating

Switching a cluster to or from a multi bucket is nothing special — it works exactly like migrating between object storage buckets: point -bucketURL at the new destination (a warpstream_multi:// URL, or a plain single-bucket URL when migrating off) and keep the previous bucket(s) in -additionalBackgroundTasksBucketURLs until the old data has drained. Existing files continue to be read from wherever they were written.