Striped Buckets

Combine several object storage buckets into one logical bucket that stripes writes across them to scale write throughput past a single bucket's limits.

A striped bucket combines several object storage buckets into a single logical bucket by striping writes across them: each object is placed on exactly one sub-bucket, chosen deterministically by hashing the object's key. It exists to scale write throughput past the request-rate ceiling of a single bucket — spreading writes across N sub-buckets multiplies the headroom for ingestion and compaction.

Striped buckets are configured through the same -bucketURL flag (WARPSTREAM_BUCKET_URL) as a normal single bucket — you just pass a warpstream_stripe:// URL instead of a plain s3:// (or other provider) URL.

Bucket striping requires WarpStream Agent v827 or above.

Striping is not a redundancy or availability feature. Each object lives on exactly one sub-bucket, so if a sub-bucket becomes unavailable its objects become unreadable — the same blast radius as a single bucket. If you want to survive the loss of a bucket or region, use a Multi Bucket (which replicates every write) instead.

When to use it

Reach for a striped bucket when a single bucket is hitting its object storage request-rate limits (for example, sustained 503 SlowDown throttling on S3) and the workload needs more write throughput than one bucket can provide. If instead you need durability across regions, use a Multi Bucket.

URL format

warpstream_stripe://$BUCKET_1_URL<>$BUCKET_2_URL<>...<>$BUCKET_N_URL

The sub-buckets are separated by <> (2 to 32 sub-buckets), and each sub-URL is constructed exactly as described on the Object Storage Configuration page.

For example, striping across three buckets:

-bucketURL "warpstream_stripe://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-east-1<>s3://bucket-c?region=us-east-1"

How it works

• Placement is deterministic: each object is hashed to one sub-bucket, so writers and readers independently agree on where an object lives — nothing extra is stored to track it.
• Reads go to the hashed sub-bucket and cost the same as a single bucket (one round-trip).
• Write resilience: if a sub-bucket's circuit breaker opens (it is throttling or failing), new writes are steered to a healthy sub-bucket so ingestion keeps making progress.

Multi buckets vs. striped buckets

	warpstream_multi://	warpstream_stripe://
Each object is…	replicated to a quorum of buckets	written to exactly one bucket
Optimizes for	durability / availability	write throughput
Survives a bucket/region loss	yes (no data loss)	no (that bucket's objects become unreadable)
Relative storage cost	~N×	~1×

Requirements and constraints

• Every WarpStream Agent must be able to read and write all of the sub-buckets.
• Each sub-bucket needs the same permissions and bucket configuration (no object retention policy, versioning, or soft deletion) as a normal WarpStream bucket. Using the same provider and type for every sub-bucket is recommended.
• Changing the set of sub-buckets is forward-only. Existing files stay in whichever sub-bucket they were originally written to, and are always read from there. Switching a single bucket to a striped bucket (or adding sub-buckets) is safe: new files are striped across the new set while old files continue to be read from where they live.

Migrating

Switching a cluster to or from a striped bucket is nothing special — it works exactly like migrating between object storage buckets: point -bucketURL at the new destination (a warpstream_stripe:// URL, or a plain single-bucket URL when migrating off) and keep the previous bucket(s) in -additionalBackgroundTasksBucketURLs until the old data has drained. The same applies when retiring a single sub-bucket from a stripe: add the retired bucket (or the previous full stripe URL) to -additionalBackgroundTasksBucketURLs so its files are still read and cleaned up while it drains.