For the complete documentation index, see llms.txt. This page is also available as Markdown.

Striped Buckets

Combine several object storage buckets into one logical bucket that stripes writes across them to scale write throughput past a single bucket's limits.

A striped bucket combines several object storage buckets into a single logical bucket by striping writes across them: each object is placed on exactly one sub-bucket, chosen deterministically by hashing the object's key. It exists to scale write throughput past the request-rate ceiling of a single bucket — spreading writes across N sub-buckets multiplies the headroom for ingestion and compaction.

Striped buckets are configured through the same -bucketURL flag (WARPSTREAM_BUCKET_URL) as a normal single bucket — you just pass a warpstream_stripe:// URL instead of a plain s3:// (or other provider) URL.

Bucket striping requires WarpStream Agent v827 or above.

When to use it

Reach for a striped bucket when a single bucket is hitting its object storage request-rate limits (for example, sustained 503 SlowDown throttling on S3) and the workload needs more write throughput than one bucket can provide. If instead you need durability across regions, use a Multi Bucket.

URL format

warpstream_stripe://$BUCKET_1_URL<>$BUCKET_2_URL<>...<>$BUCKET_N_URL

The sub-buckets are separated by <> (2 to 32 sub-buckets), and each sub-URL is constructed exactly as described on the Object Storage Configuration page.

For example, striping across three buckets:

-bucketURL "warpstream_stripe://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-east-1<>s3://bucket-c?region=us-east-1"

How it works

  • Placement is deterministic: each object is hashed to one sub-bucket, so writers and readers independently agree on where an object lives — nothing extra is stored to track it.

  • Reads go to the hashed sub-bucket and cost the same as a single bucket (one round-trip).

  • Write resilience: if a sub-bucket's circuit breaker opens (it is throttling or failing), new writes are steered to a healthy sub-bucket so ingestion keeps making progress.

Multi buckets vs. striped buckets

warpstream_multi://

warpstream_stripe://

Each object is…

replicated to a quorum of buckets

written to exactly one bucket

Optimizes for

durability / availability

write throughput

Survives a bucket/region loss

yes (no data loss)

no (that bucket's objects become unreadable)

Relative storage cost

~N×

~1×

Requirements and constraints

  • Every WarpStream Agent must be able to read and write all of the sub-buckets.

  • Each sub-bucket needs the same permissions and bucket configuration (no object retention policy, versioning, or soft deletion) as a normal WarpStream bucket. Using the same provider and type for every sub-bucket is recommended.

  • Changing the set of sub-buckets is forward-only. Existing files stay in whichever sub-bucket they were originally written to, and are always read from there. Switching a single bucket to a striped bucket (or adding sub-buckets) is safe: new files are striped across the new set while old files continue to be read from where they live.

Migrating

Switching a cluster to or from a striped bucket is nothing special — it works exactly like migrating between object storage buckets: point -bucketURL at the new destination (a warpstream_stripe:// URL, or a plain single-bucket URL when migrating off) and keep the previous bucket(s) in -additionalBackgroundTasksBucketURLs until the old data has drained. The same applies when retiring a single sub-bucket from a stripe: add the retired bucket (or the previous full stripe URL) to -additionalBackgroundTasksBucketURLs so its files are still read and cleaned up while it drains.

Last updated

Was this helpful?