> For the complete documentation index, see [llms.txt](https://docs.warpstream.com/warpstream/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/gcp-rapid-buckets-beta.md). # GCP Rapid Buckets (beta) {% hint style="danger" %} The default storage quotas for GCP rapid buckets are very low, on the order of 1 TiB per GCP project. We attempt to minimize this limitation as much as possible by writing recently ingested data to Rapid Buckets to reduce latency and then quickly tiering it to a regular regional GCS bucket for long term storage, but high volume workloads may still hit the 1TiB limit in a short period of time before the tiering has kicked in. \ If you're planning to use GCP Rapid Buckets for a high volume use-case, reach out to your GCP account representative and request a quota increase before deploying to production. {% endhint %} {% hint style="danger" %} GCP Rapid Buckets are **not** compatible with [lightning topics](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/lightning-topics.md). Enabling lightning topics on Agents configured to use Rapid Buckets may actually increase latency instead of decreasing it. {% endhint %} [GCP Rapid Buckets](https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket) are a special type of GCS bucket that is backed [Rapid Storage](https://docs.cloud.google.com/storage/docs/storage-classes#rapid) which is a GCP storage class that provides much lower latency for writes and reads, but only stores the data in a single availability zone. The WarpStream Agents have native support for Rapid Buckets and can use them to store newly written data. Combined with a reduced batch timeout, GCP Rapid Buckets can reduce the P99 latency of Produce requests to less than 150ms. {% hint style="info" %} Rapid Buckets require v815+ of the Agents. {% endhint %} ### Tradeoffs and How We Mitigate Them This latency improvement comes with tradeoffs that WarpStream helps you mitigate so that you get the best of both worlds. Rapid Buckets offer faster reads and writes, but charges more for storage. It also provides less resilience than GCS "classic", since by default it doesn't duplicate data across multiple zones. To mitigate Rapid Buckets' increased storage costs, the WarpStream Agents can use different buckets for data ingestion and data compaction. This enables the Agents to ingest data into Rapid Buckets to reduce Produce request latency, but then compact the data into a regular GCS object storage bucket to keep storage costs low. Think of this as a form of tiered storage *within* the object store itself. This is the recommended way to leverage GCP Rapid Buckets with WarpStream because the storage cost of retaining data in in Rapid Buckets is \~5.5x higher than regular object storage **before** taking replication into account. GCP Rapid Buckets only store data in a single availability zone. Therefore to prevent availability zone failures from interrupting your cluster, WarpStream will automatically replicate your ingested data across a quorum of multiple GCS Rapid Buckets deployed in different availability zones. WarpStream's multi-bucket replication makes Rapid Buckets as resilient as GCS "classic", but drives up your storage costs even further. By restricting GCP Rapid Buckets to data ingestion only, you limit the cost increase to just paying for network transfer (which is not free for Rapid Buckets like it is for "classic" GCS buckets) while saving on storage. For more details on Rapid Buckets pricing, see [GCP's documentation](https://cloud.google.com/storage/pricing). ### Configuration The first step to using GCP Rapid Buckets is to create the buckets. This can be done in the GCP console, or by using infrastructure as code like Terraform. Below is a sample Terraform block: ```hcl locals { # GCP Rapid Storage buckets are zonal, so create one bucket per zone that # your WarpStream Agents run in. # # https://cloud.google.com/storage/docs/locations region = "us-east1" rapid_storage_zones = ["us-east1-b", "us-east1-c", "us-east1-d"] } resource "google_storage_bucket" "warpstream_rapid_storage_buckets" { count = length(local.rapid_storage_zones) name = "warpstream-rapid-${local.rapid_storage_zones[count.index]}" location = local.region soft_delete_policy { # Disable soft deletion, otherwise deleted data is retained (and billed) # for 7 days by default. retention_duration_seconds = 0 } versioning { # Make sure versioning is disabled or it will massively inflate your # storage costs. enabled = false } lifecycle_rule { condition { age = 7 } action { type = "AbortIncompleteMultipartUpload" } } # Everything below is required for Rapid Storage, do not remove. storage_class = "RAPID" uniform_bucket_level_access = true hierarchical_namespace { enabled = true } custom_placement_config { # This is what pins the bucket to a single zone (the equivalent of # gcloud's --placement flag). data_locations = [local.rapid_storage_zones[count.index]] } } # roles/storage.objectUser grants object read/write/delete/list, and is also # required for the folder operations (create/delete) that hierarchical # namespace buckets use. resource "google_storage_bucket_iam_member" "warpstream_rapid_storage_object_user" { count = length(local.rapid_storage_zones) bucket = google_storage_bucket.warpstream_rapid_storage_buckets[count.index].name role = "roles/storage.objectUser" member = "serviceAccount:YOUR_SERVICE_ACCOUNT_EMAIL" } resource "google_storage_bucket_iam_member" "warpstream_rapid_storage_bucket_viewer" { count = length(local.rapid_storage_zones) bucket = google_storage_bucket.warpstream_rapid_storage_buckets[count.index].name role = "roles/storage.bucketViewer" member = "serviceAccount:YOUR_SERVICE_ACCOUNT_EMAIL" } ``` Note that we created *three* Rapid Buckets. The reason for this is that the WarpStream Agents will flush ingestion files to all Rapid Buckets buckets, and then wait for a quorum of acknowledgements before considering the data durable. In the future we will allow more flexible configurations, but for now we require that at least 3 buckets are configured and all writes must succeed to at least 2 buckets before being considered successful. Once you've created the buckets, the final step is to change the Agent configuration to write newly ingested data to a quorum of the Rapid Buckets instead of the regular object storage bucket. This is done by deleting the `-bucketURL` flag (`WARPSTREAM_BUCKET_URL` environment variable) and replacing it with two new flags: 1. `-ingestionBucketURL` (`WARPSTREAM_INGESTION_BUCKET_URL`) 2. `-compactionBucketURL` (`WARPSTREAM_COMPACTION_BUCKET_URL`) The value of `compactionBucketURL` should point to a classic regional GCS bucket configured for WarpStream, i.e. the same value as `bucketURL` in [the default object store configuration](https://docs.warpstream.com/warpstream/configuration/different-object-stores). The value of `ingestionBucketURL` should be a `<>` delimited list of Rapid Bucket URLs with a `warpstream_multi://` prefix. For example: {% code overflow="wrap" %} ``` warpstream_multi://gs://warpstream_rapid_us_east1_a<>gs://warpstream_rapid_us_east1_b<>gs://warpstream_rapid_us_east1_c ``` {% endcode %} That's it! The WarpStream Agents will automatically write newly ingested data to a quorum of the GCP Rapid Buckets and then asynchronously compact those files into the regular regional GCS bucket. The Agents will also automatically take care of deleting files whose data has completely expired from both the Rapid Buckets, and the regular object storage bucket. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/gcp-rapid-buckets-beta.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.