GCP Rapid Buckets (beta)

The default storage quotas for GCP rapid buckets are very low, on the order of 1 TiB per GCP project. We attempt to minimize this limitation as much as possible by writing recently ingested data to Rapid Buckets to reduce latency and then quickly tiering it to a regular regional GCS bucket for long term storage, but high volume workloads may still hit the 1TiB limit in a short period of time before the tiering has kicked in.

If you're planning to use GCP Rapid Buckets for a high volume use-case, reach out to your GCP account representative and request a quota increase before deploying to production.

GCP Rapid Buckets are not compatible with lightning topics. Enabling lightning topics on Agents configured to use Rapid Buckets may actually increase latency instead of decreasing it.

GCP Rapid Buckets are a special type of GCS bucket that is backed Rapid Storage which is a GCP storage class that provides much lower latency for writes and reads, but only stores the data in a single availability zone. The WarpStream Agents have native support for Rapid Buckets and can use them to store newly written data. Combined with a reduced batch timeout, GCP Rapid Buckets can reduce the P99 latency of Produce requests to less than 150ms.

Rapid Buckets require v815+ of the Agents.

Tradeoffs and How We Mitigate Them

This latency improvement comes with tradeoffs that WarpStream helps you mitigate so that you get the best of both worlds. Rapid Buckets offer faster reads and writes, but charges more for storage. It also provides less resilience than GCS "classic", since by default it doesn't duplicate data across multiple zones.

To mitigate Rapid Buckets' increased storage costs, the WarpStream Agents can use different buckets for data ingestion and data compaction. This enables the Agents to ingest data into Rapid Buckets to reduce Produce request latency, but then compact the data into a regular GCS object storage bucket to keep storage costs low. Think of this as a form of tiered storage within the object store itself.

This is the recommended way to leverage GCP Rapid Buckets with WarpStream because the storage cost of retaining data in in Rapid Buckets is ~5.5x higher than regular object storage before taking replication into account.

GCP Rapid Buckets only store data in a single availability zone. Therefore to prevent availability zone failures from interrupting your cluster, WarpStream will automatically replicate your ingested data across a quorum of multiple GCS Rapid Buckets deployed in different availability zones.

WarpStream's multi-bucket replication makes Rapid Buckets as resilient as GCS "classic", but drives up your storage costs even further. By restricting GCP Rapid Buckets to data ingestion only, you limit the cost increase to just paying for network transfer (which is not free for Rapid Buckets like it is for "classic" GCS buckets) while saving on storage. For more details on Rapid Buckets pricing, see GCP's documentation.

Configuration

The first step to using GCP Rapid Buckets is to create the buckets. This can be done in the GCP console, or by using infrastructure as code like Terraform. Below is a sample Terraform block:

locals {
  # GCP Rapid Storage buckets are zonal, so create one bucket per zone that
  # your WarpStream Agents run in.
  #
  # https://cloud.google.com/storage/docs/locations
  region              = "us-east1"
  rapid_storage_zones = ["us-east1-b", "us-east1-c", "us-east1-d"]
}

resource "google_storage_bucket" "warpstream_rapid_storage_buckets" {
  count = length(local.rapid_storage_zones)
  name     = "warpstream-rapid-${local.rapid_storage_zones[count.index]}"
  location = local.region
  soft_delete_policy {
    # Disable soft deletion, otherwise deleted data is retained (and billed)
    # for 7 days by default.
    retention_duration_seconds = 0
  }
  versioning {
    # Make sure versioning is disabled or it will massively inflate your
    # storage costs.
    enabled = false
  }
  lifecycle_rule {
    condition {
      age = 7
    }
    action {
      type = "AbortIncompleteMultipartUpload"
    }
  }
  # Everything below is required for Rapid Storage, do not remove.
  storage_class               = "RAPID"
  uniform_bucket_level_access = true
  hierarchical_namespace {
    enabled = true
  }
  custom_placement_config {
    # This is what pins the bucket to a single zone (the equivalent of
    # gcloud's --placement flag).
    data_locations = [local.rapid_storage_zones[count.index]]
  }
}

# roles/storage.objectUser grants object read/write/delete/list, and is also
# required for the folder operations (create/delete) that hierarchical
# namespace buckets use.
resource "google_storage_bucket_iam_member" "warpstream_rapid_storage_object_user" {
  count = length(local.rapid_storage_zones)
  bucket = google_storage_bucket.warpstream_rapid_storage_buckets[count.index].name
  role   = "roles/storage.objectUser"
  member = "serviceAccount:YOUR_SERVICE_ACCOUNT_EMAIL"
}

resource "google_storage_bucket_iam_member" "warpstream_rapid_storage_bucket_viewer" {
  count = length(local.rapid_storage_zones)
  bucket = google_storage_bucket.warpstream_rapid_storage_buckets[count.index].name
  role   = "roles/storage.bucketViewer"
  member = "serviceAccount:YOUR_SERVICE_ACCOUNT_EMAIL"
}

Note that we created three Rapid Buckets. The reason for this is that the WarpStream Agents will flush ingestion files to all Rapid Buckets buckets, and then wait for a quorum of acknowledgements before considering the data durable. In the future we will allow more flexible configurations, but for now we require that at least 3 buckets are configured and all writes must succeed to at least 2 buckets before being considered successful.

Once you've created the buckets, the final step is to change the Agent configuration to write newly ingested data to a quorum of the Rapid Buckets instead of the regular object storage bucket. This is done by deleting the -bucketURL flag (WARPSTREAM_BUCKET_URL environment variable) and replacing it with two new flags:

1. -ingestionBucketURL (WARPSTREAM_INGESTION_BUCKET_URL)
2. -compactionBucketURL (WARPSTREAM_COMPACTION_BUCKET_URL)

The value of compactionBucketURL should point to a classic regional GCS bucket configured for WarpStream, i.e. the same value as bucketURL in the default object store configuration.

The value of ingestionBucketURL should be a <> delimited list of Rapid Bucket URLs with a warpstream_multi:// prefix. For example:

warpstream_multi://gs://warpstream_rapid_us_east1_a<>gs://warpstream_rapid_us_east1_b<>gs://warpstream_rapid_us_east1_c

That's it! The WarpStream Agents will automatically write newly ingested data to a quorum of the GCP Rapid Buckets and then asynchronously compact those files into the regular regional GCS bucket. The Agents will also automatically take care of deleting files whose data has completely expired from both the Rapid Buckets, and the regular object storage bucket.