S3 Express

S3 Express One Zone is a tier of AWS S3 that provides much lower latency for writes and reads, but only stores the data in a single availability zone. The WarpStream Agents have native support for S3 Express and can use it to store newly written data. Combined with a reduced batch timeout, S3 express can reduce the P99 latency of Produce requests to less than 150ms.

Tradeoffs and How We Mitigate Them

This latency improvement comes with tradeoffs that WarpStream helps you mitigate so that you get the best of both worlds. S3 Express offers faster reads and writes, but charges more for storage. It also provides less resilience than S3 "classic", since by default it doesn't duplicate data across multiple zones.

To mitigate S3 Express's increased storage costs, the WarpStream Agents can use different buckets for data ingestion and data compaction. This enables the Agents to ingest data into S3 Express to reduce Produce request latency, but then compact the data into a regular object storage bucket to keep storage costs low. Think of this as a form of tiered storage within the object store itself.

This is the recommended way to leverage S3 Express with WarpStream because the storage cost of retaining data in in S3 Express is ~7x higher than regular object storage before taking replication into account.

As the name implies, S3 Express One Zone only stores data in a single availability zone. Therefore to prevent availability zone failures from interrupting your cluster, WarpStream will automatically replicate your ingested data across a quorum of multiple S3 Express single-zone buckets.

WarpStream's multi-bucket replication makes S3 Express as resilient as S3 "classic", but drives up your storage costs even further. By restricting S3 Express to data ingestion only, you limit the cost increase to API calls and throughput while saving on storage. For more details on S3 Express One Zone pricing, see AWS's documentation.

Configuration

The first step to using S3 Express is to create the buckets. This can be done in the AWS console, or by using infrastructure as code like Terraform. Below is a sample Terraform block:

locals {
  # S3 Express may not be available in every zone in a region. This
  # is fine though because we don't get billed for inter-zone networking
  # between EC2 and S3 Express buckets. You can see the list of available
  # zone IDs here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-Endpoints.html
  s3_express_zones_ids = ["use1-az4", "use1-az5", "use1-az6"]
}

resource "aws_s3_directory_bucket" "warpstream_s3_express_buckets" {
  count = length(local.s3_express_zones_ids)

  # AZ has to be encoded in this exact format, see docs:
  # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_directory_bucket
  bucket          = "warpstream_s3_express--${local.s3_express_zones_ids[count.index]}--x-s3"
  data_redundancy = "SingleAvailabilityZone"
  type            = "Directory"

  location {
    name = local.s3_express_zones_ids[count.index]
    type = "AvailabilityZone"
  }
}

data "aws_region" "current" {}
data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "warpstream_s3_express_buckets" {
  statement {
    effect = "Allow"

    actions = [
      "s3:ListBucket",
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject",
      "s3express:CreateSession"
    ]

    resources = concat([
      for bucket in aws_s3_directory_bucket.warpstream_s3_express_buckets[*].bucket :
      "arn:aws:s3express:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:bucket/${bucket}"
      ],
      [
        for bucket in aws_s3_directory_bucket.warpstream_s3_express_buckets[*].bucket :
        "arn:aws:s3express:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:bucket/${bucket}/*"
      ]
    )
  }
}

resource "aws_iam_role_policy" "warpstream_s3_express_buckets" {
  name = "warpstream-s3express"
  role = "YOUR ROLE ID"

  policy = data.aws_iam_policy_document.warpstream_s3_express_buckets.json
}

Note that in addition to creating the S3 express buckets, you'll also want to add an S3 express endpoint your VPC. This is free, and will ensure that you don't pay internet egress fees for your S3OZ traffic, as well as reduce latency.

resource "aws_vpc_endpoint" "s3_express" {
  vpc_id       = $VPC_ID
  service_name = "com.amazonaws.$REGION.s3express"
  route_table_ids = $ROUTE_TABLE_IDS
}

Note that we created three S3 Express directory buckets. The reason for this is that the WarpStream Agents will flush ingestion files to all S3 Express directory buckets, and then wait for a quorum of acknowledgements before considering the data durable. In the future we will allow more flexible configurations, but for now we require that at least 3 buckets are configured and all writes must succeed to at least 2 buckets before being considered successful.

In addition to creating the buckets, you'll also need to grant your WarpStream Agents' IAM role one extra permission: s3express:CreateSession.

Once you've created the buckets, and updated the WarpStream Agent IAM role, the final step is to change the Agent configuration to write newly ingested data to a quorum of the S3 Express directory buckets instead of the regular object storage bucket. This is done by deleting the -bucketURL flag (WARPSTREAM_BUCKET_URL environment variable) and replacing it with two new flags:

1. -ingestionBucketURL (WARPSTREAM_INGESTION_BUCKET_URL)

2. -compactionBucketURL (WARPSTREAM_COMPACTION_BUCKET_URL)

The value of compactionBucketURL should point to a classic S3 bucket configured for Warpstream, i.e. the same value as bucketURL in the default object store configuration.

The value of ingestionBucketURL should be a <> delimited list of S3 Express bucket directory URLs with a warpstream_multi:// prefix. For example:

warpstream_multi://s3://warpstream_s3_express--us-east-1a--x-s3?region=us-east-1<>s3://warpstream_s3_express--us-east-1d--x-s3?region=us-east-1<>s3://warpstream_s3_express--us-east-1f--x-s3?region=us-east-1

That's it! The WarpStream Agents will automatically write newly ingested data to a quorum of the S3 Express directory buckets, and then asynchronously compact those files into the regular object storage bucket. The Agents will also automatically take care of deleting files whose data has completely expired from both the S3 Express directory buckets, and the regular object storage bucket.