Serverless Clusters

WarpStream Serverless clusters are entirely managed by WarpStream. In addition to the fully-managed control plane, which is used by all cluster types, the Agents for Serverless clusters run in WarpStream's environment, and use object storage in WarpStream's cloud account.

While Serverless cluster operations are similar to BYOC clusters, there are some known differences you should be aware of.

Dedicated Kafka Headers

If you are using a Serverless cluster, then the Kafka headers starting with “_ws” are reserved for WarpStream. Concretely this means that:

• Produce requests containing records with at least one header starting with this prefix will be rejected.

• Those headers are used for internal purposes only and will never be returned to users when records are fetched.

Ratelimits

Severless clusters are rate-limited as follows:

Global ratelimits (per virtual cluster)

• Produce Compressed bytes: 20Mib/s

• Produce Uncompressed bytes: 20Mib/s

• Fetch Uncompressed bytes: 80Mib/s

• New Connections Rate: 1k/s

• Produce Requests Rate: 6k/s

• General Requests Rate: 1k/s

• Fetch Parallel Requests: 360

• Consumer Rebalance Timeout: 2m

Connection Ratelimits (per connection)

• Connection throughput: 64 MiB/s.

• Number of Connections: 2k

Request Ratelimits

• Max TCP Request Size: 32MiB

• Max Produce Request Uncompressed Bytes: 20 MiB