Skip to content
WarpStream Documentation

Maximum Record Size

Overview of how WarpStream enforces maximum record size limits and how to adjust them at the topic, cluster, or Agent level.

WarpStream enforces a limit on the maximum size of each record written to a topic. This limit defaults to 32 MB (32000000 bytes) on BYOC clusters and can be adjusted at the topic level, the cluster (broker) level, or the Agent level.

In open source Apache Kafka, message.max.bytes (broker) and max.message.bytes (topic) limit the size of a compressed record batch sent by a producer.

In WarpStream, maximum record size limits apply to each individual uncompressed record (its varint-framed byte length in the Kafka record format, including key, value, headers, and per-record metadata).

When a producer sends a record that exceeds the effective limit, WarpStream rejects the Produce request with Kafka error MESSAGE_TOO_LARGE (error code 10) and emits the produce_records_too_large diagnostic.

WarpStream supports configuring the maximum record size at three levels, evaluated in strict precedence order:

Precedence Scope Configuration Default Valid Values
1 (highest) Topic max.message.bytes Unset (inherits from cluster or Agent) [0, ...] (bytes)
2 Cluster (broker) message.max.bytes Unset (inherits from Agent) [0, ...] (bytes)
3 (lowest) Agent -maxProduceRecordSizeBytes / WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES 32000000 (32 MB) 1048576 (1 MiB) to 268435456 (256 MiB)
  • Higher-precedence settings override lower ones in either direction. If max.message.bytes is set on a topic, WarpStream uses that value for the topic even if it is larger or smaller than cluster-level message.max.bytes or the Agent flag. Likewise, if topic-level max.message.bytes is unset and cluster-level message.max.bytes is set, the cluster-level value overrides the Agent flag.
  • Unset configs are omitted from DescribeConfigs. When max.message.bytes (topic) or message.max.bytes (broker) is not explicitly set, WarpStream omits it from DescribeConfigs responses because the effective fallback depends on the Agent configuration.
  • Setting 0 does not reset to default. Explicitly setting max.message.bytes=0 or message.max.bytes=0 enforces a 0-byte limit and rejects all produced records. To fall back to the next level in the precedence chain, delete/unset the configuration key instead.

Set max.message.bytes on a topic using any Kafka client or tool supporting AlterConfigs / IncrementalAlterConfigs (such as kafka-configs), the Topics REST API, or the Terraform provider. Delete the max.message.bytes config key to revert the topic back to the cluster or Agent limit.

Set message.max.bytes at the broker/cluster level using any Kafka client or tool supporting broker AlterConfigs / IncrementalAlterConfigs (such as kafka-configs). This applies to all topics in the cluster that do not explicitly set max.message.bytes. Delete the message.max.bytes config key to fall back to the Agent limit.

Agent Default (-maxProduceRecordSizeBytes)

Section titled “Agent Default (-maxProduceRecordSizeBytes)”

To change the default across Agents (applicable when neither max.message.bytes nor message.max.bytes is set), configure the Agent flag or environment variable and restart the Agents:

# Set the Agent default to 64 MiB (allowed range: 1 MiB to 256 MiB)
export WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES=67108864

Or pass the CLI flag directly:

warpstream agent -maxProduceRecordSizeBytes=67108864 ...

See the Agent Configuration Reference for all Agent flags.

Raising WarpStream’s server-side limit is only half the configuration—your producers and consumers must also allow records of that size:

  • Producer clients:
    • Java Kafka client: Increase max.request.size (and buffer.memory if needed).
    • librdkafka: Increase message.max.bytes.
    • franz-go: Increase kgo.ProducerBatchMaxBytes(...).
  • Consumer clients:
    • Ensure fetch.max.bytes and max.partition.fetch.bytes (or your client’s equivalents) are at least as large as the largest uncompressed record you expect to consume. WarpStream interprets Fetch byte limits as uncompressed bytes (see Compression).
    • On the Agent side, -kafkaMaxFetchRequestBytesUncompressedOverride and -kafkaMaxFetchPartitionBytesUncompressedOverride default to 256 MiB. If you have lowered either flag, keep it at least as large as your maximum record size.

See Tuning for Performance for client configuration examples.

Even if max.message.bytes or message.max.bytes is set to a larger value, the following transport and product limits still apply:

  • Kafka TCP request frame size (BYOC): WarpStream Agents enforce a maximum Kafka TCP request frame size of 256 MiB (268435456 bytes). Requests larger than 256 MiB on the wire cause the Agent to close the TCP connection.
  • Serverless clusters: Serverless clusters enforce fixed request limits (Max TCP Request Size: 32 MiB and Max Produce Request Uncompressed Bytes: 20 MiB). See Serverless Clusters.
  • HTTP Produce APIs: The native HTTP Produce API and Datadog HTTP Log Intake endpoints enforce a 64 MiB request body limit (after decompression), in addition to enforcing the effective maximum record size on each produced record. See HTTP Endpoints.