Maximum Record Size
Overview of how WarpStream enforces maximum record size limits and how to adjust them at the topic, cluster, or Agent level.
WarpStream enforces a limit on the maximum size of each record written to a topic. This limit defaults to 32 MB (32000000 bytes) on BYOC clusters and can be adjusted at the topic level, the cluster (broker) level, or the Agent level.
How It Differs from Apache Kafka
Section titled “How It Differs from Apache Kafka”In open source Apache Kafka, message.max.bytes (broker) and max.message.bytes (topic) limit the size of a compressed record batch sent by a producer.
In WarpStream, maximum record size limits apply to each individual uncompressed record (its varint-framed byte length in the Kafka record format, including key, value, headers, and per-record metadata).
When a producer sends a record that exceeds the effective limit, WarpStream rejects the Produce request with Kafka error MESSAGE_TOO_LARGE (error code 10) and emits the produce_records_too_large diagnostic.
Configuration Levels and Precedence
Section titled “Configuration Levels and Precedence”WarpStream supports configuring the maximum record size at three levels, evaluated in strict precedence order:
| Precedence | Scope | Configuration | Default | Valid Values |
|---|---|---|---|---|
| 1 (highest) | Topic | max.message.bytes |
Unset (inherits from cluster or Agent) | [0, ...] (bytes) |
| 2 | Cluster (broker) | message.max.bytes |
Unset (inherits from Agent) | [0, ...] (bytes) |
| 3 (lowest) | Agent | -maxProduceRecordSizeBytes / WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES |
32000000 (32 MB) |
1048576 (1 MiB) to 268435456 (256 MiB) |
Precedence Rules
Section titled “Precedence Rules”- Higher-precedence settings override lower ones in either direction. If
max.message.bytesis set on a topic, WarpStream uses that value for the topic even if it is larger or smaller than cluster-levelmessage.max.bytesor the Agent flag. Likewise, if topic-levelmax.message.bytesis unset and cluster-levelmessage.max.bytesis set, the cluster-level value overrides the Agent flag. - Unset configs are omitted from
DescribeConfigs. Whenmax.message.bytes(topic) ormessage.max.bytes(broker) is not explicitly set, WarpStream omits it fromDescribeConfigsresponses because the effective fallback depends on the Agent configuration. - Setting
0does not reset to default. Explicitly settingmax.message.bytes=0ormessage.max.bytes=0enforces a0-byte limit and rejects all produced records. To fall back to the next level in the precedence chain, delete/unset the configuration key instead.
How to Adjust the Limit
Section titled “How to Adjust the Limit”Per-Topic (max.message.bytes)
Section titled “Per-Topic (max.message.bytes)”Set max.message.bytes on a topic using any Kafka client or tool supporting AlterConfigs / IncrementalAlterConfigs (such as kafka-configs), the Topics REST API, or the Terraform provider. Delete the max.message.bytes config key to revert the topic back to the cluster or Agent limit.
Cluster-Wide (message.max.bytes)
Section titled “Cluster-Wide (message.max.bytes)”Set message.max.bytes at the broker/cluster level using any Kafka client or tool supporting broker AlterConfigs / IncrementalAlterConfigs (such as kafka-configs). This applies to all topics in the cluster that do not explicitly set max.message.bytes. Delete the message.max.bytes config key to fall back to the Agent limit.
Agent Default (-maxProduceRecordSizeBytes)
Section titled “Agent Default (-maxProduceRecordSizeBytes)”To change the default across Agents (applicable when neither max.message.bytes nor message.max.bytes is set), configure the Agent flag or environment variable and restart the Agents:
# Set the Agent default to 64 MiB (allowed range: 1 MiB to 256 MiB)export WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES=67108864Or pass the CLI flag directly:
warpstream agent -maxProduceRecordSizeBytes=67108864 ...See the Agent Configuration Reference for all Agent flags.
Aligning Kafka Client Settings
Section titled “Aligning Kafka Client Settings”Raising WarpStream’s server-side limit is only half the configuration—your producers and consumers must also allow records of that size:
- Producer clients:
- Java Kafka client: Increase
max.request.size(andbuffer.memoryif needed). - librdkafka: Increase
message.max.bytes. - franz-go: Increase
kgo.ProducerBatchMaxBytes(...).
- Java Kafka client: Increase
- Consumer clients:
- Ensure
fetch.max.bytesandmax.partition.fetch.bytes(or your client’s equivalents) are at least as large as the largest uncompressed record you expect to consume. WarpStream interprets Fetch byte limits as uncompressed bytes (see Compression). - On the Agent side,
-kafkaMaxFetchRequestBytesUncompressedOverrideand-kafkaMaxFetchPartitionBytesUncompressedOverridedefault to256 MiB. If you have lowered either flag, keep it at least as large as your maximum record size.
- Ensure
See Tuning for Performance for client configuration examples.
Related Hard Limits
Section titled “Related Hard Limits”Even if max.message.bytes or message.max.bytes is set to a larger value, the following transport and product limits still apply:
- Kafka TCP request frame size (BYOC): WarpStream Agents enforce a maximum Kafka TCP request frame size of 256 MiB (
268435456bytes). Requests larger than256 MiBon the wire cause the Agent to close the TCP connection. - Serverless clusters: Serverless clusters enforce fixed request limits (
Max TCP Request Size: 32 MiBandMax Produce Request Uncompressed Bytes: 20 MiB). See Serverless Clusters. - HTTP Produce APIs: The native HTTP Produce API and Datadog HTTP Log Intake endpoints enforce a 64 MiB request body limit (after decompression), in addition to enforcing the effective maximum record size on each produced record. See HTTP Endpoints.