# Maximum Record Size

Overview of how WarpStream enforces maximum record size limits and how to adjust them at the topic, cluster, or Agent level.

WarpStream enforces a limit on the maximum size of each record written to a topic. This limit defaults to **32 MB** (`32000000` bytes) on BYOC clusters and can be adjusted at the topic level, the cluster (broker) level, or the Agent level.

## How It Differs from Apache Kafka

In open source Apache Kafka, `message.max.bytes` (broker) and `max.message.bytes` (topic) limit the size of a **compressed record batch** sent by a producer.

In WarpStream, maximum record size limits apply to each **individual uncompressed record** (its varint-framed byte length in the Kafka record format, including key, value, headers, and per-record metadata).

:::note
WarpStream enforces this limit per individual record rather than per producer batch because WarpStream Agents decompress incoming batches, write records into WarpStream's file format, and re-batch and re-compress them when serving consumers. Since producer batch sizes do not map 1:1 to consumer batch sizes, bounding individual record size prevents consumers from choking on oversized records.
:::

When a producer sends a record that exceeds the effective limit, WarpStream rejects the `Produce` request with Kafka error `MESSAGE_TOO_LARGE` (error code `10`) and emits the `produce_records_too_large` diagnostic.

## Configuration Levels and Precedence

WarpStream supports configuring the maximum record size at three levels, evaluated in strict precedence order:

| Precedence | Scope | Configuration | Default | Valid Values |
| --- | --- | --- | --- | --- |
| 1 (highest) | Topic | `max.message.bytes` | Unset (inherits from cluster or Agent) | `[0, ...]` (bytes) |
| 2 | Cluster (broker) | `message.max.bytes` | Unset (inherits from Agent) | `[0, ...]` (bytes) |
| 3 (lowest) | Agent | `-maxProduceRecordSizeBytes` / `WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES` | `32000000` (32 MB) | `1048576` (1 MiB) to `268435456` (256 MiB) |

### Precedence Rules

* **Higher-precedence settings override lower ones in either direction.** If `max.message.bytes` is set on a topic, WarpStream uses that value for the topic even if it is larger or smaller than cluster-level `message.max.bytes` or the Agent flag. Likewise, if topic-level `max.message.bytes` is unset and cluster-level `message.max.bytes` is set, the cluster-level value overrides the Agent flag.
* **Unset configs are omitted from `DescribeConfigs`.** When `max.message.bytes` (topic) or `message.max.bytes` (broker) is not explicitly set, WarpStream omits it from `DescribeConfigs` responses because the effective fallback depends on the Agent configuration.
* **Setting `0` does not reset to default.** Explicitly setting `max.message.bytes=0` or `message.max.bytes=0` enforces a `0`-byte limit and rejects all produced records. To fall back to the next level in the precedence chain, delete/unset the configuration key instead.

## How to Adjust the Limit

:::note
Prefer topic-level `max.message.bytes` or cluster-level `message.max.bytes` when you want to change the limit dynamically without restarting Agents, or when only specific topics need a different cap.
:::

### Per-Topic (`max.message.bytes`)

Set `max.message.bytes` on a topic using any Kafka client or tool supporting `AlterConfigs` / `IncrementalAlterConfigs` (such as `kafka-configs`), the [Topics REST API](/warpstream/reference/api-reference/topics/update-topic), or the [Terraform provider](/warpstream/agent-setup/infrastructure-as-code/terraform-provider). Delete the `max.message.bytes` config key to revert the topic back to the cluster or Agent limit.

### Cluster-Wide (`message.max.bytes`)

Set `message.max.bytes` at the broker/cluster level using any Kafka client or tool supporting broker `AlterConfigs` / `IncrementalAlterConfigs` (such as `kafka-configs`). This applies to all topics in the cluster that do not explicitly set `max.message.bytes`. Delete the `message.max.bytes` config key to fall back to the Agent limit.

### Agent Default (`-maxProduceRecordSizeBytes`)

To change the default across Agents (applicable when neither `max.message.bytes` nor `message.max.bytes` is set), configure the Agent flag or environment variable and restart the Agents:

```bash
# Set the Agent default to 64 MiB (allowed range: 1 MiB to 256 MiB)
export WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES=67108864
```

Or pass the CLI flag directly:

```bash
warpstream agent -maxProduceRecordSizeBytes=67108864 ...
```

See the [Agent Configuration Reference](/warpstream/kafka/advanced-agent-deployment-options/agent-configuration) for all Agent flags.

## Aligning Kafka Client Settings

Raising WarpStream's server-side limit is only half the configuration—your producers and consumers must also allow records of that size:

* **Producer clients:**
  * **Java Kafka client:** Increase `max.request.size` (and `buffer.memory` if needed).
  * **librdkafka:** Increase `message.max.bytes`.
  * **franz-go:** Increase `kgo.ProducerBatchMaxBytes(...)`.
* **Consumer clients:**
  * Ensure `fetch.max.bytes` and `max.partition.fetch.bytes` (or your client's equivalents) are at least as large as the largest uncompressed record you expect to consume. WarpStream interprets Fetch byte limits as **uncompressed** bytes (see [Compression](/warpstream/kafka/reference/compression#difference-with-kafka-for-fetch-requests)).
  * On the Agent side, `-kafkaMaxFetchRequestBytesUncompressedOverride` and `-kafkaMaxFetchPartitionBytesUncompressedOverride` default to `256 MiB`. If you have lowered either flag, keep it at least as large as your maximum record size.

See [Tuning for Performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) for client configuration examples.

## Related Hard Limits

Even if `max.message.bytes` or `message.max.bytes` is set to a larger value, the following transport and product limits still apply:

* **Kafka TCP request frame size (BYOC):** WarpStream Agents enforce a maximum Kafka TCP request frame size of **256 MiB** (`268435456` bytes). Requests larger than `256 MiB` on the wire cause the Agent to close the TCP connection.
* **Serverless clusters:** Serverless clusters enforce fixed request limits (`Max TCP Request Size: 32 MiB` and `Max Produce Request Uncompressed Bytes: 20 MiB`). See [Serverless Clusters](/warpstream/kafka/reference/serverless-clusters#ratelimits).
* **HTTP Produce APIs:** The native HTTP Produce API and Datadog HTTP Log Intake endpoints enforce a **64 MiB** request body limit (after decompression), in addition to enforcing the effective maximum record size on each produced record. See [HTTP Endpoints](/warpstream/kafka/reference/protocol-and-feature-support/http-endpoints).