Client Configuration Auto-tuning

How to allow the Agents to auto-tune settings

Auto-tuning consumer settings to optimize fetch throughput

WarpStream differs from tradition Kafka in a couple ways

1. WarpStream has higher latency.

2. Agents interpret fetch limits in terms of uncompressed bytes instead of compressed bytes like Kafka does.

3. WarpStream presents to the Kafka clients as if all of the topic-partitions are owned by a single Agent but some Kafka clients only issue one concurrent fetch request per broker by default.

As such, sometimes customers need to tune their consumer setting to achieve better performance when migrating to WarpStream. To reduce the amount of manual tuning, the Agents support an opt-in feature that automatically adjusts the settings related to the amount of data to be returned in a fetch request on your behalf. This can be enabled using the -autoTuneFetchLimits command-line flag or the WARPSTREAM_AUTO_TUNE_FETCH_LIMITS environment variable. If you prefer to manually tune your Kafka clients see this page for guidance.