Auto-tuning consumer settings to optimize fetch throughput
WarpStream differs from tradition Kafka in a couple ways
WarpStream has higher latency.
Agents interpret fetch limits in terms of uncompressed bytes instead of compressed bytes like Kafka does.
As such, sometimes customers need to tune their consumer setting to achieve better performance when migrating to WarpStream. To reduce the amount of manual tuning, the Agents automatically adjust the settings related to the amount of data to be returned in a fetch request on your behalf.
In most cases, this is fine, but in some cases it can lead to excessive memory usage in the consumer clients. To disable this feature at the Agent level, use the -autoTuneFetchLimits=false flag or set the environment variable WARPSTREAM_AUTO_TUNE_FETCH_LIMITS=FALSE .
Separately, you can also disable this feature on a per-client basis by adding the ws_dfat=trueclient ID feature to your clients.
