# Agent Configuration Reference

## Required Command Line Flags and Environment Variables

All WarpStream Agent configurations can be set via command-line flags or environment variables. Command-line flags take precedence over environment variables.

<table><thead><tr><th width="168">Flag</th><th>Environment Variable</th><th>Description</th></tr></thead><tbody><tr><td><code>bucketURL</code></td><td><code>WARPSTREAM_BUCKET_URL</code></td><td>See the <a href="../../agent-setup/different-object-stores">dedicated documentation section</a></td></tr><tr><td><code>agentKey</code></td><td><code>WARPSTREAM_AGENT_KEY</code></td><td>WarpStream Agent Key obtained from the WarpStream admin console.</td></tr><tr><td><code>defaultVirtualClusterID</code></td><td><code>WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID</code></td><td>WarpStream Virtual Cluster ID obtained from the WarpStream admin console.</td></tr><tr><td><code>region</code></td><td><code>WARPSTREAM_REGION</code></td><td>WarpStream virtual cluster's control plane region. Can be obtained from the WarpStream admin console.</td></tr></tbody></table>

## Optional Command Line Flags and Environment Variables

All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.

<table><thead><tr><th width="221.98657718120808">Flag</th><th>Environment Variable</th><th>Description</th></tr></thead><tbody><tr><td><code>apiKey</code></td><td><code>WARPSTREAM_API_KEY</code></td><td>Backward-compatible alias of agentKey</td></tr><tr><td><code>agentGroup</code></td><td><code>WARPSTREAM_AGENT_GROUP</code></td><td>Name of the 'group' that the Agent belongs to. This feature is used to isolate groups of Agents that belong to the same logical cluster, but should not communicate with each other because they're deployed in separate cloud accounts, vpcs, or regions. Leave blank to indicate the Agent belongs to the default group.</td></tr><tr><td><code>heartbeatEvery</code></td><td><code>WARPSTREAM_HEARTBEAT_EVERY</code></td><td>How often the agent should heartbeat the WarpStream backend. Recommended to not modify this.</td></tr><tr><td><code>httpPort</code></td><td><code>WARPSTREAM_HTTP_PORT</code></td><td>The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc) (default 8080).</td></tr><tr><td><code>enableKafka</code></td><td><code>WARPSTREAM_ENABLE_KAFKA</code></td><td>Enable kafka server (default true).</td></tr><tr><td><code>kafkaPort</code></td><td><code>WARPSTREAM_KAFKA_PORT</code></td><td>The port the Agent will listen on for Kafka client TCP connections (default 9092).</td></tr><tr><td><code>kafkaFetchCompression</code></td><td><code>WARPSTREAM_KAFKA_FETCH_COMPRESSION</code></td><td>Compression type to use for Fetch responses: none, gzip, snappy, lz4 (by default), zstd. This is only used if no compression is set explicitly, or if 'agent' type compress.</td></tr><tr><td><code>kafkaMetadataRefreshInterval</code></td><td><code>WARPSTREAM_KAFKA_METADATA_REFRESH_INTERVAL</code></td><td>Period of time at which topic metadata is refreshed. Unlike Kafka, this metadata cache refresh also affects the timestamp type associated with a stream (default 1m0s).</td></tr><tr><td><code>kafkaHandleConsumerGroupsInBackend</code></td><td><code>WARPSTREAM_KAFKA_HANDLE_CONSUMER_GROUPS_IN_BACKEND</code></td><td>Handle consumer group 'JoinGroup' and 'SyncGroup' requests in the backend instead of in the agent. When handled in the backend, the 'Rebalance Timeout' is always set to 10 seconds, whereas in the agent, it will be determined by client specifications. Enabling this option offers the advantage of reduced error potential and seamless integration of backend improvements and bug fixes. However, exercise caution when enabling it for large consumer groups, as a 10-second rebalance timeout may lead to extended rebalancing times and consequently, prolonged consumption pauses. Warning: Ensure uniformity within your agent pool regarding this setting. Having a mix of enabled and disabled settings may lead to rebalancing issues and potential disruptions.</td></tr><tr><td><code>kafkaHighCardinalityMetrics</code></td><td><code>WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS</code></td><td>Whether to emit metrics with high cardinality tags. When set to true, it enables detailed tracking at a granular level, such as metrics for individual fetch and produce operations on a per-topic basis. Use with caution as high cardinality can significantly increase the amount of data collected, potentially impacting performance.</td></tr><tr><td><code>kafkaCloseIdeConnAfter</code></td><td><code>WARPSTREAM_KAFKA_CLOSE_IDLE_CONN_AFTER</code></td><td>Close idle connections after the number of duration specified by this config (default 10m0s).</td></tr><tr><td><code>kafkaMaxFetchRequestBytesUncompressedOverride</code></td><td><code>WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE</code></td><td>Maximum number of uncompressed bytes that can be fetched in a single fetch request (default 128MiB).</td></tr><tr><td><code>kafkaMaxFetchPartitionBytesUncompressedOverride</code></td><td><code>WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE</code></td><td>Maximum number of uncompressed bytes that can be fetched in a single fetch request for a single topic-partition (default 128MiB).</td></tr><tr><td><code>fileCacheSizeBytes</code></td><td><code>WARPSTREAM_FILE_CACHE_SIZE_BYTES</code></td><td>Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers.<br><br>Defaults to 0.5GiB/vCPU if omitted.</td></tr><tr><td><code>fileCacheExtraReplicas</code></td><td><code>WARPSTREAM_FILE_CACHE_EXTRA_REPLICAS</code></td><td>Number of extra replicas for the distributed file cache. Helps improve availability and reduce errors when Agents shutdown ungracefully. You can override this to 0, but do not increase this value above 1 unless you know what you're doing.</td></tr><tr><td><code>gracefulShutdownDuration</code></td><td><code>WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION</code></td><td>Amount of time to wait after receiving SIGTERM before exiting to allow graceful removal from service discovery (default 1m0s).</td></tr><tr><td><code>maxConcurrentRequestPerCPU</code></td><td><code>WARPSTREAM_MAX_CONCURRENT_REQUEST_PER_CPU</code></td><td>Maximum number of concurrent requests (per CPU) allowed by the Kafka server.</td></tr><tr><td><code>enableClusterWideEnvironment</code></td><td><code>WARPSTREAM_ENABLE_CLUSTER_WIDE_ENVIRONMENT</code></td><td>Whether the cluster wide environment should be enabled.</td></tr><tr><td><code>clusterWideEnvironmentPort</code></td><td><code>WARPSTREAM_CLUSTER_WIDE_ENVIRONMENT_PORT</code></td><td>The default port to use for the cluster wide environment (default 9999).</td></tr><tr><td><code>ingestionBucketURL</code></td><td><code>WARPSTREAM_INGESTION_BUCKET_URL</code></td><td>Object storage URL to use for data ingestion (produce requests).</td></tr><tr><td><code>compactionBucketURL</code></td><td><code>WARPSTREAM_COMPACTION_BUCKET_URL</code></td><td>Object storage URL to use for files created by compactions.</td></tr><tr><td><code>batchTimeout</code></td><td><code>WARPSTREAM_BATCH_TIMEOUT</code></td><td>Controls the maximum amount of time the WarpStream Agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa.<br><br>Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency.<br><br>Defaults to 250ms, minimum is 50ms.</td></tr><tr><td><code>batchMaxSizeBytes</code></td><td><code>WARPSTREAM_BATCH_MAX_SIZE_BYTES</code></td><td>Controls the maximum number of bytes that will be buffered by the WarpStream Agents before flushing it to object storage. Increasing this value reduces object storage API costs for workloads that write more than uncompressed 16MiB/s/Agent, but increases latency, and vice versa.<br><br>Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency.<br><br>Defaults to 4MiB, minimum is 1MiB, maximum is 64MiB.</td></tr><tr><td><code>batcherMaxInflightBytesPerCPU</code></td><td><code>WARPSTREAM_BATCHER_MAX_INFLIGHT_BYTES_PER_CPU</code></td><td>Maximum number of inflight bytes per CPU from Produce requests that have not yet been flushed to object storage that can be in memory at once before the Agent will begin backpressuring.</td></tr><tr><td><code>batcherMaxInflightFilesPerCPU</code></td><td><code>WARPSTREAM_BATCHER_MAX_INFLIGHT_FILES_PER_CPU</code></td><td>Maximum number of inflight files per CPU from Produce requests that have not yet been flushed to object storage that can be in memory at once before the Agent will begin backpressuring.</td></tr><tr><td><code>metadataURL</code></td><td><code>WARPSTREAM_METADATA_URL</code></td><td>Address for WarpStream metadata backend (favor using the <code>-region</code>flag instead).</td></tr><tr><td><code>schemaRegistryURL</code></td><td><code>WARPSTREAM_SCHEMA_REGISTRY_URL</code></td><td>Address for WarpStream schema registry backend.</td></tr><tr><td><code>schemaRegistryPort</code></td><td><code>WARPSTREAM_SCHEMA_REGISTRY_PORT</code></td><td>Port to run the schema registry server on (default 9094).</td></tr><tr><td><code>region</code></td><td><code>WARPSTREAM_REGION</code></td><td>Region that the WarpStream control plane is running in. Value for your cluster can be obtained from the WarpStream console. Optional if your control plane is in <code>us-east-1</code>, otherwise must be provided.</td></tr><tr><td><code>kafkaLoadBalancingInterval</code></td><td><code>WARPSTREAM_KAFKA_LOAD_BALANCING_INTERVAL</code></td><td>Time after which the Kafka connection will be closed. This mechanism helps load balance the clients by forcing them to query the magic URL again. By resetting the connection periodically, clients are evenly distributed across available Kafka connections. (default 8760h0m0s).</td></tr><tr><td><code>kafkaInterzoneLoadBalancingInterval</code></td><td><code>WARPSTREAM_KAFKA_INTERZONE_LOAD_BALANCING_INTERVAL</code></td><td>Interval at which the Kafka connection assesses if the client-agent connection resides in the same Availability Zone (AZ). If they are not in the same AZ and there are agents available within the client's AZ, the connection is terminated. This approach encourages load balancing by prompting clients to re-query the magic URL and, consequently, connect to agents within their respective AZ. For this mechanism to function, clients should include 'waprstream_az=X' and 'warpstream_interzone_lb=true' in their clientID. (default 1m0s).</td></tr><tr><td><code>kafkaLoadBalancingDrainingTime</code></td><td><code>WARPSTREAM_KAFKA_LOAD_BALANCING_DRAINING_TIME</code></td><td>Time given to gracefully close the Kafka connection after the reconnect interval is reached.</td></tr><tr><td><code>advertiseHostnameStrategy</code></td><td><code>WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY</code></td><td><p>Which hostname strategy should be used the agent should advertise itself on. Accepted values: <code>auto-ip4</code>/<code>auto-ip6</code>/<code>local</code>/<code>custom</code>.</p><p><code>auto-ip4</code> means that it will try to automatically find an IP v4 that makes sense</p><p><code>auto-ip6</code> will do the same with an IPv6.</p><p><code>local</code> will use <code>localhost</code></p><p>If you select <code>custom</code> them you have to also define <code>advertiseHostnameCustom</code>.</p></td></tr><tr><td><code>advertiseHostnameCustom</code></td><td><code>WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM</code></td><td>Custom hostname value to advertise to service discovery for clustering purposes if the <code>custom</code> advertise strategy is selected.</td></tr><tr><td><code>requireSASLAuthentication</code></td><td><code>WARPSTREAM_REQUIRE_SASL_AUTHENTICATION</code></td><td>If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials.</td></tr><tr><td><code>enabledSASLMechanisms</code></td><td><code>WARPSTREAM_SASL_ENABLED_MECHANISMS</code></td><td>If you provide it with a comma-separated list of SASL mechanisms, only those will be enabled. Valid values are "PLAIN" and "SCRAM-SHA-512". For example, if you set WARPSTREAM_SASL_ENABLED_MECHANISMS="SCRAM-SHA-512", you will not be able to use the PLAIN mechanism to connect, only SCRAM-SHA-512.</td></tr><tr><td><code>logInterval</code></td><td><code>WARPSTREAM_LOG_INTERVAL</code></td><td>Interval for logging service status (default 15s).</td></tr><tr><td><code>enableDatadogProfiling</code></td><td><code>WARPSTREAM_ENABLE_DATADOG_PROFILING</code></td><td>Enable datadog profiling (default false).</td></tr><tr><td><code>enableDatadogTracing</code></td><td><code>WARPSTREAM_ENABLE_DATADOG_TRACING</code></td><td>Enable datadog tracing (default false).</td></tr><tr><td><code>enablePrometheusMetrics</code></td><td><code>WARPSTREAM_ENABLE_PROMETHEUS_METRICS</code></td><td>Enable prometheus metrics (default true).</td></tr><tr><td><code>enableDatadogMetrics</code></td><td><code>WARPSTREAM_ENABLE_DATADOG_METRICS</code></td><td>Enable datadog metrics (default false).</td></tr><tr><td><code>disableConsumerGroupMetrics</code></td><td><code>WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS</code></td><td>Disable the consumer group offset metrics automatically published by default (<code>warpstream_consumer_group_lag</code> and <code>warpstream_consumer_group_max_offset</code>).</td></tr><tr><td><code>disableConsumerGroupsMetricsTags</code></td><td><code>WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS</code></td><td>Comma-separated list of the tags to not expose in the consumer group offset metrics (<code>warpstream_consumer_group_lag</code> and <code>warpstream_consumer_group_max_offset</code>).</td></tr><tr><td><code>disableLogsCollection</code></td><td><code>WARPSTREAM_DISABLE_LOGS_COLLECTION</code></td><td>Disable the logs collection sent to warpstream backend (default false).</td></tr><tr><td><code>roles</code></td><td><code>WARPSTREAM_AGENT_ROLES</code></td><td>Roles that the agent should start (comma-separated) (default "proxy, jobs").</td></tr><tr><td><code>bentoBucketURL</code></td><td><code>WARPSTREAM_BENTO_BUCKET_URL</code></td><td>Bucket URL to use when fetching the bento configuration.</td></tr><tr><td><code>bentoConfigPath</code></td><td><code>WARPSTREAM_BENTO_CONFIG_PATH</code></td><td>Path in the bucket to fetch the bento configuration.</td></tr><tr><td><code>enableManagedPipelines</code></td><td><code>WARPSTREAM_ENABLE_MANAGED_PIPELINES</code></td><td>Whether data pipelines can be managed by the control plane.</td></tr><tr><td><code>availabilityZoneRequired</code></td><td><code>WARPSTREAM_AVAILABILITY_ZONE_REQUIRED</code></td><td>When enabled, the agent will synchronously try to resolve its az during startup for 1min, and will not start serving its <code>/v1/status</code> health check until it succeeds. The process will exit early if it did not manage to resolve the availability zone. Only used in <code>agent</code> mode.</td></tr><tr><td><code>kafkaTLS</code></td><td><code>WARPSTREAM_TLS_ENABLED</code></td><td>Enable TLS for Kafka client/Agent connections. Must also specify <code>tlsServerCertFile</code> and <code>tlsServerPrivateKeyFile</code>.</td></tr><tr><td><code>schemaRegistryTLS</code></td><td><code>WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLED</code></td><td>Enable TLS encryption over the schema registry port.</td></tr><tr><td><code>tlsServerCertFile</code></td><td><code>WARPSTREAM_TLS_SERVER_CERT_FILE</code></td><td>Path to the X.509 certificate file in PEM format for the server.</td></tr><tr><td><code>tlsServerPrivateKeyFile</code></td><td><code>WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE</code></td><td>Path to the X.509 private key file in PEM format for the server.</td></tr><tr><td><code>tlsClientCACertFile</code></td><td><code>WARPSTREAM_TLS_CLIENT_CA_CERT_FILE</code></td><td>Path to the X.509 certificate file in PEM format for the client certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification.</td></tr><tr><td><code>requireMTLSAuthentication</code></td><td><code>WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION</code></td><td>If set to true, the Agents will require that all Kafka clients authenticate themselves with mTLS. <code>enableTLS</code> must be set to <code>true</code>.</td></tr><tr><td><code>tlsPrincipalMappingRule</code></td><td><code>WARPSTREAM_TLS_PRINCIPAL_MAPPING_RULE</code></td><td>Regular expression to extract the ACL principal from the client certificate distinguished name. <code>requireMTLSAuthentication</code> must be set to <code>true</code>.</td></tr><tr><td><code>storageCompression</code></td><td><code>WARPSTREAM_STORAGE_COMPRESSION</code></td><td>Compression used in object store, either zstd or lz4.</td></tr><tr><td><code>externalSchemaRegistryBasicAuthUsername</code></td><td><code>WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_USERNAME</code></td><td>Username for the external schema registry.</td></tr><tr><td><code>externalSchemaRegistryBasicAuthPassword</code></td><td><code>WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_PASSWORD</code></td><td>Password for the external schema registry.</td></tr><tr><td><code>externalSchemaRegistryTlsServerCACertFile</code></td><td><code>WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_SERVER_CA_CERT_FILE</code></td><td>Path to the X.509 certificate file in PEM format for the schema registry server's certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification.</td></tr><tr><td><code>externalSchemaRegistryTlsClientCertFile</code></td><td><code>WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_CERT_FILE</code></td><td>Path to the X.509 certificate file in PEM format for the schema registry client.</td></tr><tr><td><code>externalSchemaRegistryTlsClientPrivateKeyFile</code></td><td><code>WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_PRIVATE_KEY_FILE</code></td><td>Path to the X.509 private key file in PEM format for the schema registry client.</td></tr><tr><td><code>enableSetMutexProfileFraction</code></td><td><code>WARPSTREAM_ENABLE_SET_MUTEX_PROFILE_FRACTION</code></td><td>Enable this flag to call <code>runtime.SetMutexProfileFraction</code> with the value passed along <code>-mutexProfileFraction</code>.</td></tr><tr><td><code>mutexProfileFraction</code></td><td><code>WARPSTREAM_MUTEX_PROFILE_FRACTION</code></td><td>Tune the value passed to call <code>runtime.SetMutexProfileFraction</code> (default 10).</td></tr><tr><td><code>enableSetBlockProfileRate</code></td><td><code>WARPSTREAM_ENABLE_SET_BLOCK_PROFILE_RATE</code></td><td>Enable this flag to call <code>runtime.SetBlockProfileRate</code> with the value passed along <code>-blockProfileRate</code>.</td></tr><tr><td><code>blockProfileRate</code></td><td><code>WARPSTREAM_BLOCK_PROFILE_RATE</code></td><td>Tune the value passed to call <code>runtime.SetBlockProfileRate</code> (default 100000000).</td></tr><tr><td><code>maxProduceRecordSizeBytes</code></td><td><code>WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES</code></td><td>Maximum size of a record that can be produced. Value needs to be between 4MiB and 256 MiB (default 32 MB).</td></tr><tr><td><code>disableProfileForwarding</code></td><td><code>WARPSTREAM_DISABLE_PROFILE_FORWARDING</code></td><td>Disable profile forwarding to warpstream backend. Note that if both Datadog profiling and profile forwarding are on, profile forwarding will automatically be turned off (default false).</td></tr><tr><td><code>maxProfileSize</code></td><td><code>WARPSTREAM_MAX_PROFILE_SIZE</code></td><td>Maximum number of bytes for buffering profiles in memory. Value needs to be smaller than 1 MiB (default 500 KiB). This is only used if <code>-disableProfileForwarding</code> is <code>false</code>.</td></tr><tr><td><code>zonedCIDRBlocks</code></td><td><code>WARPSTREAM_ZONED_CIDR_BLOCKS</code></td><td>A mapping of availability zones to Kafka client IPs. The mapping should be a <code>&#x3C;></code> delimited list of AZ to CIDR range pairs, where each pair starts with an AZ, a <code>@</code>, and a comma separated list of CIDR blocks for that given AZ. For example, <code>us-east-1a@10.0.0.0/19,10.0.32.0/19&#x3C;>us-east-1b@10.0.64.0/19&#x3C;>us-east-1c@10.0.96.0/19</code>.</td></tr><tr><td><code>autoTuneFetchLimits</code></td><td><code>WARPSTREAM_AUTO_TUNE_FETCH_LIMITS</code></td><td>Allow consumer fetch limits to be auto-adjusted (default true).</td></tr><tr><td>N/A</td><td><code>WARPSTREAM_AVAILABILITY_ZONE</code></td><td><p>Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata (see <a href="#availability-zone-automatic-detection">section</a> below).</p><p>We do not recommend overriding this in the general case.</p></td></tr><tr><td>N/A</td><td><code>WARPSTREAM_LOG_LEVEL</code></td><td><p>Override the log level of the WarpStream Agent from the default value of <code>info</code>. Acceptable values are <code>debug</code>, <code>info</code>, <code>warn</code>, and <code>error</code>.</p><p>Defaults to <code>info</code>.</p></td></tr><tr><td>N/A</td><td><code>WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE</code></td><td>Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address).<br><br>This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP.</td></tr></tbody></table>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/agent-configuration.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.