Links
Comment on page

Agent Configuration

Required Command Line Flags and Environment Variables

All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag
Environment Variable
Description
bucketURL
WARPSTREAM_BUCKET_URL
apiKey
WARPSTREAM_API_KEY
WarpStream API key obtained from the WarpStream admin console.
defaultVirtualClusterID
WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID
WarpStream Virtual Cluster ID obtained from the WarpStream admin console.
agentPoolName
WARPSTREAM_AGENT_POOL_NAME
(Optional) WarpStream Agent Pool name obtained from the WarpStream admin console. This argument is optional if you're using the "default" Virtual Cluster that was automatically created when you signed up for WarpStream.
However, it must be set if you're using a new Virtual Cluster that you created yourself. You can read more about this in the Agent Pools and Virtual Clusters Reference Documentation.

Optional Command Line Flags and Environment Variables

All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag
Environment Variable
Description
requireAuthentication
WARPSTREAM_REQUIRE_AUTHENTICATION
If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials.
kafkaPort
WARPSTREAM_KAFKA_PORT
The port the Agent will listen on for Kafka client TCP connections.
httpPort
WARPSTREAM_HTTP_PORT
The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc).
N/A
WARPSTREAM_AVAILABILITY_ZONE
Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata.
We do not recommend overriding this in the general case.
N/A
WARPSTREAM_LOG_LEVEL
Override the log level of the WarpStream Agent from the default value of info. Acceptable values are debug, info, warn, and error.
Defaults to info.
batchTimeout
WARPSTREAM_BATCH_TIMEOUT
Controls the maximum amount of time the WarpStream agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa. Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency. Defaults to 250ms, minimum is 50ms.
fileCacheSizeBytes
WARPSTREAM_FILE_CACHE_SIZE_BYTES
Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers. Defaults to 0.5GiB/vCPU if omitted.
reportDiscoveryIP6
WARPSTREAM_REPORT_DISCOVERY_IP6
If set to true, the WarpStream Agents will report their IP6 address instead of IP4 to the WarpStream discovery system. This is useful when running in VPCs that only support IP6, like fly.io.
N/A
WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE
Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address). This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP.
advertiseHostnameStrategy
WARPSTREAM_ADVERTISED_HOSTNAME_STRATEGY
Which hostname strategy should be used the agent should advertise itself on. Accepted values: auto-ip4/auto-ip6/local/custom.
auto-ip4 means that it will try to automatically find an IP v4 that makes sense
auto-ip6 will do the same with an IPv6.
local will use localhost
If you select custom them you have to also define advertiseHostnameCustom.
advertiseHostnameCustom
WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM
Custom hostname value to advertise to service discovery for clustering purposes if the custom advertise strategy is selected.
Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. Kinesis is a trademark of Amazon Web Services.