Comment on page

Agent Configuration

Required Command Line Flags and Environment Variables
All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag
Environment Variable
Description
bucketURL
WARPSTREAM_BUCKET_URL
See the dedicated documentation section.
apiKey
WARPSTREAM_API_KEY
WarpStream API key obtained from the WarpStream admin console.
defaultVirtualClusterID
WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID
WarpStream Virtual Cluster ID obtained from the WarpStream admin console.
agentPoolName
WARPSTREAM_AGENT_POOL_NAME
(Optional) WarpStream Agent Pool name obtained from the WarpStream admin console. 

This argument is optional if you're using the "default" Virtual Cluster that was automatically created when you signed up for WarpStream.
​
However, it must be set if you're using a new Virtual Cluster that you created yourself. 

You can read more about this in the Agent Pools and Virtual Clusters Reference Documentation.
Optional Command Line Flags and Environment Variables
All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag
Environment Variable
Description
requireAuthentication
WARPSTREAM_REQUIRE_AUTHENTICATION
If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials.
kafkaPort
WARPSTREAM_KAFKA_PORT
The port the Agent will listen on for Kafka client TCP connections.
httpPort
WARPSTREAM_HTTP_PORT
The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc).
N/A
WARPSTREAM_AVAILABILITY_ZONE
Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata.
We do not recommend overriding this in the general case.
N/A
WARPSTREAM_LOG_LEVEL
Override the log level of the WarpStream Agent from the default value of info. Acceptable values are debug, info, warn, and error.
Defaults to info.
batchTimeout
WARPSTREAM_BATCH_TIMEOUT
Controls the maximum amount of time the WarpStream agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa.

Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency.

Defaults to 250ms, minimum is 50ms.
fileCacheSizeBytes
WARPSTREAM_FILE_CACHE_SIZE_BYTES
Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers.

Defaults to 0.5GiB/vCPU if omitted.
reportDiscoveryIP6
WARPSTREAM_REPORT_DISCOVERY_IP6    
If set to true, the WarpStream Agents will report their IP6 address instead of IP4 to the WarpStream discovery system.

This is useful when running in VPCs that only support IP6, like fly.io.
N/A
WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE
Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address).

This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP.
advertiseHostnameStrategy
WARPSTREAM_ADVERTISED_HOSTNAME_STRATEGY
Which hostname strategy should be used the agent should advertise itself on. Accepted values: auto-ip4/auto-ip6/local/custom. 
​
auto-ip4 means that it will try to automatically find an IP v4 that makes sense
​
auto-ip6 will do the same with an IPv6.
​
local will use localhost
​
If you select custom them you have to also define advertiseHostnameCustom.
advertiseHostnameCustom
WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM
Custom hostname value to advertise to service discovery for clustering purposes if the custom advertise strategy is selected.

Flag	Environment Variable	Description
`bucketURL`	`WARPSTREAM_BUCKET_URL`	See the dedicated documentation section.
`apiKey`	`WARPSTREAM_API_KEY`	WarpStream API key obtained from the WarpStream admin console.
`defaultVirtualClusterID`	`WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID`	WarpStream Virtual Cluster ID obtained from the WarpStream admin console.
`agentPoolName`	`WARPSTREAM_AGENT_POOL_NAME`	(Optional) WarpStream Agent Pool name obtained from the WarpStream admin console. This argument is optional if you're using the "default" Virtual Cluster that was automatically created when you signed up for WarpStream. However, it must be set if you're using a new Virtual Cluster that you created yourself. You can read more about this in the Agent Pools and Virtual Clusters Reference Documentation.

Flag	Environment Variable	Description
`requireAuthentication`	`WARPSTREAM_REQUIRE_AUTHENTICATION`	If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials.
`kafkaPort`	`WARPSTREAM_KAFKA_PORT`	The port the Agent will listen on for Kafka client TCP connections.
`httpPort`	`WARPSTREAM_HTTP_PORT`	The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc).
N/A	`WARPSTREAM_AVAILABILITY_ZONE`	Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata. We do not recommend overriding this in the general case.
N/A	`WARPSTREAM_LOG_LEVEL`	Override the log level of the WarpStream Agent from the default value of `info`. Acceptable values are `debug`, `info`, `warn`, and `error`. Defaults to `info`.
`batchTimeout`	`WARPSTREAM_BATCH_TIMEOUT`	Controls the maximum amount of time the WarpStream agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa. Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency. Defaults to 250ms, minimum is 50ms.
`fileCacheSizeBytes`	`WARPSTREAM_FILE_CACHE_SIZE_BYTES`	Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers. Defaults to 0.5GiB/vCPU if omitted.
`reportDiscoveryIP6`	WARPSTREAM_REPORT_DISCOVERY_IP6	If set to true, the WarpStream Agents will report their IP6 address instead of IP4 to the WarpStream discovery system. This is useful when running in VPCs that only support IP6, like fly.io.
N/A	`WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE`	Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address). This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP.
`advertiseHostnameStrategy`	`WARPSTREAM_ADVERTISED_HOSTNAME_STRATEGY`	Which hostname strategy should be used the agent should advertise itself on. Accepted values: `auto-ip4`/`auto-ip6`/`local`/`custom`. `auto-ip4` means that it will try to automatically find an IP v4 that makes sense `auto-ip6` will do the same with an IPv6. `local` will use `localhost` If you select `custom` them you have to also define `advertiseHostnameCustom`.
`advertiseHostnameCustom`	`WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM`	Custom hostname value to advertise to service discovery for clustering purposes if the `custom` advertise strategy is selected.

Describe All Consumer Groups

Next - Background Information

WarpStream's Architecture

Last modified 13d ago