Agent Configuration Reference

Reference documentation for WarpStream Agent flags.

Required Command Line Flags and Environment Variables

All WarpStream Agent configurations can be set via command-line flags or environment variables. Command-line flags take precedence over environment variables.

Flag	Environment Variable	Description
bucketURL	WARPSTREAM_BUCKET_URL	See the dedicated documentation section.
agentKey	WARPSTREAM_AGENT_KEY	WarpStream Agent Key obtained from the WarpStream admin console.
defaultVirtualClusterID	WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID	WarpStream Virtual Cluster ID obtained from the WarpStream admin console.
region	WARPSTREAM_REGION	WarpStream virtual cluster's control plane region. Can be obtained from the WarpStream admin console.

Optional Command Line Flags and Environment Variables

All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.

Flag	Environment Variable	Description
kafkaPort	WARPSTREAM_KAFKA_PORT	The port the Agent will listen on for Kafka client TCP connections.
gracefulShutdownDuration	WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION	How long the Agents will wait after receiving a sigterm signal before exiting. Useful for making contain shutdowns and rolling-restarts more graceful for clients with long metadata refresh intervals.
requireSASLAuthentication	WARPSTREAM_REQUIRE_SASL_AUTHENTICATION	If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials.
requireMTLSAuthentication	WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION	If set to true, the Agents will require that all Kafka clients authenticate themselves with mTLS. enableTLS must be set to true.
tlsPrincipalMappingRule	WARPSTREAM_TLS_PRINCIPAL_MAPPING_RULE	Regular expression to extract the ACL principal from the client certificate distinguished name. requireMTLSAuthentication must be set to true.
enableTLS	WARPSTREAM_TLS_ENABLED	Enable TLS for Kafka client/Agent connections. Must also specify tlsServerCertFile and tlsServerPrivateKeyFile.
tlsServerCertFile	WARPSTREAM_TLS_SERVER_CERT_FILE	Path to the X.509 certificate file in PEM format for the server
tlsServerPrivateKeyFile	WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE	Path to the X.509 private key file in PEM format for the server.
tlsClientCACertFile	WARPSTREAM_TLS_CLIENT_CA_CERT_FILE	Path to the X.509 certificate file in PEM format for the client certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification.
httpPort	WARPSTREAM_HTTP_PORT	The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc).
N/A	WARPSTREAM_AVAILABILITY_ZONE	Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata (see section below). We do not recommend overriding this in the general case.
region	WARPSTREAM_REGION	Region that the WarpStream control plane is running in. Value for your cluster can be obtained from the WarpStream console. Optional if your control plane is in us-east-1, otherwise must be provided.
N/A	WARPSTREAM_LOG_LEVEL	Override the log level of the WarpStream Agent from the default value of info. Acceptable values are debug, info, warn, and error. Defaults to info.
batchTimeout	WARPSTREAM_BATCH_TIMEOUT	Controls the maximum amount of time the WarpStream Agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa. Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency. Defaults to 250ms, minimum is 50ms.
batchMaxSizeBytes	WARPSTREAM_BATCH_MAX_SIZE_BYTES	Controls the maximum number of bytes that will be buffered by the WarpStream Agents before flushing it to object storage. Increasing this value reduces object storage API costs for workloads that write more than uncompressed 16MiB/s/Agent, but increases latency, and vice versa. Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency. Defaults to 4MiB, minimum is 1MiB, maximum is 8MiB.
fileCacheSizeBytes	WARPSTREAM_FILE_CACHE_SIZE_BYTES	Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers. Defaults to 0.5GiB/vCPU if omitted.
reportDiscoveryIP6	WARPSTREAM_REPORT_DISCOVERY_IP6	If set to true, the WarpStream Agents will report their IP6 address instead of IP4 to the WarpStream discovery system. This is useful when running in VPCs that only support IP6, like fly.io.
N/A	WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE	Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address). This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP.
advertiseHostnameStrategy	WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY	Which hostname strategy should be used the agent should advertise itself on. Accepted values: auto-ip4/auto-ip6/local/custom. auto-ip4 means that it will try to automatically find an IP v4 that makes sense auto-ip6 will do the same with an IPv6. local will use localhost If you select custom them you have to also define advertiseHostnameCustom.
advertiseHostnameCustom	WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM	Custom hostname value to advertise to service discovery for clustering purposes if the custom advertise strategy is selected.
agentGroup	WARPSTREAM_AGENT_GROUP	Name of the 'group' that the Agent belongs to. This feature is used to isolate groups of Agents that belong to the same logical cluster, but should not communicate with each other because they're deployed in separate cloud accounts, vpcs, or regions. Leave blank to indicate the Agent belongs to the default group.
availabilityZoneRequired	WARPSTREAM_AVAILABILITY_ZONE_REQUIRED	When enabled, the agent will synchronously try to resolve its az during startup for 1min, and will not start serving its /v1/status health check until it succeeds. The process will exit early if it did not manage to resolve the availability zone. Only used in agent mode.
disableConsumerGroupMetrics	WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS	Disable the consumer group offset metrics automatically published by default (warpstream_consumer_group_lag and warpstream_consumer_group_max_offset).
disableConsumerGroupsMetricsTags	WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS	Comma-separated list of the tags to not expose in the consumer group offset metrics (warpstream_consumer_group_lag and warpstream_consumer_group_max_offset).
externalSchemaRegistryBasicAuthUsername	WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_USERNAME	Username for the external schema registry
externalSchemaRegistryBasicAuthPassword	WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_PASSWORD	Password for the external schema registry
externalSchemaRegistryTlsClientCertFile	WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_CERT_FILE	Path to the X.509 certificate file in PEM format for the schema registry client
externalSchemaRegistryTlsClientPrivateKeyFile	WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_PRIVATE_KEY_FILE	Path to the X.509 private key file in PEM format for the schema registry client.
externalSchemaRegistryTlsServerCACertFile	WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_SERVER_CA_CERT_FILE	Path to the X.509 certificate file in PEM format for the schema registry server's certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification.