Agent Configuration Reference
Reference documentation for WarpStream Agent flags.
Required Command Line Flags and Environment Variables
All WarpStream Agent configurations can be set via command-line flags or environment variables. Command-line flags take precedence over environment variables.
Flag | Environment Variable | Description |
---|---|---|
|
| See the dedicated documentation section. |
|
| WarpStream Agent Key obtained from the WarpStream admin console. |
|
| WarpStream Virtual Cluster ID obtained from the WarpStream admin console. |
|
| WarpStream virtual cluster's control plane region. Can be obtained from the WarpStream admin console. |
Optional Command Line Flags and Environment Variables
All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag | Environment Variable | Description |
---|---|---|
|
| The port the Agent will listen on for Kafka client TCP connections. |
|
| How long the Agents will wait after receiving a sigterm signal before exiting. Useful for making contain shutdowns and rolling-restarts more graceful for clients with long metadata refresh intervals. |
|
| If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials. |
|
| If set to true, the Agents will require that all Kafka clients authenticate themselves with mTLS. |
|
| Regular expression to extract the ACL principal from the client certificate distinguished name. |
|
| Enable TLS for Kafka client/Agent connections. Must also specify |
|
| Path to the X.509 certificate file in PEM format for the server |
|
| Path to the X.509 private key file in PEM format for the server. |
|
| Path to the X.509 certificate file in PEM format for the client certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification. |
|
| The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc). |
N/A |
| Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata (see section below). We do not recommend overriding this in the general case. |
|
| Region that the WarpStream control plane is running in. Value for your cluster can be obtained from the WarpStream console. Optional if your control plane is in |
N/A |
| Override the log level of the WarpStream Agent from the default value of Defaults to |
|
| Controls the maximum amount of time the WarpStream Agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa. Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency. Defaults to 250ms, minimum is 50ms. |
|
| Controls the maximum number of bytes that will be buffered by the WarpStream Agents before flushing it to object storage. Increasing this value reduces object storage API costs for workloads that write more than uncompressed 16MiB/s/Agent, but increases latency, and vice versa. Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency. Defaults to 4MiB, minimum is 1MiB, maximum is 8MiB. |
|
| Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers. Defaults to 0.5GiB/vCPU if omitted. |
| If set to true, the WarpStream Agents will report their IP6 address instead of IP4 to the WarpStream discovery system. This is useful when running in VPCs that only support IP6, like fly.io. | |
N/A |
| Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address). This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP. |
|
| Which hostname strategy should be used the agent should advertise itself on. Accepted values:
If you select |
|
| Custom hostname value to advertise to service discovery for clustering purposes if the |
|
| Name of the 'group' that the Agent belongs to. This feature is used to isolate groups of Agents that belong to the same logical cluster, but should not communicate with each other because they're deployed in separate cloud accounts, vpcs, or regions. Leave blank to indicate the Agent belongs to the default group. |
|
| When enabled, the agent will synchronously try to resolve its az during startup for 1min, and will not start serving its |
|
| Disable the consumer group offset metrics automatically published by default ( |
|
| Comma-separated list of the tags to not expose in the consumer group offset metrics ( |
|
| Username for the external schema registry |
|
| Password for the external schema registry |
|
| Path to the X.509 certificate file in PEM format for the schema registry client |
|
| Path to the X.509 private key file in PEM format for the schema registry client. |
|
| Path to the X.509 certificate file in PEM format for the schema registry server's certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification. |
Last updated