Comment on page
Agent Configuration
All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag | Environment Variable | Description |
---|---|---|
bucketURL | WARPSTREAM_BUCKET_URL | |
apiKey | WARPSTREAM_API_KEY | WarpStream API key obtained from the WarpStream admin console. |
defaultVirtualClusterID | WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID | WarpStream Virtual Cluster ID obtained from the WarpStream admin console. |
agentPoolName | WARPSTREAM_AGENT_POOL_NAME | (Optional) WarpStream Agent Pool name obtained from the WarpStream admin console.
This argument is optional if you're using the "default" Virtual Cluster that was automatically created when you signed up for WarpStream. However, it must be set if you're using a new Virtual Cluster that you created yourself.
You can read more about this in the Agent Pools and Virtual Clusters Reference Documentation. |
All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables.
Flag | Environment Variable | Description |
---|---|---|
requireAuthentication | WARPSTREAM_REQUIRE_AUTHENTICATION | If set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials. |
kafkaPort | WARPSTREAM_KAFKA_PORT | The port the Agent will listen on for Kafka client TCP connections. |
httpPort | WARPSTREAM_HTTP_PORT | The port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc). |
N/A | WARPSTREAM_AVAILABILITY_ZONE | Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata. We do not recommend overriding this in the general case. |
N/A | WARPSTREAM_LOG_LEVEL | Override the log level of the WarpStream Agent from the default value of info . Acceptable values are debug , info , warn , and error .Defaults to info . |
batchTimeout | WARPSTREAM_BATCH_TIMEOUT | Controls the maximum amount of time the WarpStream agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa.
Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency.
Defaults to 250ms, minimum is 50ms. |
fileCacheSizeBytes | WARPSTREAM_FILE_CACHE_SIZE_BYTES | Size of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers.
Defaults to 0.5GiB/vCPU if omitted. |
reportDiscoveryIP6 | WARPSTREAM_REPORT_DISCOVERY_IP6 | If set to true, the WarpStream Agents will report their IP6 address instead of IP4 to the WarpStream discovery system.
This is useful when running in VPCs that only support IP6, like fly.io. |
N/A | WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE | Overrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address).
This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP. |
advertiseHostnameStrategy | WARPSTREAM_ADVERTISED_HOSTNAME_STRATEGY | Which hostname strategy should be used the agent should advertise itself on. Accepted values: auto-ip4 /auto-ip6 /local /custom . auto-ip4 means that it will try to automatically find an IP v4 that makes sense auto-ip6 will do the same with an IPv6. local will use localhost If you select custom them you have to also define advertiseHostnameCustom . |
advertiseHostnameCustom | WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM | Custom hostname value to advertise to service discovery for clustering purposes if the custom advertise strategy is selected. |