warpstream agent
warpstream agent
The warpstream agent subcommand can be used to administer the WarpStream agents.
The following table displays the available warpstream agent arguments.
| Argument | Description |
|---|---|
-advertiseHostnameCustom | Custom hostname value to advertise to service discovery for clustering purposes if the custom advertise strategy is selected |
-advertiseHostnameStrategy | Which hostname strategy should be used the agent should advertise itself on. Auto means that it will try to automatically find an IP that makes sense. If you select custom them you have to also define -advertiseHostnameCustom. Currently supported strategy: [auto-ip4, auto-ip6, local, custom]. (default "auto-ip4") |
-agentPoolName | Agent pool to register the agent under (default "apn_default") |
-apiKey | API key for WarpStream cloud |
-batchTimeout | How frequently to flush batches. Lower values reduce latency, higher values cost more money (default 250ms) |
-bucketURL | Object storage URL to write data to |
-clusterWideEnvironmentPort | The default port to use for the cluster wide environment (default 9999) |
-defaultVirtualClusterID | Virtual cluster to default to for all requests |
-disableConsumerGroupMetrics | Disable the consumer group offset metrics automatically published by default |
-disableLogsCollection | Disable the logs collection sent to warpstream backend (enabled by default) |
-enableClusterWideEnvironment | Whether the cluster wide environment should be enabled |
-enableDatadogMetrics | Enable datadog metrics |
-enableDatadogProfiling | Enable datadog profiling |
-enableDatadogTracing | Enable kafka server (default true) |
-enableKinesis | Enable kinesis server, enabled by default (default true) |
-enablePrometheusMetrics | Enable prometheus metrics (default true) |
-fileCacheExtraReplicas | Number of extra replicas for the distributed file cache. Helps improve availability and reduce errors when Agents shutdown ungracefully. You can override this to 0, but do not increase this value above 1 unless you know what you're doing. (default 1) |
-fileCacheSizeBytes | File cache size in bytes. Defaults to 0.5GiB/vCPU if left unmodified. |
-gracefulShutdownDuration | Amount of time to wait after receiving SIGTERM before exiting to allow graceful removal from service discovery (default 1m0s) |
-heartbeatEvery | How often the agent should heartbeat the WarpStream backend. Recommended to not modify this. |
-httpPort | Port to run HTTP handlers on (Kinesis, distributed file cache, prometheus metrics, etc) (default 8080) |
-kafkaFetchCompression | Compression type to use for Fetch responses: none, gzip, snappy, lz4 (by default), zstd. This is only used if no compression is set explicitly, or if 'agent' type compress |
-kafkaHandleConsumerGroupsInBackend | Handle consumer group 'JoinGroup' and 'SyncGroup' requests in the backend instead of in the agent. When handled in the backend, the 'Rebalance Timeout' is always set to 10 seconds, whereas in the agent, it will be determined by client specifications. Enabling this option offers the advantage of reduced error potential and seamless integration of backend improvements and bug fixes. However, exercise caution when enabling it for large consumer groups, as a 10-second rebalance timeout may lead to extended rebalancing times and consequently, prolonged consumption pauses. Warning: Ensure uniformity within your agent pool regarding this setting. Having a mix of enabled and disabled settings may lead to rebalancing issues and potential disruptions. |
-kafkaInterzoneLoadBalancingInterval | Interval at which the Kafka connection assesses if the client-agent connection resides in the same Availability Zone (AZ). If they are not in the same AZ and there are agents available within the client's AZ, the connection is terminated. This approach encourages load balancing by prompting clients to re-query the magic URL and, consequently, connect to agents within their respective AZ. For this mechanism to function, clients should include 'waprstream_az=X' and 'warpstream_interzone_lb=true' in their clientID. (default 1m0s) |
-kafkaLoadBalancingDrainingTime | Time given to gracefully close the Kafka connection after the reconnect interval is reached. |
-kafkaLoadBalancingInterval | Time after which the Kafka connection will be closed. This mechanism helps load balance the clients by forcing them to query the magic URL again. By resetting the connection periodically, clients are evenly distributed across available Kafka connections. (default 8760h0m0s) |
-kafkaMetadataRefreshInterval | Period of time at which topic metadata is refreshed. Unlike Kafka, this metadata cache refresh also affects the timestamp type associated with a stream. (default 1m0s) |
-kafkaPort | Port to run the kafka server on (default 9092) |
-logInterval | Interval for logging service status (default 15s) |
-maxConcurrentRequestPerCPU | Maximum number of concurrent requests (per CPU) allowed by the Kafka server |
-metadataURL | Address for WarpStream metadata backend (default "https://api.prod.us-east-1.warpstream.com") |
-requireAuthentication | Controls whether the agents will enforce that all connections are SASL authenticated. Will automatically disable Kinesis support if enabled |
-roles | Roles that the agent should start (comma-separated) (default "proxy, jobs") |
Last updated
