# Deploy the Agents {% hint style="warning" %} Remember to review our documentation on [how to configure your Kafka client for WarpStream](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client), as well as our instructions on [tuning for performance](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/tuning-for-performance) once you're done. A few small changes in client configuration can result in 10-20x higher throughput when using WarpStream, and proper client configuration is required to leverage WarpStream's zone-aware discovery system. {% endhint %} ## Overview This page describes how to deploy the WarpStream Agents into your environments. Note that while WarpStream has several different products / cluster types: [Kafka](https://github.com/warpstreamlabs/docs/blob/master/agent-setup/deploy/broken-reference/README.md), [Schema Registry](https://github.com/warpstreamlabs/docs/blob/master/agent-setup/deploy/broken-reference/README.md), and [Tableflow](https://docs.warpstream.com/warpstream/reference/integrations/timeplus), they're all deployed using the same Agent binary. Therefore you can follow the instructions below regardless of which product you're deploying, and if there are any product-specific instructions, they'll be called out explicitly. ## Required Arguments The WarpStream Agent is completely stateless and thus can be deployed however you prefer to deploy stateless containers. For example, you could use AWS ECS or a Kubernetes Deployment. The WarpStream Docker containers can be found in the [installation docs](https://docs.warpstream.com/warpstream/getting-started/install-the-warpstream-agent#docker). However, if you're deploying WarpStream into a Kubernetes cluster, we highly recommend using our [official Helm charts](https://docs.warpstream.com/warpstream/agent-setup/infrastructure-as-code/helm-charts). Similarly, for AWS ECS we have a dedicated [terraform module](https://github.com/warpstreamlabs/terraform-aws-warpstream-ecs). The Agent has four required arguments that must be passed as command line flags: 1. `bucketURL` (not required for Tableflow) 2. `agentKey` 3. `defaultVirtualClusterID` 4. `region` For example: ```bash docker run public.ecr.aws/warpstream-labs/warpstream_agent:latest \ agent \ -bucketURL "s3://$S3_BUCKET?region=$S3_BUCKET_REGION" \ -agentKey $AGENT_KEY \ -defaultVirtualClusterID $YOUR_VIRTUAL_CLUSTER_ID -region $CLUSTER_REGION ``` The values of `agentKey` , `defaultVirtualClusterID` , and `region` can be obtained from the [WarpStream Admin Console](https://console.warpstream.com). {% hint style="info" %} Note that the entrypoint for the WarpStream docker image is a multi-command binary. For production usage, the subcommand that you want to run is just called `agent` as shown above. {% endhint %} Depending on the tool you're using to deploy/run containers, it can sometimes be cumbersome to provide additional arguments beyond the `agent` subcommand. In that case, all of the required arguments can be passed as environment variables instead: 1. `WARPSTREAM_BUCKET_URL` 2. `WARPSTREAM_AGENT_KEY` 3. `WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID` 4. `WARPSTREAM_REGION` ### Object Storage `bucketURL` is the URL of the object storage bucket that the WarpStream Agent should write to. See [our documentation](https://docs.warpstream.com/warpstream/different-object-stores#bucket-url-construction) on how to construct a proper URL for the specific object storage implementation that you're using. The `Deploy` tab in the WarpStream UI for your BYOC cluster also has a utility to help you construct a well formed URL. In addition to constructing a well-formed `bucketURL`, you'll also need to create and configure a dedicated object storage bucket for the Agents, and ensure that the Agents have the appropriate permissions to access that bucket. See [our documentation](https://docs.warpstream.com/warpstream/different-object-stores#bucket-permissions) on how to do that correctly. If you are [using a bucket prefix](https://docs.warpstream.com/warpstream/different-object-stores#using-a-bucket-prefix), please ensure that your bucketURL is in quotes to prevent any interpolation when running from the command line ### Region The `region` flag corresponds to the region that the WarpStream control plane is running in. This corresponds to the value that was selected when the BYOC cluster was created and can be obtained from the WarpStream UI. This value does not need to correspond to the cloud region that the Agents are deployed in, but you should pick the region that is closest to where your Agents are deployed to minimize latency. Currently supported regions for BYOC are: | Cloud provider | Region | URL | | -------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | AWS | `us-east-1` | | | AWS | `us-east-2` | [https://metadata.default.us-east-2.warpstream.com](https://metadata.default.us-east-1.warpstream.com) | | AWS | `us-west-2` | | | AWS | `eu-central-1` | | | AWS | `eu-west-1` | [https://metadata.default.eu-west-1.warpstream.com](https://metadata.default.eu-central-1.warpstream.com) | | AWS | `ap-southeast-1` | | | AWS | `ap-southeast-2` | [https://metadata.default.ap-southeast-2.warpstream.com](https://metadata.default.ap-southeast-1.warpstream.com) | | AWS | `ap-south-1` | [https://metadata.default.ap-south-1.warpstream.com](https://metadata.default.ap-southeast-1.warpstream.com) | | AWS | `ap-northeast-1` | [https://metadata.default.ap-northeast-1.warpstream.com](https://metadata.default.eu-central-1.warpstream.com) | | GCP | `us-central1` | | | GCP | `northamerica-northeast1` | [https://metadata.default.northamerica-northeast1.gcp.warpstream.com](https://metadata.default.us-central1.gcp.warpstream.com) | | GCP | `europe-west1` | | | GCP | `asia-south1` | | | Azure | `eastus` | | You can contact us to request a new region by sending an email to . ## Permissions and Ports The WarpStream Agents need permission to perform various different operations against the object storage bucket. Review [our object storage permissions documentation](https://docs.warpstream.com/warpstream/different-object-stores#bucket-permissions) for more details. In addition to object storage access, the WarpStream Agent will also need permission to communicate with the control plane URL (see table above) in order to write/read Virtual Cluster metadata. Raw data flowing through your WarpStream will **never** leave your cloud account, only metadata required to order batches of data and perform remote consensus. You can read more about what metadata leaves your cloud account in our [security and privacy considerations documentation](https://docs.warpstream.com/warpstream/reference/security-and-privacy-considerations). Finally, the WarpStream Agent requires 2 ports to be exposed. For simplicity, we recommend just ensuring that the WarpStream Agent can listen on ports `9092` (or `9094` in the case of schema registry) and `8080` by default; however, the section below contains more details about how each port is used and how to override them if necessary. {% tabs %} {% tab title="Kafka Port" %} Default: `9092` Override: `-kafkaPort $PORT` Disable: `-enableKafka false` This is the port that exposes the Kafka TCP protocol to Kafka clients. Only disable it if you don't intend to use the Kafka protocol at all. {% endtab %} {% tab title="HTTP Port" %} Default: `8080` Override: `-httpPort $PORT` User for inter-agent communication within a single availability zone so the Agents can form a [distributed file cache](https://docs.warpstream.com/warpstream/overview/architecture/read-path) with each other. Also used to expose Prometheus metrics. {% endtab %} {% tab title="Schema Registry Port" %} Default: `9094` Override: `-schemaRegistryPort $PORT` You can also use the env variable `WARPSTREAM_SCHEMA_REGISTRY_PORT` to override the port. This is the port that exposes the Schema Registry HTTP server. {% endtab %} {% endtabs %} ## Service discovery The `advertiseHostnameStrategy` flag allows you to choose how the agent will advertise itself in Warpstream service discovery (more details [here](https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy)). The default `auto-ip4` is a good choice for most cases in production. ## GOMAXPROCS The WarpStream Agent uses heuristics to automatically configure itself based on the available resources. The most important way this happens is by adjusting concurrency and cache sizes based on the number of available cores. The Agent uses standard operating system APIs to determine how many cores are available, and it prints this value when starting: {% code overflow="wrap" %} ```bash 2023/08/31 09:21:22 maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined ``` {% endcode %} This number is *usually* right, but it may not be right depending on how the Agent is deployed. For example, the Agent may determine the wrong value when running in [AWS ECS](https://github.com/uber-go/automaxprocs/issues/66). In general, we recommend that you manually set the `GOMAXPROCS` environment variable to the number of cores that you've made available to the Agent in your environment. For example, if you've allocated 3 cores to the Agent's container, then we recommend adding `GOMAXPROCS=3` as an environment variable. The value of `GOMAXPROCS` must be a whole number and not a fraction. We also recommend that you always assign Agent whole numbers for CPU quotas so that the Agent doesn't have fractional CPU quotas. Fractional CPU quotas can result in throttling and increased latency since the value of `GOMAXPROCS` and the number of whole cores available to the Agent won't match. ## Instance Selection While the WarpStream Agents don't store data on local disks, they do use the network heavily. Therefore we recommend using network-optimized cloud instances that provide at least 4GiB of RAM per vCPU. We also recommend using **dedicated** instances and not bin-packing the Agent containers to avoid noisy neighbor issues where another container running on the same VM as the Agents causes network saturation. In AWS, we think the `m5n` and `m6in` series are a great choice for running the Agents. In GCP, the `n4` series is a great choice with the `c4` series as a close second. We recommend running the Agents with *at least* 2 vCPUs available and providing at least 4 GiB of RAM per vCPU, therefore the `m5n.large` and `m6in.large` are the minimum recommended instance sizes in AWS. Using much larger instances is fine as well; just make sure to set the value of [GOMAXPROCS](#gomaxprocs) to ensure the Agent can make use of all the available cores even when running in a containerized environment (our helm chart does this automatically). ### Network Optimized Instances The Agent does a *lot* of networking to service Apache Kafka Produce and Fetch requests, as well as perform background compaction. The Agent uses compression and intelligent caching to minimize this, but fundamentally, WarpStream is a data-intensive system that is even more networking-heavy than Apache Kafka due to reliance on remote storage. Debugging latency caused networking bottlenecks and throttling is a *nightmare* in all cloud environments. None of the major clouds provide sufficient instrumentation or observability to understand why or if your VM's network is being throttled. Some have dynamic throttling policies that allow long bursts but suddenly degrade with no explanation. For all of these reasons, we recommend running the WarpStream Agents on network-optimized instances, which allows the Agents to saturate their CPU before saturating the network interface. That situation is easier to understand, observe, and auto-scale on. ## Auto-Scaling When running the Agent on the appropriate instance type as described above, we recommend auto-scaling based on CPU usage with a target of 50% average usage. Our internal testing workload runs the Agent at more than 75% CPU usage with little latency degradation, but choosing an appropriate threshold requires balancing the concerns of cost efficiency and responsiveness to bursts of traffic that happen faster than your auto-scaler can react. If you're using our [official helm charts](https://docs.warpstream.com/warpstream/byoc/infrastucture-as-code/helm-charts), auto-scaling can be [enabled trivially](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent#scaling). ### Zone-Specific Scaling Kubernetes will usually deploy the WarpStream Agents spread equally across multiple availability zones. When the horizontal pod autoscaler (HPA) takes action, those actions will be taken equally across all zones. This approach works well for most applications, but if you've enabled [zone-aware routing](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/configure-clients-to-eliminate-az-networking-costs) in the WarpStream Agents then this could be a problem. For example, imagine a scenario where the WarpStream Agents are deployed across zones A, B, and C, with an equal number of Agents in each zone, but zone A has three times as much client traffic as zones B and C. In this scenario, the *average* CPU utilization of all the Agents across all the zones may not be enough for Kubernetes to trigger an up-scale, but the Agents in zone A may be overloaded resulting in degradation of client performance in that zone. If your workload is susceptible to traffic imbalances across zones **and** you're running the WarpStream Agents in multiple zones **and** you've enabled zone-aware routing, then our general recommendation is to create a dedicated WarpStream Agent deployment (all belonging to the same virtual cluster) in each availability zone. This will make the cluster resilient to zonal imbalances as each Agent deployment in each zone will have its own auto-scaler, and thus be able to react to imbalances in zonal traffic independently. If you're using our [Kubernetes chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent), this is as easy as applying the WarpStream Agent chart once per availability zone. See our [chart documentation](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent#autoscaling-per-zone) on this issue for more details. By default, Kubernetes will usually deploy the WarpStream Agents spread equally across multiple availability zones. ## Automatic Availability Zone Detection By default, the WarpStream Agents will try to determine which availability zone they're running in by querying Kubernetes and/or cloud-provider native APIs. This currently works automatically in AWS, GCP, and Azure. {% hint style="info" %} Availability zone names are not consistent between different AWS accounts. If you're deploying a WarpStream cluster where Agents and Clients will be deployed in different AWS accounts, you'll want the Agents to advertise their **availability zone ID** instead of their **availability zone name**. This can be accomplished by setting the `WARPSTREAM_LOOKUP_AVAILABILITY_ZONE_ID=true` environment variable on the Agents. {% endhint %} If your Agents are advertising their availabiltiy zone as `warpstream-unset-az` in the WarpStream console, then it means they failed to determine their availability zone automatically. Check the Agent logs for a message in the form of: `error determining availability zone` which should contain a detailed error message. For instance, a known issue on AWS EKS is that the hop limit on old EKS node group is 1, preventing the call to AWS metadata from failing. Raising it to 2 should fix the issue (see [AWS doc](https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-supports-ec2-instance-metadata-service-v2/)). As a last resort, you can use the `WARPSTREAM_AVAILABILITY_ZONE` environment variable described in the table above to declare the availability zone in which your agent is running. ## Deploying Multiple Clusters in the same VPC If you plan to deploy multiple Warpstream clusters in the same VPC, you must be aware that there is a risk an IP from one agent of a given cluster could be reused by an agent of another cluster. You should familiarize yourself with [this section](https://docs.warpstream.com/warpstream/agent-setup/kubernetes-known-issues#when-an-ip-is-reused-by-another-agents-pod) to make your deployments robust to this. ## Routing Requests from the Agent Through a Proxy You might have a proxy deployed between your Agents and the WarpStream Control Plane, with the Agents unable to reach the `*.warpstream.com` Control Plane IPs directly. In this scenario, set the `WARPSTREAM_HTTP_PROXY` environment variable on the Agents to the hostname of your proxy. For example, if the hostname of your proxy was `http://internal.proxy.com` then you would set `WARPSTREAM_HTTP_PROXY=http://internal.proxy.com` as the environment variable. The Agent will then use this proxy for all Agent <> Control Plane communication, but Agent <> Agent and Agent <> Object Storage communication will continue to happen directly, bypassing the internal proxy. {% hint style="warning" %} In addition to the `WARPSTREAM_HTTP_PROXY` environment variable, WarpStream also supports a generic `HTTP_PROXY` environment variable that will impact all HTTP traffic, not just traffic between the Agents and the Control Plane. We don't recommend ever setting this value because the amount of networking performed between the WarpStream Agents, as well as between the Agents and the object store (which is also over HTTP) can be extremely high volume, and inserting a proxy in the middle of that traffic would significantly reduce the performance of the cluster. {% endhint %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.warpstream.com/warpstream/agent-setup/deploy.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.