# Agent Roles

{% hint style="warning" %}
Configuring individual Agent roles is an advanced feature. We recommend familiarizing yourself with WarpStream in its default configuration first before considering splitting roles.

Also keep in mind that at least some of your Agents **must** run the `jobs` role or your cluster will eventually cease to function until `jobs` Agents are created and they can finish processing the accumulated backlog of unprocessed jobs.
{% endhint %}

### What are Agent Roles

The WarpStream Agent has two primary roles:

1. Receiving data from Apache Kafka producers and serving data to Apache Kafka consumers
2. Running background jobs

The first role is what we call the `proxy` role. It runs three main services.

* The HTTP and TCP servers that respond to produce requests
* The HTTP and TCP servers that respond to fetch requests
* The file cache for reducing the number of object storage GET operations to respond to Fetch requests.

The second role is called the `jobs` role. It runs all of the background tasks that are necessary for a WarpStream cluster to continue functioning. It runs a few different kinds of jobs:

* Compaction jobs periodically rewrite and merge files in object storage.
* Retention jobs delete files which contain out-of-retention data.
* Publish metrics jobs that makes some WarpStream-internal metrics available over the Prometheus endpoint.

By default, all Agents run both the `proxy`and `jobs` roles, but it is possible to configure your Agents to start only a subset of these roles.

Optionally, there is one additional role that can be configured called `pipelines`. This role is used to run WarpStream Managed Data Pipelines. See [our documentation](https://docs.warpstream.com/warpstream/kafka/manage-connectors/bento) for more details about this role.

{% hint style="warning" %}
All roles (except pipelines) are necessary in a deployment of WarpStream. You cannot run only one of the roles for example.

However, you can have some Agents run a single role.
{% endhint %}

### Configuring Agent Roles

{% hint style="warning" %}
If you're upgrading an existing WarpStream cluster with traffic to enable roles, follow the [Targeting Agent Groups](#targeting-agent-groups) instructions to configure your Kafka clients to target specific roles first, then redeploy the Agents to configure them into dedicated groups.

This is especially important if you're splitting out the `proxy-produce` and `proxy-consume` roles.

If you don't change your Kafka client configuration first, then some of your producers will end up connected to the `proxy-consume` Agents and their Produce requests will be rejected with an `INVALID_REQUEST` error message. Similarly, some of your consumers will end up connected to the `proxy-produce` Agents and their Fetch requests will be rejected as well.
{% endhint %}

Use the `-roles` command line flag or the `WARPSTREAM_AGENT_ROLES` environment variable to configure the roles that an Agent should run.

Valid values are:

* `"proxy"` to run only the `proxy`role, and to advertise that this Agent is able to process all Kafka protocol requests. You can connect your clients to this agent to produce records or to consume records. In the case of Schema Registry agents, using this role means the agent can handle all API requests.
* `"proxy-produce"` to run only the `proxy-produce` role which is a subset of the `proxy` role and indicates that this Agent will process Produce requests, but not Fetch requests. This role is not applicable for Schema Registry clusters.
* `"proxy-consume"` to run only the `proxy-consume` role which is a subset of the `proxy` role and indicates that this Agent will process Fetch requests, but not Produce requests. This role is not applicable for Schema Registry clusters.
* `"jobs"` to run only the jobs role.
* `"pipelines"` to run only the pipelines role (only applicable if using the [Managed Data Pipelines](https://docs.warpstream.com/warpstream/kafka/manage-connectors/bento) product).

You can also combine values, for example:

* `"proxy-consume,jobs"` runs the `jobs` role and the `proxy-consume` role.
* `"proxy,jobs,pipelines"` runs all roles.

{% hint style="danger" %}
Make sure at least some of your Agents are running the `jobs` role. This role performs compaction and clean up of deleted files in object storage, and without it your cluster will eventually cease to function entirely until `jobs` Agents are added and they can finish processing the accumulated backlog of unprocessed jobs.
{% endhint %}

A simple example using our Docker container:

```
docker run public.ecr.aws/warpstream-labs/warpstream_agent_linux_amd64:latest \
    agent \
    -bucketURL mem://mem_bucket \
    -apiKey $YOUR_API_KEY \
    -defaultVirtualClusterID $YOUR_VIRTUAL_CLUSTER_ID
    -roles "jobs"
```

### Targeting Agent Roles

{% hint style="warning" %}
If you don't configure the `warpstream_proxy_target` feature on your Kafka client IDs, then your clients will connect to any Agent running a proxy role, regardless of whether its `proxy-consume` or `proxy-produce`. This may break your application as the `proxy-consume` Agents will reject Produce requests and the `proxy-produce` Agents will reject Fetch requests.
{% endhint %}

If some of your Agents are running with either the `proxy-consume` or the `proxy-produce` role then you will need to update your Apache Kafka client configuration to indicate which set of Agents you want to target based on whether your client is a Producer or Consumer.

* For a Kafka consumer application that needs to target Agents running the `proxy-consume` role, add `,warpstream_proxy_target=proxy-consume` to the end of your client\_id.
* For a Kafka producer application that needs to target Agents running the `proxy-produce` role, add `,warpstream_proxy_target=proxy-producer` to the end of your client\_id.

For more information about how to configure your Apache Kafka client with additional WarpStream features like role target, see our [Configuring Kafka Client ID Features](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features) documentation.

### Filtering Job Types

{% hint style="danger" %}
Job type filtering is an advanced feature that should not be manually configured unless absolutely necessary. It is primarily designed to be used in pre-defined configurations constructed by the WarpStream team. Please reach out to the WarpStream team before using this feature in your own deployments.
{% endhint %}

Requires Agent Version: `v760+`

When running agents with the `jobs` role, you can control which types of background jobs an agent handles using the `-jobSelector` command line flag or the `WARPSTREAM_JOB_SELECTOR` environment variable. This is useful for isolating different types of jobs onto dedicated Agent deployments so that they can be isolated and/or scaled independently. For example, running query jobs on separate agents from where compaction or orbit jobs run to keep interactive user queries isolated from your online workloads.

{% hint style="warning" %}
The `-jobSelector` flag requires the `jobs` role. If you specify a job selector without including `jobs` in `-roles`, the agent will fail to start.
{% endhint %}

The job selector accepts a comma-separated list of job categories:

| Category  | Description                     |
| --------- | ------------------------------- |
| `orbit`   | Orbit jobs                      |
| `query`   | Query engine jobs               |
| `metrics` | Cluster metrics publishing jobs |

Specify one or more categories to only run those job types. Prefix categories with `-` to exclude them instead:

```
# Only run orbit jobs
warpstream agent -roles jobs -jobSelector "orbit"

# Run all jobs except orbit
warpstream agent -roles jobs -jobSelector "-orbit"

# Only run orbit and query jobs
warpstream agent -roles jobs -jobSelector "orbit,query"
```

You cannot mix include and exclude syntax in the same selector (e.g., `orbit,-query` is invalid).

If no job selector is specified, the agent handles **all** job types. Existing deployments are unaffected.

{% hint style="danger" %}
Every job type must be covered by at least one running agent. If no agent is configured to handle a particular job type, those jobs will fail to run and break your cluster. When using job selectors, ensure that the union of all your agents' selectors covers all job categories.
{% endhint %}

A common deployment pattern is to run separate agent groups for different job types:

```
# Group 1: Dedicated orbit agents
warpstream agent \
  -roles jobs \
  -jobSelector "orbit" \
  -agentGroup orbit-jobs

# Group 2: Dedicated query agents
warpstream agent \
  -roles jobs \
  -jobSelector "query" \
  -agentGroup query-jobs

# Group 3: All other jobs (everything except orbit and query)
warpstream agent \
  -roles jobs \
  -jobSelector "-orbit,-query" \
  -agentGroup other-jobs
```

This allows you to independently scale and tune the resources for each workload. The control plane automatically routes jobs to agents that have registered for the appropriate job types.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.