Agent Groups
How to split Agents for a cluster into different "groups".
Last updated
How to split Agents for a cluster into different "groups".
Last updated
Agent Groups are distinct sets of Agents that all belong to the same logical cluster. Groups enable a single logical cluster to be split into many different "groups" that are isolated at the network / service discovery layer.
For example, consider the scenario where a single logical WarpStream cluster is "flexed" across multiple VPCs, regions, or even cloud providers:
For example, in the diagram above producer and consumer clients running in vpc_1 will only ever connect to Agents running in group_vpc_1. Similarly, producers and consumer running in vpc_2 will only ever connect to Agents running in group_vpc_2. However, since both Agent Groups belong to the same logical virtual cluster and have access to the same object storage bucket, clients in each VPC will be able to write and read data for all topics and partitions, even those that were created by clients / Agents running in a completely different VPC!
Agent groups are a powerful abstraction that enable a variety of use-cases:
Isolating specific producers or consumers to dedicated Agent Groups to avoid noisy neighbors.
"Flexing" a single logical cluster across multiple VPCs, regions, or even clouds providers without resorting to complex VPC peering setups.
Configuring Agent Groups is simple. Just add the -agentGroup $GROUP_NAME flag to your Agent deployment. For example, if you wanted to flex a single logical WarpStream cluster across two Kubernetes clusters running in different VPCs:
Alternatively, you can set the WARPSTREAM_AGENT_GROUP environment variable instead.
When your Kafka client connects to an Agent in a specific group, the WarpStream service discovery system will ensure that your client only connects to other Agents in the same group. This means that in order to take advantage of the Agent group functionality, you need to ensure that the bootstrap URL you configure in your client will only ever resolve to Agents in the correct group.
Unfortunately, the convenience bootstrap URL hosted by WarpStream and displayed in the console is not "agent group aware" and will randomly return Agent IP addresses from different groups. Therefore if you're using the Agent Group functionality, you should not use the convenience bootstrap URL and instead use a URL that is specific to the agent group your application wants to target.
For example, if you're using the official WarpStream Kubernetes chart, you can just use the Kubernetes service name generated by the chart as the bootstrap URL.
