Comment on page
Agent Pools and Virtual Clusters
This page describe what Agent Pools and Virtual Clusters are, and how they interact.
Virtual clusters represent discrete "logical" units of data and metadata. Virtual Clusters are the mechanism via which WarpStream isolates and scales metadata operations.
Agent Pools, on the other hand, represent pools of available Agent containers and hardware resources. Agent Pools are the mechanism via which WarpStream isolates and scales physical resources like Agent CPU, Memory, and Network bandwidth.
The current implementation of WarpStream enforces a 1:1 mapping between Virtual Clusters and Agent Pools such an Agent can only operate on behalf of a single tuple of
[agent_pool, virtual_cluster] at any given moment.However, future versions of WarpStream will allow the two to be decoupled such that a single Agent Pool can be used to produce, fetch, and execute jobs on behalf of many different Virtual Clusters. This will enable many different Virtual Clusters to share the same physical resources (Agent containers/cpu/memory/network) while still maintaining logical separation of their data and metadata.
Every WarpStream account is created with a default Virtual Cluster and Agent Pool called
apn_default. If you omit the agentPoolName flag and WARPSTREAM_AGENT_POOL_NAME environment variable from your Agent configuration, then it will default to using the apn_default pool.However, if you create additional Virtual Clusters in the WarpStream Admin console and want to configure a pool of agents to use that new virtual cluster, you'll also have to configure your agents with the name of the new agent pool that is associated with the new Virtual Cluster by adding either the
agentPoolName flag or WARPSTREAM_AGENT_POOL_NAME environment variable to your Agent configuration. If you fail to do this, your WarpStream Agent will refuse to start and will emit an error log to inform you that it is misconfigured.