LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Port Forwarding (K8s)
    • Orbit
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
Powered by GitBook
On this page
  • Agent Groups
  • Configuring Agent Goups
  • Targeting Agent Groups

Was this helpful?

  1. BYOC
  2. Advanced Agent Deployment Options

Agent Groups

How to split Agents for a cluster into different "groups".

PreviousAgent RolesNextProtect Data in Motion with TLS Encryption

Last updated 3 months ago

Was this helpful?

Agent Groups

Agent Groups are distinct sets of Agents that all belong to the same logical cluster. Groups enable a single logical cluster to be split into many different "groups" that are isolated at the network / service discovery layer.

For example, consider the scenario where a single logical WarpStream cluster is "flexed" across multiple VPCs, regions, or even cloud providers:

In the diagram above producer and consumer clients running in vpc_1 will only ever connect to Agents running in group_vpc_1. Similarly, producers and consumer running in vpc_2 will only ever connect to Agents running in group_vpc_2. However, since both Agent Groups belong to the same logical virtual cluster and have access to the same object storage bucket, clients in each VPC will be able to write and read data for all topics and partitions, even those that were created by clients / Agents running in a completely different VPC!

Agent groups are a powerful abstraction that enable a variety of use-cases:

  1. Isolating specific producers or consumers to dedicated Agent Groups to avoid noisy neighbors.

  2. "Flexing" a single logical cluster across multiple VPCs, regions, or even clouds providers without resorting to complex VPC peering setups.

Configuring Agent Goups

Configuring Agent Groups is simple. Just add the -agentGroup $GROUP_NAME flag to your Agent deployment. For example, if you wanted to flex a single logical WarpStream cluster across two Kubernetes clusters running in different VPCs:

# In Kubernetes cluster 1
warpstream agent -virtualClusterID $CLUSTER_ID -agentGroup group-1

# In Kubernetes cluster 2
warpstream agent -virtualClusterID $CLUSTER_ID -agentGroup group-2

Alternatively, you can set the WARPSTREAM_AGENT_GROUP environment variable instead.

Agent group names may only contain lowercase letters, numbers, and dashes (-).

Targeting Agent Groups

When your Kafka (or Schema Registry) client connects to an Agent in a specific group, the WarpStream service discovery system will ensure that your client only connects to other Agents in the same group. This means that in order to take advantage of the Agent group functionality, you need to ensure that the bootstrap URL you configure in your client will only ever resolve to Agents in the correct group.

Kubernetes

Targeting specific Agent Groups in Kubernetes is easy. Each Agent group will have its own helm deployment, and therefore will have its own Kubernetes service. Use the Kubernetes service name that corresponds to the Agent Group that you want to target as the bootstrap URL for your Kafka client.

Non-Kubernetes

Most non-Kubernetes deployments use WarpStream's hosted convenience bootstrap URL for service discovery. These URLs are visible in the WarpStream UI under the "Connect" tab in the cluster view and generally look something like this:

api-XXXX-XXXX-XXXX-XXXX-XXXX.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092 This bootstrap URL is not group-aware by default and therefore may return Agent I.P addresses for any agent group. To target a specific agent group, modify the bootstrap URL like this:

api-XXXX-XXXX-XXXX-XXXX-XXXX.group$GROUP_NAME.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092.

For example, if the agent group name was: "test-group-foo", then the bootstrap URL would become:

api-XXXX-XXXX-XXXX-XXXX-XXXX.grouptest-group-foo.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092