Configure Clients to Eliminate AZ Networking Costs

How to configure your Kafka clients to keep all traffic zone-local.

With WarpStream, there are no Availability Zone (AZ) networking costs between agents. This means you can produce and consume data from different AZs without incurring additional networking expenses. The same applies to agent-client communication: WarpStream eliminates AZ networking costs, allowing you to connect clients only to agents within the same AZ.

Requirements for Zonal Alignment of Kafka clients

To ensure your Kafka clients connect to agents within the same availability zone, you need to ensure there is at least one agent in the same availability zone as your clients and provide the Kafka client’s availability zone. There are two ways to provide the availability zone information:

1. Specifying the availability zone in your client ID

2. Mapping subnets to availability zones in the Agent configuration

Specifying the availability zone in your client ID

Append the following value to your Kafka client's ClientID: ws_az=<your-az>. This flag indicates the AZ in which the client is operating.

Example

Here is an example of how to set up the clientID with the AZ flag:

availabilityZone := lookupAZ()
clientID := fmt.Sprintf("ws_az=%s",availabilityZone)

Our warpstream-go library has sample code that demonstrates how to query for your application's availability zone in every major cloud.

Mapping subnets to availability zones in the Agent configuration

Pass a subnet mapping to the agents via the -zonedCIDRBlockscommand-line flag or the WARPSTREAM_ZONED_CIDR_BLOCKS environment variable. This mapping allows the agents to determine which zone the Kafka client is sending traffic from. The value needs to be a <> delimited list of availability zone to CIDR range pairs, where each pair starts with an AZ, a @, and a comma separated list of CIDR blocks representing the range of IPs used by the Kafka clients in that given AZ. For example, us-east-1a@10.0.0.0/19,10.0.32.0/19<>us-east-1b@10.0.64.0/19<>us-east-1c@10.0.96.0/19 indicates that the Kafka clients with IPs that match 10.0.0.0/19 and 10.0.32.0/19 belong to us-east-1a, those with IPs that match 10.0.64.0/19 belong to us-east-1b, and those with IPs that match 10.0.96.0/19 belong to us-east-1c.

Note that if an AZ is appended to the Kafka client's ClientID and a subnet mapping is also provided to the agents but the values conflict with each other, the AZ from the ClientID will be used as the source of truth.