Configure Clients to Eliminate AZ Networking Costs

How to configure your Kafka clients to keep all traffic zone-local.

With WarpStream, there are no Availability Zone (AZ) networking costs between agents. This means you can produce and consume data from different AZs without incurring additional networking expenses. The same applies to agent-client communication: WarpStream eliminates AZ networking costs, allowing you to connect clients only to agents within the same AZ.

Requirements for Zonal Alignment of Kafka clients

To ensure your Kafka clients connect to agents within the same availability zone, you need to ensure there is at least one agent in the same availability zone as your clients and provide the Kafka client’s availability zone. There are two ways to provide the availability zone information:

1. Specifying the availability zone in your client ID

2. Mapping subnets to availability zones in the Agent configuration

Specifying the availability zone in your client ID

Append the following value to your Kafka client's ClientID: ws_az=<your-az>. This flag indicates the AZ in which the client is operating.

Example

Here are examples using various kafka libraries of how to set up the clientID with the AZ flag

librdkafka/confluent-kafka

Generic Configuration

client.id=application-foo,ws_az=us-east-1a

Python

availability_zone = lookup_az()
client_id = "application-foo"

p = Producer({
    'client.id': f"{client_id},ws_az={availability_zone}",
})

Golang

availabilityZone := lookupAZ()
clientID = "application-foo"

p, err := kafka.NewProducer(&kafka.ConfigMap{
	"client.id": fmt.Sprintf("%s,ws_az=%s", clientID, availabilityZone)
})
if err != nil {
	panic(err)
}

Javascript

availabilityZone = lookupAZ()
clientID = "application-foo"

const producer = new Kafka().producer({
    'client.id': `${clientID},ws_az=${availabilityZone}`
});

Java

String availabilityZone = lookupAZ()
String clientID = "application-foo"

Properties properties = new Properties();
properties.setProperty(ConsumerConfig.CLIENT_ID_CONFIG, String.format("%s,ws_az=%s", clientID, availabilityZone));
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(properties);

Franz-go

availabilityZone := lookupAZ()
clientID = "application-foo"

cl, err := kgo.NewClient(
    kgo.SeedBrokers("localhost:9092"),
    kgo.ClientID(fmt.Sprintf("%s,ws_az=%s", clientID, availabilityZone)),
)

Sarama

availabilityZone := lookupAZ()
clientID = "application-foo"

config := sarama.NewConfig()
config.ClientID = fmt.Sprintf("%s,ws_az=%s", clientID, availabilityZone)

client, err := sarama.NewConsumerGroup(brokers, groupID, config)

KafkaJS

availabilityZone = lookupAZ()
clientID = "application-foo"

const kafka = new Kafka({
  clientId: `${clientID},ws_az=${availabilityZone}`
});

To find the AZ that your application is in it is recommended to use your Cloud Provider's Metadata API, for example in AWS querying the following URL http://169.254.169.254/latest/meta-data/placement/availability-zone will give you the availability zone.

Our warpstream-go library has sample code that demonstrates how to query for your application's availability zone in every major cloud.

To learn more about the WarpStream-specific client ID features (like ws_si and ws_az) check out our documentation on configuring Kafka Client ID features.

Mapping subnets to availability zones in the Agent configuration

Pass a subnet mapping to the agents via the -zonedCIDRBlockscommand-line flag or the WARPSTREAM_ZONED_CIDR_BLOCKS environment variable. This mapping allows the agents to determine which zone the Kafka client is sending traffic from. The value needs to be a <> delimited list of availability zone to CIDR range pairs, where each pair starts with an AZ, a @, and a comma separated list of CIDR blocks representing the range of IPs used by the Kafka clients in that given AZ. For example, [email protected]/19,10.0.32.0/19<>[email protected]/19<>[email protected]/19 indicates that the Kafka clients with IPs that match 10.0.0.0/19 and 10.0.32.0/19 belong to us-east-1a, those with IPs that match 10.0.64.0/19 belong to us-east-1b, and those with IPs that match 10.0.96.0/19 belong to us-east-1c.

Note that if an AZ is appended to the Kafka client's ClientID and a subnet mapping is also provided to the agents but the values conflict with each other, the AZ from the ClientID will be used as the source of truth.