Comment on page
Configure Your Apache Kafka Client for WarpStream
This pages explains how to configure your Apache Kafka client with WarpStream.
Don't forget to review our documentation for tuning your Kafka client for maximum performance with WarpStream once you're done. A few small changes in client configuration can result in 10-20x higher throughput when using WarpStream.
WarpStream provides API-compatibility with Apache Kafka, so you can just connect your clients to the WarpStream agents by setting the WarpStream Magic URL (obtained from the WarpStream admin console) as the value in the Kafka bootstrap settings. For example, using the
librdkafka client in Go:availabilityZone := lookupAZ()
sessionID := uuid.New().String()
clientID := fmt.Sprintf(
"warpstream_session_id=%s,warpstream_az=%s,
sessionID, availabilityZone)
bootstrapServer := strings.ReplaceAll(
"api-80ba097c-d4ef-4e0b-8e86-d05b80fee6ed.AVAILABILITY_ZONE.discovery.prod-z.us-east-1.warpstream.com:9092",
"AVAILABILITY_ZONE", availabilityZone)
)
producerConfig := map[string]kafka.ConfigValue{
// Find your Magic URL template for `bootstrap.servers` on the
// details page for your Agent Pool. This one is just an example.
"bootstrap.servers": bootstrapServer,
"broker.address.family": "v4",
"log.connection.close": "false",
"client.id": clientID,
}
config := kafka.ConfigMap(producerConfig)
producer, err := kafka.NewProducer(&config)
if err != nil {
return fmt.Errorf("error initialiazing kafka producer: %w", err)
}
Developing locally? The WarpStream magic URLs don't always work with residential ISPs in the mix. Consider using
localhost:9092 as the bootstrap server for local development instead.You can read more about the WarpStream service discovery system in our Service Discovery reference documentation, but if you want to take advantage of WarpStream's zone-aware service discovery system and achieve good load balancing you must use a zone-specific magic URL in your client configuration and encode the availability zone of your application in your Kafka client's client ID using the format in the code sample above.
Follow this documentation to figure out how to determine your application's availability zone in all major cloud environments and template your clients magic URL / client ID properly.
If you just want to get up and running quickly, you can use a non-AZ specific magic URL and skip encoding your applications availability zone in the client ID, but beware that you may incur significantly higher costs due to inter-zone networking and traffic may not balance evenly across your WarpStream Agents.