Configure an Apache Kafka Client for WarpStream

This pages explains how to configure your Apache Kafka client with WarpStream.

Don't forget to review our documentation for tuning your Kafka client for maximum performance with WarpStream once you're done. A few small changes in client configuration can result in 10-20x higher throughput when using WarpStream.

WarpStream provides API-compatibility with Apache Kafka, so you can just connect your clients to the WarpStream agents by setting the WarpStream Magic URL (obtained from the WarpStream console) as the value in the Kafka bootstrap settings. For example, using the librdkafka client in Go:

availabilityZone := lookupAZ()
sessionID := uuid.New().String()

clientID := fmt.Sprintf(
	"warpstream_session_id=%s,warpstream_az=%s,
	sessionID, availabilityZone)
bootstrapServer := "api-80ba097c-d4ef-4e0b-8e86-d05b80fee6ed.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092"

producerConfig := map[string]kafka.ConfigValue{
	// Obtain your magic URL from the "connect" tab of the
	// virtual cluster view in the WarpStream console.
	"bootstrap.servers": bootstrapServer,
	"broker.address.family": "v4",
	"log.connection.close":  "false",
	"client.id": clientID,
}

config := kafka.ConfigMap(producerConfig)
producer, err := kafka.NewProducer(&config)
if err != nil {
	return fmt.Errorf("error initialiazing kafka producer: %w", err)
}

Developing locally? The WarpStream magic URLs don't always work with residential ISPs in the mix. Consider using localhost:9092 as the bootstrap server for local development instead.

You can read more about the WarpStream service discovery system in our Service Discovery reference documentation, but if you want to take advantage of WarpStream's zone-aware service discovery system and achieve good load balancing, you must encode your application's availability zone in your Kafka client's client ID using the format in the code sample above.

Follow this documentation to learn how to determine your application's availability zone in all major cloud environments and properly template your client ID.

If you just want to get up and running quickly, you can use a regular client ID and skip encoding your application's availability zone in the client ID. However, beware that you may incur significantly higher costs due to inter-zone networking.