Service Discovery
This page explains how WarpStream's custom service discovery mechanism works, as well as how it interacts with Kafka's service discovery system.
Last updated
This page explains how WarpStream's custom service discovery mechanism works, as well as how it interacts with Kafka's service discovery system.
Last updated
The "Hacking the Kafka Protocol" blog post is a good complement to this documentation page and provides some more context and background on the design decisions that were taken. It also goes into more technical detail about how the WarpStream service discovery and load balancing systems work together in tandem.
Before we discuss how WarpStream's service discovery system works, lets first go over the service discovery mechanism in Kafka. Kafka clients are usually instantiated with a list of URLs that identify a set of "bootstrap" Kafka brokers. The URLs can be raw IP addresses, or they can be hostnames that are resolved by the client to IP addresses via DNS.
Once the client has connected to the bootstrap brokers, it will issue a Metadata request for the cluster. The response to this request will contain a list of which brokers are in the cluster, what their hostname / IP addresses are, what port they're available on, which rack (or AZ) they're running in, and which topic-partitions they're the leader for.
The client will use this metadata to establish connections to all the other Kafka brokers in the cluster with which it needs to communicate. The client will always try to communicate with the leader of a given topic partition when it is producing data and one of the replicas (depending on how it is configured) when it is consuming data.
In addition, if the client is using Kafka's consumer group functionality, it will also make a FindCoordinator request to the cluster to determine which Kafka broker is the "group coordinator" for its consumer group, and then the client will establish a connection to that broker for the purposes of participating in the consumer group protocol.
Kafka brokers are inherently stateful. Therefore, the focus of the Kafka service discovery system is to ensure that clients connect to the right brokers that are responsible for the topic-partitions that they want to produce to or fetch from, as well as regularly fetching updated metadata to react to changes in topic-partition leadership.
WarpStream Agents, on the other hand, are completely stateless, and any WarpStream Agent can service any produce or fetch request. Therefore the focus of the WarpStream service discovery system is different from that of Kafka's. WarpStream's service discovery system has exactly two goals:
Keep traffic/load as evenly balanced across the WarpStream Agents as possible.
Keep communication between Kafka clients and WarpStream Agents zone-local as much as possible to minimize inter-zone networking costs.
WarpStream service discovery begins the moment the Agents are deployed. The Agents will use cloud-specific APIs to try to automatically discover which availability zone they're running in (this currently works in AWS, GCP, Azure, and Fly.io) and then publish this information along with their internal IP address to WarpStreams' service discovery system.
You can override the availability zone the Agents report to the discovery system using the WARPSTREAM_AVAILABILITY_ZONE environment variable.
You can double check the configuration child page to read more about how to override the default values used for service discovery.
Next, the Kafka clients in your applications need to be configured with the WarpStream bootstrap URL.
You also need to encode the availability zone of your application into your Kafka client's "client ID". See our documentation on "Configuring your Apache Kafka Client for WarpStream" for more details on how to do that.
Regardless, once you've configured the WarpStream bootstrap URL and client ID in your Kafka client, service discovery will proceed similarly to how it does in traditional Kafka. First, the client will use DNS to resolve the WarpStream bootstrap URL to an IP address. At this point, we don't care about zone awareness yet, so the WarpStream DNS server will return any available WarpStream Agent IP address.
Next, the client will issue a Metadata request to the WarpStream Agent identified in the previous step. The Agent will proxy the Metadata request to the WarpStream discovery service, which will return an appropriate Agent hostname / IP address that is running in the same availability zone as your client if you've properly encoded your application's availability zone into the Kafka client's client ID. If you haven't, it will return an Agent hostname / IP address from any availability zone that has live agents. This is how WarpStream "injects" zone awareness into the Kafka protocol, even for messages like Produce() which don't support "rack awareness" in the traditional Kafka protocol.
This approach also solves for load balancing. Proxying the Metadata request to the WarpStream service discovery system provides the service discovery system with an opportunity to perform load balancing. Currently, it does this using a naive (but effective) global round-robin balancing algorithm to Agents within the Availability Zone specified in the client ID, although we will make the algorithm more nuanced in the future.
Unlike Apache Kafka, WarpStream Agents are not "leaders" for individual topic-partitions. In fact, any Agent can write or read any record for any topic-partition. However, due to how the Apache Kafka protocol works, the Metadata responses need to inform the client which "brokers" are the leader for all the topic-partitions it wants to write to and read from. In addition, WarpStream needs to balance the traffic of all the different Kafka clients evenly amongst all the Agents.
By default, WarpStream accomplishes this using a unique approach that differs from Apache Kafka. Every time a client performs a Metadata request, the WarpStream service discovery system returns a view of the cluster in which a single agent is the leader for all topic-partitions.
This means that in the general case, each Kafka client is connected to only a single WarpStream Agent at a time. Load balancing is accomplished by balancing client connections instead of individual Produce / Fetch requests, and is handled automatically by the WarpStream control plane, which uses a round-robin load balancing strategy to return a different WarpStream Agent every time it receives a Metadata request from a Kafka Client.
This single_agent partition assignment strategy is the default strategy in WarpStream because it yields good performance for the vast majority of workloads. However, WarpStream does support a number of other partition assignment strategies. Check out our client configuration documentation for more details.
All of the consumer group coordinator logic is handled by the WarpStream control plane; the Agents just act as a proxy for those requests. As a result, any WarpStream Agent can be returned as the result of a call to FindCoordinator and the system would work correctly. However, to make WarpStream look more like a traditional Apache Kafka cluster, the WarpStream control plane uses a consistent hashing ring to select a special "coordinator" Agent to return as the consumer group coordinator.
This is acceptable because the amount of communication between the Kafka clients and the "group coordinator" is minimal and has almost no impact on load balancing on inter-zone networking.
