Partition Assignment Strategies

This page explains the various different partition assignment strategies in WarpStream.

Unlike Apache Kafka, WarpStream Agents are not "leaders" for individual topic-partitions. In fact, any Agent can write or read any record for any topic-partition at any time.

However, due to how the Apache Kafka protocol works, the Metadata responses need to inform the client which "brokers" are the leader for all the topic-partitions it wants to write to and read from. In addition, WarpStream needs to balance the traffic of all the different Kafka clients evenly amongst all the Agents.

This means that WarpStream needs some "strategy" for determining which Agent it should tell clients is the leader for each topic-partition. In essence, you can think of WarpStream's partition assignment strategies as different load balancing strategies with different trade-offs. The rest of this document outlines the various strategies that WarpStream supports and their respective trade-offs.

Strategies

consistent_random_jump (default, recommended)

consistent_random_jump uses an algorithm called "random jump consistent hashing" to assign topic-partitions to individual Agents. The result is that the vast majority of produce requests for a given topic-partition will be processed by the same Agent. This dramatically improves the performance of the cluster for two reasons:

First, the control plane has to process significantly less metadata.

Second, it concentrates data for each partition in a smaller number of files which reduces the need for compaction as well as the effort required by the Agents to process fetch requests for live consumers.

The downside is that this consistent hashing strategy can result in Agent hot spots if traffic for some topic-partitions is higher than others. However, the "random jump" aspect of the load balancing algorithm will detect hot-spots automatically and spread the hot topic-partitions across multiple Agents. This results in slightly less good balancing than single_agent for clusters with extreme skews, but is the right trade-off for the vast majority of workloads.

single_agent

Every time a client performs a Metadata request, the WarpStream service discovery system returns a view of the cluster in which a single agent is the leader for all topic-partitions.

This means that in the general case, each Kafka client is connected to only a single WarpStream Agent at a time. Load balancing is accomplished by balancing client connections instead of individual Produce / Fetch requests, and is handled automatically by the WarpStream control plane. The control plane uses a power of 2 random choices load balancing strategy to return a different WarpStream Agent every time it receives a Metadata request from a Kafka Client.

The single_agent strategy has two primary benefits:

First, its the strategy that results in the lowest outlier latency for Produce requests. The reason for this is simple: object storage tends to have higher outlier latency than more traditional SSD storage. As a result, single_agent minimizes each individual client's exposure to outlier latency because each client is only exposed to the outlier latency from a single WarpStream Agent and it's file flushes at any given moment.

Second, it has the most powerful load-balancing capabilities. The single_agent load balancing strategy effectively just balances Kafka client connections based on the observed load of the Agents, regardless of the shape of the underlying Kafka workload in terms of topic-partitions. In other words, the single_agent strategy results in the most even load utilization in the cluster because load is the only factor that it takes into consideration when making decisions.

The primary downside of this strategy (and why its not the default / recommended strategy) is that it spreads data for the same topic-partition across many different files created by many different Agents. This results in the cluster having to spend more resources performing compactions, as well as making it more difficult for the Agents to serve fetch requests for all of the live consumers.

Configuration

There are two ways to configure partition assignment strategies in WarpStream:

1. At the Agent level.

2. Using a client id feature.

Agent Configuration

Set the -defaultPartitionAssignmentStrategy flag or WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY environment variable on the Agents to the name of the partition assignment strategy that you want to use, I.E consistent_random_jump or single_agent .

Once set, this will apply to any Kafka clients that are connected to those Agents.

Client ID

Alternatively, the partition assignment strategy can be configured for just a single application without impacting the default behavior for other applications using a Kafka client ID feature.