# Partitions Auto-Scaler
The partitions auto-scaler is a convenience feature that automatically scales the number of partitions in a topic to ensure that the average write throughput (uncompressed bytes per second) stays below a configured threshold. This enables the partition count to increase automatically with organic traffic growth so that operators don't have to take manual actions or perform capacity planning.
For example, consider a topic called called "logs" with the following configuration values:
| Configuration | Value |
| ------------------------------------------------------------------------------------------ | --------- |
| `warpstream.partitions_auto_scaler.enabled` | `true` |
| `warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second` | `2500000` |
| `warpstream.partitions_auto_scaler.max_partition_count` | `1024` |
The logs topic has 50 partitions and an average total throughput of 100 uncompressed MB/s. If we do the math then we'll see that:
$$
100/50 == 2MB/s/partition
$$
which is below the limit of 2.5MB/s/partition. In this scenario, the partitions auto-scaler will take no action. However, if the total throughput increases to 250 uncompressed MB/s, then the throughput per partition will increase to:
$$
250/50 == 5MB/s/partition
$$
which is above the limit of 2.5MB/s. As a result, the partitions auto-scaler will detect this and begin adding additional partitions to the topic until the average throughput per partition falls back below 2.5MB/s (I.E once the topic reaches roughly 100 partitions).
The partitions auto-scaler can add partitions to a topic, but it can never remove them. As a result, it does not take action on every temporary spike in write throughput, and instead waits for throughput to exceeded the configured target consistently for a contiguous window of time (currently 15 minutes) before taking action.
The partitions auto-scaler is particularly convenient for workloads where WarpStream is being used as a highly scalable and cost-effective "pipe" where records with the same key don't always need to be written to the same partition.
## Caveats
1. This feature is *not* suitable for any workloads where records with a specific key must always be written and consumed from the same partition. The reason for this is that in the Kafka protocol, record partitioning happens *client side*, so when the number of partitions in a topic *increases*, the destination partition for each record key will change.
2. The partitions auto-scaler can *increase* the number of partitions in a topic, but it cannot *decrease* the number of partitions in a topic. The reason for this is that the Kafka protocol (and the domain model of Kafka itself) are such that it's impossible to delete partitions from a topic with active producers/consumers in a safe manner. However, in the future we will solve this problem with WarpStream by creating "Virtual Topics" that abstract over multiple individual topics.
## Configuration Options
All of the configuration options below are topic-level configuration values.
| Configuration | Description |
| ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `warpstream.partitions_auto_scaler.enabled` |
Boolean value.
Whether the partitions auto-scaler is enabled for this topic. Defaults to false.
Target throughput (in uncompressed bytes per second) for each partition in the topic. If average throughput per partition is below this value, the partitions auto-scaler will continue to add partitions to the topic until throughput per partition falls below this value or the max partitions limit is reached.
For example, if this was set to 2500000 (2.5 MB/s/partition) then the partitions auto-scaler would begin adding additional partitions once the total throughput of the topic exceeded 10MB/s (10MB / 4 partitions = 2.5MB/s/partition).
|
| `warpstream.partitions_auto_scaler.max_partitions` | Maximum number of partitions beyond which the partitions auto-scaler will not add any more partitions to the topic, regardless of how high the throughput is. |
## How to Edit Configuration
There are two ways to enable / configure partitions auto-scaling on a topic:
1. The WarpStream UI.
2. The Kafka protocol's AlterConfigs API.
### WarpStream UI
Click "Edit Configuration" in the Topics view for your cluster, and then navigate to the "Partitions Auto scaler" tab.
### AlterConfigs API
Use a Kafka client / tool / UI of your choice that allows editing topic-level configuration values to update the configuration values described in the [configuration section](#configuration-options) for your topic.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.warpstream.com/warpstream/kafka/reference/partitions-auto-scaler-beta.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.