# Introduction The official documentation for WarpStream, a diskless, Apache Kafka-compatible data streaming platform built directly on top of cloud object stores such as S3. Welcome to the documentation for WarpStream, a diskless data streaming platform designed to seamlessly integrate with cloud object stores such as S3, GCP, and Azure while remaining compatible with Apache Kafka. To learn more, we recommend reviewing our documentation in the following order: 1. [WarpStream architecture](/warpstream/overview/architecture) to learn the basic principals of how WarpStream works. 2. Run the [WarpStream demo](broken://pages/Dv6iLG3VaT9hTeWcBEGJ) to see the product in action. Finally once you're ready to deploy WarpStream to one of your live environments, you can create a free WarpStream account by going to and then follow our documentation to [deploy the Agents](/warpstream/agent-setup/deploy). # Architecture This page provides an overview of WarpStream's architecture. {% embed url="" %} Overview of WarpStream's storage engine. {% endembed %} WarpStream eases the burden of running Apache Kafka by replacing the deployment and maintenance of a physical Kafka cluster with a single stateless binary (called the Agent) that only communicates with object storage (making it [diskless Kafka](https://www.warpstream.com/blog/zero-disks-is-better-for-kafka)) like Amazon S3 and our Cloud Metadata Store. WarpStream Agents speak the Apache Kafka protocol, but unlike an Apache Kafka broker, any Agent can act as the leader for any topic, commit offsets for any consumer group, or act as the coordinator for the cluster. No Agent is special, so auto-scaling based on CPU usage or network bandwidth is trivial. Running the WarpStream Agent is as easy as running a proxy or web server, such as nginx. But hang on a second, that sounds too good to be true. How did we accomplish this if Apache Kafka requires running ZooKeeper (or using kRaft), and maintaining a cluster of stateful brokers that have to be rebalanced all the time? 1. We separate storage and compute. 2. We separate data from metadata. 3. We separate the data plane from the control plane. ### Separating Storage and Compute Separating storage and compute is a common technique for scaling modern data processing systems. It allows you to scale your compute clusters up, down, in, or out in response to load while leveraging low-cost storage managed by someone else, such as Amazon S3. It allows any compute node to process data from any file in storage instead of each node "owning" a subset of the data. Separating storage and compute allows operators to scale up or down the number of WarpStream Agents to respond to changes in load without rebalancing data. It also enables faster recovery from failures because any request can be retried on another Agent immediately. We also eliminate hotspots, where some Kafka brokers have dramatically higher load than others due to uneven amounts of data in each partition. All of these hard problems have been delegated to hyper-scale cloud provider object storage services, where tens-of-millions of human-years of effort and billions of dollars have been invested into durability, availability, and operational excellence. ### Separating Data from Metadata Another pillar of WarpStream's design is the separation of data from metadata. This is also becoming a more common technique, such as Snowflake's SQL data warehouse storing primary data in object storage and metadata in FoundationDB. Our founders [did this at Datadog too](https://www.datadoghq.com/blog/engineering/introducing-husky/) when building Husky, the system that powers storage and queries for logs, real-user monitoring, network performance monitoring, and many other products at Datadog. WarpStream uses this technique to offload metadata management from our customers' operations teams to ours. We store the metadata for every cluster in our cloud metadata store designed from scratch to only solve this specific problem, operated 24x7 by the team who wrote it. This separation also provides useful security guarantees because we cannot read the data in your topics, even if our cloud was compromised. ### Separating Data Plane from Control Plane At a high level, the data plane of a WarpStream virtual cluster is a pool of Agents connected to our cloud. Any Agent in any pool can serve any produce or consume request for topics in that virtual cluster. The control plane runs in our cloud, where we decide which Agents will be compacting your data files for optimal performance, which Agents will participate in the distributed, zone-aware object storage cache, and which Agents will scan your object storage bucket for files which are past retention and can be deleted. Our control plane enables us to deliver on our promise of an Apache Kafka-compatible streaming system that is as easy to operate as nginx by offloading the hard problems of consensus and coordination onto our fully-managed control plane, while at the same time achieving a much lower TCO with the object storage-backed data plane running on the Agents. ### WarpStream Is Not "Tiered Storage" Multiple vendors offer "Tiered Storage" with an Apache Kafka-compatible interface. This means either you or the vendor run something that looks roughly like a stateful Apache Kafka broker that periodically offloads some older data to S3. WarpStream does not work this way. The Agent does not require local disks *at all.* Data streams directly from the Agent to object storage instead of being replicated via extremely expensive cross-AZ networking at an effective cost of $0.05/GB in AWS at retail prices. That's the same cost as storing data in S3 for over 2 months! Instead, the WarpStream Agent leverages the free networking between EC2 and S3 in AWS, which just so happens to durably replicate your data along the way. Working directly on top of S3 required a complete re-architecture of the system of the ground up. Other vendors can continue to push "Tiered Storage" to the physical limit, which is all but the last record of each topic-partition stored on a stateful broker and the remaining records in object storage, but they cannot compete with the savings of never going across zonal boundaries in the first place. ### Cost and performance considerations Naively attempting to implement an Apache Kafka-compatible system on top of S3 will end up with either extremely high latency, or a huge S3 API operations bill. For example, making a file per partition every 30 seconds is likely cost effective, but 30 seconds of latency is not what Apache Kafka users expect from their system today. Making a file per partition every 100ms could provide the latency that Kafka users expect, but the minimum cost per partition per month would be roughly $130/month in S3 PUT operations alone. That doesn't even get into the cost or latency to consume so many tiny files! WarpStream Agents make a few files per second, but each file contains records from multiple topics and partitions. In the background the pool of Agents compacts those small files into larger files to make reprocessing historical data for both single partitions and whole topics both cost effective and high throughput. This compaction and batching approach achieves a reasonable tradeoff between cost and end to end latency. ### WarpStream's Architecture Everyone loves a good architecture diagram, so we've provided ours here. The important bits are the fact that the Agent Pool runs inside a customer's VPC and not in ours, and customer data is never sent outside the customer VPC. The only data transferred from the Agent pool to the WarpStream Cloud is metadata about which files belong to a given Virtual Cluster, which is a collection of topics and partitions administered together. Applications connect to the Agent Pool using standard Apache Kafka clients. #### Architecture Diagram
#### Virtual Cluster A Virtual Cluster is the metadata store for WarpStream. Each customer can create multiple isolated Virtual Clusters for separating teams or departments. Kafka API operations within a Virtual Cluster are atomic, including producing records to multiple topics and partitions. Each Virtual Cluster is a replicated state machine which stores the mapping between files in object storage and ranges of offsets in each Kafka topic-partition. Every Virtual Cluster metadata operation is journaled to our strongly-consistent log storage system before being executed by a Virtual Cluster replica and acknowledged back to the Agent, which then acknowledges the request from your client application. Within a Virtual Cluster, the Agents can optionally be configured to [each serve a specific role](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles). This feature is only possible in WarpStream due to its decoupled architecture and cloud-native design. This functionality is not possible with Apache Kafka or any proprietary distributions of Kafka. #### Cloud Services Our Cloud Services layer manages the lifecycle of each replica of your Virtual Clusters. Each Virtual Cluster has multiple replicas for high availability and is backed up to object storage itself in sync with our log storage system. Replacement replicas of your Virtual Cluster are created automatically when existing replicas fail without any human intervention based on those backups and replaying the metadata command log. The Cloud Services layer also powers our administrative control panel and the observability system for your Virtual Cluster metrics. This Cloud Services layer is an isolated deployment per region, as are the Virtual Clusters underneath it. # Service Discovery This page explains how WarpStream's custom service discovery mechanism works, as well as how it interacts with Kafka's service discovery system. {% hint style="info" %} The ["Hacking the Kafka Protocol" blog post](https://www.warpstream.com/blog/hacking-the-kafka-protocol) is a good complement to this documentation page and provides some more context and background on the design decisions that were taken. It also goes into more technical detail about how the WarpStream service discovery and load balancing systems work together in tandem. {% endhint %} ## Kafka Service Discovery Before we discuss how WarpStream's service discovery system works, lets first go over the service discovery mechanism in Kafka. Kafka clients are usually instantiated with a list of URLs that identify a set of "bootstrap" Kafka brokers. The URLs can be raw IP addresses, or they can be hostnames that are resolved by the client to IP addresses via DNS. Once the client has connected to the bootstrap brokers, it will issue a `Metadata` request for the cluster. The response to this request will contain a list of which brokers are in the cluster, what their hostname / IP addresses are, what port they're available on, which rack (or AZ) they're running in, and which topic-partitions they're the leader for. The client will use this metadata to establish connections to all the other Kafka brokers in the cluster with which it needs to communicate. The client will always try to communicate with the leader of a given topic partition when it is producing data and one of the replicas (depending on how it is configured) when it is consuming data. In addition, if the client is using Kafka's consumer group functionality, it will also make a `FindCoordinator` request to the cluster to determine which Kafka broker is the "group coordinator" for its consumer group, and then the client will establish a connection to that broker for the purposes of participating in the consumer group protocol. ## Mapping WarpStream Service Discovery to Kafka Service Discovery Kafka brokers are inherently stateful. Therefore, the focus of the Kafka service discovery system is to ensure that clients connect to the right brokers that are responsible for the topic-partitions that they want to produce to or fetch from, as well as regularly fetching updated metadata to react to changes in topic-partition leadership. WarpStream Agents, on the other hand, are completely stateless, and any WarpStream Agent can service any produce or fetch request. Therefore the focus of the WarpStream service discovery system is different from that of Kafka's. WarpStream's service discovery system has exactly two goals: 1. Keep traffic/load as evenly balanced across the WarpStream Agents as possible. 2. Keep communication between Kafka clients and WarpStream Agents zone-local as much as possible to minimize inter-zone networking costs. WarpStream service discovery begins the moment the Agents are deployed. The Agents will use cloud-specific APIs to try to automatically discover which availability zone they're running in (this currently works in AWS, GCP, Azure, and Fly.io) and then publish this information along with their internal IP address to WarpStreams' service discovery system.

A WarpStream cluster evenly distributed across 3 availability zones in us-east-1.

{% hint style="info" %} You can override the availability zone the Agents report to the discovery system using the `WARPSTREAM_AVAILABILITY_ZONE` environment variable. \ You can double check the [configuration child page](/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy) to read more about how to override the default values used for service discovery. {% endhint %} Next, the Kafka clients in your applications need to be configured with the WarpStream bootstrap URL.
You also need to encode the availability zone of your application into your Kafka client's "client ID". See [our documentation on "Configuring your Apache Kafka Client for WarpStream"](/warpstream/kafka/configure-kafka-client) for more details on how to do that. Regardless, once you've configured the WarpStream bootstrap URL and client ID in your Kafka client, service discovery will proceed similarly to how it does in traditional Kafka. First, the client will use DNS to resolve the WarpStream bootstrap URL to an IP address. At this point, we don't care about zone awareness yet, so the WarpStream DNS server will return any available WarpStream Agent IP address. Next, the client will issue a `Metadata` request to the WarpStream Agent identified in the previous step. The Agent will proxy the `Metadata` request to the WarpStream discovery service, which will return an appropriate Agent hostname / IP address that is running in the same availability zone as your client [*if you've properly encoded your application's availability zone into the Kafka client's client ID*](/warpstream/kafka/configure-kafka-client)*.* If you haven't, it will return an Agent hostname / IP address from *any* availability zone that has live agents. This is how WarpStream "injects" zone awareness into the Kafka protocol, even for messages like `Produce()` which don't support "rack awareness" in the traditional Kafka protocol. This approach also solves for load balancing. Proxying the `Metadata` request to the WarpStream service discovery system provides the service discovery system with an opportunity to perform load balancing. Currently, it does this using a naive (but effective) global round-robin balancing algorithm to Agents within the Availability Zone specified in the client ID, although we will make the algorithm more nuanced in the future. ## Partition Assignment See our dedicated [partition assignment strategy documentation](/warpstream/kafka/reference/partition-assignment-strategies). ## Group Coordinator Selection All of the consumer group coordinator logic is handled by the WarpStream control plane; the Agents just act as a proxy for those requests. As a result, any WarpStream Agent can be returned as the result of a call to `FindCoordinator` and the system would work correctly. However, to make WarpStream look more like a traditional Apache Kafka cluster, the WarpStream control plane uses a consistent hashing ring to select a special "coordinator" Agent to return as the consumer group coordinator. This is acceptable because the amount of communication between the Kafka clients and the "group coordinator" is minimal and has almost no impact on load balancing on inter-zone networking. # Write Path The following page is a summary of WarpStream's approach to persisting data produced from clients. {% hint style="info" %} For a more in-depth discussion of the WarpStream write path, refer to [Unlocking Idempotency with Retroactive Tombstones](https://www.warpstream.com/blog/unlocking-idempotency-with-retroactive-tombstones) on the WarpStream blog. {% endhint %} ## Introduction The WarpStream write path is designed to be cost-efficient, especially for higher-throughput workloads (often 5-10x more than Apache Kafka), with performance characteristics that are aligned with the requirements of real-time data systems. At the same time, WarpStream was also designed to be an order of magnitude easier to *operate* than Apache Kafka. It accomplishes this by leveraging object storage directly, *with no intermediary disks.* However, keeping both costs and latency low while leveraging object storage as the *only* storage is a tall order. The rest of this document will explain the architectural decisions that make this possible. ## Write Path There are three key aspects to WarpStream's write path: 1. WarpStream automatically aligns producer clients with Agents running in the same Availability Zone as the producer. 2. Every WarpStream Agent can write data for any topic-partition (there are no topic-partition leaders). 3. Data is always stored durably in object storage and committed to the WarpStream control plane before successfully acknowledging a produce request. This design avoids 100% of cross-AZ data transfer fees because producer clients never have to write data to Agents in another AZ, and replication is handled by object storage. This results in a dramatically better cost profile and higher durability guarantees than Apache Kafka. #### Buffering WarpStream Agents buffer `Produce()` requests from multiple producer clients and partitions, and then write these records in batches to object storage. By default, the Agents will buffer data for 250ms, or until 8 MiB of data has been accumulated, whichever comes first. After the file is written to object storage, the Agent commits the file metadata to the WarpStream Metadata Store, and then acknowledges all of the `Produce()` requests in the batch back to the clients. WarpStream never acknowledges writes until data is durably persisted in object storage *and* committed to the metadata store.
Unlike Apache Kafka, WarpStream does not use a naive file-per-partition strategy as that would require writing many small files to object storage or buffering data in-memory for an unacceptably long period of time. Instead, the WarpStream Agents create individual files that contain data from many different topic-partitions, which keeps both costs and latency low.

An illustrative set of buffered Produce() requests, which are aggregated in the Agent before being flushed to object storage

Leveraging object storage directly, with no intermediary disks or WAL, greatly simplifies WarpStream's architecture and operational requirements: 1. Partitions never have to be rebalanced. 2. The WarpStream Agents can be instantly scaled up and back down with zero data shuffling. 3. Disruption or loss of even 100% of the WarpStream Agents will result in loss of 0 acknowledged data. ## File Compaction In the background, WarpStream Agents compact files in object storage, which enables merging batches for the same topic-partition in different files together, improving IO access patterns for historical replays. It also provides the opportunity to reorganize the data to improve locality by topic-partition. See the documentation of the [read path](/warpstream/overview/architecture/read-path) to learn how WarpStream avoids making a large number of GET requests to object storage APIs, regardless of the number of partitions and consumers. {% hint style="info" %} Also, refer to Minimizing S3 API Costs with Distributed map on our blog for a detailed discussion of WarpStream's cost and performance optimizations using object storage primitives. {% endhint %} ## Durability Produce requests are not acknowledged until the data is persisted in the object store and the metadata is committed to WarpStream's metadata store. This is similar to Kafka's acks=all semantics, however cloud object storage systems such as Amazon S3 have much better durability guarantees than what can be accomplished with Apache Kafka. By eliminating local storage, WarpStream is able to provide stronger durability guarantees than what can be achieved using triply replicated SSDs. ## Ordering and Idempotency WarpStream maintains the exact same ordering guarantees and idempotency guarantees as Apache Kafka. As with Kafka, messages produced to a specific topic partition in WarpStream are appended to the log in the order that they are sent, and consumers will read the messages in the order that they are stored in the log. To maintain ordering within a topic-partition, while still enabling a single topic-partition to be appended to from dozens of different Agents, WarpStream determines the order of writes upon *committing* a batch to the WarpStream Metadata Store, not when the data is flushed to object storage. This approach enables the Agents to massively parallelize writes, while still maintaining the same ordering and idempotency guarantees as Apache Kafka. To learn more about how this works, check out the [Unlocking Idempotency with Retroactive Tombstones blog post](https://www.warpstream.com/blog/unlocking-idempotency-with-retroactive-tombstones). # Read Path The following page describes the mechanism for serving Fetch requests from WarpStream. {% hint style="info" %} For a detailed discussion of the WarpStream read path, review [Minimizing S3 API Costs with Distributed mmap](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-distributed-mmap) on the WarpStream blog. {% endhint %} ## Introduction WarpStream Agent maintains cost efficiency for reads by ensuring that the Agents only load data from object storage once per availability zone and minimize `GET` requests to object storage by loading data in large chunks. To accomplish this goal, WarpStream Agents maintain a per-Availability Zone cache that behaves like a distributed [mmap](https://en.wikipedia.org/wiki/Mmap), which completely decouples the number of partitions and consumers from the number of object storage `GET` requests that the Agents have to perform to serve consumer clients' `Fetch()` requests. ## Read Path In order to efficiently distribute load between Agents, WarpStream uses a consistent hashing ring between the Agents, which enables each Agent to maintain responsibility for the location of a subset of data in a given topic. When a WarpStream Agent receives a `Fetch()` request from a client, the first step is to find the Agent that is responsible for the file, and route the request to that Agent. Then, the responsible Agent pages the file chunk into memory. This in-memory cache can then be used to serve `Fetch()` requests to clients. A cache is maintained per Availability Zone, which ensures that `Fetch()` requests do not need to cross zonal boundaries in order to be fulfilled. While this results in slightly more `GET` requests to object storage, the avoidance of cross-AZ data transfer fees more than makes up for this increase. In addition, the Agents serve subsequent `Fetch()` requests from the cache, which further reduces the impact of the per-AZ `GET` requests.
In the illustration above, Agent 1 initially receives a `Fetch()` request from a client for a particular topic, partition, and offset. Agent 1 forwards the request to Agent 2 because Agent 2 is responsible for the file that contains the data requested by the client. Agent 2 pages the relevant chunk of the file into memory, which can now be returned to the client that made the `Fetch()` request. Subsequently, two more requests are received, first by Agent 3 and then by Agent 2. Because Agent 2 already has the chunk of the relevant file in memory, and both requests are fetching data located in File 3, Agent 2 serves the `Fetch()` requests from memory, eliminating the need to read it out of object storage again. {% hint style="info" %} Refer to [Minimizing S3 API Costs with Distributed mmap](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-distributed-mmap) on our blog for a detailed discussion of WarpStream's cost and performance optimizations using object storage primitives. {% endhint %} This approach results in a workload cost profile that is both more cost-efficient than what is possible with Kafka brokers with local disks, and cost-efficient with respect to usage of cloud object storage. ## Ordering Any WarpStream Agent can serve any `Fetch()` request from any client. WarpStream does not have a concept of leader partitions. However, WarpStream still maintains Kafka's ordering guarantees.
In the above example, the client makes a `Fetch()` request for messages beginning with offset 306. The Agent queries the Metadata Store for the file(s) and batches in which these offsets are contained. In order to maintain Kafka's ordering semantics, the Metadata Store returns an ordered list of files and batches that the Agent should read. Because writes to object storage are [decoupled from metadata commits](/warpstream/overview/architecture/write-path), the files may be written to object storage out of order, but the source of truth for ordering is held in the Metadata Store, so clients are served data in the correct order. ## Follower Fetching Whereas Kafka requires careful considerations and complex configuration to reduce cross-AZ networking costs, WarpStream is designed from first principles to avoid cross-AZ traffic. {% hint style="info" %} For a detailed discussion of WarpStream's zone-aware routing and AZ-specific load balancing, please review [Hacking the Kafka PRoTocOL](https://www.warpstream.com/blog/hacking-the-kafka-protocol) on the WarpStream blog. {% endhint %} By default, Kafka consumers read from the leader partition, regardless of where the broker hosting that partition is located. To help reduce cross-AZ traffic between consumers and brokers, Apache Kafka has introduced a feature to enable consumer clients to [fetch from the closest replica](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica), regardless of whether it is the leader partition. This feature enables consumers to avoid cross-AZ data transfer fees by aligning `Fetch()` requests to the broker's availability zone. However, Follower Fetching is a complex feature to use in real-world situations. To use Follower Fetching, the consumer must be aligned with at least one zone where Kafka is deployed. To maintain high availability, the consumer must be distributed across multiple AZs. In many cases, this would require a significant amount of work, and in all cases it results in a more complex deployment. For example, a consumer for a given application may be deployed in a single AZ, and if Follower Fetching is enabled, this consumer will read only from the broker in that AZ. If the follower partition falls out of the ISR, consumers will be unable to consume records whose offsets exist locally but are not yet known to be committed or whose offsets are known to be committed but do not yet exist locally. Both of these situations will manifest as unavailability. This is a common case in organizations with a shared Kafka cluster that many application teams use to source data. In the worst case, consumer applications may be deployed in an Availability Zone that has *no* overlap with the Kafka cluster, which results in 100% misalignment between the consumer and Kafka. Due to the cost of replicating data across AZs and the complexity of maintaining local replicas, Kafka clusters normally span a maximum of three AZs. Simply stretching the cluster to another AZ to ensure zone alignment with the consumer normally negates the cost savings achieved by zone alignment and increases the complexity of managing the cluster. These complications are due to Kafka's replication protocol, which replicates data between brokers' local disks. In addition, because brokers are stateful systems and changing where the brokers are deployed is nontrivial, once a decision is made regarding which AZs to deploy the brokers in, it is difficult to change. Because WarpStream Agents are stateless, and WarpStream is designed specifically to avoid cross-AZ data transfer fees, achieving zone alignment between Agents and clients is trivial. In the case where the consumer has partial alignment with the cluster, WarpStream incurs zero cross-AZ data transfer charges because there is no concept of a "leader" partition. Consumers can fetch data from any partition, from any Agent, and reads occur within the same AZ due to the [caching approach](#read-path) discussed above. In the case where there is zero alignment between the consumer and the cluster, users can trivially deploy Agents in the correct Availability Zone. # Life of a Request (Simplified) This page explains the flow for a Kafka Produce or Fetch request from client to WarpStream Agent to WarpStream Cloud and back again. ### Produce Before we can `Produce` to a topic, our client calls the `Metadata` Kafka API on our behalf. This API returns the leader broker for each topic-partition. Because any Agent can handle any Produce or Metadata request, each client can have a different view of which Agent is acting as leader broker for every topic-partition. After discovering who the leader is for our desired topics (which, in WarpStream's case, is an arbitrary agent in the same AZ), our client can begin producing records. Because we only return a single leader for every topic-partition in the cluster, every Produce request can contain messages for every topic-partition our client writes to. Let's assume that our client only writes to a single topic for now to make things easier to understand. WarpStream Produce requests arrive on an arbitrary Agent and are batched together with other `Produce` requests from other clients happening concurrently. Once the Agent's batching interval has elapsed, or the Active Buffer is full, the Active Buffer is serialized to WarpStream's file format and passed to the Flush Queue to be written to object storage.
Once the file is flushed to object storage, the metadata for that file is sent to the WarpStream Cloud and routed to the Virtual Cluster hosting that topic and the other topics that have data contained within that file. The Virtual Cluster assigns offsets to every batch of records and returns those back to the Agent, which then fans out responses back to waiting clients. The client that issued the `Produce` request will not receive an acknowledgement from the Agent until this entire sequence has completed and the data has been durably persisted in object storage and committed to WarpStream Cloud. WarpStream Agents *never* acknowledge data until it has been durably persisted. ### Fetch `Fetch`, similar to `Produce`, depends on previously calling the `Metadata` to discover the topic-partition leaders. Calling WarpStream's bootstrap URL will direct your client to an Agent in your availability zone to serve your `Fetch` request. Let's assume for simplicity that your client is only reading a single topic-partition. Your request will be processed by some Agent, and that Agent will send a command to the WarpStream Cloud that eventually will be processed by your Virtual Cluster. This command instructs your Virtual Cluster to return the set of files that contain data starting with the offset in your `Fetch` request at that moment in time. Once the Agent processing your `Fetch` has the "pointers" to the set of files containing the data you're going to read, it fans out IO requests to other Agents in the same AZ. All the Agents in each AZ participate in a zone-aware distributed file caching layer to reduce the number of object storage GET requests for the most recent set of files. The Agent can respond to your `Fetch` request and begin sending the bytes back over the wire once all the "pointers" for the offset you've requested have been read from the distributed file cache.
# Change Log Contains a history of changes made to the Agent. {% code title="Subscribe to the RSS Feed" overflow="wrap" %} ``` https://console.warpstream.com/agent-changelog.rss ``` {% endcode %} {% hint style="info" %} By default all Agent upgrades are designed to be seamless and backwards compatible. Occasionally we have to make make breaking changes which we document in the [migrations sub-page](/warpstream/overview/change-log/migrations) where you can see if any breaking changes were made between the version you're running and the version you're upgrading to. {% endhint %} ## Change Log #### Release v832 August 19, 2026 * Tableflow: * Add the ability to create more row groups per Parquet file for ingestion. * Relax ingestion file size constraint. * Add the following metrics: * The `agent_tableflow_parquet_file_num_rows` distribution metric tracks the number of rows per Parquet file. * The `agent_tableflow_ingestion_live_partitions` gauge tracks the number of distinct Iceberg partitions created during ingestion. * Prevent deleted source topics from blocking Orbit fetches for healthy topics. * Stop paging the file-cache invariant `single actor stream returned N bytes, but M were requested` when the Fetch is already canceled. A canceled request could close the stream while it was still being copied, which looked like a short stream. * `warpstream local`: add `-saslUser`, `-superUsers`, and `-enableACLs` so integration tests can start a local cluster with named SASL users and Kafka ACL enforcement. Also fix `-requireSASLAuthentication` storing/printing the password as `REDACTED`. * Fix a bug in the HTTP fetch endpoints where the pooled fetch buffers were returned to the buffer pool before the HTTP response encoder had finished reading them (the JSON response references the buffers zero-copy). Under concurrent load, another fetch could re-lease the buffers and overwrite them mid-encode, corrupting the HTTP fetch response. The buffers are now only released after the response has been fully encoded and written. * Bump build to Go 1.26.6 and `golang.org/x/mod` to v0.40.0 to fix CVE-2026-39821, CVE-2026-46600, CVE-2026-33818, CVE-2026-56853, CVE-2026-56862, CVE-2026-56859, CVE-2026-56864, and CVE-2026-56865. * Add `blob_store_write_duration` metric, tagged by `bucket_url`, reporting per-sub-bucket write latency for striped buckets. #### Release v831 August 14, 2026 * Tableflow: * Add the `WARPSTREAM_TABLEFLOW_ASSUME_ROLE_ARN` and optional `WARPSTREAM_TABLEFLOW_ASSUME_ROLE_DURATION_MINUTES` environment variables for a shared STS AssumeRole used by both S3 bucket access and AWS Glue catalog sync. When set, the Tableflow role supersedes `WARPSTREAM_BUCKET_ASSUME_ROLE_*` for bucket access. * Fix a bug where the bucket URL `prefix` query parameter wasn't being respected when AssumeRole is used. * Stop logging an error when setting or clearing a connection's read deadline fails because the connection was already closed. This just means the client disconnected. * Add the `loading_queue_used_count`, `loading_queue_discard_count`, and `loading_queue_load_outcome` (tagged by `outcome`) metrics, all tagged by `queue` (`prepared_upload`, `jobs`, or `file_id`), so it's possible to see how many items each of the agent's loading queues handed out, aged out unused, and failed to load. #### Release v830 August 13, 2026 * Tableflow: * A `schema` can now be defined inline in schema registry mode and will be used as the schema used to write records in Parquet files. This is handy for defining transforms that change of the shape of the data being inserted into the table. * The `error handling connection, closing it` log now includes the `client_id` of the Kafka client that was using the connection. * Remove noisy log when the context is canceled for speculative reads. * The auto migration forwarded produce metrics `warpstream.agent_kafka_produce_forwarded_records_counter`, `warpstream.agent_kafka_produce_forwarded_compressed_bytes_counter` and `warpstream.agent_kafka_produce_forwarded_compressed_bytes` are now tagged by `outcome`, so records that the source Kafka cluster rejected are reported alongside the ones it accepted. Possible values are `success`, `error`, and `unverified` (used when producing with `acks=0`, where the source cluster sends no response and the Agent cannot confirm the records were accepted). #### Release v829 August 12, 2026 * Tableflow: * Add the `-tableflowHighCardinalityDistributionMetrics` flag (`WARPSTREAM_TABLEFLOW_HIGH_CARDINALITY_DISTRIBUTION_METRICS` environment variable) to control whether high cardinality per table distribution metrics are emitted. Defaults to false. * Fix pure-Go `Lz4Block` `CompressAppend` so it appends to the destination prefix instead of discarding it, which could corrupt Kafka record-batch headers written before compression. * Fix flexible Kafka response framing so reused connection write buffers always emit an empty response-header tagged-fields count. A stale byte (for example after SASL handshake v0) could make clients fail to decode later flexible responses such as Fetch. * Fix a race that could assign the same Kafka connection ID to an external connection and an in-process direct-dialer connection, and refuse to serve a prefetched Fetch response when the claimer's Fetch API version does not match the originator's. * Fix noisy "async gcs rapid storage writer close failed" logs with `context canceled` during Rapid Storage uploads by decoupling the writer context from caller cancelation while preserving the deadline. * The `metrics` subcommand now supports OIDC workload identity federation via the `-workloadIdentityTokenSource` flag (and `WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE` environment variable), matching the `agent` subcommand. When set, `-enableControlPlanePrometheusEndpoint` is forced to `true` because the exchanged agent token is only accepted by the control plane's prometheus endpoint. * Enable public Agent releases to receive control-plane-generated Events on Kafka clusters, including diagnostics and Orbit migration events. * Size the batcher's prepared upload queue and load concurrency off the agent's CPU quota instead of the host's CPU count, so containerized agents no longer over-create prepared upload files. * Stop deleting prepared upload files that lose the race for a spot in the batcher's queue. They now wait for the next free spot, which removes the object store create and delete each one used to cost. #### Release v828 August 10, 2026 * Tableflow: * Add the `agent_tableflow_parquet_file_size_uncompressed` and `agent_tableflow_parquet_file_size_compressed` distribution metrics for Parquet files, tagged by `source` (`ingestion`, `compaction`, or `deletion`). * Add the `agent_tableflow_parquet_file_num_row_groups`, `agent_tableflow_row_group_size_uncompressed`, and `agent_tableflow_row_group_size_compressed` distribution metrics for Parquet row groups, tagged by `source` (`ingestion`, `compaction`, or `deletion`). * Add support for multi-bucket catalogs in BigLake. * Add experimental support for upsert tables (`table_type: upsert`): records are deduplicated by Kafka record key within each source partition (highest offset wins) and records with null values delete the key from the table. * Fetch requests that include an unknown or invalid topic ID (for example, a recently deleted topic referenced by TopicID only) now return a partition-level error for that topic instead of failing the entire Fetch. Valid topics in the same request continue to return data, matching Apache Kafka behavior. #### Release v827 August 4, 2026 * Add diagnostic to detect when the Orbit auto migration client has connectivity issues with the source cluster. * Fix incorrect `failed to lookup AZ from CIDR mapping` warnings for the Agent's own internal clients (like the query engine). * Improve auto migration's logic to detect if a produce request is transactional by actually checking if the record batch has the transactional flag in its attributes. * Upgrade Bento to v1.20.0. * Add the user-provided `credential_name` to ACL denial logs. * `warpstream local`: allow clients to use session / rebalance / transaction timeouts below the normal broker minimums so unit tests against local can run faster (previously rejected). * Skip shadow ACL denial noise for in-process direct-dialer clients (e.g. the agent's query engine / events UI). * Accept OAuth bearer tokens on the HTTP Fetch and Produce APIs. When `saslOauthIssuerURL` and `saslOauthAudience` are configured, HTTP clients can authenticate with `Authorization: Bearer ` in addition to HTTP Basic auth; bearer tokens are validated via the same OAUTHBEARER path used by the Kafka wire protocol. * Migrate direct imports from `github.com/hamba/avro/v2` to `github.com/iskorotkov/avro/v2` `v2.33.1` to address `CVE-2026-46384`, `CVE-2026-46385`, and `GHSA-mx64-mj3q-7prj`. Transitive `hamba/avro` usage remains until dependent modules move off it. * Set `MaxMapAllocSize` on Tableflow Avro decode of untrusted Kafka payloads to mitigate map-allocation DoS (GHSA-mx64-mj3q-7prj). * Added **bucket striping** (`warpstream_stripe://$URL_1<>$URL_2<>...<>$URL_N`, 2–32 sub-buckets): an object-storage wrapper that scales writes past a single bucket's request-rate ceiling by striping objects across sub-buckets. Placement is deterministic — each object lives on exactly one sub-bucket, chosen by hashing its key. Writes fail over to another sub-bucket when the target's circuit breaker is open; reads go to the hashed sub-bucket and fall back to the others on NotFound. Unlike `warpstream_multi://` (which replicates for read availability), striping stores each object once, so a sub-bucket outage makes its objects unreadable — the same blast radius as a single bucket. Retiring a sub-bucket by shortening the URL strands dead files on it unless the retired bucket (or the previous stripe URL) is added to `additionalBackgroundTasksBucketURLs`. #### Release v826 July 29, 2026 * Tableflow: * Introduce a maximum uncompressed row size limit for Tableflow ingestion, measured after projection and transforms. Records that exceed the limit are handled based on the configured DLQ policy. * Use a more precise memory measurement for Tableflow ingestion buffering. * Added new metric `warpstream_orbit_auto_migration_unproxied_source_writes_num_records`: counts the number of detected unproxied records to the source cluster that bypassed the WarpStream proxy. * tags: `topic=` * Add `agent_group` to root metric tags so metrics like `warpstream.agent_kafka_request_latency` and `warpstream_consumer_group_lag` are groupable by agent group in Datadog. * Fix SASL OAUTHBEARER rejecting non-RS256 tokens (e.g. ES512) when the OAuth 2.x authorization server is discovered via RFC 8414 and does not advertise `id_token_signing_alg_values_supported`. * Fix events queries returning an internal error instead of a "no events data found" message when an event topic was created moments earlier and the stream metadata cache had not caught up yet. * Adds support for auto-migrating idempotent producers via Orbit. #### Release v825 July 24, 2026 * Add Confluent REST Proxy v2-compatible and multi-topic HTTP produce endpoints. * Tableflow: Fix ingestion trying to flush an empty file (`failed to add record to buffer: failed to sort and flush: cannot flush empty record batch`). * Bump `google.golang.org/grpc` to `v1.82.1` to fix `GHSA-hrxh-6v49-42gf`. #### Release v824 July 22, 2026 * Managed Data Pipelines: when the number of pipelines Agents changes, scale pipeline instance concurrency up or down instead of stopping and restarting all pipelines. * Fix a throughput bottleneck during Orbit auto-migration `PROXY`. #### Release v823 July 21, 2026 * Fixes a panic that impacted the `demo` command, the `playground` command, and agents connected to a Tableflow cluster (`panic: failed to initialize duckdb client dependencies`). * OAUTHBEARER SASL: fall back to `/.well-known/oauth-authorization-server` (RFC 8414) when OpenID Connect discovery at `/.well-known/openid-configuration` fails, so identity providers that only implement the RFC 8414 discovery endpoint can be used. * Tableflow: Add support for custom table names. When a table is renamed, the tagging of logs for that table will start to use the new name. #### Release v822 July 20, 2026 * Tableflow: * Reject decimal values that exceed the column type's declared precision with an error instead of panicking during ingestion. * Release sorting and column statistics tracking for Iceberg decimal types. * Fix false-positive "Cannot Access Tableflow Schema Registry" health diagnostic against WarpStream BYOC Schema Registry (and other registries that do not implement `GET /mode`). * Remove the legacy AWS SDK v1 from agent binaries to prevent false-positive CVE-2020-8911 reports. * GCS Rapid Storage: flush on close instead of waiting for a full synchronous Close, reducing write finalize latency. * Add a `delete-broker-config` CLI sub-command (`warpstream cli delete-broker-config --config-name `) that resets a broker config to its default via `IncrementalAlterConfigs` with a delete op. #### Release v821 July 14, 2026 * Tableflow: * Store decimals with precision greater than 18 as FIXED\_LEN\_BYTE\_ARRAY in Parquet. * Add a health diagnostic that checks connectivity to schema registry integrations. * Upgrade Bento for the `parse_big_decimal` bloblang method (Kafka Connect / Debezium decimal decoding). * OIDC workload identity federation now works on multi-region clusters (and across single-region cluster migrations): the agent holds a short-lived token per region and presents the one minted by whichever region it is talking to. * Promote `warpstream cli-beta` to `warpstream cli` — it is now the default Kafka CLI. `warpstream cli-beta` still works as a deprecated alias (prints a deprecation warning on stderr) and will be removed in a future release. The previous `warpstream cli` command has been renamed to `warpstream cli-old` and will be retired eventually; `warpstream kcmd` continues to work as before. Note: this is potentially a breaking change since some commands are different and most of the flags are different. #### Release v820 July 10, 2026 * Tableflow: * Add Schema Registry integration for Avro schemas. * Enable LZ4 compression in benchmkark-producer CLI command and tweak defaults to be more sane. * Fix a data race when speculative object storage uploads finish concurrently. #### Release v819 July 10, 2026 * Tableflow: * Emit diagnostic and event when the subject is not found in the Schema Registry. * Upgrade Bento to v1.19.0 (requires Go 1.26.5). * Bump build to go `1.26.5`. #### Release v818 July 9, 2026 * Added a new `warpstream local` command that runs a fully in-memory, self-contained WarpStream cluster (Kafka + Schema Registry + Tableflow) with no connection to the WarpStream control plane. Intended as a replacement for `warpstream playground` in CI pipelines. * Automatically run `warpstream local` when `warpstream demo` or `warpstream playground` is started in CI, with a warning that local mode is designed for CI environments. * Agent: dynamically tune the direct load prefetch concurrency based on observed CPU usage instead of using a fixed value of 16. Underloaded agents can still serve up to 64MiB of concurrent prefetch per request, but as CPU usage rises the concurrency automatically steps down to avoid the agent DDOSing itself with prefetch work it can't transmit in time, which previously caused wasted GET requests and slow recovery under high load. * Orbit: add support for JSON key/value secrets in Cloud Provider's Secrets Manager. A secret reference in the Orbit config can now specify an optional `key` to extract a single value from a secret stored as a JSON object (the standard AWS Secrets Manager key/value pattern). AWS secrets stored as binary (`SecretBinary`) are now supported as well. * Ensure `Close()` is called on the pooled buffer after all replicas complete on the quorum-success path of multi-bucket `PutBytes`. * Add cluster-level default partition auto scaler settings (`warpstream.default.partitions_auto_scaler.*` broker configs) that are applied to newly created topics, so a cluster can enable partition auto scaling by default for all new topics. #### Release v817 July 8, 2026 * Tableflow: * Expose Schema Registry writer schema ID and version as `warpstream.sr.writer_schema_id` and `warpstream.sr.writer_schema_version` to ingested records before applying Bento transforms so that the transform logic can condition on those properties. * Make BigLake integration automatically detect catalogs with credential vending mode and send `X-Iceberg-Access-Delegation` header on Iceberg REST calls if so. #### Release v816 July 6, 2026 * Allow Orbit to chunk requests when copying consumer group offsets. * Schema Registry: include Confluent-compatible `guid` values in schema API responses and support looking up schemas by GUID with `GET /schemas/guids/{guid}`. #### Release v815 July 2, 2026 * Add support of `-disableConsumerGroupMetrics` (env var `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS`) and `-disableConsumerGroupsMetricsTags` (env var `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS`) in agent `metrics` mode to offer the same knobs as in regular agent mode over cluster level metrics cardinality. By default `-disableConsumerGroupsMetricsTags` is set to empty string (the default value for regular agents is `partition`). * Tableflow: * Add support for OAuth as an authentication mechanism for source Kafka clusters. * Fix Go runtime CPU utilization reporting so stale high samples decay when the Go runtime has not emitted a fresh CPU metrics snapshot. * Auto enable bucket pre-warming when the bucket is of type Rapid Storage. * Reduce default value of GOGC for Agents running only pipelines role from 1000 to 200. #### Release v814 July 1, 2026 * Fixed a bug in the Prometheus metrics scraping path where the `warpstream_max_offset` and `warpstream_min_offset` metrics were summed across partitions instead of taking the max/min when the `partition` tag is disabled. * Fix the metric double publishing problem introduced in `v808` when the dedicated metrics mode was enabled (the regular agents would keep publishing the same metrics). * Improve produce latency for GCP Rapid Buckets by pre-warming upload streams. * Add a new `-additionalBackgroundTasksBucketURLs` flag and `WARPSTREAM_ADDITIONAL_BACKGROUND_TASKS_BUCKET_URLS` environment variable that accepts a comma-separated list of bucket URLs that should have background tasks run on them like scanning for dead files to delete or ripcord files to ingest. This is important for customers who are migrating from one bucket to another. The new flag supersedes the existing `-additionalDeadscannerBucketURLs` flag and `WARPSTREAM_ADDITIONAL_DEADSCANNER_BUCKET_URLS` environment variable. * Add support for OIDC workload identity federation: when `WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE` is set, the agent obtains a platform OIDC token and exchanges it with the control plane for a short-lived agent token, removing the need for a static agent key. Currently `aws` (STS) and `gcp` (metadata server) sources are supported, but other platforms may be added in the future. Supported for single-region clusters only. * Add an opt-in file ID allocator (behind the `use_range_file_id_allocator` flag, default off) that hands out IDs from several discontiguous reserved ranges to spread object-storage key prefixes more evenly and reduce metadata-layer hot-spotting. The key layout is unchanged. The `WARPSTREAM_USE_RANGE_FILE_ID_ALLOCATOR` environment variable (`true`/`false`) is the final per-Agent override for the flag. #### Release v813 June 29, 2026 * Tableflow: * Update internal compaction API usage. * Fix bug where the schema registry integration feature was not ungated properly. * Add support of Cloud Provider's Secrets Manager in TableFlow config. * Add a new flag `-skipBucketsPermissionCheck` (env var `WARPSTREAM_SKIP_BUCKETS_PERMISSION_CHECK`) that allows to skip the permission check for buckets when starting an agent. The default value is false. * Improve DNS resolution of S3 addresses by automatically injecting a `.` at the end of any S3 endpoint we try to connect to. This is a workaround because the AWS SDK does not support it to this day . * Re-enable GCS direct connectivity client by default. * Disable some connection-level backpressure mechanisms when TLS is enabled because once we've paid the price of accepting a TLS connection and doing the formal handshake, paradoxically, backpressuring that connection by closing it due to high load makes the situation worse as the client will just try to re-establish the connection and the TLS handshake is extremely expensive. * Stop playground agents from logging query engine state refresh errors when query engine credentials are not required. #### Release v812 June 22, 2026 * Move orbit consumer offset copying job entirely into the agents. #### Release v811 June 22, 2026 * Tableflow: * Add Schema Registry integration for Protobuf schemas. * Fix Tableflow ingestion getting stuck (`no ranges were processed, possibly due to timeout`) on source topics written by transactional producers (e.g. CDC), where the offset at the high watermark is a transaction control marker that a read-committed consumer never receives. * Add support of Cloud Provider's Secrets Manager in Orbit config. * Schema Registry: prevent an incompatible type change of a `oneof` field when that field's `oneof` block is renamed or dissolved. This matches a bug fix in Confluent's protobuf compatibility that will be released in Confluent Platform 8.3.1. #### Release v810 June 19, 2026 * Tableflow: * Support sorting and column statistics on `float` and `double` columns. * Schema Registry: Match Confluent's protobuf compatibility behavior by allowing a type change on a `oneof` field is now allowed when that field's `oneof` block is renamed or dissolved. #### Release v809 June 18, 2026 * Orbit now creates empty source topics with a non-zero log end offset at that offset in WarpStream. This fixes a bug where an empty source topic could be migrated to WarpStream at offset 0 instead of preserving its source log end offset. * Add a `cli-beta validate-pipeline-config` command for validating managed data pipeline config files locally. Usage: `warpstream cli-beta validate-pipeline-config --config-file pipeline.yaml`. * Restrict which clients see support for KIP-714 client metrics APIs. The GetTelemetrySubscriptions and PushTelemetry endpoints will no longer be advertised to clients older than this version. #### Release v808 June 18, 2026 * Tableflow: * Add support for sorting and column statistics tracking for Iceberg tables. * Fix time-based partition transform on an Avro date field. * Update the publish metrics job implementation to scrape metrics from the control plane Prometheus endpoint. This should not change any of the existing metric, but will add a few that were only available in the control plane Prometheus endpoint. #### Release v807 June 12, 2026 * Orbit Auto Migration is now available. #### Release v806 June 12, 2026 * Forward produce backpressure metrics for observability. * Add support for OAuth as an authentication mechanism for source clusters in Orbit. * Tableflow: Validate properly the required fields nested under untyped maps via `additionalProperties` for JSON ingestion. * Bump agent base image (`cgr.dev/chainguard/wolfi-base`) to pick up busybox fix for `CVE-2023-39810`. #### Release v805 June 10, 2026 * KIP-714 client metrics are now generally available to all customers. #### Release v804 June 9, 2026 * Add global sort duration in batcher to flush\_end log. * Fix a bug where the agent could have problems listing ripcord files if there were too many of them and your bucket URL was long, because the generated output was too big. * Fix Orbit and Tableflow source connections silently dropping SASL when TLS is enabled without mTLS. #### Release v803 June 8, 2026 **Warning**: This version drops SASL when connecting to an Orbit or Tableflow source cluster over TLS without mTLS. It is fixed in v804. Please upgrade to v804 directly and skip v803. * Fix panic when emitting the client availability zone mismatch diagnostic during interzone load balancing. * Upgrade Bento to v1.18.1. * Fix Orbit and Tableflow source cluster TLS configuration to allow setting a private CA (`mtls_server_ca_cert_env`) without requiring mTLS. * Schema Registry: the `error_code` for incompatible schema responses now returns `40901` (was `409`) to align with Confluent's documented API. The HTTP status code remains `409`. * Schema Registry: Avro schemas with invalid field defaults are now rejected at registration time (aligns with Confluent SR 8.2.1 behavior). Existing schemas registered before this change are unaffected. * Add batching for JoinGroup and SyncGroup requests via RSM batch commands, reducing the number of RSM proposals when many consumers join/sync concurrently. #### Release v802 June 4, 2026 * Bump build to go `1.26.4` to fix `CVE-2026-42504` * Add an agent flag to override max inflight fetch compressed bytes per CPU. * Upgrade Bento to v1.18.0 * Tableflow: * Fix bug for `dlq_keep_settings.retention`: we now allow for `d` and `w` units for days and weeks respectively. * Fix bug that considered null values set within transforms as non-null when fields were marked as required. * Tableflow agents can now create Iceberg table metadata even when they cannot access external blob store buckets directly. This can be enabled by setting the `tableflowMetadataSyncMode` flag to `proxy`. * Support converting between booleans and strings, and between integers and strings, in both directions when mapping records to the destination schema. * **Breaking Change (AWS Glue):** Fix a bug in the AWS Glue integration that caused us to hit the TableVersions limit when syncing the table. This fixes the error `Number of TABLE_VERSION resources exceeds the limit 100000 per TABLE`. Note that agents will now require additional roles: `glue:GetTableVersions` and `glue:BatchDeleteTableVersion`. * Fix some tags on the event emitted when sending records to the DLQ. #### Release v801 May 29, 2026 * Deprecate a legacy code path for computing Data Lake sorting bounds and column statistics. * Fix events scheduler writing a corrupt active state when an event type was disabled while events were globally enabled. Agents now drop records with stream ID 0 instead of poisoning the file batch. * Tableflow: Implement basic validation for required / nullable fields in JSON. #### Release v800 May 27, 2026 * Tableflow: * Fix bug on transforms: Bento transforms replacing the full root object instead of modifying it field by field are now handled correctly (e.g. `root = { ...}` now works). * Fix `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` reporting full retention as the lag for partitions with no committed offset. The metric now uses the age of the oldest surviving record (bounded by retention). #### Release v799 May 27, 2026 * Fix events queries failing with `UNSUPPORTED_SASL_MECHANISM` on clusters that restrict `enabledSASLMechanisms` to exclude `PLAIN`. #### Release v798 May 26, 2026 * Fix a query engine bug that could cause Agent panics for some sorted and limited queries. * Fix false positive diagnostic for cross az traffic when the query engine was used. * Fix false positive diagnostic for small fetch timeout when the query engine was used. * Return clear error when schema registry request body size exceeds limit. #### Release v797 May 19, 2026 * Tableflow: * Fix decoding of Protobuf `oneof` fields (introduced in v796, upgrade directly to v797 if you use a configuration with Protobuf schemas). * Recreate BigLake tables when the Iceberg table UUID changes. * Fix DNS-backed Agent file cache retries so failed replicas can fall back to other resolved Agents. * Add per-pipeline log level controls for managed data pipelines. * Fix event queries on agents with SASL enabled. #### Release v796 May 18, 2026 **Warning**: There is a problem for Tableflow protobuf schemas with that version fixed in v797. Please upgrade to v797 directly and skip v796 if you use Tableflow with a protobuf schema. * Upgrade the embedded version of Bento to v1.17.0. * Start supporting regional migrations. * Add a rate limiter in the metrics mode to throttle metrics emission. This avoids flooding the Datadog agent when Datadog metrics are enabled. * Adds an experimental agent flag that can be used to reduce E2E latency for workloads that do not produce new records consistently on all partitions all the time. * Tableflow: * Breaking change for very old agent versions using protobuf, they would need to upgrade to version 796+. * Add support for field type remapping #### Release v795 May 14, 2026 * Forward schema registry metrics for observability. * Tableflow: Fix the Tableflow configuration used in the demo mode. * Reduced telemetry sent from Agents to control plane. #### Release v794 May 12, 2026 * Add a diagnostic for invalid regex subscriptions in modern consumer group heartbeat requests. * Bump `golang.org/x/net` to `v0.54.0` to fix `CVE-2026-33814` * Detect when Kafka clients request one availability zone via `ws_az` but produce or fetch against an Agent in a different availability zone and fire a diagnostic. #### Release v793 May 11, 2026 * Unless customers specify a `-metadataURL`/`WARPSTREAM_METADATA_URL` the agents will start using 3 different "zonal" endpoints to communicate with WarpStream - meaning one per WarpStream control plane availability zone. The previous behavior can be put back by setting the environment variable `WARPSTREAM_AGENT_ENABLE_ZONAL_URLS` to false. Removes a dependency on inconsistent behavior across cloud providers when load balancing across multiple availability zones and increases the product's resiliency to zonal outages significantly. * Bump build to go `1.26.3` * Fixed CVEs: CVE-2026-39820 / CVE-2026-33811 / CVE-2026-42499 / CVE-2026-39820 / CVE-2026-33811 / CVE-2026-39836 * Agent will now enforce read-only Schema Registry credentials and only allow read schema operations for read-only credentials. #### Release v792 May 8, 2026 * Add optional PROXY protocol v2 support on the Kafka listener via `kafkaProxyProtocol` (or `WARPSTREAM_KAFKA_PROXY_PROTOCOL`). When enabled, every connection must include a v2 PROXY header before any other bytes; v1 headers are rejected. Operators can also set `kafkaProxyProtocolPrincipalTLVType` (a hex byte in the PP2 user-defined range `0xE0`–`0xEF`) to source the connection's ACL principal from a custom TLV in the PROXY header — the TLV value must already include the `User:` prefix and overrides any principal that would otherwise be derived from mTLS. * Tableflow: * Fix a bug in ingestion that was not respecting some types from the input schema. * Fix ingestion of date types for Avro input schemas. * Fix ingestion of Avro map types with optional values when the deprecated way of specifying schemas is used. * Fix several correctness bugs in the Protobuf decoder so records now align with the proto3 spec (`oneof` last-wins, merging of repeated singular message fields, strict wire-type validation). #### Release v791 May 6, 2026 * Fix a bug in the agent metrics mode that would cause it to stall when any of the metrics scraped contained a comma in its labels. * Mark file cache closed-pipe logs as debug. #### Release v790 May 5, 2026 * Tableflow: Make ingestion handle partition evolution. * Bump our Chainguard Wolfi Docker image to `79af0917ba7ac066ebf1f99e5967ee9e77e6a350facec5be43a7a236c705cf10` to fix `CVE-2026-5450` and `CVE-2026-5928`. * Fix a bug in the query engine when querying across events that mix integers and floats in the same field name. * Fix a bug in the query when combining multiple filters like: filter a=="b" | filter c=="d" where only the latter filter would be applied. #### Release v789 April 30, 2026 * Fix a bug where object store errors due to networking issues (DNS or TCP timeouts, context cancellation, etc) were misclassified as NotFound errors, causing red-herring invariant violations. * Tableflow: Fix ingestion of list types with optional elements. * Public agent docker images and release tarballs now ship `notices.txt` (auto-generated from the binary's transitive Go dependencies) and `non_ibm_license.txt` at the container/archive root, for IBM third-party legal compliance. * Fix the reason for a decision made during compaction in a log from being wrong. * Treat "unexpected EOF" as a retriable error for long-running compactions. #### Release v788 April 29, 2026 * Bump `github.com/aws/smithy-go` to `v1.25.1`. This fixes a very slow memory leak associated with using S3 Express One Zone in the AWS SDK. #### Release v787 April 28, 2026 * Query engine: disable the admin RPC metadata cache because it causes a bug with inconsistent results when querying events. * Query engine: fix `::int`, `::float`, and `::str` casts in event queries, and make integer/float comparisons work in filters. * Agents will now automatically backpressure Kafka protocol requests when their CPU is >= 98% for a sustained period of time. This threshold can be adjusted by setting the `WARPSTREAM_HIGH_CPU_BACKPRESSURE_THRESHOLD_PERCENT` environment variable and disabled by setting it to 0. * All Datadog metrics now include `virtual_cluster_id`, `agent_id`, and `agent_version` tags. #### Release v786 April 28, 2026 * Emit cluster-level Prometheus-style metrics (consumer group lag, topic details, diagnostics, tableflow state, ...) as `cluster_metrics` CloudEvents whenever events are enabled on the cluster. Emission is independent of the Datadog push: events are still emitted even when `DisableAllMetrics` or `DisableConsumerGroupMetrics` is set. Events are always at topic-level granularity (partition tag is always stripped) to keep the per-job event volume bounded. The event's `metric_name` field carries the canonical `warpstream_`-prefixed name (e.g. `warpstream_consumer_group_lag`), identical to the Prometheus scrape name and the `describe_cluster_metrics` API response, so the same string identifies a metric regardless of how it is queried. * Tableflow: improve DLQ health reporting so records routed to the DLQ are no longer surfaced as skipped, reducing misleading health warnings and making it clearer when data was retained for later replay. * Improve query engine performance by caching metadata lookups and aligning query time boundaries. * Move modern consumer group heartbeat regex matching and topic `DESCRIBE` ACL filtering from saasy into the agent, and forward the resolved regex stream IDs to saasy for statemachine application. * Allow reserved keywords like topic to parse as part of .-deleted field identifiers in the query engine. #### Release v785 April 23, 2026 * Emit cluster-level Prometheus-style metrics (consumer group lag, topic details, diagnostics, tableflow state, ...) as `cluster_metrics` CloudEvents whenever events are enabled on the cluster. Emission is independent of the Datadog push: events are still emitted even when `DisableAllMetrics` or `DisableConsumerGroupMetrics` is set. Events are always at topic-level granularity (partition tag is always stripped) to keep the per-job event volume bounded. * Make metrics pod use external HTTP client to have longer connect timeouts. * Make metrics pod re-use HTTP client across requests to reduce time spent on establishing new connections. * Change `consoleURL` parameter name to `apiURL` . * Fix a bug that would cause the original error to be dropped from error messages and logs when hedging requests with the fast retrier. * Fix bug in query parser for handling sub expressions. * Make tableflow events all have the table name and table UUID fields set properly. * Fix panic in query engine related to nested expression and or statement. * Emit a tableflow event when the offset scraper job handler fails. * Enable the agent query engine by default and require `disableQueryEngine` to opt out of query handling. * Removed the `OBJECT_STORAGE_PUBLIC_NETWORK_PATH` diagnostic (added in v765). * Added a diagnostic that fires when a produce request contains records for both lightning and classic topics, which removes the latency benefit of the lightning topic. #### Release v784 April 20, 2026 * Fix a memory leak when using the `metrics` agent mode. * Bump `github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream` to `v1.7.8`, `github.com/aws/aws-sdk-go-v2/service/kinesis` to `v1.43.5`, `github.com/aws/aws-sdk-go-v2/service/lambda` to `v1.89.0` and `github.com/aws/aws-sdk-go-v2/service/s3` to `v1.99.0` to fix `GHSA-xmrv-pmrh-hhx2`. * WarpStream Agents will no longer report the availability zone that they're running in as the value of the Rack field in Broker metadata in the response of Metadata RPCs. Instead, they will always report their Rack as: "warpstream-fake-rack". The reason for this is that WarpStream's service discovery system tracks availability zones of clients using client ID features, not this Rack field. In addition, when clients enable rack-awareness for consumers in their Kafka clients without properly configuring WarpStream's zone-aware service discovery system, the presence of the availability zone in the Rack field in the Broker's metadata, combined with the fact that it will change from time to time due to how WarpStream's partition assignment strategies work, may result in an excessive number of consumer group rebalances. As a result, since this field has no value in WarpStream clusters, we're hard-coding the value of Rack so that misconfigured clients will not experience excessive consumer group rebalances. * Tableflow: * Fix ingestion of Avro fixed and time types. * Performance improvement when sorting is enabled on tables. #### Release v783 April 16, 2026 * Tableflow: * Reject invalid combinations of `WARPSTREAM_BUCKET_URL`, `WARPSTREAM_INGESTION_BUCKET_URL` and `WARPSTREAM_COMPACTION_BUCKET_URL` with an explicit error message for datalake agents with events enabled. * Bump `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp` to `v1.43.0` for `CVE-2026-39882` vulnerability. * Fix connection reuse on the query engine. * Fix a bug that didn't allow users to manually disable events for tableflow clusters. #### Release v782 April 14, 2026 * Tableflow: add `warpstream.tableflow_dlq_records_counter` agent metric tagged by `topic` and `strategy` (`skip` or `keep`) to track records handled by the DLQ during ingestion. Replaces the previous `warpstream.ingestion_job_dlq_skip_strategy_counter` which only covered the skip strategy. * Improve query engine performance by fixing connection reuse. * Improve produce latency by parallelizing record sorting in the agent batcher hot path. * Fix Avro schema validation to handle references using short name within a namespace properly. * Bump `go.opentelemetry.io/otel/sdk` to `v1.43.0` for `CVE-2026-39883` vulnerability. * Improve how runtime CPU measurements are taken to be more precise and not persist inaccurate values for long period of time by occasion. * Improved agent feature versioning scheme to auto-compute latest released version from feature declarations, eliminating manual maintenance and potential for version bump errors * Query engine: fix distributed aggregation for aliased `count()` and `count(expr)` so merge/final stages consume the correct intermediate count columns. * CLI `alter-broker-config` now surfaces per-broker error responses instead of silently reporting success. * Support setting `warpstream.default.topic.type` via AlterConfigs and IncrementalAlterConfigs. * Unknown warpstream-prefixed broker and topic configs now return an error instead of being silently ignored. #### Release v781 April 9, 2026 * **Breaking Change (GCP):** WarpStream Agents now depend on `storage.buckets.get` access for all configured GCP (gcs\://) bucket URLs. This can be overriden by setting the environment variable `WARPSTREAM_GCS_PERFORM_BUCKET_STORAGE_CLASS_LOOKUP=false` . * Allow configuring the `tracer` block in managed data pipeline configs. * Automatically disable idempotent writes for managed data pipelines writing to WarpStream clusters via `kafka_franz_warpstream` output blocks. * Fix non-determinism in generating schema hash for unrecognized key in Avro schemas. * Bump build to go `1.26.2` * Add support for GCP Rapid Buckets. * Optimize metrics code, particularly when emitting Datadog metrics. * Fix the docker image base to properly use the multi-arch wolfi-base image #### Release v780 April 8, 2026 * Reduce max diagnostic groups per type from 10 to 5 to lower payload size sent to the control plane. * Remove spammy error log when diagnostic groups exceed the per-type limit. * Forward sampled flush\_event logs for observability. * Emit forwarded flush synchronous duration metric. * Bump our Chainguard Wolfi Docker image to `79af0917ba7ac066ebf1f99e5967ee9e77e6a350facec5be43a7a236c705cf10` to fix `CVE-2026-4437`. * Bump `github.com/go-jose/go-jose/v4` to `v4.1.4` to fix `GHSA-78h2-9frx-2jm8`. * Prevent a single fetch request from triggering thousands of concurrent HTTP requests if it's querying data for many topic-partitions whose individual data is spread across many files. * Install CURL in Agent image so it can be used in health checks for services like ECS that rely on container having command line utilities available. * Tableflow: tableflow events are generally available and agent-side events handling is enabled by default. This is a **breaking change** since tableflow agents now require a bucket URL to be configured (either `WARPSTREAM_BUCKET_URL` or both `WARPSTREAM_INGESTION_BUCKET_URL` and `WARPSTREAM_COMPACTION_BUCKET_URL`). Agents without bucket URLs will fail to start. * Enable events when bootstrapping demo and playground clusters. * Added SLOW\_CONSUMER push diagnostic: detects consumers that are too slow between fetch requests using a dynamic threshold (baseline + 1s per MB of previous response size) and emits the diagnostic after 3 consecutive slow fetches on a connection. * Fix GOMEMLIMIT parsing to support byte suffixes like "24GiB" and "1000MB" instead of requiring plain integers. * Fix GOGC, GOMAXPROCS, and GOMEMLIMIT heartbeat reporting to not report a value on parse error. Previously a parse error would silently report 0. #### Release v779 April 1, 2026 * Fix bug that prevented query engine from running queries successfully on Agents configured to only run the Jobs role. #### Release v778 April 1, 2026 * Add principal impersonation via record headers. Configured impersonator principals can produce messages on behalf of other users by setting a designated record header, enabling ACL evaluation as the impersonated principal. * Fix a bug that caused events queries to fail on agents with mTLS enabled. * Enable dual-mode Prometheus native histograms alongside classic histograms for improved metric accuracy. * Query engine: route query jobs only to agents with the query engine enabled and fail fast if an agent is configured to accept only query jobs without enabling the query engine. * Replace mutex-protected `rand.Rand` instances with lock-free top-level `rand` functions in certain hot paths. * Fix a bug that was preventing events from being emitted/enabled in public Agent builds. * Enable agent-side events handling and query engine automatically for Kafka agents. * Expose a `/debug/pprof/trace` endpoint for Go runtime traces * Tableflow: fix a bug where compaction failed on certain parquet file layouts with gaps between column chunks, leaving small files unmerged. * Tableflow: add `type: "record_ingestion_failed"` to logs corresponding to decoding errors during ingestion. * Tableflow: add `warpstream.tableflow_ingestion_lag_seconds` and `warpstream.tableflow_query_lag_seconds` metrics. Ingestion lag measures how far behind the ingestion process is from the source Kafka topic. Query lag measures the time from when data is produced to Kafka until it is queryable (ingestion lag + catalog sync delay). #### Release v777 March 26, 2026 * Fix schema validation in playground by passing missing schema registry URL to config. #### Release v776 March 26, 2026 * Tableflow: * Add `warpstream.tableflow_partition_offset_lag` gauge for Tableflow ingestion offset lag in records. When topic or partition tags are omitted, offset lag is summed across partitions (like `consumer_group_lag`); time lag remains the max per scope. #### Release v775 March 25, 2026 * Add topic, partition, and streamID to orbit offset error messages for easier debugging. #### Release v774 March 25, 2026 * Cache metrics in fetch and produce Kafka handlers to reduce tagset allocations when updating metrics. * Tableflow: * Fix panic in demo mode by adding missing `input_schema` to demo config. * Add topic tag to `agent_kafka_produce_with_offset_uncompressed_bytes_counter` metric when high cardinality metrics are enabled. #### Release v773 March 24, 2026 * Don't swallow circuit breaker errors and make it clear in error logs that are showing cached circuit breaker errors what is happening. * For instance consumers could start failing when some Kafka topics are deleted but are still trying to be accessed, because the metadata response would be incorrect * Don't count structured errors from the control plane (400s) as failures in the control plane client circuit breakers. #### Release v772 March 23, 2026 * Fixed IPv6 address parsing. * Tableflow: * Make the `TABLEFLOW_` prefix optional for environment variables used to specify Kafka cluster credentials. * Fix BigQuery integration for Tableflow tables failing to update when the Iceberg schema evolves (e.g., new columns added). * Add support for ConsumerGroupDescribe API (key 69, KIP-848) for the modern consumer group protocol. Disabled by default. * Make the `ORBIT` prefix optional for environment variables used to specify Kafka cluster credentials. * Fix the `empty_response` tag value for the `warpstream.agent_kafka_fetch_single_attempt_outcome` metric. * Bump `google.golang.org/grpc` to `v1.79.3` for `CVE-2026-33186` vulnerability. * Bump our Chainguard Wolfi Docker image to `ce84795834de56c47f27b6be64388628677d18e799e9b5a05fe6538aaa17bc79` to fix `CVE-2026-2673`. * Upgrade Bento to `v1.16.1` to get Datadog and Bigtable outputs support. * Query engine: stop clamping query tail offsets from cached stream metadata so Event Explorer queries do not intermittently miss recent time buckets. * Add Datadog-compatible HTTP log intake endpoints on Agents so Datadog log batches can be routed into WarpStream topics. * Add a zstd produce fallback decoder path so Agents can accept some batches that fail `github.com/DataDog/zstd` with `unexpected EOF` but succeed with `klauspost/compress/zstd`. #### Release v771 March 19, 2026 * Fix the metrics prefix pushed by the agent in `metrics` mode: in previous versions we would add back a "warpstream\_" prefix to all metrics, which was redundant. #### Release v770 March 18, 2026 * Fix: re-include `nc` in the warpstream-agent image so that healthchecks can be performed properly on ECS. #### Release v769 March 18, 2026 * Tableflow: * Add BigLake Metastore integration via Iceberg REST Catalog API for GCP data lake tables. * Add Hive Metadata Store integration #### Release v768 March 12, 2026 * Disable GCS direct connectivity client by default as it is causing a regression in some GCP regions. #### Release v767 March 11, 2026 * Fix `tls_insecure_skip_verify` not being applied for TLS+SASL connections without mTLS in Tableflow and Orbit source clusters. * Prevent Agents from trying to issue file cache requests against Agents that no longer exist when fetching old file extents by enforcing a maximum cache staleness in the activation cache. #### Release v766 March 11, 2026 * Bump build to go `1.26.1` * Tableflow: * Enhance logging when metadata upload fails. * Add a `module` attribute to job logs to group them logically. * Fix the command to query the Tableflow table logged by the demo agent. * Fail agent startup if the ripcord mode is used for an agent connecting to a Tableflow or a Schema Registry cluster. * Add observability by forwarding agent produce error outcomes. * Switch agent Docker base images from Alpine to Chainguard Wolfi * Misc: New way of releasing static agent features, this should be a transparent change. * Add `create-acls`, `describe-acls`, and `delete-acls` clibeta commands for managing ACLs from the CLI, providing a `kafka-acls.sh` equivalent. #### Release v765 March 5, 2026 * Tableflow: Add retries to the bucket access checker and remove extra slash used in the request URL. * Add `agent_version` label/tag to all metrics emitted by the agents. * Bump `github.com/docker/cli` to `v29.2.1` for `CVE-2025-15558` vulnerability. * Add L7 proxy detection diagnostic for inter-agent traffic. When a load balancer or reverse proxy is detected between agents, a new high-severity diagnostic is surfaced to help users identify and remove proxy interference. * Added new `OBJECT_STORAGE_PUBLIC_NETWORK_PATH` diagnostic that detects when S3 or Azure Blob Storage endpoints resolve to public IPs, indicating a missing VPC endpoint (or equivalent). This helps identify unnecessary networking costs from object storage traffic going over the public internet. * Fix a bug where some log attributes were grouped into a `!BADKEY` key. #### Release v764 March 3, 2026 * Tableflow: make Protobuf type `google.protobuf.Timestamp` usable as a custom partitioning field. #### Release v763 February 27, 2026 * Increase the maximum allowed value for `batchMaxSizeBytes` (max uncompressed batch flush size) from 64MiB to 128MiB. * Improve Schema Registry to properly handle trailing slashes in URL paths (e.g., `/config/` now works the same as `/config`). #### Release v762 \[do not use] {% hint style="info" %} This version has a known bug that causes some records to be incorrectly skipped by consumers. Do not use it. {% endhint %} February 26, 2026 * Make sure BYOC schema registry response's Content-Type is application/json if the request's Accept header is application/json. * Fixed a bug where the demo agent would get stuck waiting for the Tableflow Iceberg table to be created. * Optimizes fetch to use less CPU and IO when fetching a single record, or when `max.partition.fetch.bytes` is low. #### Release v761 February 25, 2026 * Add PSI (Pressure Stall Information) metrics collection for CPU, memory, and I/O pressure. New metrics: `pressure_cpu_some_pct`, `pressure_cpu_full_pct`, `pressure_memory_some_pct`, `pressure_memory_full_pct`, `pressure_io_some_pct`, `pressure_io_full_pct`. These metrics report the percentage of time tasks were stalled due to resource contention over the last sampling interval (0-100%). * Update orbit to cover for protocol incompatibility in some systems which implement kafka api #### Release v760 February 24, 2026 * Tableflow: Performance improvement on ingestion saving 5 to 10% of CPU usage. * Add `metrics` category to `--jobSelector` flag, allowing agents to run only metrics publishing jobs (e.g., `--jobSelector metrics`). #### Release v759 February 24, 2026 * Add `--jobSelector` flag (env: `WARPSTREAM_JOB_SELECTOR`) to specify which job types an agent handles. #### Release v758 February 20, 2026 * Tableflow: * Fix ingestion of Avro and JSON when lists or map values contain decimal fields. * Make Proto unsigned integers (`uint32`, `uint64`, `fixed32`, `fixed64`) convert into decimals. Before that, making a Protobuf backward-compatible change that changed a `uint32` into a `uint64` (or vice-versa) was forbidden because it would end in a non-backward-compatible Iceberg change (from `long` to `string` or vice-versa). Now it's possible. This is a **breaking** change for Protobuf tables as the Iceberg column type changes for those uints. * Add support for `google.protobuf.Timestamp` type in Protobuf. It gets converted to an Iceberg `timestamptz`. * Fix time\_since\_last\_top\_level\_fetch\_ms to avoid negative values when there are concurrent fetch requests. * Improve the "Cross AZ Kafka Clients" diagnostic to show more than 1 clientId. * Fix a bug in the instance type detection logic for Kubernetes environments that could cause it to fail to detect the instance type in some cases and return "unknown" instead. * Add log message for mTLS authentication that includes certificate serial number and validity period in order to help monitor client certificate expirations. * This functionality is disabled by default and can be controlled with the `-logMTLSCertificateInfo` flag or `WARPSTREAM_LOG_MTLS_CERTIFICATE_INFO` environment variable. #### Release v757 February 19, 2026 * Added a new agent flag `-injectPrincipalHeader` (env var `WARPSTREAM_INJECT_PRINCIPAL_HEADER`) that, when set to a header key name, injects the kafka principal username as a header in each produced record. This allows downstream consumers to identify which user produced each record. * Fixes a bug in the ripcord deletion queue that could cause the "last opened sequence" to grow until the agent is restarted, because the agent silently discards files that should be deleted in some circumstances. * Query engine: Fix a bug that caused fields escaped with backticks to not be queryable properly. * Tableflow: * Emit the correct logical type annotation for time fields in Parquet files. * Fix ingestion of Avro records with decimal fields. #### Release v756 February 17, 2026 * Reject all traffic to HTTP fetch endpoints if mTLS is enabled on the Agents. * Validate SASL credentials in HTTP fetch endpoints even if SASL authentication is not required by the Agents. * Tableflow: fix regression requiring the IngestionBucketURL to be set for datalake clusters (new unreleased feature). #### Release v755 February 12, 2026 * Add new /v1/kafka/fetch HTTP/JSON endpoint for manually issuing Fetch requests against the Agents. * Add convenience endpoint `GET /v1/kafka/fetch_single_record?topic=TOPIC&partition=PARTITION&offset=OFFSET` for fetching a single record by offset. * Add convenience endpoint `GET /v1/kafka/topics/{topic}/partitions/{partition}/records/{offset}` as a REST-style alternative for fetching a single record by offset. * Due to false positives, skip the Unregistered WarpStream Environment Variables diagnostic if the agents is deployed as a Kubernetes service with a name that begins with "warpstream\_" or "warpstream-". #### Release v754 February 12, 2026 * Fixed a bug introduced in v746 where the Agents could get deadlocked after experiencing tends of thousands of errors. #### Release v753 February 11, 2026 * \[experimental] Add `-disableOutOfRangeOrbit` flag (env: `WARPSTREAM_DISABLE_OUT_OF_RANGE_ORBIT`): when enabled, orbit-managed topics return `KAFKA_STORAGE_ERROR` instead of `OFFSET_OUT_OF_RANGE` in the fetch path, preventing clients from resetting offsets when data has not yet been replicated from the source cluster. #### Release v752 February 11, 2026 * Redact agent key from startup logs. * Tableflow: Fix rare bug in parquet footer length tracking that could cause compaction jobs to fail with "unsupported thrift type" errors when reading input files. #### Release v751 February 9, 2026 * Fix default value for `-batchMaxCompressedSizeBytes` to be 16MB instead of a value higher than the max. #### Release v750 February 9, 2026 **Warning**: There is a problem with that version fixed in v751. Please upgrade to v751 directly and skip v750. * **Flushing behaviour update** * Before v750, the agent would flush a new file if it was over your configured `-batchMaxSizeBytes` value, and its default value was 4MiB. This means that the agent would not create files that contain more than 4i of uncompressed data by default. * After this release, by default, the Agent will create a new file if it estimates that it's going to be more than 1MiB after compression by default. * In more details: * This version introduces a `-batchMaxCompressedSizeBytes` flag (and the corresponding WARPSTREAM\_BATCH\_MAX\_COMPRESSED\_SIZE\_BYTES environment variable). * If you define just `-batchMaxSizeBytes`, then `-batchMaxCompressedSizeBytes` will not be used, the agent will use exactly your uncompressed bytes limit to choose when to create a new file. * If you define just `-batchMaxCompressedSizeBytes`, then `-batchMaxSizeBytes` will be automatically set to a very high value (64MiB) * If you define no flags, then `-batchMaxCompressedSizeBytes` will be set to 1MiB and `-batchMaxSizeBytes` will be set to 64MiB. * If you define both, a new file will be created when it becomes higher than either limit. * SyncGroup handler now returns validation errors (invalid group ID, missing member ID, etc.) embedded in the response instead of as a top-level error. This improves compatibility with Kafka clients and ensures these expected protocol errors are not logged/metriced as server errors. * Tableflow: * Fix BigQuery integration bug, where orphaned tables were preventing table recreation. * Update ingestion to handle empty topics gracefully * Fix timestamp field indexing for JSON format: timezone is not required anymore for fields with type timestamp. * Fix bug in batcher which would increase the number of files flushed when disable\_only\_flush\_after\_timeout is set to true. * Include virtual cluster ID in auto-generated client IDs for Managed Data Pipelines. This ensures it works with `VirtualClusterIDClientStrictValidation` enabled. * Improve performance of metrics code by swapping a mutex for an RWMutex. * Bump franz-go package from v1.18.1 to v1.20.6. * Add low severity diagnostic to detect misspelled environment variables beginning with `WARPSTREAM_`. * Fix noisy ACL denial logs for "all topics" metadata requests. When clients refresh metadata for all topics, unauthorized topics are now silently filtered without logging denials (per Kafka protocol behavior). #### Release v749 February 2, 2026 * Tableflow: Support `skip_raw_record_values` that doesn't store the raw record values in the produced data files. * Bump build to go `1.25.6` to fix vulnerability `CVE-2025-61726`. #### Release v748 January 30, 2026 * Return an error when MTLS is enabled (without `-spiffeMTLSAuthentication`) but the agent is unable to determine the username from the client certificate subject. * Add a new push diagnostic that fires when the timeout on produce requests is less than 5 seconds. This helps identify clients with misconfigured timeout settings that may cause issues with WarpStream's object storage-based architecture. * Allow clients to reauthenticate via SASL even when the control plane is down for Ripcord mode. * Tableflow: * Add support for custom partitioning. * Add support for compressing data files, default being snappy. * New metrics: * `warpstream.tableflow_partitions_count`: Total number of table partitions in the cluster. * `warpstream.tableflow_partitions_limit`: Maximum number of table partitions allowed in the cluster. #### Release v747 January 29, 2026 * Buffer audit logs agent side in case of failure to publish them. * Patch CVE-2025-15467 & CVE-2025-69419 vulnerabilities in libssl3/libcrypto3. * Tableflow: * Fix decoding of JSON records with date fields serialized in the "YYYY-MM-DD" format. * Update how we track source data timestamp metadata (kafka timestamps) for more efficient retention enforcement. #### Release v746 January 27, 2026 * Flushes will wait for the entire batch timeout even if the batch max size bytes limit is exceeded. Files will be split and flushed in parallel in accordance to the batch max size bytes limit. #### Release v745 January 23, 2026 * Add support for SASL `OAUTHBEARER` authentication mechanism in the WarpStream Agent for Kafka clients that support it. * New flags and environment variables: * `-saslOAuthIssuerURL` / `WARPSTREAM_SASL_OAUTH_ISSUER_URL`: The OAuth issuer URL used to validate tokens presented by clients. * `-saslOAuthAudience` / `WARPSTREAM_SASL_OAUTH_AUDIENCE`: The expected audience claim in the OAuth tokens presented by clients. * When these settings are configured, the Agent will accept OAuth tokens from clients and validate them against the specified issuer and audience. * ACL rules can be created with OAuth principals in the format `User:` to allow or deny access based on the subject claim in the token. * Tableflow: Add support for Protobuf schemas. #### Release v744 January 21, 2026 * Fix issue with lightning topics configuration validation in previous versions that would prevent the agents from starting on some machines (introduced in v737) * Tableflow: Add memory safeguard to prevent potential OOM when ingesting data spanning many partitions. #### Release v743 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} January 20, 2026 * Tableflow: Optimize ingestion by seeking to offsets that correspond to records to speed up the ingestion of topic with large offset gaps in the beginning. * Returns the "invalid topic" error rather than the "kafka storage error" when attempting to fetch from a topic name that does not exist. This is usually gated from the kafka client though. * The WarpStream agent can now create [WarpStream lightning topics](https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/lightning-topics). * The WarpStream agent can now be started in ripcord mode, with `-enableRipcord` or by setting the `WARPSTREAM_ENABLE_RIPCORD` environment variable to `true`. See [the documentation](https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/ripcord) for more details. #### Release v742 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} January 19, 2026 * Add additional attributes i.e. `client_id` and `username` to shadow ACL logs. * Enable SASL Authentication automatically for Managed Data Pipelines when ACL shadowing is enabled. * Put back tag `virtual_cluster_id` and `virtual_cluster_name` to `agent_kafka_request_outcome` metric that got removed by mistake in v740. * Fix bug in ACL Shadowing to stop emitting false-positive deny diagnostics. #### Release v741 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} January 15, 2026
* Update playground url to point to the `api.warpstream.com` instead of `console.warpstream.com`. * Please upgrade to this release for playground functionality to continue working. * Rename `-consoleURL` to `-apiURL` flags in playground mode. #### Release v740 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} January 14, 2026 * Downgrade gosnowflake dependency to remove glibc error related to cgo execution. * Fail to start tableflow agents if incorrect roles are provided / configured. * Add `warpstream_agent_acl_denied` metric to track ACL denials. * Treat more kafka server errors as `canceled` instead of `error` if the errors are indicative of the client disconnecting. #### Release v739 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} **Note:** This release contains a glibc/cgo error due to the Bento upgrade. Please use v740 instead. January 12, 2026 * Return COORDINATOR\_LOAD\_IN\_PROGRESS instead of KAFKA\_STORAGE\_ERROR as the generic error code for consumer group coordinator RPCs (JoinGroup, SyncGroup, Heartbeat, LeaveGroup, DescribeGroups, DeleteGroups, ListGroups, OffsetDelete, OffsetFetch) and TxnOffsetCommit. This improves compatibility with the Java consumer client which will automatically retry COORDINATOR\_LOAD\_IN\_PROGRESS errors for these RPCs, but not KAFKA\_STORAGE\_ERROR. * Tableflow: Fix BigQuery integration to handle table recreation. When a datalake table is recreated with a new UUID, the BigQuery external table is now automatically dropped and recreated to point to the new metadata location. Previously, updates would fail because BigQuery validates that both old and new metadata files exist. * Merge batcher related metrics into single metrics with a `name` tag: * `warpstream.xxx_batcher_batches_count` -> `warpstream.batcher_batches_count` * `warpstream.xxx_batcher_batches_distribution` -> `warpstream.batcher_batches_distribution` * `warpstream.xxx_batcher_called` -> `warpstream.batcher_called` * Fix bug in Fetch logic that could cause massive amounts of overfetching if clients (like librdkafka) sent fetch requests where the value of partition max bytes was >> the value of fetch max bytes for the entire request. * Reduce default value of `kafkaMaxFetchRequestBytesUncompressedOverride` and `kafkaMaxFetchPartitionBytesUncompressedOverride` from 1GiB to 256MiB. Now that the fetch code has pre-fetching logic, requesting huge amounts of data in a single fetch request is not nearly as useful so we should not allow it by default. * Upgrade Bento to v1.14.1 to fix a regression where functions with optional arguments were not being honored. #### Release v738 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} January 5, 2026 * Add `describe-broker-configs`, `describe-groups`, `describe-log-dirs`, and `describe-topic-configs` commands to the `warpstream agent clibeta` command for inspecting broker, consumer group, log directory, and topic configuration respectively. * Add `client_az` tag to metrics `warpstream_agent_kafka_fetch_compressed_bytes_counter` and `warpstream_agent_kafka_produce_compressed_bytes_counter`. The `client_az` tag prioritizes using the availability zone set in the client ID, otherwise it uses the resolved client ID from the Agent's subnet mapping. If neither is provided, then a value of `none` is set. * Bump `github.com/eclipse/paho.mqtt.golang` to fix `CVE-2025-10543` #### Release v737 {% hint style="warning" %} This version of the agent panics at startup on some machines with a high CPU count. Upgrade to v744 or higher if you have this problem. The exact error message is {% code overflow="wrap" %} ``` panic: error starting agent: error validating config: error validating ripcord config: error validating delete queue config: QueueSize must be less than or equal to 1M, it will use too much memory ``` {% endcode %} {% endhint %} December 22, 2025 * Ignore GCS context canceled error in circuit breakers to prevent context cancelation from accidentally opening circuit breakers inappropriately. * Tableflow: * Add BigQuery integration support for external Iceberg tables. Tables can now be automatically created and updated in BigQuery when new metadata is committed. * Add "stop" DLQ mode for Tableflow, which blocks ingestion of invalid records instead of skipping them. "skip" mode will remain the default for now, but a later release will make "stop" the default. #### Release v736 December 17, 2025 * Treat context.Canceled errors in ACL code as WARN instead of ERROR since it just indicates that the client disconnected, not that anything is wrong. * Add pipeline\_name tag to Managed Data Pipeline metrics * Emit logs for shadow ACL denials. #### Release v735 December 11, 2025 * Add support for spiffe URIs in mTLS authentication between kafka clients and ACLs * Add support for spiffe URIs in ACLs * ACLs can now be created with spiffe URIs as the principal * `User:spiffe://example.org/service` to allow a specific trust domain and workload ID. * `User:spiffe://example.org/*` to allow a specific trust domain and any workload ID. * spiffe super users can be created by creating cluster credentials with spiffe URI as the name. * Improve the logic for determining when to delay partition assignment changes for individual clients. * Fix ACL handling for `ANY` resource type. #### Release v734 December 9, 2025 * Add support for internal SPIFFE trust domain and workload ID for mTLS authentication between agents. * Trust domain and workload ID can be specified via new flags (`-internalSpiffeTrustDomain` and `-internalSpiffeWorkloadID`) and environment variables (`WARPSTREAM_INTERNAL_SPIFFE_TRUST_DOMAIN` and `WARPSTREAM_INTERNAL_SPIFFE_WORKLOAD_ID`). * If set, certificates without the specified trust domain and workload identity will be rejected. #### Release v733 December 8, 2025 * Tableflow: * Write always null count statistics, even if they are zero, in the parquet data statistics, to make it work with Query Engines that have strict requirements of reading those statistics. * Bump build to go `1.25.5` to fix vulnerability `CVE-2025-61729`. #### Release v732 December 3, 2025 * Improve logic for controlling delays between polling for new records in the Fetch code. This change dramatically reduces P99 E2E latency for some workloads. * Upgrade Bento version to `v1.13.1` in order to include fix on `parquet_encode` processor: it now allows for column names starting with an underscore. #### Release v731 November 28, 2025 * Adds a `-enabledSASLMechanisms` flag in the agent (and a corresponding `WARPSTREAM_ENABLED_SASL_MECHANISMS` environment variable). * If you provide it with a comma-separated list of SASL mechanisms, only those will be enabled. Valid values are `PLAIN` and `SCRAM-SHA-512`. * For example, if you set `WARPSTREAM_ENABLED_SASL_MECHANISMS=SCRAM-SHA-512`, you will not be able to use the `PLAIN` mechanism to connect, only `SCRAM-SHA-512`. * Fixes a bug preventing the agent from starting successfully on hosts with IP v6 only (assuming they start with `-advertiseHostnameStrategy=auto-ip6` or setting the env var `WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY=auto-ip6`) * Tableflow: * Fix AVRO record decoding for optional nested structs. This resolves validation errors like "missing required field" that could occur during ingestion when using AVRO schemas with optional nested records. #### Release v730 November 26, 2025 * Prevent partition assignments for each client from being able to change on every Metadata refresh. This dramatically improves load-balancing behavior when many clients have synchronized Metadata refresh intervals. * Tableflow: * Add support for arbitrary partition transforms during ingestion. * Make transforms work with Avro encoded data. * Validate 'required' fields are present for JSON records during ingestion. #### Release v729 November 24, 2025 * Tableflow: fix a bug in data cleanup where some data files that were compacted away or out of retention weren't being removed from the blob storage correctly. * Fix the agent\_roles metric tag to be deterministic. #### Release v728 November 21, 2025 * Perform ACL shadowing when ACLs are configured but disabled, and surface the result as a diagnostic. This helps users detect and fix invalid ACL rules even if ACLs are not currently enforced. * Use much more up to date Agent load information in the partition assignment strategies that use consistent hashing. This should dramatically improve load balancing for those strategies as previously the information that was used could be up to 1m stale and now it should never be more than a few seconds stale. * Add support for custom Bento transforms in Tableflow ingestions jobs. * Add support for dropping records as part of custom Bento transforms in Tableflow ingestion jobs. * Update `demo` command to demonstrate custom Bento tranforms for Tableflow. * Reduce log level from ERROR to WARN for failure to upload a profile. * Treat "connection timed out" errors when reading requests from Kafka client connections the same as "idle connection closed" from a logging perspective (reduces error log spam). * Reduce log level from ERROR to WARN when background prefetches fail. #### Release v727 November 19, 2025 * Re-enable watching for changes to the bucket URLs so that the Agents can refresh them if they change through overrides on the cluster settings page. * Return COORDINATOR\_LOAD\_IN\_PROGRESS instead of KAFKA\_STORAGE\_ERROR as the generic error code for the OffsetCommit RPC. This improves compatibility with the Java consumer client which will automatically retry COORDINATOR\_LOAD\_IN\_PROGRESS errors for that RPC, but not KAFKA\_STORAGE\_ERROR. #### Release v726 November 18, 2025 * Enable a stricter check when `warpstream_agent_group`/`ws_agent_group`/`ws_ag` is set in the client. If an Agent in group A receives a request from a client that indicated its intended target is Agents in group b, then the Agent in group A will reject the request with an error before closing the connection. This prevents issues that can occur where clients end up connected Agents in the wrong group due to IP reuse in high-churn environments like Kubernetes * Only consider internal errors in the `control_plane_errors` agent diagnostic. * Bump `github.com/dvsekhvalnov/jose2go` to `1.8.0` for `CVE-2025-63811` vulnerability. * Add gated support to ignore flush size in the batcher until flush timeout is hit. #### Release v725 November 14, 2025 * Enable efficient consumer group rebalances, by not sending member's metadata multiple times for a single JoinGroup request. * Add `-disableAzLookupWarnings` flag and `WARPSTREAM_DISABLE_AZ_LOOKUP_WARNINGS` environment variable to disable warnings when availability zone lookup fails via CIDR blocks. Additionally, these warnings are now throttled to a maximum of 1 log per minute to prevent log spam when clients intentionally connect from outside configured CIDR ranges while using the `WARPSTREAM_ZONED_CIDR_BLOCKS` configuration. * Attempt to use GCS direct connectivity (gRPC) automatically with GCS buckets by default, and fallback to standard HTTP when its unavailable. This significantly reduces the P99/max latencies of object storage operations in GCP. * Also added a new flag to fail the Agent on startup if they can't establish direct connectivity on GCP eligible datacenters: `gcsDirectConnectivityRequired` / `WARPSTREAM_GCS_DIRECT_CONNECTIVITY_REQUIRED` * Fix the `time_since_last_top_level_fetch_ms` attribute in `sample_fetch_statistics` debug log which is inaccurate when the fetch is performed during prefetching. Also adds `is_prefetch` to the `sample_fetch_statistics`. * Improve performance of metadata handler by switching to a faster api to list streams. * Unregister agent from service discovery during graceful shutdown. * Add diagnostic to detect when an agent does not shutdown cleanly. * Disable prefetching when the duration between individual consumer fetch requests is longer than the period of time that prefetched results will be held in memory before being GC'd if they're not consumed. This helps prevent the prefetching logic from putting additional load on the Agents when they're already overloaded. * Auto-tune the topic metadata cache refresh interval based on the number of topics in the cache so that workloads with a high number of Agents and topics do not generate excessive amounts of HTTP traffic. * Tableflow: Migrate compaction job to stop using a deprecated version of an internal API. #### Release v724 November 6, 2025 * Add support of a new `warpstream_cluster_id` client ID (see [documentation](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_cluster_id)). * Add a new `enableConfluentComponents` setting to enable the use of confluent cloud connectors with warpstream clusters. * Add new default concurrency limit of 16 concurrent Metadata requests/vCPU. This limit can be changed using the `-maxConcurrentMetadataRequestsPerCPU` flag or `WARPSTREAM_MAX_CONCURRENT_METADATA_REQUESTS_PER_CPU` environment variable. Setting the value to zero disables the concurrency limit. * Improve partial error handling in the batcher. * Bump build to go `1.25.3` to fix vulnerability `CVE-2025-58187`. * Configure maximum staleness in the AssumeRole cache so that role tokens are refreshed proactively in the critical path after long periods of idleness instead of just passively in the background after being read. This prevents spurious `api error BadRequest: Bad Request` errors from occurring during some infrequent background processes like file cleanup. * Bump github.com/opencontainers/runc to `1.3.3` for `CVE-2025-31133` and `CVE-2025-52565` vulnerabilities. #### Release v723 November 4, 2025 * Bump build to go `1.25.2` to fix vulnerability `CVE-2025-47912`. * Reduce CPU and allocations when processing Metadata requests for specific topics in clusters with a large number of topics/partitions. * Fix the metric `agent_kafka_produce_records_counter` to have the right tags when `EnableHighCardinalityMetrics` is enabled. #### Release v722 October 31, 2025 * Added a cache to AWS assume role credentials to prevent rate limiting in large workloads. * Tableflow: Improve ingestion telemetry reporting. * Schema registry: Add support for `Protobuf`. * Schema registry: Add support for schema normalization. #### Release v721 October 29, 2025 * Tableflow: Reduces agent memory usage during table compaction by using buffers pooling. * Fix a bug that would sometimes cause the Agents to reject Produce requests during shutdown with errors like "error getting allocated file ID: context deadline exceeded: 10.002999653s" that would result in latency spikes and error log spam. * Add a diagnostic to detect when the Agent file descriptor limit is less than the configured connection limit. * Add the ability to use the Bento `opensnowcat` processor in managed data pipelines. #### Release v720 October 27, 2025 * Bump Bento to v1.12.1 which does proper error handling for GCP BigQuery output. * Auto tune TCP buffer size based on observed number of connections. This makes it so well behaved workloads with low connection counts benefit from large buffers and minimal syscalls, but workloads with extremely high connection counts work too without requiring excessive amounts of memory. * Increase maximum allowed message size from 128 MiB to 256 MiB. * Change the log level of "successfully fetched credentials for assume role provider" from `info` to `analytics` * Split produce batch into per shard batches in the batcher. #### Release v719 October 23, 2025 * Tableflow: Fix ingestion of json records when the json key contains non-alphanumeric characters. * Reduce the amount of polling/networking done between the Agents and control plane for Agents running in `demo` and `playground` modes * Increase default maximum number of connections per vCPU from 8\_192 to 32\_768. * `WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES` can now support record size from 1 MiB to 128 MiB with default at 32 MiB. * Fix a bug in the loading cache that may return transient invalid results, resulting in errors such as "stream metadata cache returned not exist which should never happen". * Turn `current_buffer_stats` log into a debug log. #### Release v718 October 21, 2025 * Fix JSON Schema Validation to handle the `$defs` keyword for draft version `2020-12`. #### Release v717 October 16, 2025 * Tune number of allowed retries for blob store operations during compactions slightly. * Add a cache that makes partition assignment strategies that use consistent hashing much more performant when the volume of Metadata requests is extremely high. * Increase default number of inflight files for Producing from 16/vCPU to 64/vCPU. This makes it much less likely that the Agents will start backpressuring due to hitting the inflight files limit before hitting the inflight bytes limit which is much more important and accurate. * Increase the default ratelimit for max processed bytes per second/vCPU from 50MiB/s to 100MiB/s for Agents running only the `pipelines` role to promote more CPU usage and better auto-scaling. #### Release v716 October 14, 2025 * Reduce the amount of observability metadata we send back to the control plane on fetch and produce. * Changed default Tableflow HTTP port from 10001 to 8081 in demo/playground mode. * Upgrade the embedded version of bento to v1.11.0 * Tableflow: * Made Tableflow HTTP port in demo/playground mode overrideable using the `-tableflowInternalHTTPPort` flag. * Add a `TABLEFLOW_` prefix to the environment variables for passing cluster credentials. #### Release v715 October 13, 2025 * Accept JSON Schemas with draft versions 04, 06, and 07 for schema validation. #### Release v714 October 13, 2025 * Improve the performance of GCS client when direct connectivity is enabled by reducing the number of in-memory clients/connections that are in use at once. * Add new diagnostic for when a consumer gets an offset out of range error. * Bump our Alpine base image to version `3.22.2` to fix vulnerability `CVE-2025-9230` #### Release v713 October 10, 2025 * Bump our Alpine base image to version `3.22.1` to fix vulnerability `CVE-2025-9230` #### Release v712 October 8, 2025 * Fix spanner data plane bug. #### Release v711 (this release introduced a spanner data plane bug which has been fixed in 712) October 8, 2025 * Improve Spanner data plane performance. * Make `warpstream playground` command use less CPU when the Agent is idle (no traffic). * Improve the accuracy of the CPU utilization as measured by the Agents. #### Release v710 October 6, 2025 * Tableflow: Add AWS Glue Data Catalog integration to automatically register Iceberg tables and update metadata location. #### Release v709 (backwards incompatible metrics change) October 3, 2025 * Add support for Spanner and SQLite as a blob storage backends through `spanner://` and `sqlite://` URLs. * Tableflow: Fix indexing of uuid types. * Disable emitting high cardinality per-topic distribution/histogram metrics by default. See the new flag/environment variable added below to re-enable these metrics. Impacted metrics: * `warpstream_agent_kafka_produce_with_offset_uncompressed_bytes_bucket` * `warpstream_agent_kafka_produce_uncompressed_bytes_bucket` * `warpstream_agent_kafka_produce_compressed_bytes_bucket` * `warpstream_agent_kafka_produce_uncompressed_bytes_bucket` * `warpstream_agent_kafka_produce_compressed_bytes_bucket` * `warpstream_agent_kafka_fetch_uncompressed_bytes_bucket` * `warpstream_agent_kafka_fetch_compressed_bytes_bucket` * Add new `kafkaHighCardinalityDistributionMetrics` and `WARPSTREAM_KAFKA_HIGH_CARDINALITY_DISTRIBUTION_METRICS` environment variable that when set to true enables emitting high cardinality per-topic distribution metrics that are 10-20x higher cardinality than the regular counter metrics. Defaults to false. * Fix a bug in the pre-fetcher that would sometimes generate invalid/corrupt responses for fetch requests that requested data for a single-topic partition when the client retried a failed fetch request while the Agent was under extremely heavy load. The prefetcher is remotely disabled in previous versions, so this bug doesn't affect any version. * Return ErrNotLeaderOrFollower error instead of InvalidRequest error when processing a Produce or Fetch request on an Agent that is only running the proxy-consume or proxy-produce roles respectively. This should help some clients refresh their metadata and get routed to the right Agent more quickly when they're connected to the wrong Agent role due to I.P address reuse. * Add new environment variables `WARPSTREAM_GCS_GRPC_CONNECTION_POOL_SIZE` and `WARPSTREAM_GCS_ALLOW_DIRECT_CONNECTIVITY` to control new experimental GCS direct connectivity feature. #### Release v708 September 29, 2025 * Add subject `version` attribute to schema registry log request. #### Release v707 September 25, 2025 * Make bucket URLs overridable through the Cluster Settings page on the console, to swap out faulty data planes without having to restart the agents. #### Release v706 September 24, 2025 * Fix bug which prevented fetch size auto tuning when prefetching. * Tolerate leading/trailing quotes and whitespace in client IDs when parsing client ID features. * Add `WARPSTREAM_LOOKUP_AVAILABILITY_ZONE_ID` environment variable that looks up the availability zone ID instead of availability zone name when running Agents in AWS. This is useful for preventing inter-zone networking fees in WarpStream clusters where the Agents and clients are running in different AWS accounts. * Improved the diagnostic that detects when Agents are split into different proxy roles, but Kafka clients haven't been configured to target a specific proxy role. * When registering an incompatible schema in the schema registry, the error response now contains the list of violations explaining why it was rejected. * Tableflow: Add support for compacted and transactional topics. #### Release v705 September 19, 2025 * Add `-schemaRegistryEnableLogRequest` flag and `WARPSTREAM_SCHEMA_REGISTRY_ENABLE_LOG_REQUEST` environment variable that enables the schema registry to log every request it receives. #### Release v704 September 18, 2025 * Multi-region: Add support for the "Reduced Quorum Required Bucket" setting, which, on dual-bucket data planes running in a degraded (reduced quorum) mode, specifies which bucket *must* succeed when writing records to object storage. This improves the resiliency of dual-bucket data planes running in a degraded mode. * Tableflow: Fix a bug where certain records could be ingested twice when there is a large amount of uningested records. * Diagnostics metrics: update `warpstream.diagnostic_failure` labels * Replace single `severity` tag with boolean tags: `severity_low`, `severity_medium`, `severity_high`, `severity_critical`. * For successful diagnostics the severity\_\* tags are all `false`; for failing diagnostics exactly one matching severity\_\* tag is `true`. #### Release v703 September 17, 2025 * Add `-jobsRunnerSlotsAmplification` flag and `WARPSTREAM_JOBS_RUNNER_SLOTS_AMPLIFICATION` environment variable that can be used to increase the number of concurrent jobs that the Agents will run per vCPU. Defaults to `1.0`. * Treat "i/o timeout" as a resumable error during compactions. This prevents an entire compaction from failing due to a single killed connection. * Add reduced quorum support for dual-bucket data planes, so they can operate in a degraded mode during a regional incident. #### Release v702 September 16, 2025 * Bump Bento version to v1.10.1. * Emit a warning log if writing responses takes too long due to head of line blocking or slow TCP throughput. * Improved the logic in the job runner during Agent shutdowns so that Agents always notify the control plane that they weren't able to complete running a job before exiting so that the control plane can reschedule the job on another Agent without waiting for the full timeout. * Tableflow: Enhance Agent telemetry for investigating ingestion issues. * Add new `/debug/speedtest/download` endpoint to Agent HTTP server for measuring the maximum rate at which an Agent is able to transfer bytes over the network to clients. #### Release v701 September 12, 2025 * Make error message in diagnose-connection command more clear that the -tls flag may be missing. * Detect when TLS-related flags are set in CLI, but `-tls` or `-enable-tls` is not set and return an error. * Diagnostics metrics: standardize labels and binary values for easier querying * `diagnostic_name` is now lower\_snake\_case; `diagnostic_type` is lowercase * Deprecated: `warpstream.diagnostic_status` (kept temporarily for compatibility). New metric: `warpstream.diagnostic_failure` (gauge is 1 when failing and 0 when successful). The old metric will be removed in a few versions. #### Release v700 September 11, 2025 * Fix a bug where usages of the `prefix` query parameter in the bucket URL without a trailing `/` would be transformed incorrectly. * Fix a bug where the `WARPSTREAM_DISABLE_S3_CHECKSUMS` environment variable isn't respected for multi-part uploads. #### Release v699 September 10, 2025 * Fix internal bug which accidentally limits partition fetch concurrency. * Make `consistent_random_jump` the new default partition assignment strategy. You can set the environment variable `WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY` to `single_agent` to revert to the previous default behavior. * Tableflow: Improve timestamp conversion when indexing JSON messages and fix indexing of date types for Avro messages. #### Release v698 September 8, 2025 * Fail to start if a private IP address to advertise cannot be detected in auto-ip4 mode instead of defaulting to localhost. * Include the remote network address (IP and port) in diagnostics that identify cross-availability zone Kafka clients. * Fix a bug where setting `WARPSTREAM_DISABLE_S3_CHECKSUMS` to true doesn't really disable checksumming. #### Release v697 September 4, 2025 * Add `WARPSTREAM_DISABLE_S3_CHECKSUMS` environment variable that when set to `true` disables checksumming in the S3 client to improve compatibility with some S3-compatible object stores that are not actually S3 and haven't been updated yet to support S3's latest checksumming scheme. * Fix bug where the agent was unable to start in multiregion mode when using static metadata URLs. #### Release v696 September 3, 2025 * Fix schema registry `GET /subjects/(string: subject)/versions/(versionId: version)/schema` and `GET /schemas/ids/{int: id}/schema` endpoints to return properly formatted schema. * Enable single partition prefetching by default. This behavior can be disabled by setting the environment variable `WARPSTREAM_ENABLE_PREFETCHING_FOR_SINGLE_PARTITION_WORKLOADS=false`. * Adds support for Orbit fetch v2 for increased Orbit performance. #### Release v695 September 3, 2025 * Reverted: "Increased the default value of maximum inflight fetch compressed bytes allowed for all role configurations by 50% (due to the new more efficient fetch code path)." * Improved prefetching logic for single partition workloads to result in 99% hit ratios in most reasonable use cases. * Updated default of `-gracefulShutdownDuration` & `WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION` to 300s (5 minutes) * Added the `WARPSTREAM_HTTP_PROXY` environment variable to allow configuring a HTTP proxy for the Agent that will act only on `*.warpstream.com` requests rather than using `HTTP_PROXY` which would act on all requests. #### Release v694 August 27, 2025 * Split internal agent to agent listener to it's own port (default 8443) when TLS is enabled. * Fix ingestion of maps with optional values for Tableflow. #### Release v693 August 26, 2025 * Improve accuracy of some distribution metrics, like `agent_segment_batcher_flush_file_size_uncompressed_bytes` and `agent_segment_batcher_flush_file_size_compressed_bytes`. * Fix some metrics to include key tags: `virtual_cluster_id`, `agent_group`, and `agent_roles`. * Add support for TLS and mTLS for the internal agent to agent communication. * Upgrade to Bento v1.10.0. With this, `aws_dynamodb` output also supports deleting items from a dynamodb table. #### Release v692 August 25, 2025 * Add a new environment variable `WARPSTREAM_ENABLE_PREFETCHING_FOR_SINGLE_PARTITION_WORKLOADS` that enables background pre-fetch for consumers that only fetch data for a single partition at a time (like Spark). This dramatically reduces latency (and as a result, improves throughput) for these consumers. * Fix the client id used by the tableflow agent and orbit for consuming to work with Kafka agents with roles configured. * Add severity tag to diagnostic metrics (`warpstream_diagnostic_status`), enabling filtering by `severity` (low/medium/high/critical). #### Release v691 August 21, 2025 * Fix a bug where the schema registry agent doesn't return the right content type (application/vnd.schemaregistry.v1+json). #### Release v690 August 20, 2025 * Fix a bug where the agent occasionally returns validation error ("file has already been created once already") when a client produces to an agent that hasn't received produce requests in a while. #### Release v689 August 20, 2025 * Dramatically reduce read amplification for some workloads consuming historical data from small topics. * Bump gocloud.dev dependency to pick up a S3 path-style URLs fix. #### Release v688 (This version is incompatible with S3 path-style URLs, which are enabled when `s3ForcePathStyle=true` is specified in the bucket URL. Please upgrade to v689.) August 19, 2025 * Bump gocloud.dev dependency to pick up a data race fix. #### Release v687 August 18, 2025 * Add read uncommitted support to Orbit agent handler. * Update logic for reloading the Agent key from a file to fix issue of not reloading when symlinks are used. #### Release v686 August 18, 2025 * Fixes a bug in service discovery where agents deployed with different roles could cause no brokers to be returned. Previously, if an availability zone contained agents but none had the required produce/consume/both role, the system would return an empty broker list instead of falling back to other zones. This release resolves the issue by properly implementing zone fallback logic. * Fixes a bug in the fetch diagnostic that was causing it to fire even when there was no backpressure going on. #### Release v685 August 12, 2025 * Enhance Agent telemetry for investigating performance issues. #### Release v684 August 12, 2025 * Add new agent flag `checkBucketAccessRetryCount` (and the `WARPSTREAM_CHECK_BUCKET_ACCESS_RETRY_COUNT` env variable) to control the maximum times the agent retries object storage permissions checks during startup. * If a produce request contains topics both owned by Orbit and not, allow the topics not owned by Orbit to produce successfully. #### Release v683 August 11, 2025 * Enable new fetch code path that is lower latency and uses significantly less memory. This new fetch code path should also reduce memory usage in consumer client applications. To revert to the old fetch code path set the environment variable `WARPSTREAM_FETCH_SINGLE_KAFKA_BATCH=true`. * Fixed a bug where Agents running with the `proxy` role had 50% less maximum inflight fetch compressed bytes allowed than intended. * Increased the default value of maximum inflight fetch compressed bytes allowed for all role configurations by 50% (due to the new more efficient fetch code path). * Increased default value of `WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE` environment variable and `kafkaMaxFetchRequestBytesUncompressedOverride` flag to 1GiB (due to the new more efficient fetch code path). * Increased default value of `WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE` environment variable and `kafkaMaxFetchPartitionBytesUncompressedOverride` flag to 1GiB (due to the new more efficient fetch code path). #### Release v682 August 7, 2025 * Enable topics metadata cache by default. This increases the latency of CreateTopic and DeleteTopic requests to \~1s, but dramatically reduces the latency of Metadata operations which improves latency and compatibility with many Kafka clients and tools. #### Release v681 August 5, 2025 * Enhance the behavior of the Kafka lag related metrics emitted by the Agent when the `topic` cardinality is disabled. We would emit the same metric several times with different values, whereas we now emit a single metric with the sum of the lag across all partitions of the topic - or the max for the time lag metric. * Double the maximum number of outstanding files/bytes that are allowed per vCPU for producing when Agents are running only the `proxy-produce` and `proxy-consume` roles in addition to doing it when the Agents are running only the `proxy-produce` role. This will make it easier to increase CPU usage enough that auto-scaling kicks in with this combination of roles. * Double the the maximum number of inflight compressed bytes for fetching when Agents are running only the `proxy-produce` and `proxy-consume` roles in addition to doing it when the Agents are running only the `proxy-consume` role. This will make it easier to increase CPU usage enough that auto-scaling kicks in with this combination of roles. * Use FastRetry when loading directly from object storage without going through the file cache to reduce outlier latency. * Increase parallelism and reduce chunk size when fetching directly from object storage without going through the file cache. Reduces Fetch latency for large fetch requests. * Improved object pooling to reduce memory usage for large Fetch requests. * Add defaults for Bento's output `kafka_franz` component: `max_buffered_records` (`1000000`, i.e. 1 million records) and `max_message_bytes` (`16MiB`). This will prevent rate-limiting the throughput for heavy workloads. #### Release v680 July 31, 2025 * Fix `cli-beta` commands not properly parsing client tls certificates #### Release v679 July 30, 2025 * Fix unit of flag `-overridePipelinesSharedRateLimitPerVCPU` flag and `WARPSTREAM_OVERRIDE_PIPELINES_SHARED_RATE_LIMIT_PER_VCPU` environment variable: it was incorrectly in bytes/agent/s. Now it is in bytes/vCPU/s. * **breaking metrics change**: All metrics on Datadog will now be prefixed with `warpstream.` instead of `warpstream_`. You can fall back to the previous behavior by setting the `WARPSTREAM_DATADOG_NORMALIZER_PREFIX_WITH_DOT` environment variable to `false`. This change comes along the official release of our Datadog integration, making all the Warpstream Agent metrics free if you install the integration (and use the new naming convention). * Bump Bento version to v1.9.1 to fix an issue with multipart uploads to S3 outputs. #### Release v678 July 23, 2025 * Add jks support for Orbit. See Orbit docs. #### Release v677 July 23, 2025 * Import Bento crypto package to allow using the `parse_jwt_es256` method in managed data pipelines. * Add an `agent_kafka_fetch_records_counter` metric that counts the number of records present in all of the data pages the agent fetched to respond to fetches. * Tag native Bento metrics emitted by the agent with `pipeline_id`. #### Release v676 July 23, 2025 * Fix the `warpstream_agent_kafka_produce_records_counter` metric for orbit topics when `WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS` is true. #### Release v675 July 14, 2025 * Allow individual fetch requests to fetch up to 1GiB of uncompressed data per topic-partition, per fetch request, increased from 512 MiB. #### Release v674 July 14, 2025 * Remove more access control log spam by applying the same rule that was introduced in v673 to certain edge cases. #### Release v673 July 11, 2025 * Fix: Cleaned up access control logs. ACL failures for metadata requests are now only logged when a user explicitly queries for a specific topic they are not authorized to see, preventing log spam from general "list all" queries. #### Release v672 July 9, 2025 * Use a dedicated context for AWS calls to assume roles to ensure we always allow up to 15s to attempt to assume the role from AWS. #### Release v671 July 7, 2025 * Switch to ZSTD storage engine compression by default. * Add new diagnostic for when Agent backpressure produce/fetch/connections. * Add new diagnostic for when clients produce to a topic that is still being actively managed / replicated by Orbit. * Tag all logs with the "agent\_id" field. * Add support for new experimental partition assignment strategies: `consistent_spread_bounded_load`, `consistent_random_jump`, `consistent_pods`. * Bump Bento version to v1.9.0. As part of the update, `gcp_bigquery_write_api` can receive messages formatted as protobuf (if specified via `message_format`). #### Release v670 June 30, 2025 * Improve Schema Linking to better handle pipeline recreations. #### Release v669 June 26, 2025 * Improve the shutdown lock behavior to unregister from service discovery before acquiring the shutdown lock. #### Release v668 June 25, 2025 * Add support for `-enableShutdownLock` flag and `WARPSTREAM_ENABLE_SHUTDOWN_LOCK` environment variable that when set to true will force Agents to acquire a lock (per agent group) when shutting down. This is helpful when using clients like Sarama that are extremely sensitive to changes in partition assignments to prevent any disruption as a result of too many partition assignments shifting at once as Agents are rolled / killed. #### Release v667 June 24, 2025 * Add support for lookup schema endpoint (POST /subjects/{subject}) for WarpStream's BYOC Schema Registry. This endpoint checks if a schema has already been registered under the specified subject. * Automatically detect when Agent CPU usage is high and opportunistically capture CPU/heap profiles to help with debugging. * Add new `warpstream_agent_kafka_produce_records_counter` metric that can be used to measure the number of records written in the Agents on a per topic basis when the high cardinality metrics flag is enabled. * Add new `-enableMetadataTopicsCache` flag and `WARPSTREAM_ENABLE_METADATA_TOPICS_CACHE` environment variable that when set to true makes Metadata requests much faster (10s of microseconds on average) at the expense of external consistency (topics created recently may not appear in Metadata requests for up to 1s). #### Release v666 June 18, 2025 * Fix bug related to `WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY` being used in combination with `consistent_spread` value. #### Release v665 June 17, 2025 * Batch DescribeGroups requests for applications that aggresively monitor consumer groups. * Add `-defaultPartitionAssignmentStrategy` flag and `WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY` environment variable that allows overriding the default partition assignment strategy used for clients with no client-level override. Default value is `single_agent` (same behavior as before). #### Release v664 June 16, 2025 * Restore the previous behavior of `ws_pas=equal_spread` and introduced a new `ws_pas=equal_spread_v2` that handles workloads with a large number of topics with a low number of partitions much better. * Improve the error message when an agent can't find it's own private IP. The error will now log the found IP address and the accepted private IP ranges. * Add a new partition assignment strategy `ws_pas=consistent_spread` that uses a consistent hash ring to assign partitions to individual Agents resulting in significantly smoother data processing for large clusters during rolling deployments. #### Release v663 June 12, 2025 * Add diagnostic to detect when Kafka Clients issue Fetch Requests with a fetch size bigger than the limit set in the Agent. * Add flag `batchMaxCompressedSizeBytes` and environment variable `WARPSTREAM_BATCH_MAX_COMPRESSED_SIZE_BYTES` to define the max compressed size (similar to the already-existing flag `batchMaxSizeBytes` and environment variable `WARPSTREAM_BATCH_MAX_SIZE_BYTES` that allow to define a target based on the uncompressed size. Note: this is not a hard limit, it is based on estimations so the actual compressed size may be a bit higher than the setting. * Enforce strict mode (i.e. `error handling.strategy: reject`) when running bento pipelines. This means that Bento will reject all batches containing messages with errors, propagating a `nack` to the input layer (instead of attempting to send message batches that contain messages with errors to the configured sink). #### Release v662 June 9, 2025 * Add support for confluent processors like `schema_registry_decode` to managed data pipelines. #### Release v661 June 9, 2025 * Enable acl deny logging by default. * Change the behavior of equal\_spread partition assignment strategy to handle a wider variety of workloads. Previously it worked well if all topics in the cluster had number of partitions >> than the number of Agents in the cluster, but worked extremely poorly if there were many topics in the cluster with a very small number of partitions. * Bump Bento version to v1.8.0. As part of the update, the kafka\_franz input would force reconnect when it sees an unknown topic/partition error (i.e. when a topic is recreated). #### Release v660 June 6, 2025 * Updated cli-beta benchmark #### Release v659 June 4, 2025 * Improve how Agents handle Fetch requests when there are partial errors (i.e. some of the partition requests are invalid). * Improve how Agents preload topic metadata to prevent cold-start problems for Producers that produce to dozens or 100s of topics in a single Produce request. * Fix bug in BatchMaxSizeBytes validation (max allowed: 16MB). #### Release v658 May 30, 2025 * Add diagnostic to detect when Agents have hit connection limit. * Add metrics for throughput and latency of RPCs between Agents and control plane. * Bring back the `warpstream_error_count` metric that corresponds to the number of error logs. * Be smarter about when to send CPU/Heap profiles back to the control plane to try and capture more interesting profiles. * Add support for hot reloading TLS server certificates, now certificates can be changed without having to restart WarpStream agents. #### Release v657 May 22, 2025 * Diagnostic to detect when Kafka Clients issue Fetch requests with very low timeout. * Add flag `httpsProxyCACertFile` and environment variable `WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE` to support enterprise MITM HTTPS Proxies. * If using a MITM HTTPS Proxy users may see this error `Post "https://metadata.default.${region}$.${cloud}$.warpstream.com/api/v1/agent/agentpool": tls: failed to verify certificate: x509: certificate signed by unknown authority` because the agent doesn't trust the HTTPS Proxy's CA. This flag allows the user to set the CA to the internal WarpStream HTTPS client as a trusted certificate authority. * Add flag `tlsProfile` and environment variable `WARPSTREAM_TLS_PROFILE` to change TLS MinVersion, Curves and Ciphers. * See the [Protect Data in Motion with TLS Encryption](/warpstream/kafka/manage-security/protect-data-in-motion-with-tls-encryption) documentation for exact details. #### Release v656 May 20, 2025 * Fix the diagnostic for unavailable storage bucket: it triggered on some errors that were not bucket errors in AWS. * Modified the record size limit resolution logic to prioritize in the following order: 1) topic-level `max.message.bytes`, 2) cluster-level `message.max.bytes`, 3) agent-level default. This replaces the previous behavior which used the minimum value between topic and cluster limits. * Allow Orbit to fetch more than 100MiB per kafka fetch request. #### Release v655 May 20, 2025 * Increase default value for `WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE` from 128MiB to 256MiB. * Increase default value for `WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE` from 128MiB to 256MiB. * Adds a diagnostic that triggers when there are errors talking to the cloud storage bucket (either because the storage bucket is rate limiting reads/writes or because of a temporary problem connecting to it). * Significantly improve the performance of consumer workloads that are fetching large amounts of compacted data. #### Release v654 May 15, 2025 * Cleanly NACK jobs that failed to run during agents shutdown. * Improve the logic for long-polling fetch requests after detecting that there is no new data for the set of topic-partitions that are being fetched. #### Release v653 May 9, 2025 * Added cluster-level (`message.max.bytes`) and topic-level (`max.message.bytes`) limits. In case of Warpstream these limits apply to individual records, instead of record batches. This is because OSS Kafka doesn't process individual records, it processes batches, but WarpStream operates on records directly. So there is no relationship between the batch size sent by producers and the size of batches received by consumers, and therefore, limiting record sizes will prevent consumers from choking on any individual records that are too large. * The effective limit is the minimum value across topic-level, cluster-level, and agent-level configurations. #### Release v652 May 8, 2025 * Added optional prometheus metrics server for `benchmark-producer` and `benchmark-consumer` commands. #### Release v651 May 7, 2025 * Added a producer benchmark tool `benchmark-producer` to the `cli-beta` commands. * Added a consumer benchmark tool `benchmark-consumer` to the `cli-beta` commands. * Add a 15s socket deadline for every socket read/write to ensure we detect when goroutines are stuck reading from a bad connection. * Tune HTTP connection pools to be smaller and dont allow idle connections to remain in the pool for as long. * Report how long each job has run to the control plane. #### Release v650 May 2, 2025 * Add new `-tlsBlobURL` flag and `WARPSTREAM_TLS_BLOB_URL` environment variable to load TLS files from a blob store bucket. * Increase default connection limit per vCPU from 4096 to 8192. * Add metric `warpstream_agent_active_pipeline_instances` (tags: `pipeline_id`) that tracks the number of active pipeline instances run by a given agent. #### Release v649 Apr 29, 2025 * Add a cache for api versions. This will reduce excessive rpcs to the control plane. * When an agent has the "pipelines" role AND any other role, Bento pipelines are rate-limited (default: 5MB/vCPU). When the agent has only the "pipelines" role, the rate limit is higher (50 MB/vCPU). * Add new `-overridePipelinesSharedRateLimitPerVCPU` flag and `WARPSTREAM_OVERRIDE_PIPELINES_SHARED_RATE_LIMIT_PER_VCPU` environment variable to override the default rate limit shared across all pipelines. #### Release v648 April 18, 2025 * Fixes an invariant violation error that could be emitted if you tried to fetch transactional records before the beginning of a partition. * Add support for ratelimiting the number of bytes that can be read from input streams during compactions to reduce networking microbursts. * Tune the maximum number of concurrent input streams that can be opened during compactions based on the overall size of the compaction to reduce networking microbursts. * Add a new flag for a preview feature, `-multiregion`. This tells the agent that the control plane runs across two regions, which will have the agent talk to the leader region while keeping track of leadership. * Enable the Agent file cache to begin backpressuring sooner than before. This prevents the Agents from OOMing some scenarios when they're completely overloaded and will now backpressure traffic more appropriately. * Allow reducing batch timeout to 25ms now that S3OZ is so much cheaper. * Add a new `warpstream_control_plane_utilization` metric (tagged by cluster). A value of 1.0 (100%) means the control plane is fully saturated and can no longer keep up with incoming requests. * Fixes a race condition in schema validation when there are concurrent produces. * Orbit now has an estimated time lag. The time lag is defined as `time.Since(X)` where `X` is the timestamp of the source record that is most recently copied by Orbit. The time lag will be emitted with the `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` metric for the Orbit consumer group. #### Release v647 April 10, 2025 * Fix a bug where S3 multi-part uploads from the produce path and the schema migrator would fail with "InvalidPart: One or more of the specified parts could not be found. The part may not have been uploaded, or the specified entity tag may not match the part's entity tag". The github.com/aws/aws-sdk-go-v2/config version was upgraded to 1.29.9 from 1.27.43 in Agent v643 but the upgrade is backwards-incompatible. Specifically, multi-part uploads that don't specify a checksum algorithm will fail (see reports [here](https://github.com/aws/aws-sdk-go-v2/discussions/2960#discussioncomment-12077210) and [here](https://github.com/aws/aws-sdk-go-v2/issues/3005#issuecomment-2641575808)). #### Release v646 (Bad version, please upgrade to v647) April 9, 2025 * Orbit ignores topic id in fetch responses if kafka version < 3.1. #### Release v645 (Bad version, please upgrade to v647) April 8, 2025 * Orbit makes metadata requests using topic name when max supported request version is < 12. #### Release v644 (Bad version, please upgrade to v647) April 8, 2025 * Add flags `batch-size` and `sleep-between-batch` for the `delete-all-topics-unsafe` command to allow configuring the batch size and sleep duration between batch deletions. * Allow Agents to advertise IPs in range 240.0.0.0/8–255.0.0.0/8 automatically for customers using those in their private subnets. #### Release v643 (Bad version, please upgrade to v647) April 8, 2025 * Make the `delete-all-topics-unsafe` command iterative. #### Release v642 April 7, 2025 * Add `tls_insecure_skip_verify` to orbit and schema registry migrator config #### Release v641 April 7, 2025 * Treat "connection timed out" error as resumable for object storage GET requests to prevent long running compactions from failing. * Fixes a bug where the agent could get stuck in some circumstances when reading data after a gap of more than 4B records created either by orbit or compacted topics compaction. #### Release v640 April 2, 2025 * Tune down the interval for batching internal API calls for describing the topics in a cluster. * Reduce memory usage for S3 clients when performing large numbers of concurrent DELETE operations. * Double number of LOW priority jobs that can run concurrently. * Create `file-reader` and `file-scrubber` cli-beta commands to read and scrub warpstream agent files. * Add new `-additionalDeadscannerBucketURLs` flag and `WARPSTREAM_ADDITIONAL_DEADSCANNER_BUCKET_URLS` environment variable to allow specifying additional bucket URLs that should be scanned for dead objected to be deleted from the object store. This enables more seamless migrations from one object storage bucket to another without having to manually cleanup the dead files in the old object storage bucket. * Add new `mtls_server_ca_cert_env` field to Orbit configuration to allow pointing to a file containing PEM encoded public keys of the certificate authorities that sign your server certificates. #### Release v639 * Add support for WarpStream Schema Linking. * Fix a rare panic when fetch auto-tuning is disabled explicitly and the kafka clients issue very small requests. * Bump github.com/golang-jwt/jwt/v5 dependency for CVE-2025-30204 vulnerability. * Return LeaderNotAvailable instead of BrokerNotAvailable for some transient / retriable Metadata errors to improve client compatibility. * Change the verbosity of deadscanner "deadscanner\_progress" logs when they hit a "not found" error. * Allow Agents to run more L0/L1 compactions in parallel which helps prevent temporary spikes of L0/L1 lag when large L2 compactions are running. #### Release v638 March 21, 2025 * Add metric `agent_kafka_source_cluster_connections_counter` that counts the number of connections made by agents to the source cluster (via Orbit) #### Release v637 March 21, 2025 * Add `consumer-group-lag` command to `cli-beta`. * Increase the default value of GOMEMLIMIT to 3GiB/vCPU from 2GiB/vCPU. For properly scaled clusters this should have no impact on memory usage, but it should prevent excessive garbage collection from occurring when there are a lot of inflight fetch requests for overloaded clusters. * Ignore context.DeadlineExceeded errors for circuit breaking purposes when streaming GET requests from object store to avoid opening the circuit breaker when the Agents are overloaded, but the underlying object store is fine. * Reduce log spam for context.Canceled errors. * Use 4MiB pages by default instead of 16MiB pages. * Agent handles sasl handshakes in the agent itself. This reduces excessive rpcs to the control plane. * Orbit will prioritize fetching partitions by the order defined by the control plane. #### Release v636 March 18, 2025 * Release analyzers in the Agent to feed Warpstream Diagnostics: Diagnostics continuously analyzes your clusters to identify potential problems, cost inefficiencies, and ways to make things better. It looks at the health and cost of your cluster and gives detailed explanations on how to fix and improve them. * Add flag `enableACLLogs` and env var `WARPSTREAM_ENABLE_ACL_LOGS` to enable ACL logging. * As of this release Produce, Fetch, Metadata, JoinGroup, SyncGroup, DeleteRecords, InitProducerID Kafka API calls are logged if they are denied. More APIs will be added over time as we update the Control Plane. * Reduce allocations in the fetch code path to improve performance. * Fixes a very rare case where the agent would improperly categorize a single record as committed when it was aborted, with isolation\_level=read\_committed. #### Release v635 March 12, 2025 * Enable a new mechanism in the Agents for cleaning up deleted files from object storage. This new mechanism makes it much easier for high volume clusters to keep up with object deletion, and also reduces costs by dramatically reducing the amount of LIST and HEAD requests that the Agent make. * Reduces amount of data Orbit sends to the control plane to create topics. #### Release v634 March 12, 2025 * Add support for using IMDSv2 to resolve instance types for Agents running in AWS. #### Release v633 March 11, 2025 * Fixed a bug where setting an agent flag to an empty string by explicitly setting an environment variable to an empty string (i.e. `ENV_VAR=""`) doesn't work as expected as it uses the default value instead. This affects the following flags: * If you set `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS` to an empty string, it used to default to disabling the `partition` tag. Now, if you set the env variable to an empty string, it doesn't disable any tags for the consumer group offset metrics. * If you set `WARPSTREAM_AGENT_ROLES` to an empty string, it used to default to `proxy, jobs`. Now, if you set `WARPSTREAM_AGENT_ROLES` to an empty string it means no roles are selected. Note that unless a role is provided via flags like `enableManagedPipelines`, empty roles is not allowed. * Add `begin_fetch_at_latest_offset` to Orbit topic mappings which forces Orbit to fetch from the latest topic partition offset in the source cluster, for the first fetch of each topic partition. #### Release v632 March 10th, 2025 * If the `partition` tag is disabled via the `disableConsumerGroupsMetricsTags` flag (or the `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS` environment variable), the `warpstream_max_offset` metric will be the max of the topic's partitions' max offsets. * Increase `WARPSTREAM_KAFKA_CLOSE_IDLE_CONN_AFTER` to 1 hour to decrease the frequency of `broken pipe` errors on some kafka clients. * Reduce log spam for some context canceled errors. #### Release v631 March 7th, 2025 * Implement support for the v0 JoinGroup API in consumer group management, allowing older clients to join groups. #### Release v630 March 6th, 2025 * If the `partition` tag is disabled via the `disableConsumerGroupsMetricsTags` flag (or the `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS` environment variable): * The `consumer_group_lag` metric will be the sum of the consumer group lag across the topic's partitions. * The `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` metric will be the max of the estimated lag across the topic's partitions. * Increase default connection idle timeout from 10m to 15m. * Improve Agent logging (remove spurious logs, emit sampled logs for idle connections being closed for improved observability) #### Release v629 Feb 28th, 2025 * The agent now accepts to create batches that span more than 4B offsets. This is useful for heavily compacted topics. * Include warpstream proxy target in client ID for kafka\_franz\_warpstream blocks in managed data pipelines so that managed data pipelines works when Agents are split into proxy-consume and proxy-produce roles with no Agents running both roles simultaneously. * Add MTLS support orbit. #### Release v628 Feb 19th, 2025 * Improve visibility of ACL authorization failures. * Return ErrKafkaStorageError instead of ErrThrottlingQuotaExceeded for backpressure errors from Produce requests because librdkafka doesn't retry ErrThrottlingQuotaExceeded requests. * Fix bug in PREFIXED ACLs, where sometimes were not correctly applied. #### Release v627 Feb 18th, 2025 * Change the default fetch size for kafka\_franz in managed data pipelines from 100MiB to 50MiB and the per-fetch fetch size from 50MiB to 25MiB. * If sasl/mTLS auth is performed using a cluster credentials name, with or without the ccn\_ prefix, allow both the credentials name and the credentials username to be used as the kafka principal for acls. Previously, if a client performed auth using a credentials name, the credentials username was used as the kafka principal, which prevented valid acls matches. #### Release v626 February 18th, 2025 * Upgrade Agents to go 1.23.6. * Bump our Alpine base image to `3.21.3` to fix some vulnerabilities * Fix a rare panic happening on slow nodes when the publishing metric job would take too long to be handled #### Release v625 February 12th, 2025 * Add batching of internal API calls for describing the topics in a cluster. #### Release v623 February 10th, 2025 * Add `diagnose-record` to `cli-beta`, this command prints out diagnostic information about a specific record * Reduce default values for fetch auto-tuning to be less aggressive so it causes less memory pressure on consumer clients if they embed multiple Kafka consumers within the same application. * Prevent a tombstone from being deleted if it is the last record in a topic partition. Some workload use the offsets of the records consumed to determine if a consumer is caught up with the high watermark offset, and not deleting the tombstone allows the tombstone record to be consumed. #### Release v622 February 6th, 2025 * Change the default value of `autoTuneFetchLimits` from `false` to `true`. This makes it so the WarpStream Agents automatically auto-tune the fetch limits of Kafka consumer client applications. This dramatically improves the consumer performance of consumer applications that have not explicitly tuned their application for performance with WarpStream. * Add a new `ws_dfat` client ID feature that allows consumer clients to disable fetch auto-tuning if needed. * Add flag `--consume-from-beginning` to the CLI `console-consumer` to allow consuming from the beginning of the topic partition. * Enforce that agent group names only contain lowercase letters, numbers, and dashes. * Add `cli-beta` subcommand that changes how cli flags work to have per cli subcommand flags instead of adding them all to every cli command. * Add `console-consumer` to `cli-beta` which adds new features for printing out the offset of the message and the message key. Also add `max-messages` flag to limit the number of messages to consume. * Fix a bug where the agent would return an empty response that the sarama kafka client would not know how to parse in some cases in transactional or compacted topics. * New metrics: * `warpstream_num_records`: This is a gauge that indicates the number of records being stored per topic * tags: `topic` #### Release v621 February 5th, 2025 * Add support for the `min.compaction.lag.ms` topic-level configuration. This configuration allows users to specify a minimum time that must pass before a segment is eligible for compaction. This configuration is useful for ensuring that data is not compacted too soon after being written in a topic that has compact or compact,delete as its cleanup policy. * Fix a rare bug where incorrect data could be returned when using transactions in a compacted topic. #### Release v620 February 3rd, 2025 * Allow Orbit to work with older kafka clusters(pre 3.1 which was when topic ids were added to kafka fetch request responses). If Orbit is used without topic ids, then it is possible that a topic was deleted and recreated in the source cluster with the exact same name which might lead to inconsistencies in Orbit. This is easily solved by deleting the topic in the target cluster, and it will be automatically recreated by Orbit. #### Release v619 February 2nd, 2025 * Only retry Metadata errors that have retryable error codes. #### Release v618 Jan 31st, 2025 * Allow blob storage retries to retry GET and multi-part upload requests in compaction code to prevent long-running compactions from failing due to transient errors. * Improve error handling of the Metadata RPC to convert 500s and transient retryable errors to Kafka error LeaderNotAvailable instead of KafkaStorageError. This improves compatibility with librdkafka and prevents issues where librdkafka would refuse to accept additional records into its producer queue after receiving a KafkaStorageError in a Metadata response. #### Release v617 Jan 28th, 2025 * Improve the consistent hashing ring used in the file cache by increasing the number of virtual shards and switching hashing algorithms. Agents will automatically wait until all of the other Agents in the cluster are upgraded to the latest version and then switch hash ring implementations at approximately the same time to minimize disruption during the rollout. * Removed the `allowHighCardinalityConsumerGroupTags` flag and changed the default values of `disableConsumerGroupsMetricTags` to `partition` . This is a potentially **breaking** change for observability, but was done to prevent cluster with large numbers of topic-partitions from overloading customer's metrics systems and bills. The `partition` tag for consumer group metrics can be re-enabled by setting the `disableConsumerGroupsMetricTags` value to an empty string. * Add `cli certificate-to-dn` helper command to convert a client certificate to a DN. #### Release v616 Jan 24th, 2025 * Fix JKS certificate loading to be according to the JKS spec, specifically load certificates from DER instead of PEM formatting. #### Release v615 Jan 24th, 2025 * Fixed a bug where the BYOC schema registry incorrectly returns a schema incompatibility error when removing a field whose default value is `null`. #### Release v614 Jan 23nd, 2025 * Fix naming of `tlsJavaKeystorePasword` and `tlsJavaTruststorePasword` to be `tlsJavaKeystorePassword` and `tlsJavaTruststorePassword` . #### Release v613 Jan 22nd, 2025 * Orbit now copies internal kafka topics if these topics match the regex specified in the Orbit config. #### Release v612 Jan 16th, 2025 * Added flags to support Java Keystores and Truststores to make migration from Kafka easier * `tlsJavaKeystoreFile`, `tlsJavaKeystorePasword`, `tlsJavaKeystoreKeyPasword`, `tlsJavaTruststoreFile`, `tlsJavaTruststorePasword`. * Java Keystore passwords are not used to encrypt the keystore, only to confirm it's integrity. In most Java implementations the keystore password and private key password must be the same which negates any encryption advantages. We recommend using the same security controls when using Java Keystores that you would use with unencrypted certificate keys. * Added support for using IMDSv2 to configure `DD_AGENT_HOST` for Agents running in AWS. If v2 is not available we automatically fall back to v1. * Improved schema validation error message to return more detailed reason for why invalid record is rejected. * Added the `-schemaValidationURL` flag for schema registry to replace the deprecated `-schemaRegistryURL` flag (still supported). #### Release v611 Jan 10th, 2025 * Add `tlsServerPrivateKeyPasswordFile` flag to enable using encrypted TLS private keys. * The encrypted key given via `tlsServerPrivateKeyFile` must be in `PKCS#8` format. * The password file must contain a single line which is the private key's password. * Honor `DD_AGENT_HOST` environment variable if its already set. * Allow TLS to be used with Orbit even if SASL is not configured. * Add support for Apache Kafka transactions. Read our announcement at . #### Release v610 Jan 8th, 2025 * **Basic Authentication Support for BYOC Schema Registry**: * Add `schemaRegistryBasicAuth` flag to enable basic authentication for schema registry. * Support the `GET /subjects/(string: subject)/versions` endpoint in schema registry to allow getting a list of versions registered under the specified subject. * Bump golang.org/x/net dependency for CVE-2024-45338 vulnerability. #### Release v609 Dec 19th, 2024 * Add `-requireSASLAuthentication` and `-requireMTLSAuthentication` to playground mode. They where missed when these flags where added to the agent in `v549`. #### Release v608 Dec 18th, 2024 * Add support for using DynamoDB as the agent's backing store alongside S3 and S3 Express One Zone. #### Release v607 Dec 18th, 2024 * Fix an issue where tombstones could be incorrectly handled, and showed as messages with empty values. * Fix a rare issue where the agent would fail to connect to a S3 bucket on startup in some configurations. #### Release v606 Dec 11th, 2024 * Make Orbit much more efficient and able to achieve higher throughput more easily. * Double the ratelimit for the maximum inflight fetch compressed bytes per core (now that fetch is significantly more efficient). * Make bento logging less noisy by limiting each managed pipeline to one bento log per second or less. #### Release v605 Dec 10th, 2024 * Avoid noisy error logs when instance type discovery fails. * Fix macOS Agent binaries to not segfault about a missing library #### Release v604 Dec 9nd, 2024 * Add the ability to load the agent key from a file using the `-agentKeyPath` flag or `WARPSTREAM_AGENT_KEY_PATH` environment variable. The agent will reload the file when the file is updated so the key can be changed without restarting the agent. * Upgrade to latest Bento which adds ability to ratelimit pipelines based on bytes instead of messages, and improving GCP bigquery output logging. * Switch to C++ library for lz4 compression instead of pure Go. #### Release v603 Dec 2nd, 2024 * Preserves the original DN from the TLS cert to be used as the ACL principal. Previously, while the DN was semantically correct, the order of its elements wasn't necessarily preserved. This leads to the principal set during auth to not match the principal set when the ACL is created. * Fix a log in the managed data pipelines feature that was logging a cluster-specific credential. * Add support for SASL SCRAM 256/512 as an authentication mechanism for source clusters in Orbit. #### Release v602 Nov 25th, 2024 * Retry object storage permissions check up to 3 times on startup to avoid false positives related to dial timeouts * **Added BYOC Schema Registry Support**: * The agent now supports hosting a BYOC Schema Registry. * Use the `-schemaRegistryPort` flag (or the `WARPSTREAM_SCHEMA_REGISTRY_PORT` env variable) to specify the port (default 9094) to run the schema registry server on. * Use the `-schemaRegistryTLS` flag (or the `WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLED` env variable) to enable tls over the schema registry server. * New metrics: * `warpstream_agent_schema_registry_inflight_connections`: number of currently inflight / active connections to schema registry server. * tags: `schema_registry_operation`, `outcome` * `warpstream_agent_schema_registry_request_latency`: latency (seconds) for processing each Schema registry request. * tags: `schema_registry_operation`, `outcome` * `warpstream_agent_schema_registry_outcome`: outcome (success, error, etc) for each Schema Registry request. * tags: `schema_registry_operation`, `outcome` * `warpstream_agent_schema_registry_request_bytes_counter`: number of bytes of incoming requests. * tags: `schema_registry_operation` * `warpstream_agent_schema_registry_response_bytes_counter`: number of bytes of outgoing response. * tags: `schema_registry_operation`, `outcome` * `warpstream_schema_versions_count`: Total number of schema versions in the schema registry cluster. * `warpstream_schema_versions_limit`: Maximum number of schema versions allowed in the cluster. #### Release v601 Nov 21st, 2024 * Limit maximum number of inflight bytes for fetch as compressed instead of uncompressed, and do it *before* issuing the fetch requests. This prevents excessive read amplifications when the Agents are highly loaded. * Switch from offheap to onheap cache for file cache for better eviction policy. * Fix a bug that was preventing the Agents from loading large IOs directly from object storage leading to excessive cache churn. * Remove some prefetcher restrictions to improve the performance of large fetch requests with many topic-partitions. #### Release v600 Nov 20th, 2024 * Fixed DescribeCluster API to correctly override hostname * Fixed a bug where the demo/playground agent was incorrectly using the environment variable. #### Release v599 Nov 20th, 2024 * Only restart Bento pipelines when their deployed configurations are modified. Previously, modifying one configuration restarted all deployed pipelines. * Increase default timeout for loading data into the file cache from 5s to 15s. This has no impact on tail latencies for healthy clusters due to the fast retry mechanism, but dramatically improves the ability of overloaded clusters to make progress. * Cap number of speculative/fast retries to 2% at most. This prevents excessive read amplification in some scenarios where the Agents are overloaded. * Upgrade Agents to go 1.23.3 to eliminate contention in HTTP client. #### Release v598 Nov 6th, 2024 * Removed noisy error log triggered when producing with ACKS=0. Previously, clients and agents generated a fake invariant error: 'Previous sent message is not current sequence -1' due to an unordered sequence number. * Allow Agents to create ingestion files with up to 64MiB of uncompressed data, increased from 16 MiB. * Orbit encrypts communication with source clusters using TLS. #### Release v597 Nov 6th, 2024 * **Default Agent Group:** When the `-agentGroup` parameter is not explicitly set, the agent will now default to a group named 'default'. Previously, if a Kafka client connects to an agent without a group specified, it could end up connecting to agents in other groups. With this change, clients connected to an agent with no group specified will now only see agents in the 'default' group. * **Note:** This change is backwards compatible; during rollout, agents with no group set and those in the 'default' group will be treated as a single group. * Add support for [SASL handshake v0](https://kafka.apache.org/protocol.html#sasl_handshake). #### Release v596 Nov 1st, 2024 * Add delay + jitter + randomization to the order/timing in which Bento pipelines are started to avoid pipelines synchronizing with each other and doing all of their work at the same time. * Add support for Orbit to the Agents. See: #### Release v595 October 30th, 2024 * Return success + offsets/timestamps instead of DuplicateSequenceError + offsets/timestamps when a duplicate batch is detected via the idempotent producer functionality to mirror Kafka's behavior. * Upgrade to latest version of Bento with more improvements for parquet encoder (bug fixes + support for int8/int16). #### Release v594 October 28th, 2024 * Upgrade to latest version of Bento with improved handling of decimals and floats in parquet encoder. #### Release v593 October 27th, 2024 * Fixes metrics called "warpstream\_agent\_segment\_batcher\_flush\_xxx" who were reporting their successes with "flush\_cause:buffer\_full" as errors. * Emit Bento metrics in Agents when managed data pipelines is enabled. All Bento metrics will be prefixed with: `warpstream_bento`. * Upgrade to latest version of Bento with ability to encode maps/lists in the parquet encoder. #### Release v592 October 23nd, 2024 * Upgrade to latest Bento version, which improves error logging for the GCP BigQuery output. #### Release v591 October 22nd, 2024 * Add support for isolating managed data pipelines to different groups of pipelines Agents using the `managedPipelinesGroupName` flag and `WARPSTREAM_MANAGED_PIPELINES_GROUP_NAME` environment variable. [Read the docs](/warpstream/kafka/manage-connectors/bento#pipeline-groups) * Make manage data pipelines product work automatically, even when the WarpStream Agents are advertising the hostname of a load balancer to the Kafka protocol by making the Kafka metadata handler always return the Agents actual IP addresses to the managed data pipelines product. * Upgrade to latest Bento version which adds support for interpolating table name in GCP BigQuery output, as well as support for GCP's new [Storage Write API for BigQuery](https://warpstreamlabs.github.io/bento/docs/components/outputs/gcp_bigquery_write_api). Also adds support for writing to [Redshift clusters in AWS](https://warpstreamlabs.github.io/bento/cookbooks/redshift/). #### Release v590 October 17th, 2024 * Make the Agents use the pure GO DNS resolver to work around a concurrency [issue](https://github.com/golang/go/issues/63567) with `setenv` and `getenv`. * Agents will now automatically tune their GC and backpressure settings automatically based on which roles are configured. `proxy-consume` Agents will have larger file caches, `proxy-produce` Agents will buffer more data before backpressuring, `pipelines` Agents will GC less aggressively, etc #### Release v589 October 16th, 2024 * Add fetch limits auto-tuning to the Agents, which automatically adjusts fetch request size limits when the Agents deem necessary. To enable this feature, use the flag `-autoTuneFetchLimits` or the env variable `WARPSTREAM_AUTO_TUNE_FETCH_LIMITS`. By default this is disabled. #### Release v588 October 11th, 2024 * Agents configured with specific roles will now reject requests that they're not supposed to handle. For example, proxy-consume Agents will reject Fetch requests and proxy-produce Agents will reject Produce requests. * WarpStream Agents will now automatically avoid communicating with each other if multiple clusters are deployed in the same VPC, even if I.P addresses are quickly recycled between the two clusters (due to high container churn). * Increased default batching interval for Metadata requests to 250ms. * Upgrade managed data pipelines to latest version of Bento, and automatically tune franz\_kafka blocks to fetch data from source Kafka/WarpStream clusters faster. #### Release v587 September 27th, 2024 * Increase the default value of GOMEMLIMIT from 1GiB/vCPU to 1.5GiB/vCPU. This should result in less GC overhead for highly loaded Agents. * Automatically disable profile forwarding if Datadog profiling is turned on. #### Release v586 September 25th, 2024 * Change the default value of `disableProfileForwarding`/`WARPSTREAM_DISABLE_PROFILE_FORWARDING` to false, which means that the Agents will start forwarding profiles to WarpStream's control plane. #### Release v585 September 24th, 2024 * Add support for [scheduling blocks](/warpstream/kafka/manage-connectors/bento#warpstream-block) in managed data pipelines #### Release v584 September 23rd, 2024 * **Add support for AWS Glue Schema Registry schema validation** * Users can now use schemas stored in their AWS Glue Schema Registry to validate records. * Add new topic-level configuration `warpstream.schema.registry.type` to specify what type of schema registry the agent should fetch the remote schemas from. Supported values include `"STANDARD"` and `"AWS_GLUE"`. Defaults to `"STANDARD"` * Add `-zonedCIDRBlocks` flag and `WARPSTREAM_ZONED_CIDR_BLOCKS` env variable as an alternative way to provide information on the Kafka client's availability zone. The value is a mapping of availability zones to Kafka client IPs and should be a <> delimited list of AZ to CIDR range pairs, where each pair starts with an AZ, a @, and a comma separated list of CIDR blocks for that given AZ. For example, `us-east-1a@10.0.0.0/19,10.0.32.0/19<>us-east-1b@10.0.64.0/19<>us-east-1c@10.0.96.0/19`. #### Release v583 September 18th, 2024 * Bump the otel dependency to fix spurious error logs introduced in v582 #### Release v582 September 18th, 2024 * Increase default clean shutdown interval from 60s to 80s. * Increase the max number of connections per CPU (from 2048 to 4096) #### Release v581 September 12th, 2024 * Bump build to go `1.22.7` to fix some stdlib vulnerabilities #### Release v580 September 11th, 2024 * Add agent profile forwarding to WarpStream's control plane. Note that this feature cannot be enabled together with Datadog profiling and is not supported for self-hosted control planes. The following flags are added for this feature. * `disableProfileForwarding`/`WARPSTREAM_DISABLE_PROFILE_FORWARDING`: this flag can be used to disable profile forwarding. By default this is set to true and no profiles will be forwarded. * `maxProfileSize`/`WARPSTREAM_MAX_PROFILE_SIZE`: max number of bytes allowed for buffering profiles in memory. * Bump our Alpine base image to `3.20.3` to fix some vulnerabilities #### Release v579 September 9th, 2024 * Add `-maxProduceRecordSizeBytes` flag and `WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES` env variable to override the maximum uncompressed size of a record that can be produced. #### Release v578 August 30th, 2024 * Add "ratelimited" outcome for S3-backed blob storage metrics `warpstream_blob_store_*` * Increased maximum allowed ingestion file size from 8MiB to 16MiB * Improve buckets for distribution metrics for `warpstream_agent_segment_batcher_flush_file_size_uncompressed_bytes` and `warpstream_agent_segment_batcher_flush_file_size_compressed_bytes` * Fixed a bug where we returned the wrong error code when a produced record was larger than the maximum allowed size (32MiB) #### Release v577 August 25th, 2024 * Fix bug in backpressure system that would cause Agents to get stuck in backpressure state. #### Release v576 August 22nd, 2024 * Fix bug affecting librdkafka library's cooperative rebalance, when adding more clients to a consumer group. * Add new startup flags to enable mutex and block profiling: * `-enableSetMutexProfileFraction`/`WARPSTREAM_ENABLE_SET_MUTEX_PROFILE_FRACTION`: enable this flag to call 'runtime.SetMutexProfileFraction' with the value passed along -mutexProfileFraction. * `-mutexProfileFraction`/`WARPSTREAM_MUTEX_PROFILE_FRACTION`: tune the value passed to call 'runtime.SetMutexProfileFraction * `-enableSetBlockProfileRate`/`WARPSTREAM_ENABLE_SET_BLOCK_PROFILE_RATE`: enable this flag to call 'runtime.SetBlockProfileRate' with the value passed along -blockProfileRate * `-blockProfileRate`/`WARPSTREAM_BLOCK_PROFILE_RATE`: tune the value passed to call 'runtime.SetBlockProfileRate' #### Release v575 August 18th, 2024 * Report CPU usage using a moving average instead of point-in-time to improve accuracy. * Improve performance when a cluster has many clients polling partitions that are written to infrequently. #### Release v574 August 15th, 2024 * Increase default limits for backpressuring produce requests by 400% #### Release v573 August 14th, 2024 * Disable automatic retries in underlying blob storage clients (AWS / GCP) so we can control blob storage retries at the application layer. * Switch to C++ library for ZSTD instead of pure Go (2-3x faster). #### Release v572 August 6th, 2024 * Improved the Agents ability to backpressure Produce requests. * Added back-pressure support for Fetch requests in addition to Produce requests. #### Release v571 August 2nd, 2024 * Improved Agent backpressuring for Produce requests so that Agents will begin throttling Produce request and individual connections when they have too much producer data buffered in memory instead of OOMing. * Agents now return `KafkaStorageError` instead of `RequestTimedOut` for some timeout-related errors in the Fetch path. This improves behavior with Java consumer clients which treat `KafkaStorageError` as retryable, but not `RequestedTimedOut`. * Fixed a bug in `demo` / `playground` mode where the requested hostname strategy was not being used. * Tagged all Agent metrics with the `agent_roles` , `agent_group` , and `virtual_cluster_id` tags to make debugging advanced deloyments easier. #### Release v570 July 17th, 2024 * Fixed a bug that would cause idempotent producer out of sequence errors when idempotent producer was enabled on some clients. Data was never committed out of order, but the previous implementation would force the client to retry more than necessary, resulting in high latency or reduced throughput * **Added Schema Validation Support**: * Check out [documentation](https://docs.warpstream.com/warpstream/configuration/schema-registry-beta) for more details #### Release v569 July 5th, 2024 * Add a new `warpstream_max_offset` metric tagged by topic/partition (that can be controlled with the existing `disableConsumerGroupsMetricsTags` flag). * The existing `warpstream_consumer_group_max_offset` metric is deprecated as it shares the same value across consumer groups and it can be replaced with the new metric mentioned above. * Add the `topic` label/tag to `warpstream_agent_kafka_produce_uncompressed_bytes_counter` that was missing it. #### Release v568 July 3rd, 2024 * Various small performance improvements (batching, allocations, etc). * Batch Metadata and FindCoordinator request RPCs between the Agents and the control plane. Helpful for workloads with a large number of consumer/producer clients. #### Release v567 June 19th, 2024 * Allow individual fetch requests to fetch up to 128MiB of uncompressed data per topic-partition, per fetch request, increased from 32 MiB. * Add `-kafkaMaxFetchRequestBytesUncompressedOverride` flag and `WARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDE` env variable to override maximum number of uncompressed bytes that can be fetched in a single fetch request. * Add `-kafkaMaxFetchPartitionBytesUncompressedOverride` and `WARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDE` env variable to override maximum number of uncompressed bytes that can be fetched for a single topic-partition in a single fetch request. * Introduced the metric `warpstream_consumer_group_generation_id` with the tags `consumer_group`. This metric indicates the generation number of the consumer group, incrementing by one with each rebalance. It serves as an effective indicator for detecting occurrences of rebalances. * Added metric `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` (tags: `consumer_group`, `topic`, `partition`). Provides an estimated lag in seconds (equivalent to `warpstream_consumer_group_lag` but in time units instead of offsets), calculated via interpolation/extrapolation. **Caution:** Very coarse; unsuitable for precise end-to-end latency measurement. * **Renamed Flags and Environment Variables**: * `benthosBucketURL` (`WARPSTREAM_BENTHOS_BUCKET_URL`) is now `bentoBucketURL` (`WARPSTREAM_BENTO_BUCKET_URL`): Bucket URL to use when fetching the Bento configuration. * `benthosConfigPath` (`WARPSTREAM_BENTHOS_CONFIG_PATH`) is now `bentoConfigPath` (`WARPSTREAM_BENTO_CONFIG_PATH`): Path in the bucket to fetch the Bento configuration. #### Release v566 June 5th, 2024 * Fix broken formatting of logs that were sometimes `logfmt` when they should've been JSON. #### Release v565 June 5th, 2024 * Improve the Agent's ability to handle certain pathological compactions where an input stream is not read from for so long that the object store closes the connection. The Agent will now detect this scenario and issue a new GET request to resume the compaction instead of failing it entirely and never making progress. #### Release v564 May 26th, 2024 * Improve performance of file cache for high throughput workloads by improving how batching is handled. * Add support for more Benthos components: `[awk, jsonpath, lang, msgpack, parquet, protobuf, xml, zstd]`. * Fix rare memory leak. * Fix rare bug that would cause circuit breakeres to get stuck in open / half-open state forever. #### Release v563 May 22nd, 2024 * Fixed a bug in the Agent that prevented users from starting playgrounds or demos. #### Release v562 May 19th, 2024 #### **Agent** * Make Agent batch size configurable using `-batchMaxSizeBytes` flag and `WARPSTREAM_BATCH_MAX_SIZE_BYTES` environment variable. * Metrics * Add new Agent metric (gauge) that indicates state of consumer group: `warpstream_consumer_group_state`. Metric has two tags: `consumer_group` and `group_state`. * Add new Agent metric (gauge) that indicates the number of members in each consumer group: `warpstream_consumer_group_num_members`. Metric has one primary tag: `consumer_group`. * Add new Agent metric (gauge) that indicates the number of topics in each consumer group: `warpstream_consumer_group_num_topics`. Metric has one primary tag: `consumer_group`. * Add new Agent metric (gauge) that indicates the number of partitions in each consumer group: `warpstream_consumer_group_num_partitions`. Metric has one primary tag: `consumer_group`. * Add new Agent metric (gauge) that indicates the total number of topics in the cluster: `warpstream_topic_count`. * Add new Agent metric (gauge) that indicates the limit for the total number of topics in the cluster: `warpstream_topic_count_limit`. * Add new Agent metric (gauge) that indicates the total number of partitions in the cluster: `warpstream_partition_count`. * Add new Agent metric (gauge) that indicates the limit for the total number of partitions in the cluster: `warpstream_partition_count_limit`. #### Release v559 May 13th, 2024 #### **Agent** * Add support for WarpStream Managed Data Pipelines * Change the default number of uncompressed bytes that will be be buffered before flushing a file to object storage from 8MiB to 4MiB #### Release v558 May 10th, 2024 #### **Agent** * Support adopting AWS roles for providing access to S3 via environment variables instead of a query argument in the bucket URL. * Tuned circuit breakers to be less aggressive by default, and configured the retry mechanisms to avoid retrying circuit breaker errors. * **New Metrics** * `warpstream_circuit_breaker_open`: Records the number of times the circuit breaker is opened, tagged by the circuit breaker name to identify the associated operations. - tags: `name` * `warpstream_circuit_breaker_close`: Records the number of times the circuit breaker is closed, tagged by the circuit breaker name to identify the associated operations. * tags: `name` * `warpstream_circuit_breaker_halfopen`: Records the number of times the circuit breaker is half-opened, tagged by the circuit breaker name to identify the associated operations. * tags: `name` * `warpstream_circuit_breaker_error`: Records the number of times the circuit breaker blocks an operation, tagged by the circuit breaker name to identify the associated operations. * tags: `name` #### Release v557 May 9th, 2024 #### **Agent** * **Enhanced Buckets:** Refined the buckets for `warpstream_agent_kafka_fetch_uncompressed_bytes`, `warpstream_agent_kafka_fetch_compressed_bytes`, `warpstream_agent_kafka_produce_compressed_bytes` and `warpstream_agent_kafka_produce_uncompressed_bytes` to ensure they are more precise. The previous bucket configuration was too sparse. * **Fixed Prometheus Counters:** Resolved an issue with Prometheus counters previously were not appropriately incrementing the value. #### Release v556 May 9th, 2024 #### **Agent** * Tune circuit breakers to be less aggressive, and don't retry requests that failed due to a circuit breaker error. * **New metrics**: * `warpstream_circuit_breaker_open`: Records the number of times the circuit breaker is opened, tagged by the circuit breaker name to identify the associated operations. * tags: `name` * `warpstream_circuit_breaker_close`: Records the number of times the circuit breaker is closed, tagged by the circuit breaker name to identify the associated operations. * tags: `name` * `warpstream_circuit_breaker_halfopen`: Records the number of times the circuit breaker is half-opened, tagged by the circuit breaker name to identify the associated operations. * tags: `name` * `warpstream_circuit_breaker_error`: Records the number of times the circuit breaker blocks an operation, tagged by the circuit breaker name to identify the associated operations. * tags: `name` #### Release v555 May 8th, 2024 #### **Agent** * **New metrics**: * `warpstream_agent_kafka_fetch_uncompressed_bytes_counter`: Tracks the count of uncompressed bytes fetched. Although a histogram version (`warpstream_agent_kafka_fetch_uncompressed_bytes`) already exists, it may not provide an accurate count in some metrics systems. That's why we've made this data directly available as a counter. * tags: `topic` (requires enabling high-cardinality metrics) * `agent_kafka_produce_uncompressed_bytes_counter`: Tracks the count of uncompressed bytes produced. While the histogram version (`warpstream_agent_kafka_produce_uncompressed_bytes`) is available, some metrics systems might not accurately count it. So, we're also offering this data directly as a counter. * tags: `topic` (requires enabling high-cardinality metrics) #### Release v554 April 30th, 2024 #### **Agent** * Replaces all instances of Ristretto cache with a simple LRU cache, dramatically reduces the number of caches misses in topic metadata related caches. #### Release v553 April 30th, 2024 #### **Agent** * **Rename metrics**: * `warpstream_agent_kafka_inflight_conn` has been renamed to `warpstream_agent_kafka_inflight_connections` * `warpstream_agent_kafka_inflight_request` has been renamed to `warpstream_agent_kafka_inflight_requests` #### Release v552 April 29th, 2024 #### **Agent** * Dramatically improve the performance of fetch requests that query 100s or 1000s of partitions in a single fetch request. * Add speculative retries for blob storage reads in the fetch path (in addition to existing speculative retries for blob storage writes in the write path). #### Release v551 April 26th, 2024 #### **Agent** * Convert an error log (that did not actually represent an error) to an info log to reduce error spam. #### Release v550 April 23nd, 2024 #### Agent * **High-Cardinality Metrics:** Introduced a new agent configuration for enabling high-cardinality metrics. To activate this feature, use the command-line flag `-kafkaHighCardinalityMetrics` or set the environment variable `WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS=true`. * **Deprecated metrics**: * `warpstream_agent_kafka_fetch_bytes_sent` * `warpstream_agent_segment_batcher_flush_file_size_counter_type` * `warpstream_agent_segment_batcher_flush_file_size` * `blob_store_list_latency` * `blob_store_list_count` * `blob_store_delete_latency` * `blob_store_delete_count` * `blob_store_put_bytes_latency` * `blob_store_put_bytes_count` * `blob_store_put_stream_latency` * `blob_store_put_stream_count` * `blob_store_get_bytes_latency` * `blob_store_get_bytes_count` * `blob_store_get_stream_latency` * `blob_store_get_stream_count` * `blob_store_get_bytes_range_latency` * `blob_store_get_bytes_range_count` * `blob_store_get_stream_range_latency` * `blob_store_get_stream_range_count` * **New metrics**: * `warpstream_agent_kafka_fetch_uncompressed_bytes`: Tracks the total uncompressed bytes fetched, replacing `warpstream_agent_kafka_fetch_bytes_sent`. * tags: `topic` (requires enabling high-cardinality metrics) * `warpstream_agent_kafka_produce_uncompressed_bytes`: Tracks the number of uncompressed bytes produced. * tags: `topic` (requires enabling high-cardinality metrics) * `warpstream_agent_segment_batcher_flush_file_size_uncompressed_bytes`: Tracks the uncompressed size of files stored after batching, serving as a replacement for `warpstream_agent_segment_batcher_flush_file_size`. * `warpstream_blob_store_operation_latency`: Tracks the latency and count of individual object storage operations. It's a replacement for all deprecated `blob_store_` metrics. Now all the different operations are tagged within the same metric. * tags: `operation,outcome` #### Release v549 April 22nd, 2024 #### Agent 1. Adds support for authenticating Kafka clients using mTLS. A `-requireMTLSAuthentication` flag is added, and the previous `-tlsVerifyClientCert` flag has been deprecated. A new `-requireSASLAuthentication` flag is added, and the previous `-requireAuthentication` flag is deprecated. When authenticating using mTLS, the Distinguished Name(DN) from the client certificate is used as the Kafka Principal. The `-tlsPrincipalMappingRule` flag can be used to specify a Regex to extract a principal from the DN. For example, the rule `CN=([^,]+)` will extract the Common Name(CN) from the DN, and use that as the ACL principal. #### Release v548 April 17th, 2024 #### Agent 1. Fix a rare bug where some compactions could fail if you had tombstone expiration enabled and very little data to compact. #### Release v547 April 11nd, 2024 #### Agent 1. Fix playground mode against latest control plane signup constraints #### Release v545 April 4nd, 2024 #### Agent 1. Added support to new regions. There is a new "-region" (or `WARPSTREAM_REGION` env variable) that you can leverage to pick among the supported regions - you can use the console to see the current list. #### Release v544 April 2nd, 2024 #### Agent 1. Added support for TLS/mTLS for Warpstream Agent and kafka client connections. The `-kafkaTLS`(env `WARPSTREAM_TLS_ENABLED=true`), `-tlsServerCertFile`(env `WARPSTREAM_TLS_SERVER_CERT_FILE=`), `tlsServerPrivateKeyFile`(env `WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE=`), `-tlsVerifyClientCert`(env `WARPSTREAM_TLS_VERIFY_CLIENT_CERT=true`), `-tlsClientCACertFile`(env `WARPSTREAM_TLS_CLIENT_CA_CERT_FILE=`) flags were added which respectively are used to enable tls, pass the TLS server certificate to the Agent, pass the TLS server private key to the Agent, optionally enable client certificate verification, and to optionally pass the TLS root certificate authority certificate file to the server. #### Release v543 March 29th, 2024 #### Agent 1. Adding a new "availabilityZoneRequired" (or env variable "WARPSTREAM\_AVAILABILITY\_ZONE\_REQUIRED") flag. When enabled, the agent will synchronously try to resolve its availability zone during startup for 1 min, and will not start serving its /v1/status health check until it succeeds. The process will exit early if it did not manage to resolve the availability zone. #### Release v542 March 26th, 2024 #### Agent 1. Fixed a race condition during startup that could make some agents advertise their availability zone as "WARPSTREAM\_UNSET\_AZ" #### Release v541 March 25th, 2024 #### Agent 1. Added beta support for [benthos](https://www.benthos.dev/) in the WarpStream Agents #### Release v539 March 15th, 2024 #### Agent 1. Adding a new "agentGroup" parameter to define the name of the 'group' that the Agent belongs to. This feature is used to isolate groups of Agents that belong to the same logical cluster, but should not communicate with each other because they're deployed in separate cloud accounts, vpcs, or regions. By default the agent belongs to the default group. #### Release v538 March 6th, 2024 #### Agent 1. Fixed a rare panic in the Agents caught by Antithesis #### Release v537 March 1st, 2024 #### Agent 1. Kafka "compacted topics" are generally available. #### Release v536 February 28th, 2024 #### Agent 1. Fix a bug with the `warpstream playground` command, introduced in v535. #### Release v535 February 26th, 2024 #### Agent 1. Make agent pool name argument completely optional, even when using non-default clusters. 2. Fixed a memory leak that would cause some workloads to use an excessive amount of memory over time. 3. Add support for S3 express and separating the "ingestion" bucket from the "compaction" bucket so data can be landed into low-latency storage and then immediately compacted into lower cost storage. 4. Add speculative retries to file flushing which dramatically reduces outlier latency for Produce requests. #### Release v534 #### Agent **Bug fixes** 1. Fixed another bug in the fetch logic that would result in the Agent returning empty batches in some scenarios when transient network errors occurred. This did not cause any correctness issues, but would make librdkafka refuse to proceed in some scenarios and block consumption of some partitions. #### Release v533 #### Agent **Bug fixes** 1. Fixed a bug in the fetch logic that would result in the Agent returning empty batches in some scenarios depending on the client configuration. This did not cause any correctness issues, but would make librdkafka refuse to proceed in some scenarios and block consumption of some partitions. #### Release v532 #### Agent **New Features** 1. Improves roles reporting to Warpstream control plane so that they can be properly rendered in our console. #### Release v531 #### Agent **New Features** 1. Supports automatic availability zone detection in kubernetes reading the node zone label (requires version `0.10.0` of our [helm charts](https://github.com/warpstreamlabs/charts)). #### Release v530 #### Agent **New Features** 1. Compatibility with CreatePartitions Kafka API for updating partitions count. #### Release v529 #### Agent **New Features** 1. Treat grade nat as private IP addresses, and allow advertising it. **Performance improvements** 1. Improve caching in the agent. 2. Improve batching of requests to the warpstream metadata backend. 3. Other general performance improvements. #### Release v526 #### Agent **New Features** 1. Full-support for ACLs using SASL credentials. #### Release v525 #### Agent **Bug Fixes and performance Improvements** 1. Improved object pooling in RPC layer to reduce memory usage. 2. Add transparent batching to the file cache RPCs to dramatically reduce the number of inter-agent RPCs for high partition workloads. 3. Enhanced Apache Kafka compatibility by refining the handling of watermarks in Fetch responses. #### Release v524 #### Agent **Dynamic Consumer Group Rebalance Timeout** 1. Partially handle consumer group requests (JoinGroup and SyncGroup) in the agent, to use the clients' rebalance timeout, instead of the previous 10s default. This enhancement will minimize unnecessary rebalance attempts in larger consumer groups with numerous members, due to short timeouts. #### Release v523 #### Agent **Bug Fixes and performance Improvements** 1. Fixed a bug in the Fetch() code that was not setting the correct topic ID in error responses which made some Kafka clients emit warning logs when this happened. 2. Fixed a bug in the "roles" feature that was causing Agents with the "produce" role to still participate in the distributed file cache. Now only Agents with the "consume" role will participate in the file cache, as expected. #### Release v522 #### Agent **Bug Fixes and performance Improvements** 1. Circuit breakers will now return example errors for clarity. 2. Fetch() code path will now handle failures more gracefully by returning incremental results in more scenarios which improves the system's ability to recover under load. 3. Fix a memory leak in the in-memory file cache implementation. #### Release v521 #### Agent **New Features** 1. Docker images are now multi-arch, our documentation and official kubernetes charts has been updated accordingly. 2. Introduced circuit breakers around object storage access. 3. Finer control over agent roles: it is now possible to split between the `proxy-consume` and `proxy-produce` roles, our documentation has been updated as well. #### Release v520 #### Agent This release is the first phase of a two-phase upgrade to WarpStream's internal file format. This release adds support for reading the upgraded file format. You **MUST** upgrade all Agents to this version before moving from any version < v520 to any version > than v520. #### Release v518 #### Agent **New Features** 1. Support kafka Headers: if you produce messages containing Kafka headers, they will now be automatically persisted to your cloud object storage, and will be read when fetching. 2. Revisit the flags and configuration knobs to choose how the agents advertise themselves in Warpstream service discovery. Our documentation has been updated accordingly. 3. Agent nodes can now be configured to run dedicated roles - see [splitting roles documentation](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles). #### Control Plane **New Features** 1. Fully support kafka `ListOffsets` protocol: you can now look for partition offsets based on timestamps. #### Release v517 #### Agent **Bug Fixes and performance Improvements** 1. Fixed a bug related to the handling of empty (but not null) values in records in the Fetch implementation. #### Release v516 #### Agent **New Features** 1. The agent will now report a sample of its error logs back to Warpstream control plane. It should ease troubleshooting and help us identify issues earlier. This can be disabled with the flag `disableLogsCollection` or the environment variable `WARPSTREAM_DISABLE_LOGS_COLLECTION`. **Bug Fixes and performance Improvements** 1. Added batching in the metadata calls made during Kafka `Fetch`, improving memory usage along the way. #### Control Plane **New Features** 1. Added support for Kafka's `InitProducerID` protocol message, and the idempotent producer functionality in general. Requires upgrading to a version of the Agents that is >= `v515`. 2. Added support for Kafka `ListOffsets` with positive timestamps value (until now only negative values for special cases were supported) #### Release v515 #### Agent **New Features** 1. The Agents will now report the lag / max offsets for every active consumer group as standard metrics. The metrics can be found as `warpstream_consumer_group_lag` and `warpstream_consumer_group_max_offset` . 2. The Agents will now report the number of files at each compaction level so that user's can monitor whether they are experiencing compaction lag. These metrics can be found as `warpstream_files_count` and the level is tagged with the name `compaction_level`. **Bug Fixes and performance Improvements** 1. File cache is now partitioned by `` instead of just ``. This spreads the load for fetching data for large files more evenly amongst all the Agents. 2. Added some logic in the file cache to detect when certain parts of the cache are experiencing high churn and reduce the default IO size for paging in data from object storage. This helps avoid filling the cache with data that won't be read. 3. Fixed a bug in the file cache that was causing it to significantly \*over\* fetch data in some scenarios. This did not cause any correctness problems, but it wasted network bandwidth and CPU cycles. 4. Modified the implementation of the Kafka `Fetch` method to return incremental results when it experiences a retryable error mid-fetch. This makes the Agents much better at recovering from disruption and catching consumer lag incrementally. 5. Added some pre-fetching logic into the Kafka `Fetch` method so that when data for a single partition is spread amongst many files the Agent doesn't get bottlenecked making many single-threaded RPCs. This mostly helps increase the speed at which individual partitions can be "caught up" when lagging. 6. Increased the default maximum file size created at ingestion time from 4MiB to 8MiB. This improves performance for extremely high volume workloads. 7. Added replication to the Agent file cache so that if an error is experienced trying to load data from the file cache on the Agent node that is "responsible" for a chunk of data, the client can retry on a different node. This helps minimize disruption when Agents shutdown ungracefully. 8. Agents now report their CPU utilization to the control plane. We will use this information in the future to improve load balancing decisions. CPU utilization can be view in the WarpStream Admin console now as well. 9. Improved the performance of deserializing file footers. 10. Standardized prometheus metric names prefixes. 11. Added a lot more metrics and instrumentation, especially around the blob storage library and file cache. #### Control Plane **New Features** 1. Added support for the Kafka protocol message `DeleteTopics` . **Bug Fixes and Performance Improvements** 1. We added intelligent throttling / scheduling to the deadscanner scheduler. This scheduler is responsible for scheduling jobs that run in the Agent to scan for "dead files" in object storage and delete them. Previously these jobs could run with high frequency and rates which would interfere with live workloads. In addition, they could also result in very high object storage API requests costs due to excessive amounts of `LIST` requests. The new implementation is much more intelligent and automatically tunes the frequency to avoid disrupting the live workload and incurring high API request fees. # Migrations #### Release v802 If you are using the [AWS Glue integration in Tableflow](https://docs.warpstream.com/warpstream/tableflow/catalogs-and-query-engines/aws-glue), two new actions are needed in your IAM policy: * `glue:GetTableVersions` * `glue:BatchDeleteTableVersion` #### Release v793 If you are whitelisting the WarpStream control plane endpoint (like `metadata.default.us-east-1.warpstream.com`) you need to add additional endpoints before migrating. The exact format is: ``` .metadata...warpstream.com ``` With the following fields: * `prefix` will be `alpha`, `beta` and `gamma` , meaning you need to whitelist 3 different values (always the 3 values, no matter what the region/cloud provider is) * `region` is what you picked in the console UI or through the API, like `us-east-1` or `eastus` * `cloud_provider` is the cloud provider of the region you pickred in the console UI or through the API. We support: `aws` / `gcp` / `azure` Concretely let's say you picked `aws`/`us-east-1` in the console, you want to whitelist the 3 endpoints: * `alpha.metadata.us-east-1.aws.warpstream.com` * `beta.metadata.us-east-1.aws.warpstream.com` * `gamma.metadata.us-east-1.aws.warpstream.com` # Install the WarpStream Agent / CLI This page contains instructions for installing the WarpStream Agent via CURL, Docker, or raw binaries. WarpStream packages the Agent and a utility CLI into a single binary. The WarpStream Agent / CLI can be installed in one of three ways: 1. With our Docker container from our public Docker registry. 2. With our installation script. 3. Downloading the binary for your platform directly. {% tabs %} {% tab title="Installation Script" %} ```bash curl https://console.warpstream.com/install.sh | bash ``` This option is recommended for local development. {% hint style="info" %} Make sure to follow the instructions at the end of the installation script to update your bashrc/zshrc file. {% endhint %} {% endtab %} {% tab title="Docker" %} {% hint style="info" %} We have special docker tags `latest` and `latest-stable` in our ECR repos. `latest` is always the latest published version, while `latest-stable` corresponds to the most recent version that is at least one month old. {% endhint %} We host our public Docker registry using Amazon ECR. Our docker image is multi arch: linux x86-64 and arm-64 are supported. {% embed url="" %} **Quick Run Commands (Playground)** Port `8080` for Kinesis/Prometheus metrics, port `9092` for Apache Kafka-compatible API. In playground mode, the Agent will advertise its IP address as localhost instead of the Docker internal IP address by default. This will allow you to connect to the agent from a client running locally. You can change this behavior with the `advertiseHostnameStrategy` flag. If you're trying to run the WarpStream Agent as part of a larger `docker-compose` setup, then check out our reference on [how to run WarpStream in docker-compose](/warpstream/reference/integrations/use-the-agent-in-docker-compose) as you'll need to configure a different set of environment variables for service discovery to work properly. {% code title="playground" overflow="wrap" %} ```bash docker run -p 8080 -p 9092:9092 public.ecr.aws/warpstream-labs/warpstream_agent:latest playground ``` {% endcode %} **Quick Run Commands (Demo)** {% code title="demo" overflow="wrap" %} ```bash docker run public.ecr.aws/warpstream-labs/warpstream_agent:latest demo ``` {% endcode %} {% endtab %} {% tab title="Binaries" %} The links below are for the latest version of the WarpStream Agent binary. Latest version will always be the latest version published, while latest stable is the latest version that has been published for at least one month.
PlatformLatestLatest Stable
linux/amd64DownloadDownload
linux/arm64DownloadDownload
darwin(macOS)/amd64DownloadDownload
darwin(macOS)/arm64DownloadDownload
If you want to download the WarpStream Agent binary as of a specific version, use the URL templates below to select your architecture and then replace the value of `$VERSION` with the version tag from the [change log](/warpstream/overview/change-log).
PlatformURL
linux/amd64https://warpstream-public-us-east-1.s3.amazonaws.com/warpstream_agent_releases/warpstream_agent_linux_amd64_$VERSION.tar.gz
linux/arm64https://warpstream-public-us-east-1.s3.amazonaws.com/warpstream_agent_releases/warpstream_agent_linux_arm64_$VERSION.tar.gz
darwin(macOS)/amd64https://warpstream-public-us-east-1.s3.amazonaws.com/warpstream_agent_releases/warpstream_agent_darwin_amd64_$VERSION.tar.gz
darwin(macOS)/arm64https://warpstream-public-us-east-1.s3.amazonaws.com/warpstream_agent_releases/warpstream_agent_darwin_arm64_$VERSION.tar.gz
{% endtab %} {% tab title="Brew" %} ``` brew tap warpstreamlabs/homebrew-tap brew install warpstreamlabs/homebrew-tap/warpstream ``` This option is recommended for local development on Linux/MacOS. The tap can be found [here](https://github.com/warpstreamlabs/homebrew-tap). {% endtab %} {% endtabs %} Once installed, you can run `warpstream demo` in your terminal which will: 1. Automatically sign you up for a temporary account that is valid for 12 hours. 2. Run an in-memory Kafka producer that will produce small JSON documents to a stream periodically. 3. Run an in-memory Kafka consumer that consumes the JSON documents and prints them to the standard console. 4. Open up your web browser to the WarpStream console so you can explore the WarpStream U.I and feature set in more detail. # Run the Agents Locally Instructions on how to run the WarpStream Agent locally for testing / development purposes. First, [install the WarpStream Agent](/warpstream/getting-started/install-the-warpstream-agent) for your platform. ## Playground vs. Local There are two ways to run WarpStream locally: 1. `playground` mode 2. `local` mode `playground` mode is designed for interactive local development. It signs up for a temporary WarpStream account that is valid for a few hours, starts a local Agent with an embedded Kafka cluster, Schema Registry cluster, and Tableflow cluster, and uses the real hosted WarpStream control plane so that you have full access to WarpStream's featureset, API, and UI to explore as much of the product as possible. If you just want to test something manually, or explore the product, this is the best option. `local` mode is designed for non-interactive local development like automated CI environments. Unlike `playground` mode, `local` mode has no dependency on WarpStream's hosted control plane and instead uses a fake in-memory control plane in the Agent binary itself. This makes it suitable for CI environments where hundreds or even thousands of ephemeral WarpStream clusters need to be spawned simultaneously without being subjected to ratelimits. {% hint style="warning" %} Both `playground` and `local` mode store data in memory, so any data written will no longer be accessible once the process exits. They're also heavily ratelimited in terms of the amount of Kafka traffic they can handle and are not suitable at all for benchmarking. {% endhint %} ## Docker {% tabs %} {% tab title="Playground" %} {% code overflow="wrap" %} ```bash docker run -p 8080 -p 9092:9092 -p 9094:9094 public.ecr.aws/warpstream-labs/warpstream_agent:latest playground ``` {% endcode %} {% endtab %} {% tab title="Local" %} {% code overflow="wrap" %} ```bash docker run -p 8080 -p 9092:9092 -p 9094:9094 public.ecr.aws/warpstream-labs/warpstream_agent:latest local ``` {% endcode %} {% endtab %} {% endtabs %} Once the docker container is running, there will be a Kafka TCP server listening on port 9092 and a Schema Registry HTTP server listening on port 9094. This means you're ready to run any application locally that expects to connect to Kafka, and it'll connect to WarpStream instead if you set the bootstrap URL to `localhost:9092`. You can also replace the URL of your schema registry clients to `localhost:9094` and it'll connect to WarpStream's Schema Registry instead. If you encounter any problems connecting an application running *outside* of Docker to the WarpStream agent running inside of Docker, follow [our instructions below for diagnosing connection issues](#diagnosing-connection-issues). ## Standalone Binary Alternatively, if you installed the standalone WarpStream Agent binary and it is in your `PATH`, you can just run: {% tabs %} {% tab title="Playground" %} {% code overflow="wrap" %} ```bash warpstream playground ``` {% endcode %} {% endtab %} {% tab title="Local" %} {% code overflow="wrap" %} ```bash warpstream local ``` {% endcode %} {% endtab %} {% endtabs %} {% hint style="warning" %} Both `playground` and `local` mode store data in memory, so any data written will no longer be accessible once the process exits. {% endhint %} Once that completes, run the following command to test the Kafka connection: {% code overflow="wrap" %} ```bash warpstream kcmd -type diagnose-connection -bootstrap-host localhost -bootstrap-port 9092 ``` {% endcode %} If that succeeds, then you're ready to run any application locally that expects to connect to Apache Kafka, and it'll connect to WarpStream instead if you set the bootstrap URL to `localhost:9092`. If the diagnostic command returns an error, follow the provided instructions to diagnose and fix it. To test the Schema Registry connection, you can send a request to the server with `curl` as follows: ```bash curl -X POST "http://localhost:9094/subjects/foo/versions" \ -H "Content-Type: application/json" \ -d '{"schema": "{\"type\":\"long\"}"}' ``` If you receive a valid response, such as `{"id":1}`, then you're ready to run any application locally that expects to connect to a schema registry by replacing schema registry URL with `localhost:9094`. ## Diagnosing Connection Issues The WarpStream Agent binary ships with a utility for diagnosing connection issues. However, diagnosing connection issues cannot be done in a general purpose manner from within a Docker container. Therefore even if you're running the Agent in a Docker container locally, you'll need to follow our ["Installation Script" Agent installation instructions](/warpstream/getting-started/install-the-warpstream-agent) to install the raw WarpStream Agent binary locally before proceeding. Once the binary is installed, run the following command to test the connection: {% code overflow="wrap" %} ```bash warpstream kcmd -type diagnose-connection -bootstrap-host localhost -bootstrap-port 9092 ``` {% endcode %} If that succeeds, then you're ready to run any application locally that expects to connect to Apache Kafka, and it'll connect to WarpStream instead if you set the bootstrap URL to `localhost:9092`. If the diagnostic command returns an error, follow the provided instructions to diagnose and fix. # "Hello World" for Apache Kafka This page goes through the basic "Hello World" Kafka functionality (produce and consume) with WarpStream. First, [install the WarpStream Agent](/warpstream/getting-started/install-the-warpstream-agent) for your platform. ### Running the WarpStream Agent ```shell warpstream playground ``` The `playground` command will start an Agent on `localhost`, store all the data for the Agent in memory, and sign up for a temporary account for you to play around with. ### Create a Topic Let's create a topic using the Apache Kafka client built-in to the WarpStream Agent. We're going to assume in this tutorial you're running the Agent as specified above on localhost, so we'll omit the arguments to specify the Kafka bootstrap URL. Open a second terminal and issue the following command: ``` warpstream kcmd --type create-topic --topic helloworld2 ``` If you don't receive an error back, your request to create the topic succeeded. ### Add Records To The Topic Now let's write a record to the topic. The `kcmd` tool writes records with a constant key of "hello", and we'll write two records with the payload of "`world`" as our example (`,,` ) is the delimiter). 
warpstream kcmd --type produce --topic helloworld2 --records "world,,world"
You should receive output in your terminal that looks something like this: ``` result: partition:0 offset:0 value:"world" result: partition:0 offset:1 value:"world" ``` ### Read Records From The Topic ``` warpstream kcmd --type fetch --topic helloworld2 --offset 0 ``` Now you should see your records you wrote previously printed to the console. The key is always "hello" and the value field is the same "world" string we wrote before. You can repeat the process of writing another record and running `kcmd --type fetch` again to read and write more records. ``` result: partition:0 offset:0 key:"hello" value:"world" result: partition:0 offset:1 key:"hello" value:"world" ``` **And that's it!** You've successfully set up a WarpStream Agent Pool to power your Virtual Cluster, created a topic, and processed some data through it. You can now move on to trying to create a real application which reads or writes from the topic, or connect your existing tools like Flink to WarpStream. # Deploy the Agents How to deploy the Agents. {% hint style="warning" %} Remember to review our documentation on [how to configure your Kafka client for WarpStream](/warpstream/kafka/configure-kafka-client), as well as our instructions on [tuning for performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) once you're done. A few small changes in client configuration can result in 10-20x higher throughput when using WarpStream, and proper client configuration is required to leverage WarpStream's zone-aware discovery system. {% endhint %} ## Overview This page describes how to deploy the WarpStream Agents into your environments. Note that while WarpStream has several different products / cluster types: [Kafka](https://github.com/warpstreamlabs/docs/blob/master/agent-setup/deploy/broken-reference/README.md), [Schema Registry](https://github.com/warpstreamlabs/docs/blob/master/agent-setup/deploy/broken-reference/README.md), and [Tableflow](/warpstream/reference/integrations/timeplus), they're all deployed using the same Agent binary. Therefore you can follow the instructions below regardless of which product you're deploying, and if there are any product-specific instructions, they'll be called out explicitly. ## Required Arguments The WarpStream Agent is completely stateless and thus can be deployed however you prefer to deploy stateless containers. For example, you could use AWS ECS or a Kubernetes Deployment. The WarpStream Docker containers can be found in the [installation docs](/warpstream/getting-started/install-the-warpstream-agent#docker). However, if you're deploying WarpStream into a Kubernetes cluster, we highly recommend using our [official Helm charts](/warpstream/agent-setup/infrastructure-as-code/helm-charts). Similarly, for AWS ECS we have a dedicated [terraform module](https://github.com/warpstreamlabs/terraform-aws-warpstream-ecs). The Agent has four required arguments that must be passed as command line flags: 1. `bucketURL` 2. `agentKey` 3. `defaultVirtualClusterID` 4. `region` For example: ```bash docker run public.ecr.aws/warpstream-labs/warpstream_agent:latest \ agent \ -bucketURL "s3://$S3_BUCKET?region=$S3_BUCKET_REGION" \ -agentKey $AGENT_KEY \ -defaultVirtualClusterID $YOUR_VIRTUAL_CLUSTER_ID -region $CLUSTER_REGION ``` The values of `agentKey` , `defaultVirtualClusterID` , and `region` can be obtained from the [WarpStream Admin Console](https://console.warpstream.com). {% hint style="info" %} Note that the entrypoint for the WarpStream docker image is a multi-command binary. For production usage, the subcommand that you want to run is just called `agent` as shown above. {% endhint %} Depending on the tool you're using to deploy/run containers, it can sometimes be cumbersome to provide additional arguments beyond the `agent` subcommand. In that case, all of the required arguments can be passed as environment variables instead: 1. `WARPSTREAM_BUCKET_URL` 2. `WARPSTREAM_AGENT_KEY` 3. `WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID` 4. `WARPSTREAM_REGION` ### Object Storage `bucketURL` is the URL of the object storage bucket that the WarpStream Agent should write to. See [our documentation](/warpstream/agent-setup/different-object-stores#bucket-url-construction) on how to construct a proper URL for the specific object storage implementation that you're using. The `Deploy` tab in the WarpStream UI for your BYOC cluster also has a utility to help you construct a well formed URL. In addition to constructing a well-formed `bucketURL`, you'll also need to create and configure a dedicated object storage bucket for the Agents, and ensure that the Agents have the appropriate permissions to access that bucket. See [our documentation](/warpstream/agent-setup/different-object-stores#bucket-permissions) on how to do that correctly. If you are [using a bucket prefix](/warpstream/agent-setup/different-object-stores#using-a-bucket-prefix), please ensure that your bucketURL is in quotes to prevent any interpolation when running from the command line ### Region The `region` flag corresponds to the region that the WarpStream control plane is running in. This corresponds to the value that was selected when the BYOC cluster was created and can be obtained from the WarpStream UI. This value does not need to correspond to the cloud region that the Agents are deployed in, but you should pick the region that is closest to where your Agents are deployed to minimize latency. Currently supported regions for BYOC are: | Cloud provider | Region | URL | | -------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | AWS | `us-east-1` | | | AWS | `us-east-2` | [https://metadata.default.us-east-2.warpstream.com](https://metadata.default.us-east-1.warpstream.com) | | AWS | `us-west-2` | | | AWS | `eu-central-1` | | | AWS | `eu-west-1` | [https://metadata.default.eu-west-1.warpstream.com](https://metadata.default.eu-central-1.warpstream.com) | | AWS | `ap-southeast-1` | | | AWS | `ap-southeast-2` | [https://metadata.default.ap-southeast-2.warpstream.com](https://metadata.default.ap-southeast-1.warpstream.com) | | AWS | `ap-south-1` | [https://metadata.default.ap-south-1.warpstream.com](https://metadata.default.ap-southeast-1.warpstream.com) | | AWS | `ap-northeast-1` | [https://metadata.default.ap-northeast-1.warpstream.com](https://metadata.default.eu-central-1.warpstream.com) | | GCP | `us-central1` | | | GCP | `northamerica-northeast1` | [https://metadata.default.northamerica-northeast1.gcp.warpstream.com](https://metadata.default.us-central1.gcp.warpstream.com) | | GCP | `europe-west1` | | | GCP | `asia-south1` | | | Azure | `eastus` | | You can contact us to request a new region by sending an email to . ## Permissions and Ports The WarpStream Agents need permission to perform various different operations against the object storage bucket. Review [our object storage permissions documentation](/warpstream/agent-setup/different-object-stores#bucket-permissions) for more details. In addition to object storage access, the WarpStream Agent will also need permission to communicate with the control plane URL (see table above) in order to write/read Virtual Cluster metadata. Raw data flowing through your WarpStream will **never** leave your cloud account, only metadata required to order batches of data and perform remote consensus. You can read more about what metadata leaves your cloud account in our [security and privacy considerations documentation](/warpstream/reference/security-and-privacy-considerations). Finally, the WarpStream Agent requires 2 ports to be exposed. For simplicity, we recommend just ensuring that the WarpStream Agent can listen on ports `9092` (or `9094` in the case of schema registry) and `8080` by default; however, the section below contains more details about how each port is used and how to override them if necessary. {% tabs %} {% tab title="Kafka Port" %} Default: `9092` Override: `-kafkaPort $PORT` Disable: `-enableKafka false` This is the port that exposes the Kafka TCP protocol to Kafka clients. Only disable it if you don't intend to use the Kafka protocol at all. {% endtab %} {% tab title="HTTP Port" %} Default: `8080` Override: `-httpPort $PORT` User for inter-agent communication within a single availability zone so the Agents can form a [distributed file cache](/warpstream/overview/architecture/read-path) with each other. Also used to expose Prometheus metrics. {% endtab %} {% tab title="Schema Registry Port" %} Default: `9094` Override: `-schemaRegistryPort $PORT` You can also use the env variable `WARPSTREAM_SCHEMA_REGISTRY_PORT` to override the port. This is the port that exposes the Schema Registry HTTP server. {% endtab %} {% endtabs %} ## Service discovery The `advertiseHostnameStrategy` flag allows you to choose how the agent will advertise itself in Warpstream service discovery (more details [here](/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy)). The default `auto-ip4` is a good choice for most cases in production. ## GOMAXPROCS The WarpStream Agent uses heuristics to automatically configure itself based on the available resources. The most important way this happens is by adjusting concurrency and cache sizes based on the number of available cores. The Agent uses standard operating system APIs to determine how many cores are available, and it prints this value when starting: {% code overflow="wrap" %} ```bash 2023/08/31 09:21:22 maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined ``` {% endcode %} This number is *usually* right, but it may not be right depending on how the Agent is deployed. For example, the Agent may determine the wrong value when running in [AWS ECS](https://github.com/uber-go/automaxprocs/issues/66). In general, we recommend that you manually set the `GOMAXPROCS` environment variable to the number of cores that you've made available to the Agent in your environment. For example, if you've allocated 3 cores to the Agent's container, then we recommend adding `GOMAXPROCS=3` as an environment variable. The value of `GOMAXPROCS` must be a whole number and not a fraction. We also recommend that you always assign Agent whole numbers for CPU quotas so that the Agent doesn't have fractional CPU quotas. Fractional CPU quotas can result in throttling and increased latency since the value of `GOMAXPROCS` and the number of whole cores available to the Agent won't match. ## Instance Selection While the WarpStream Agents don't store data on local disks, they do use the network heavily. Therefore we recommend using network-optimized cloud instances that provide at least 4GiB of RAM per vCPU. We also recommend using **dedicated** instances and not bin-packing the Agent containers to avoid noisy neighbor issues where another container running on the same VM as the Agents causes network saturation. In AWS, we think the `m5n` and `m6in` series are a great choice for running the Agents. In GCP, the `n4` series is a great choice with the `c4` series as a close second. In production, we recommend running the Agents with *at least* 4 vCPUs available and providing at least 4 GiB of RAM per vCPU, therefore the `m5n.large` and `m6in.large` are the minimum recommended instance sizes in AWS. Using much larger instances is fine as well; just make sure to set the value of [GOMAXPROCS](#gomaxprocs) to ensure the Agent can make use of all the available cores even when running in a containerized environment (our helm chart does this automatically). ### Network Optimized Instances The Agent does a *lot* of networking to service Apache Kafka Produce and Fetch requests, as well as perform background compaction. The Agent uses compression and intelligent caching to minimize this, but fundamentally, WarpStream is a data-intensive system that is even more networking-heavy than Apache Kafka due to reliance on remote storage. Debugging latency caused networking bottlenecks and throttling is a *nightmare* in all cloud environments. None of the major clouds provide sufficient instrumentation or observability to understand why or if your VM's network is being throttled. Some have dynamic throttling policies that allow long bursts but suddenly degrade with no explanation. For all of these reasons, we recommend running the WarpStream Agents on network-optimized instances, which allows the Agents to saturate their CPU before saturating the network interface. That situation is easier to understand, observe, and auto-scale on. ## Auto-Scaling When running the Agent on the appropriate instance type as described above, we recommend auto-scaling based on CPU usage with a target of 50% average usage. Our internal testing workload runs the Agent at more than 75% CPU usage with little latency degradation, but choosing an appropriate threshold requires balancing the concerns of cost efficiency and responsiveness to bursts of traffic that happen faster than your auto-scaler can react. If you're using our [official helm charts](https://docs.warpstream.com/warpstream/byoc/infrastucture-as-code/helm-charts), auto-scaling can be [enabled trivially](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent#scaling). ### Zone-Specific Scaling Kubernetes will usually deploy the WarpStream Agents spread equally across multiple availability zones. When the horizontal pod autoscaler (HPA) takes action, those actions will be taken equally across all zones. This approach works well for most applications, but if you've enabled [zone-aware routing](/warpstream/kafka/configure-kafka-client/configure-clients-to-eliminate-az-networking-costs) in the WarpStream Agents then this could be a problem. For example, imagine a scenario where the WarpStream Agents are deployed across zones A, B, and C, with an equal number of Agents in each zone, but zone A has three times as much client traffic as zones B and C. In this scenario, the *average* CPU utilization of all the Agents across all the zones may not be enough for Kubernetes to trigger an up-scale, but the Agents in zone A may be overloaded resulting in degradation of client performance in that zone. If your workload is susceptible to traffic imbalances across zones **and** you're running the WarpStream Agents in multiple zones **and** you've enabled zone-aware routing, then our general recommendation is to create a dedicated WarpStream Agent deployment (all belonging to the same virtual cluster) in each availability zone. This will make the cluster resilient to zonal imbalances as each Agent deployment in each zone will have its own auto-scaler, and thus be able to react to imbalances in zonal traffic independently. If you're using our [Kubernetes chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent), this feature is already built-in and simply needs to be enabled. See our [chart documentation](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent#autoscaling-per-zone) instructions on how to enable it. ## Automatic Availability Zone Detection By default, the WarpStream Agents will try to determine which availability zone they're running in by querying Kubernetes and/or cloud-provider native APIs. This currently works automatically in AWS, GCP, and Azure. {% hint style="info" %} Availability zone names are not consistent between different AWS accounts. If you're deploying a WarpStream cluster where Agents and Clients will be deployed in different AWS accounts, you'll want the Agents to advertise their **availability zone ID** instead of their **availability zone name**. This can be accomplished by setting the `WARPSTREAM_LOOKUP_AVAILABILITY_ZONE_ID=true` environment variable on the Agents. {% endhint %} If your Agents are advertising their availabiltiy zone as `warpstream-unset-az` in the WarpStream console, then it means they failed to determine their availability zone automatically. Check the Agent logs for a message in the form of: `error determining availability zone` which should contain a detailed error message. For instance, a known issue on AWS EKS is that the hop limit on old EKS node group is 1, preventing the call to AWS metadata from failing. Raising it to 2 should fix the issue (see [AWS doc](https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-supports-ec2-instance-metadata-service-v2/)). As a last resort, you can use the `WARPSTREAM_AVAILABILITY_ZONE` environment variable described in the table above to declare the availability zone in which your agent is running. ## Deploying Multiple Clusters in the same VPC If you plan to deploy multiple Warpstream clusters in the same VPC, you must be aware that there is a risk an IP from one agent of a given cluster could be reused by an agent of another cluster. You should familiarize yourself with [this section](/warpstream/agent-setup/deploy/kubernetes-known-issues#when-an-ip-is-reused-by-another-agents-pod) to make your deployments robust to this. ## A Note on Load Balancers and Proxies There are roughly four network paths where a load balancer / proxy can be introduced in the WarpSteram architecture. These should be avoided whenever possible, but the rest of this section will outline in which scenarios it's acceptable and which scenario it's not. ### Between Clients and the Agents In general, it's **much** better to allow Kafka clients to connect directly to the Agents for two reasons: 1. It allows us to localize writes of batches for the same topic-partitions to a small subset of Agents which improves data locality, and as a result, massively improves performance throughout the system. You can read more about this in our documentation about [partition assignment strategies](/warpstream/kafka/reference/partition-assignment-strategies). 2. It allows us to balance the load of long-lived client connections across the Agents based on their actual observed load. As a result, introducing a load balancer or proxy between your Kafka clients and the WarpStream Agents can result in dramatically reduced performance, latency, and throughput as well as huge load imbalances between the Agents. In general, many use-cases where you want to introduce a load balancer / proxy between the Kafka clients and WarpStream Agents are better solved with [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups). That said, there are some scenarios where it's unavoidable and it is generally acceptable as long as your workload is not super high throughput. See our [documentation on advanced network architectures](/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy) for more details. ### Between the Agents (inter-agent communication) Load balancers / proxies between the Agents themselves are usually introduced accidentally, or as a result of a general company-wide requirement to use a service-mesh for all inter-service communication. This is understandable, but you should avoid it all costs when deploying WarpStream for two reasons: 1. Inter-agent communication powers [WarpStream's distributed file cache](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-distributed-mmap). As a result, it is very important that when Agent A tries to reach Agent B that its RPC actually arrives on Agent B and not some unrelated Agent C which will be unable to serve the request (and will in fact refuse it with an error). 2. Inter-agent communication can be extremely high volume (hundreds of MiB/s or even GiBs/s). This traffic is all zone-local so you will not incur any inter-zone networking fees, but introducing a proxy for this volume of traffic almost always causes huge performance problems, OOMs in the proxy itself, and resource starvation for the Agents as the proxy / side-car steals resources away from the Agents themselves. ### Between the Agents and the WarpStream Control Plane Traffic between the WarpStream Agents and the WarpStream control plane is minimal, so if necessary proxying that traffic through an internal proxy is acceptable, although you should avoid it as an additional potential failure domain if possible. In this scenario, set the `WARPSTREAM_HTTP_PROXY` environment variable on the Agents to the hostname of your proxy. For example, if the hostname of your proxy was `http://internal.proxy.com` then you would set `WARPSTREAM_HTTP_PROXY=http://internal.proxy.com` as the environment variable. The Agent will then use this proxy for all Agent <> Control Plane communication, but Agent <> Agent and Agent <> Object Storage communication will continue to happen directly, bypassing the internal proxy. {% hint style="warning" %} In addition to the `WARPSTREAM_HTTP_PROXY` environment variable, WarpStream also supports a generic `HTTP_PROXY` environment variable that will impact all HTTP traffic, not just traffic between the Agents and the Control Plane. We don't recommend ever setting this value because the amount of networking performed between the WarpStream Agents, as well as between the Agents and the object store (which is also over HTTP) can be extremely high volume, and inserting a proxy in the middle of that traffic would significantly reduce the performance of the cluster. {% endhint %} ### Between the Agents and the Object Store We don't ever recommend doing this for the same reasons we don't ever recommend using a proxy for inter-agent communication: the networking between the Agent and the object store is free, but extremely high volume and introducing a proxy will almost always result in huge performance problems, OOMs in the proxy itself, and resource starvation for the Agents as the proxy / side-car steals resources away from the Agents themselves. # Kubernetes Known Issues ## When running in EKS Availability Zone is Unset or Wrong ### Symptom In the WarpStream UI for the cluster you see `warpstream-unset-az` set as the availability zone of the agent and/or errors in the agent logs similar to the following: {% code overflow="wrap" %} ``` {"time":"2025-04-02T22:23:46.467567362Z","level":"ERROR","msg":"failed to determine availability zone","git_commit":"32d51900b2423718b692a0edd29b08b11b7dd74e","git_time":"2025-04-02T18:53:04Z","git_modified":false,"go_os":"linux","go_arch":"arm64","process_generation":"081c0596-25c3-4147-88d5-d4416cb6a998","hostname_fqdn":"warp-agent-default-67d9795854-wrwh8","hostname_short":"warp-agent-default-67d9795854-wrwh8","private_ips":["10.0.115.97"],"num_vcpus":3,"kafka_enabled":true,"virtual_cluster_id":"vci_bc62be92_d3ba_4b0c_90e8_4e7bc621a693","module":"agent_azloader","error":{"message":"awsECSErr: missing metadata uri in environment (ECS_CONTAINER_METADATA_URI_V4), likely not running in ECS\nawsEC2Err: error getting metadata: operation error ec2imds: GetMetadata, canceled, context deadline exceeded\ngcpErr: error getting availablity zone: \nazureErr: error getting location: \nk8sErr: unable to get node information: nodes \"i-025487767185742f1\" is forbidden: User \"system:serviceaccount:warpstream:warpstream0-agent\" cannot get resource \"nodes\" in API group \"\" at the cluster scope"}} ``` {% endcode %} ### Context The WarpStream Agents try to use various methods to determine which availability zone the agent is running in. When it can't determine the availability zone it falls back to `warpstream-unset-az` and logs error messages. ### Problem AWS by default prevents EKS pods from contacting the metadata service to prevent instance metadata leaks. While this is good security practice for normal instances, it prevents services within EKS from querying information about the instance. ### Solution #### Option A Use our [Helm Chart](/warpstream/agent-setup/infrastructure-as-code/helm-charts) to deploy WarpStream. The helm chart with it's default configuration will create a Kubernetes `ClusterRole` and `ClusterRoleBinding` which allows the WarpStream pods to lookup get node they are running on within the Kubernetes API and find the availability zone from node labels. #### Option B Create the appropriate ClusterRole, ClusterRoleBinding, and ServiceAccount so the WarpStream agent can get availability zone information from the Kubernetes API ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: warpstream-agent namespace: ${your-namespace} --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: warpstream-agent namespace: ${your-namespace} rules: - apiGroups: - "" resources: - pods verbs: - get - watch - list --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: warpstream-agent namespace: ${your-namespace} subjects: - kind: ServiceAccount name: warpstream-agent namespace: ${your-namespace} roleRef: kind: Role name: warpstream-agent apiGroup: rbac.authorization.k8s.io --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: warpstream-agent rules: - apiGroups: - "" resources: - nodes verbs: - get - watch - list --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: warpstream-agent roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: warpstream-agent subjects: - kind: ServiceAccount name: warpstream-agent namespace: ${your-namespace} ``` Then on your WarpStream deployment set the pod service account to `warpstream-agent`. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: warpstream-agent namespace: ${your-namespace} spec: selector: matchLabels: app.kubernetes.io/app: warpstream-agent template: metadata: labels: app.kubernetes.io/app: warpstream-agent spec: containers: - args: - agent ... image: public.ecr.aws/warpstream-labs/warpstream_agent:latest ... serviceAccount: warpstream-agent ``` #### Option C Modify your EKS Node Launch Template configuration to set `http-put-response-hop-limit` to 2. This will allow the pods running on a EKS instance to connect to the AWS metadata service to find the availability zone. ## When running in Kubernetes WarpStream pods end up in the same zone or node ### Symptom Some or all of your WarpStream pods end up running in the same availability zone or on the same Kubernetes node instead of being evenly spread out. ### Context When running workloads in Kubernetes it will try it's best to make sure pods from the same deployment are evenly spread across all nodes and availability zones, however this isn't always possible. ### Problem Depending on Kubernetes cluster configuration and other workloads on the cluster Kubernetes may not evenly deploy WarpStream pods across zones or nodes. Some Kubernetes deployments prioritize bin-packing rather then high availability of workloads. This varies by Kubernetes distribution and is not always configurable. ### Solution Use Kubernetes `topologySpreadConstraints` and `podAntiAffinity` to force Kubernetes to spread WarpStream pods evenly across zones and nodes. If your WarpStream pods are using our [Helm Chart](/warpstream/agent-setup/infrastructure-as-code/helm-charts) you can set the following in your helm values: ```yaml topologySpreadConstraints: # Try to spread pods across multiple zones - maxSkew: 1 # +/- one pod per zone topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule # minDomains is only available in Kubernetes 1.30+ # Remove this field if you are on an older Kubernetes # version. # When possible set to the number of available # availability zones in your cluster. minDomains: 3 # Label Selector to select the warpstream deployment labelSelector: matchLabels: app.kubernetes.io/name: warpstream-agent app.kubernetes.io/instance: warpstream-agent # Set to your helm release name affinity: # Make sure pods are not scheduled on the same node to prevent bin packing podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: # Label Selector to select the warpstream deployment - labelSelector: matchLabels: app.kubernetes.io/name: warpstream-agent app.kubernetes.io/instance: warpstream-agent # Set to your helm release name topologyKey: kubernetes.io/hostname ``` ## When an IP is reused by another agent's pod ### Symptom Kafka requests will either fail or target the wrong Kafka cluster. For instance a produce request aiming at cluster A could actually end up being processed in cluster B and the data will never be visible from cluster A resulting in data loss. ### Context If you deploy multiple Warpstream Agents k8s deployments in the same VPC, then it is totally possible that the IP of an agent that is going away - for instance during a scale down - is going to be re-used by another agent spawning up. And this new agent does not necessarily belong to the same k8s deployment, nor is connected to the same Warpstream cluster. Let's consider the following scenario, with both clusters A and B deployed in the same VPC: * agent with IP `10.0.104.73` shuts down and it belonged to a kubernetes deployment connected to cluster A * quickly after, a new pod is starting in a kubernetes deployment connected to cluster B, and kubernetes allocates the same IP to is * a kafka application that is configured to be connected to cluster A still has `10.0.104.73` in its DNS cache and is opening a new connection to send a produce request to it. * the connection is established fine, but the agent receiving the request actually belongs to cluster B. Auto-topic creation is on, so it just creates the topic, and the produce request is processed and acknowledged. * the kafka application receives an ack and is happy, it will keep this connection opened and send more requests through it * if the same application is consuming data from the same topic, it will never see it ### Solution Agent to agent communication is already protected against this. This kind of communication only happens on the read path (more info in [this](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-distributed-mmap) blogpost) and all agents will reject requests not targeting the right virtual cluster that is sent along each internal HTTP request. However there is nothing built-in the Kafka protocol for this, and it requires clients participation to be totally safe. The most straightforward way to do it is to leverage the `warpstream_cluster_id` [client ID](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_cluster_id) from your clients so the agent can reject the connection if they do not match. Another way is to [use SASL credentials](/warpstream/kafka/manage-security/sasl-authentication): as those are unique across clusters, if a Kafka client tries to connect to an IP thinking it still belongs to cluster A, an agent belonging to cluster B will reject the connection, and the client will retry on another IP. Alternatively, if you enable TLS, this will also completely mitigate the issue as the SSL certificate won't match and the client will get an SSL error which should cause it to retry. On top of that, if you are using agent groups, you will want to specify the `warpstream_agent_group` [client ID](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_agent_group) in your client ID with the correct agent group: this way if you connect to an agent from the same cluster but the wrong group, you will be redirected to another agent from the correct group. # Rolling Restarts and Upgrades This page explains how rolling restarts and upgrades work in WarpStream. ## Graceful Shutdown The WarpStream Agents perform a graceful shutdown routine when they receive a `sigterm` signal. The graceful shutdown routine strives to minimize disruption to the cluster and your Kafka clients. By default, this graceful shutdown process takes 300 seconds (5 minutes) to complete. However, many container orchestration frameworks will not wait that long for a container to shutdown gracefully. For example, in Kubernetes the default graceful termination window is 30 seconds. We recommend increasing this value to 600 seconds (10 minutes). In Kubernetes, the configuration value for this is called `terminationGracePeriodSeconds`. In addition, for this graceful shutdown to work smoothly with your application, we recommend setting the metadata refresh interval on your client to 1 minute. See our [client tuning documentation](/warpstream/kafka/configure-kafka-client/tuning-for-performance) for more details. If you want to change the graceful shutdown duration, you can tune the value of the `-gracefulShutdownDuration` flag or `WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION` environment variable on your Agents. The value will be parsed as a duration (I.E `30s` and `5m` are both valid values). If you do this, you may also need to update the graceful termination window in Kubernetes or whatever software you're using to schedule your containers as described above. ## Rolling Restarts Rolling restarts are handled by WarpStream gracefully as long as the Agent containers are allowed to complete their graceful shutdown process, and clients are appropriately tuned to refresh their cluster metadata frequently as describe in the section above. ## Rolling Upgrades Rolling upgrades behave just like rolling restarts and leverage the same graceful shutdown mechanism. The only difference is that the Agent docker image tag is changed. Upgrades are always backwards compatible and no manual upgrade steps are required except deploying the new Docker image version in an incremental manner. # Object Storage Configuration This page describes how to properly configure object storage for BYOC Agent deployments. We highly recommend running the WarpStream Agent with a dedicated bucket for isolation; however, the WarpStream Agent will only write/read data under the `warpstream` prefix. {% hint style="warning" %} You should use a [VPC Endpoint](https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html) (or the equivalent in your Cloud Service Provider) to ensure the network traffic between the WarpStream Agent and your Object Storage bucket does not incur any data transfer cost, such as the cost incurred by using a NAT Gateway. {% endhint %}
{% hint style="danger" %} The WarpStream Agent manages all data in the object storage `warpstream` directory. It is extremely important that you allow it to do so alone and never delete files from the `warpstream` directory manually. Manually deleting files in the `warpstream` directory will effectively "brick" a virtual cluster and require that it be recreated from scratch. {% endhint %} ## Bucket URL Construction The `bucketURL` flag is the URL of the object storage bucket that the WarpStream Agent should write to. See the table below for how to configure it for different object store implementations. Note that the WarpStream Agents will automatically write all of their data to a top-level `warpstream` prefix in the bucket. In addition, each cluster will write its data to a cluster-specific prefix (derived from the cluster ID) within the `warpstream` prefix so multiple WarpStream clusters and schema registries can share the same object storage bucket without issue.

An S3 bucket with 16 different cluster prefixes under the top-level warpstream prefix.

{% tabs %} {% tab title="AWS S3" %} Format: `s3://$BUCKET_NAME?region=$BUCKET_REGION` Example: `s3://my_warpstream_bucket_123?region=us-east-1` The WarpStream Agent embeds the official AWS Golang SDK V2 so authentication/authorization with the specified S3 bucket can be handled in [any of the expected ways, like using a shared credentials file, environment variables, or simply running the Agents in an environment with an appropriate IAM role with Write/Read/Delete/List permissions on the S3 bucket.](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html#specifying-credentials) **Assume Role** If you want to use an `AssumeRole` provider to authenticate, you can add the `WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_DEFAULT` environment variable to your Agent. For example: {% code overflow="wrap" %} ```bash WARPSTREAM_BUCKET_ASSUME_ROLE_ARN_DEFAULT=arn:aws:iam::103069001423:role/YourRoleName ``` {% endcode %} **Manually Providing Credentials** In general, we recommend using IAM roles whenever possible. However, if you want to provide object storage credentials manually then you'll need to set the following environment variables: {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY ``` {% endcode %} Environment variables can be set in our K8s chart using the `extraEnvs` and `extraEnvsFrom` fields in the [charts values.yaml](https://github.com/warpstreamlabs/charts/blob/main/charts/warpstream-agent/values.yaml). {% endtab %} {% tab title="GCP GCS" %} Format: `gs://$BUCKET_NAME` Example: `gs://my_warpstream_bucket_123` The WarpStream Agent embeds the official GCP Golang SDK so authentication/authorization with the storage bucket can be handled [in any of the expected ways](https://github.com/googleapis/google-cloud-go#authorization). {% hint style="warning" %} By default the WarpStream Agents use [gRPC direct connectivity](https://docs.cloud.google.com/storage/docs/direct-connectivity) to achieve the best performance and lowest latency. Direct connectivity has some networking [requirements](https://docs.cloud.google.com/storage/docs/direct-connectivity#requirements) that may not be desired in all environments and does not support [GCP Private Service Connect](https://docs.cloud.google.com/vpc/docs/private-service-connect). If required, direct connectivity can be disabled by setting the `WARPSTREAM_GCS_ALLOW_DIRECT_CONNECTIVITY` environment variable to `false`, however your agents may not be able to achieve maximum performance and lowest latency. {% endhint %} {% endtab %} {% tab title="Azure Blob Storage" %} Format: `azblob://$CONTAINER_NAME?storage_account=$STORAGE_ACCOUNT` Example: `azblob://my_warpstream_container_123?storage_account=my_storage_account_456` The WarpStream Agent embeds the official Azure Golang SDK which expects one of the two following environment variables to be set: `AZURE_STORAGE_KEY` or `AZURE_STORAGE_SAS_TOKEN`. Alternatively, you can use an Azure AD / managed identity / service principal. {% endtab %} {% tab title="Memory" %} {% hint style="danger" %} For testing and local development only. All data will be lost once the Agent shuts down. {% endhint %} Example: `mem://my_memory_bucket` {% endtab %} {% tab title="File" %} {% hint style="danger" %} For testing and local development only. The file store implementation is **not** robust. {% endhint %} Format: `file://$PATH_TO_DIRECTORY` Example: `file:///tmp/warpstream_tmp_123` {% endtab %} {% endtabs %} ### S3-compatible Object Stores (MinIO, Ceph, R2, Oracle Cloud, Tigris, etc) If you're using an "S3 compatible" object storage service other than Amazon S3, such as MinIO, Ceph Object Gateway, Cloudflare R2, Oracle Cloud Object Storage, Linode Object Storage, or Alibaba Object Storage, you will need to manually provide credentials as environment variables. You must also configure the S3 client to construct the appropriate URL based on the API compatibility. Detailed instructions for each provider are listed below: {% tabs %} {% tab title="MinIO" %} If you have a MinIO docker container running locally on your machine on port 9000, you can run the Agent like this after creating an Access Key in the MinIO UI: 
AWS_ACCESS_KEY_ID="wKghTMkQrFqszshHJcop" \
AWS_SECRET_ACCESS_KEY="MpMO9GFMaoIFFYd8cZi5gyk5SAjwleEbkZOSxIXv" \
warpstream demo \
-bucketURL "s3://<your-bucket>?region=us-east-1&s3ForcePathStyle=true&endpoint=http://127.0.0.1:9000""
The MinIO team has a [more detailed integration guide](https://blog.min.io/streamlining-data-streaming-a-guide-to-warpstream-and-minio/) on their website as well. Note that the region query argument is a no-op, but required to pass validation in the S3 SDK. {% endtab %} {% tab title="Ceph" %} WarpStream connects to [Ceph Object Gateway (RGW)](https://docs.ceph.com/en/latest/radosgw/s3/) through its S3-compatible API. Create a dedicated bucket and an RGW user with read, write, delete, and list access to that bucket, then provide the user's S3 access key and secret key to the Agent. Use path-style addressing unless your RGW deployment is configured for [virtual-hosted-style access](https://docs.ceph.com/en/latest/radosgw/s3/commons/#bucket-and-host-name). The bucket URL has the following format: `s3://?region=us-east-1&s3ForcePathStyle=true&endpoint=` The endpoint must include the scheme and port, if applicable. The region is required by the AWS SDK but is not used by RGW. For example: {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="YOUR_CEPH_ACCESS_KEY" \ AWS_SECRET_ACCESS_KEY="YOUR_CEPH_SECRET_KEY" \ warpstream demo \ -bucketURL "s3://warpstream?region=us-east-1&s3ForcePathStyle=true&endpoint=https://rgw.example.com" ``` {% endcode %} For an Agent deployment, configure the same values with environment variables: ```yaml config: bucketURL: "s3://warpstream?region=us-east-1&s3ForcePathStyle=true&endpoint=https://rgw.example.com" extraEnvs: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: ceph-s3-credentials key: access-key - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: ceph-s3-credentials key: secret-key ``` Configure the Ceph bucket according to the requirements in [Bucket Configuration](#bucket-configuration): do not enable object versioning, object lock, or a retention policy. Configure an S3 lifecycle rule to abort incomplete multipart uploads after seven days, and ensure that [RGW lifecycle processing](https://docs.ceph.com/en/latest/radosgw/config-ref/#lifecycle-settings) is enabled on at least one RGW daemon in each zone. Some Ceph versions do not support the checksum headers sent by recent AWS SDK releases. If the Agent reports an `XAmzContentSHA256Mismatch` error while checking bucket access, set the following environment variable: ```bash WARPSTREAM_DISABLE_S3_CHECKSUMS=true ``` {% endtab %} {% tab title="Cloudflare R2" %} 1. Create an account with [Cloudflare](https://dash.cloudflare.com). 2. Create an R2 bucket. 3. Create an R2 access token.
{% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" \ AWS_SECRET_ACCESS_KEY="XXX" \ warpstream demo -bucketURL "s3://warpstream-demo-for-fun?s3ForcePathStyle=true®ion=auto&endpoint=https://XXX.r2.cloudflarestorage.com" ``` {% endcode %}
Note that if you run multiple WarpStream Agents this way in non-demo mode, then by default they need to be running on the same internal network. The reason for this is that if the Agents believe they're all running in the same "availability zone", they will attempt to form a distributed cache with each other to reduce R2 API GET requests. However, if you wish to run multiple Agents in separate networks / regions, but still allow them to function as a single "Kafka Cluster", assign each one a dedicated availability zone. For example, Agent 1: {% code overflow="wrap" %} ```bash WARPSTREAM_AVAILABILITY_ZONE="personal_laptop_chicago" \ AWS_ACCESS_KEY_ID="XXX" \ AWS_SECRET_ACCESS_KEY="XXX" \ warpstream agent -bucketURL "s3://warpstream-demo-for-fun?s3ForcePathStyle=true®ion=auto&endpoint=https://XXX.r2.cloudflarestorage.com" ``` {% endcode %} Agent 2: {% code overflow="wrap" %} ```bash WARPSTREAM_AVAILABILITY_ZONE="personal_laptop_nashville" \ AWS_ACCESS_KEY_ID="XXX" \ AWS_SECRET_ACCESS_KEY="XXX" \ warpstream agent -bucketURL "s3://warpstream-demo-for-fun?s3ForcePathStyle=true®ion=auto&endpoint=https://XXX.r2.cloudflarestorage.com" ``` {% endcode %} This signals to each Agent that they should not attempt to communicate with each other directly over the local network, and that each one should behave as if it were running in a different availability zone. However, data will still be able to be streamed from Chicago to Nashville (or vice versa) because the Agents will use R2 as "the network". The net result of this is a "multi-region" Cluster that can read and write all topics/partitions from multiple regions at the same time. {% endtab %} {% tab title="Linode" %} 1. Create an account with [Akamai Linode](https://login.linode.com/signup) 2. Create a Bucket in [Object Storage](https://cloud.linode.com/object-storage/buckets) in the region of your choice. 3. Create a set of Object Storage [Access Keys](https://cloud.linode.com/object-storage/access-keys) To set up a WarpStream Agent with Linode Object Storage, you need your access key, secret key, and the bucket URL. The bucket URL for Akamai Linode's S3-compatible storage requires the following structure:\ \ `s3://?s3ForcePathStyle=true&endpoint=.linodeobjects.com®ion=` **Demo** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" warpstream demo -bucketURL "s3://?s3ForcePathStyle=true&endpoint=.linodeobjects.com®ion=" ``` {% endcode %} **Playground** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" warpstream playground -bucketURL "s3://?s3ForcePathStyle=true&endpoint=.linodeobjects.com®ion=" ``` {% endcode %} **Docker Configuration Example** ```bash docker run -it --rm \ -e AWS_ACCESS_KEY_ID="YOUR_LINODE_BUCKET_ACCESS_KEY" \ -e AWS_SECRET_ACCESS_KEY="YOUR_LINODE_BUCKET_SECRET_KEY" \ public.ecr.aws/warpstream-labs/warpstream_agent \ agent \ -bucketURL "s3://?s3ForcePathStyle=true&endpoint=.linodeobjects.com®ion=" \ -agentKey "YOUR_AGENT_KEY" \ -defaultVirtualClusterID "YOUR_VIRTUAL_CLUSTER_ID" \ -region "YOUR_CLUSTER_REGION" ``` Make sure to replace the following placeholders in the commands above: * `XXX`: Your Linode Object Storage access and secret keys. * ``: The name of your bucket in Linode Object Storage. * ``: The ID for your object storage cluster, such as us-ord-1. This value must be used for both the endpoint and region parameters. To deploy agents in a Kubernetes cluster, use the official [Helm chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent) and add the following configurations to the `values.yaml` file. ```yaml extraEnv: [ { name: "AWS_ACCESS_KEY_ID", value: "YOUR_LINODE_BUCKET_ACCESS_KEY" }, { name: "AWS_SECRET_ACCESS_KEY", value: "YOUR_LINODE_BUCKET_SECRET_KEY" } ] ``` {% endtab %} {% tab title="Alibaba Cloud" %} 1. Create an account with [Alibaba Cloud](https://account.alibabacloud.com/register/intl_register.htm). 2. Create a set of AccessKeys by creating a [RAM User](https://www.alibabacloud.com/help/en/ram/create-a-ram-user-1#task-187540). 3. Create a Bucket in Object Storage Service (OSS) in the region of your choice. To set up a WarpStream Agent with Alibaba Cloud OSS, you need your AccessKey ID, AccessKey Secret, and the bucket URL. The bucket URL for Alibaba Cloud's S3-compatible storage requires the following structure, explicitly setting it to use virtual-hosted-style URLs: `s3://?endpoint=oss-.aliyuncs.com®ion=&s3ForcePathStyle=false` **Demo** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" warpstream demo -bucketURL "s3://?endpoint=oss-.aliyuncs.com®ion=&s3ForcePathStyle=false" ``` {% endcode %} **Playground** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" warpstream playground -bucketURL "s3://?endpoint=oss-.aliyuncs.com®ion=&s3ForcePathStyle=false" ``` {% endcode %} **Docker Configuration Example** ```bash docker run -it --rm \ -e AWS_ACCESS_KEY_ID="YOUR_ALI_ACCESS_KEY_ID" \ -e AWS_SECRET_ACCESS_KEY="YOUR_ALI_ACCESS_KEY_SECRET" \ public.ecr.aws/warpstream-labs/warpstream_agent \ agent \ -bucketURL "s3://?endpoint=oss-.aliyuncs.com®ion=&s3ForcePathStyle=false" \ -agentKey "YOUR_AGENT_KEY" \ -defaultVirtualClusterID "YOUR_VIRTUAL_CLUSTER_ID" \ -region "YOUR_CLUSTER_REGION" ``` Placeholders Make sure to replace the following placeholders in the commands above: * `XXX`: Your Alibaba Cloud AccessKey ID and AccessKey Secret. * ``: The name of your bucket in Alibaba Cloud OSS. * ``: The ID for your object storage region. To deploy agents in a Kubernetes cluster, use the official [Helm chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent) and add the following configurations to the `values.yaml` file. ```yaml extraEnv: [ { name: "AWS_ACCESS_KEY_ID", value: "YOUR_ALI_ACCESS_KEY_ID" }, { name: "AWS_SECRET_ACCESS_KEY", value: "YOUR_ALI_ACCESS_KEY_SECRET" } ] ``` {% endtab %} {% tab title="IBM COS" %} 1. Create an [IBM Cloud Platform Account](https://cloud.ibm.com/). 2. Create an [instance of IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-provision). 3. Create a bucket with the [level of resiliency](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints) you want. 4. Create a set of Object Storage [API Keys](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage#gs-bucket-policy). To set up a WarpStream Agent with IBM Cloud Object Storage (COS), you need your access key, secret key, and the bucket URL. The bucket URL for IBM COS's S3-compatible storage requires the following structure:\ \ `s3://?endpoint=s3..cloud-object-storage.appdomain.cloud®ion=` **Demo** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" warpstream demo -bucketURL "s3://?endpoint=s3..cloud-object-storage.appdomain.cloud®ion=" ``` {% endcode %} **Playground** {% code overflow="wrap" %} ```bash AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" warpstream playground -bucketURL "s3://?endpoint=s3..cloud-object-storage.appdomain.cloud®ion=" ``` {% endcode %} **Docker Configuration Example** ```bash docker run -it --rm \ -e AWS_ACCESS_KEY_ID="YOUR_IBM_BUCKET_ACCESS_KEY" \ -e AWS_SECRET_ACCESS_KEY="YOUR_IBM_BUCKET_SECRET_KEY" \ public.ecr.aws/warpstream-labs/warpstream_agent \ agent \ -bucketURL "s3://?endpoint=s3..cloud-object-storage.appdomain.cloud®ion=" \ -agentKey "YOUR_AGENT_KEY" \ -defaultVirtualClusterID "YOUR_VIRTUAL_CLUSTER_ID" \ -region "YOUR_CLUSTER_REGION" ``` Make sure to replace the following placeholders in the commands above: * `XXX`: Your IBM Service ID Object Storage API access and secret keys. * ``: The name of your bucket in IBM Cloud Object Storage. * ``: The ID for your object storage cluster, such as us-east. This value must be used for both the endpoint and region parameters. To deploy agents in a Kubernetes cluster, use the official [Helm chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent) and add the following configurations to the `values.yaml` file. ```yaml extraEnv: [ { name: "AWS_ACCESS_KEY_ID", value: "YOUR_IBM_BUCKET_ACCESS_KEY" }, { name: "AWS_SECRET_ACCESS_KEY", value: "YOUR_IBM_BUCKET_SECRET_KEY" } ] ``` {% endtab %} {% endtabs %} ### Using a Bucket Prefix If you want the WarpStream Agents to store data in a specific prefix in the bucket, you can add the prefix as a query argument to the bucket URL. The prefix must terminate with a "/". For example: ``` s3://my_warpstream_bucket_123?region=us-east-1&prefix=my_prefix/ ``` ## Combining Multiple Buckets The `bucketURL` can also point at several object storage buckets treated as one logical bucket, using a wrapper scheme: * [**Multi Buckets**](/warpstream/agent-setup/different-object-stores/multi-buckets) (`warpstream_multi://`) — **replicate** every write across the buckets for durability and availability (for example, to survive a full region's object storage outage). * [**Striped Buckets**](/warpstream/agent-setup/different-object-stores/striped-buckets) (`warpstream_stripe://`) — **stripe** writes so each object lives on exactly one bucket, to scale write throughput past a single bucket's request-rate ceiling. ## Bucket Configuration {% hint style="danger" %} The WarpStream bucket should not have any of the following features enabled on it: 1. Object retention policy 2. Object versioning 3. Soft deletion An object retention policy could lead to data corruption if the cloud provider deletes a file that the WarpStream cluster still considers active. Object versioning and soft deletion will lead to massive storage cost inflation due to the fact that WarpStream periodically compacts (rewrites) data in the background. {% endhint %} WarpStream will manage the lifecycle of the objects, including deleting objects that have been compacted or have expired due to retention. Do not configure a retention policy on your bucket, and make sure that object versioning and object soft deletion are disabled. We do however recommend configuring a lifecycle policy for cleaning up aborted multi-part uploads. This will prevent failed file uploads from the WarpStream Agent from accumulating in the bucket forever and increasing your storage costs. Below is a sample Terraform configuration for various different cloud providers: {% tabs %} {% tab title="AWS" %} ```hcl resource "aws_s3_bucket" "warpstream_bucket" { bucket = "my-warpstream-bucket-123" tags = { Name = "my-warpstream-bucket-123" Environment = "staging" } } resource "aws_s3_bucket_metric" "warpstream_bucket_metrics" { bucket = aws_s3_bucket.warpstream_bucket.id name = "EntireBucket" } resource "aws_s3_bucket_lifecycle_configuration" "warpstream_bucket_lifecycle" { bucket = aws_s3_bucket.warpstream_bucket.id # Automatically cancel all multi-part uploads after 7d so we don't accumulate an infinite # number of partial uploads. rule { id = "7d multi-part" status = "Enabled" abort_incomplete_multipart_upload { days_after_initiation = 7 } } # No other lifecycle policy. The WarpStream Agent will automatically clean up and # deleted expired files. } resource "aws_s3_bucket_versioning" "warpstream_bucket_versioning" { bucket = aws_s3_bucket.warpstream_bucket.id versioning_configuration { # Make sure versioning is disabled or it will massively inflate your storage costs. status = "Disabled" } } ``` {% endtab %} {% tab title="GCP" %} ```hcl resource "google_storage_bucket" "warpstream_bucket" { name = "my-warpstream-bucket-123" location = "$REGION" labels = { Name = "my-warpstream-bucket-123" Environment = "staging" } lifecycle_rule { condition { age = 7 } action { type = "AbortIncompleteMultipartUpload" } } soft_delete_policy { # Make sure soft deletion is disabled or it will massively inflate your storage costs. retention_duration_seconds = 0 } versioning { # Make sure versioning is disabled or it will massively inflate your storage costs. enabled = false } } ``` {% endtab %} {% tab title="Azure" %} ```terraform resource "azurerm_storage_container" "warpstream_container" { name = "my-warpstream-container-123" storage_account_id = "$STORAGE_ACCOUNT_ID" container_access_type = "private" } ``` {% endtab %} {% endtabs %} ## Bucket Permissions In addition to configuring the WarpStream buckets, you'll also need to make sure the Agent containers have the appropriate permissions to interact with the bucket. {% tabs %} {% tab title="AWS" %} Specifically, the Agents need permission to perform the following operations: * `PutObject` * To create new files. * `GetObject` * To read existing files. * `DeleteObject` * So the Agents can enforce retention and cleanup of pre-compaction files. * `ListBucket` * So the Agents can enforce retention and cleanup of pre-compaction files. Below is an example Terraform configuration for an AWS IAM policy document that provides WarpStream with the appropriate permissions to access a dedicated S3 bucket: ```hcl data "aws_iam_policy_document" "warpstream_s3_policy_document" { statement { sid = "AllowS3" effect = "Allow" actions = [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:ListBucket" ] resources = [ "arn:aws:s3:::my-warpstream-bucket-123", "arn:aws:s3:::my-warpstream-bucket-123/*" ] } } ``` {% endtab %} {% tab title="GCP" %} The easiest way to configure bucket access in GCP is with the `roles/storage.objectUser` and `roles/storage.bucketViewer` role like so: ```hcl resource "google_storage_bucket_iam_member" "warpstream_bucket_object_user" { bucket = "my-warpstream-bucket-123" role = "roles/storage.objectUser" member = "$PRINCIPAL" } resource "google_storage_bucket_iam_member" "warpstream_bucket_viewer" { bucket = "my-warpstream-bucket-123" role = "roles/storage.bucketViewer" member = "$PRINCIPAL" } ``` However, if you need more granular permission sets, then WarpStream requires at least the following: * `storage.objects.create` * `storage.objects.delete` * `storage.objects.get` * `storage.objects.list` * `storage.multipartUploads.*` * `storage.buckets.get` {% endtab %} {% tab title="Azure" %} The easiest way to configure bucket access in Azure is with the `Storage Blob Data Contributor` role like so: ```terraform resource "azurerm_role_assignment" "warpsteam_blob_contributor" { scope = azurerm_storage_container.warpstream_container.resource_manager_id role_definition_name = "Storage Blob Data Contributor" principal_id = "$PRINCIPAL_ID" } ``` {% endtab %} {% endtabs %} ## Migrating Between Object Storage Buckets {% hint style="info" %} the `-additionalBackgroundTasksBucketURLs` flag and `WARPSTREAM_ADDITIONAL_BACKGROUND_TASKS_BUCKET_URLS` environment variable require Agent version v814 or above.
Older versions of the Agent can still migrate buckets by using the `-additionalDeadscannerBucketURLs` flag or `WARPSTREAM_ADDITIONAL_DEADSCANNER_BUCKET_URLS` environment variable instead. {% endhint %} If you need to migrate a WarpStream cluster from one object storage bucket to another, follow these steps: 1. Make sure that the Agents [have permission](#bucket-permissions) to perform operations on both the old bucket and the new bucket. 1. Note if you're using the [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) functionality, you need to do this step for all Agent Groups before proceeding to step 2. In other words, **all** Agents in **all** groups of the cluster must be able to access the old bucket and the new bucket before the new bucket can start being used. 2. Deploy the Agents with [the `bucketURL` flag](#bucket-url-construction) set to the new bucket instead of the old one. This will cause the Agents to write all new files (both for ingestion and compaction) to the new bucket while still allowing them to read historical data from the old bucket. 1. You'll also need to set the `-additionalBackgroundTasksBucketURLs` flag or `WARPSTREAM_ADDITIONAL_BACKGROUND_TASKS_BUCKET_URLS` environment variable in the Agents to point to the old bucket so that the Agents continue to scan the old bucket for dead files to delete and ripcord sequences to ingest. 3. Wait until there are no more data files under the `warpstream` prefix in the old bucket. For example, if you were migrating from AWS S3 bucket `foo` to AWS S3 bucket `bar` then you would redeploy the Agents from this configuration: ```bash WARPSTREAM_BUCKET_URL=s3://foo?region=us-east-1 ``` To this configuration: ```bash WARPSTREAM_ADDITIONAL_BACKGROUND_TASKS_BUCKET_URLS=s3://foo?region=us-east-1 WARPSTREAM_BUCKET_URL=s3://bar?region=us-east-1 ``` Then wait until all the files in the `foo` bucket under the `warpstream` prefix had been deleted. Once all the files had been deleted, you would then deploy the Agents one final time with this configuration: ```bash WARPSTREAM_BUCKET_URL=s3://bar?region=us-east-1 ``` ## Kubernetes Workload Identity for Bucket Access When running in Kubernetes in AWS, Azure, or GCP it is recommended to use Workload Identity to delegate access from the WarpStream Agent pods to the Object Storage bucket. This simplifies management of the object storage credentials and minimizes the risk of credential leaks. {% tabs %} {% tab title="AWS EKS" %} Documentation: \ \ Example Terraform ```hcl data "aws_iam_policy_document" "eks_service_account" { statement { effect = "Allow" principals { type = "Federated" identifiers = [var.eks_oidc_provider_arn] } actions = ["sts:AssumeRoleWithWebIdentity"] condition { test = "StringEquals" variable = "${replace(var.eks_oidc_issuer_url, "https://", "")}:sub" values = ["system:serviceaccount:${var.kubernetes_namespace}:warpstream-agent"] } condition { test = "StringEquals" variable = "${replace(var.eks_oidc_issuer_url, "https://", "")}:aud" values = ["sts.amazonaws.com"] } } } resource "aws_iam_role" "eks_service_account" { name = "warpstream-agent" assume_role_policy = data.aws_iam_policy_document.eks_service_account.json } data "aws_iam_policy_document" "eks_service_account_s3_bucket" { statement { effect = "Allow" actions = [ "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", ] resources = [ "arn:aws:s3:::${var.bucketName}", "arn:aws:s3:::${var.bucketName}/*", ] } } resource "aws_iam_role_policy" "eks_service_account_s3_bucket" { name = "warpstream-agent-s3" role = aws_iam_role.eks_service_account.id policy = data.aws_iam_policy_document.eks_service_account_s3_bucket.json } ``` Example Configuration on our [Helm Chart](/warpstream/agent-setup/infrastructure-as-code/helm-charts) ```yaml config: bucketURL: s3://my-bucket-name serviceAccount: annotations: "eks.amazonaws.com/role-arn": "arn:aws:iam::XXXXXXXXXXXX:role/warpstream-agent" ``` {% endtab %} {% tab title="Azure" %} Documentation: \ \ Example Terraform ```hcl resource "azurerm_user_assigned_identity" "warpstream_agent" { name = "warpstream-agent" resource_group_name = var.resource_group_name location = var.location } resource "azurerm_federated_identity_credential" "identity_credential" { name = "warpstream-agent" resource_group_name = var.resource_group_name audience = ["api://AzureADTokenExchange"] issuer = var.aks_oidc_issuer_url parent_id = azurerm_user_assigned_identity.warpstream_agent.id subject = "system:serviceaccount:${var.kubernetes_namespace}:warpstream-agent" } resource "azurerm_role_assignment" "reader_and_data_assigned_identity" { scope = var.azure_container_resource_manager_id role_definition_name = "Storage Blob Data Contributor" principal_id = azurerm_user_assigned_identity.warpstream_agent.principal_id } ``` Example Configuration on our [Helm Chart](/warpstream/agent-setup/infrastructure-as-code/helm-charts) ```yaml config: bucketURL: azblob://my-bucket-name serviceAccount: annotations: "azure.workload.identity/client-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" ``` {% endtab %} {% tab title="GCP" %} Documentation: \ \ Example Terraform ```hcl resource "google_service_account" "warpstream_agent" { account_id = "warpstream-agent" display_name = "Service Account Kubernetes" } resource "google_storage_bucket_iam_member" "gcs_access" { bucket = var.bucket_name role = "roles/storage.objectAdmin" member = "serviceAccount:${google_service_account.warpstream_agent.email}" } resource "google_project_iam_member" "gcs_access_token_creator" { project = var.project role = "roles/iam.serviceAccountTokenCreator" member = "serviceAccount:${google_service_account.warpstream_agent.email}" } resource "google_service_account_iam_binding" "workload_identity_binding" { service_account_id = google_service_account.warpstream_agent.name role = "roles/iam.workloadIdentityUser" members = [ "serviceAccount:${var.project_id}.svc.id.goog[${var.kubernetes_namespace}/warpstream-agent]", ] } ``` Example Configuration on our [Helm Chart](/warpstream/agent-setup/infrastructure-as-code/helm-charts) ```yaml config: bucketURL: gcs://my-bucket-name serviceAccount: annotations: "iam.gke.io/gcp-service-account": "warpstream-agent@xxxxxxxxxx.iam.gserviceaccount.com" ``` {% endtab %} {% endtabs %} # Multi Buckets Combine several object storage buckets into one logical bucket that replicates writes across them for durability and availability. A **multi bucket** combines several object storage buckets into a single logical bucket by **replicating** every write across them. It is a durability and availability feature: because each file is written to a quorum of buckets, the cluster can lose a full bucket (for example, an entire cloud region's object storage) without losing data or availability. Multi buckets are configured through the same `-bucketURL` flag (`WARPSTREAM_BUCKET_URL`) as a normal single bucket — you just pass a `warpstream_multi://` URL instead of a plain `s3://` (or other provider) URL. {% hint style="info" %} If your goal is to scale **write throughput** past a single bucket's request-rate ceiling rather than to add redundancy, see [Striped Buckets](/warpstream/agent-setup/different-object-stores/striped-buckets) instead. Multi buckets replicate every object (N× storage and write cost); striped buckets store each object once. {% endhint %} ## URL format ``` warpstream_multi://$BUCKET_1_URL<>$BUCKET_2_URL<>...<>$BUCKET_N_URL ``` The sub-buckets are separated by `<>`, and each sub-URL is constructed exactly as described on the [Object Storage Configuration](/warpstream/agent-setup/different-object-stores#bucket-url-construction) page. Any combination of providers and regions is allowed, though most deployments use the same provider across regions. For example, three buckets spread across three AWS regions: {% code overflow="wrap" %} ```bash -bucketURL "warpstream_multi://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-west-2<>s3://bucket-c?region=us-east-2" ``` {% endcode %} ## How it works * **Writes** are sent to all sub-buckets and acknowledged as soon as a quorum (a majority) succeed, so a single slow or unavailable bucket does not block writes. * **Reads** can be served from any sub-bucket that holds the file, so losing one bucket does not make data unreadable. The net effect is that a multi bucket tolerates the loss of a minority of its sub-buckets with no data loss and no downtime. ## Primary use case: multi-region data planes The most common reason to use a multi bucket is to spread a cluster's data plane across multiple cloud regions so it can survive a region-wide object storage outage with a Recovery Point Objective of 0. See [Multi-Region Clusters](/warpstream/kafka/advanced-agent-deployment-options/multi-region) for how this pairs with a multi-region control plane. ## Requirements * Every WarpStream Agent must be able to read and write **all** of the sub-buckets. * Each sub-bucket needs the same [permissions](/warpstream/agent-setup/different-object-stores#bucket-permissions) and [bucket configuration](/warpstream/agent-setup/different-object-stores#bucket-configuration) (no object retention policy, versioning, or soft deletion) as a normal WarpStream bucket. ## Migrating Switching a cluster to or from a multi bucket is nothing special — it works exactly like [migrating between object storage buckets](/warpstream/agent-setup/different-object-stores#migrating-between-object-storage-buckets): point `-bucketURL` at the new destination (a `warpstream_multi://` URL, or a plain single-bucket URL when migrating off) and keep the previous bucket(s) in `-additionalBackgroundTasksBucketURLs` until the old data has drained. Existing files continue to be read from wherever they were written. # Striped Buckets Combine several object storage buckets into one logical bucket that stripes writes across them to scale write throughput past a single bucket's limits. A **striped bucket** combines several object storage buckets into a single logical bucket by **striping** writes across them: each object is placed on exactly one sub-bucket, chosen deterministically by hashing the object's key. It exists to scale **write throughput** past the request-rate ceiling of a single bucket — spreading writes across N sub-buckets multiplies the headroom for ingestion and compaction. Striped buckets are configured through the same `-bucketURL` flag (`WARPSTREAM_BUCKET_URL`) as a normal single bucket — you just pass a `warpstream_stripe://` URL instead of a plain `s3://` (or other provider) URL. {% hint style="info" %} Bucket striping requires WarpStream Agent **v827** or above. {% endhint %} {% hint style="warning" %} Striping is **not** a redundancy or availability feature. Each object lives on exactly one sub-bucket, so if a sub-bucket becomes unavailable its objects become unreadable — the same blast radius as a single bucket. If you want to survive the loss of a bucket or region, use a [Multi Bucket](/warpstream/agent-setup/different-object-stores/multi-buckets) (which replicates every write) instead. {% endhint %} ## When to use it Reach for a striped bucket when a single bucket is hitting its object storage request-rate limits (for example, sustained `503 SlowDown` throttling on S3) and the workload needs more write throughput than one bucket can provide. If instead you need durability across regions, use a [Multi Bucket](/warpstream/agent-setup/different-object-stores/multi-buckets). ## URL format ``` warpstream_stripe://$BUCKET_1_URL<>$BUCKET_2_URL<>...<>$BUCKET_N_URL ``` The sub-buckets are separated by `<>` (**2 to 32** sub-buckets), and each sub-URL is constructed exactly as described on the [Object Storage Configuration](/warpstream/agent-setup/different-object-stores#bucket-url-construction) page. For example, striping across three buckets: {% code overflow="wrap" %} ```bash -bucketURL "warpstream_stripe://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-east-1<>s3://bucket-c?region=us-east-1" ``` {% endcode %} ## How it works * **Placement** is deterministic: each object is hashed to one sub-bucket, so writers and readers independently agree on where an object lives — nothing extra is stored to track it. * **Reads** go to the hashed sub-bucket and cost the same as a single bucket (one round-trip). * **Write resilience**: if a sub-bucket's circuit breaker opens (it is throttling or failing), new writes are steered to a healthy sub-bucket so ingestion keeps making progress. ## Multi buckets vs. striped buckets | | `warpstream_multi://` | `warpstream_stripe://` | | ----------------------------- | --------------------------------- | -------------------------------------------- | | Each object is… | replicated to a quorum of buckets | written to exactly one bucket | | Optimizes for | durability / availability | write throughput | | Survives a bucket/region loss | yes (no data loss) | no (that bucket's objects become unreadable) | | Relative storage cost | \~N× | \~1× | ## Requirements and constraints * Every WarpStream Agent must be able to read and write **all** of the sub-buckets. * Each sub-bucket needs the same [permissions](/warpstream/agent-setup/different-object-stores#bucket-permissions) and [bucket configuration](/warpstream/agent-setup/different-object-stores#bucket-configuration) (no object retention policy, versioning, or soft deletion) as a normal WarpStream bucket. Using the same provider and type for every sub-bucket is recommended. * **Changing the set of sub-buckets is forward-only.** Existing files stay in whichever sub-bucket they were originally written to, and are always read from there. Switching a single bucket to a striped bucket (or adding sub-buckets) is safe: new files are striped across the new set while old files continue to be read from where they live. ## Migrating Switching a cluster to or from a striped bucket is nothing special — it works exactly like [migrating between object storage buckets](/warpstream/agent-setup/different-object-stores#migrating-between-object-storage-buckets): point `-bucketURL` at the new destination (a `warpstream_stripe://` URL, or a plain single-bucket URL when migrating off) and keep the previous bucket(s) in `-additionalBackgroundTasksBucketURLs` until the old data has drained. The same applies when retiring a single sub-bucket from a stripe: add the retired bucket (or the previous full stripe URL) to `-additionalBackgroundTasksBucketURLs` so its files are still read and cleaned up while it drains. # Set up Monitoring ## Diagnostics First things first, you're not alone! The WarpStream team is constantly monitoring your cluster and when we find anomalies we create [Diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) that will alert you proactively in our UI or in our [Hosted Prometheus Endpoint](/warpstream/agent-setup/monitor-the-warpstream-agents/hosted-prometheus-endpoint). These are the most common problems we have found in operation and we are constantly adding to catalog of available diagnostics. For those that need more granular detail, keep reading... ## Logging By default, the WarpStream Agent is configured to run with log level `info` . However, this can be changed with the `WARPSTREAM_LOG_LEVEL` environment variable. For example, if the `info` level logs are too noisy for you, you can set `WARPSTREAM_LOG_LEVEL=warn`. The WarpStream Agents have an additional special log level called `analytics` that can be enabled by setting `WARPSTREAM_LOG_LEVEL=analytics`. This enables extremely detailed JSON logging that can be loaded into a logging system that supports analytics to slice and dice Agent log events and obtain a deep understanding of the workload. However, this feature emits a lot of logs, so keep that in mind before enabling it. ## Metrics The WarpStream agents expose a traditional Prometheus metrics endpoint that can be scraped by most popular tools. Prometheus metrics will automatically be exposed on the Agent "internal port" which by default is port `8080`. If you set an explicit port override, then you'll need to update your Prometheus scrape configuration port as well. All WarpStream Agent metrics begin with the `warpstream_` prefix. ### Recommended Metrics & Alerting The WarpStream system is simple by design so there is less to monitor. If you are coming from open source Kafka, this should be a breath of fresh air.\ \ In [Important Metrics and Logs](/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs) you will find all you need for monitoring the agent and to make sure your cluster is operational. If you would like to see the complete list of metrics, you can access those from the agent directly with `$IP:8080/metrics` While there are not many alerts needed, we also provide a [Recommended List of Alerts](/warpstream/agent-setup/monitor-the-warpstream-agents/recommended-list-of-alerts) where you will find a list of key metrics for which you should configure alerts to detect issues in your agent effectively. {% hint style="warning" %} Some of the metrics, particularly the consumer group metrics, can become very high cardinality if the cluster contains a lot of topics or partitions. You can learn more in [Monitoring Consumer Groups](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups) if you need to reduce the cardinality or disable them entirely {% endhint %} ## Client Metrics (KIP-714) The metrics above are emitted by the WarpStream Agents themselves. WarpStream Virtual Clusters also support [KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) client metrics. These are producer- and consumer-side metrics generated by your Kafka clients. Configure one or more client metrics subscriptions on a Virtual Cluster and matching clients will push internal metrics to the Agents on a schedule, with no application code changes. The data lands in the cluster's events stream and is queryable from the Events Explorer. See [Client Metrics (KIP-714)](/warpstream/kafka/configure-kafka-client/client-metrics-kip-714) for setup and examples. ## Health Check The Agent exposes an HTTP health check endpoint at `$IP:8080/v1/status`. A successful response is the string `OK` with a `200` status code. ## MCP Server The WarpStream Console exposes an [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server that you can connect to from AI-powered IDEs like Cursor, Claude Code, or Windsurf. Once connected, you can troubleshoot your cluster using natural language — ask questions like "are there any errors in my cluster in the last hour?" or "what diagnostics are failing?" and your AI assistant will query the cluster for you. See [MCP Server](/warpstream/reference/mcp-server) for setup instructions. # Diagnostics Diagnostics are Warpstream's feedback system, designed to provide actionable insights about the health, cost, and performance of your cluster. Each diagnostic measures specific conditions, evaluates their impact, and provides suggestions to address potential issues. Check out this video to learn more about Diagnostics: {% embed url="" %} ## Dimensions of a Diagnostic Each diagnostic has the following dimensions: * **Type**: The category of the diagnostic, such as `Health` or `Cost`. * **Name**: The component or system that the diagnostic evaluates. * **Successful**: Indicates whether the diagnostic has passed. * **Severity**: The impact level of the diagnostic, ranging from `Low` to `Critical`. * **Muted**: Specifies whether the diagnostic is muted and should not generate alerts. ## Viewing Diagnostics Diagnostics can be accessed in two ways: 1. **Console UI**: The diagnostics are displayed in the Warpstream console interface, providing a visual overview of the cluster's health and performance. 2. **Metrics** (New Version): Diagnostics are also exposed as agent metrics, facilitating integration with existing monitoring systems. Each diagnostic generates a `warpstream_diagnostic_failure` gauge metric, where a value of 1 signifies a failing diagnostic and 0 indicates a healthy status. This metric includes the following descriptive tags: * `diagnostic_name`: The specific name of the diagnostic check. Unlike the old version, the name is now normalized to snake case for consistency. * `diagnostic_type`: The functional category of the diagnostic (e.g., health, cost). * `severity_low`: 1 if the diagnostic failure severity is 'low', 0 if otherwise. * `severity_medium`: 1 if the severity is 'medium', 0 if otherwise. * `severity_high`: 1 if the severity is 'high', 0 if otherwise. * `severity_critical`: 1 if the severity is 'critical', 0 if otherwise. * `muted`: Indicates whether the diagnostic check has been temporarily suppressed (muted). {% hint style="success" %} Diagnostics metrics are available both **in the agents and** exposed via the **Prometheus endpoint** for easy scraping by monitoring tools. {% endhint %} ## Examples of Diagnostics Catalog Below is some of our diagnostics with a brief description of what each check covers. For the full list go to our console and check the Health tab for any cluster | Diagnostic | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------ | | ACL Denied | Detects access denials by ACLs (principal/resource/operation/API); verify privileges. | | Agent Version | Detects agents running older versions; recommends upgrading to stay within supported and optimized releases. | | Cluster Load per Group Role | Checks average and P90 CPU load per agent group/role combination to spot hotspots. | | Consumer Groups Configs | Detects risky consumer group timeouts (rebalance/session/heartbeat) that can trigger rebalances. | | Cross AZ Kafka Clients | Detects clients connecting across AZs (missing AZ hints); increases latency and cross‑AZ costs. | | Embedded Pipeline | Detects pipelines running on agents that also run other roles; recommends dedicated pipeline agents. | | Instance Networking | Detects non network‑optimized instance types; recommends network‑optimized for better throughput/latency. | | Kafka Client Version | Detects clients using old API versions; upgrade to modern Kafka client versions. | | Known Bad Agent Versions | Detects agents running versions with known issues; recommends upgrading beyond affected range(s). | | Known Bad Kafka Clients | Detects client libraries/versions with known issues or poor idempotent performance. | | Mixed Agent Versions | Detects multiple agent versions co‑existing for too long; standardize versions across agents. | | Missing Roles | Ensures at least one agent runs critical roles: `jobs`, `proxy-produce`, and `proxy-consume`. | | Partitions Limit | Warns when approaching the cluster partitions limit. | | Pipeline Not Runnable | Pipelines are running but there are no agents with the `pipelines` role in the relevant agent group(s). | | Produce Batches | Warns when batches per second are very high, indicating suboptimal client batching/idempotence. | | Stuck Consumer | Detects consumers that have not progressed for a while (lagging at a fixed offset). | | Tableflow tables Limit | Warns when approaching the Tableflow tables limit in a Datalake cluster. | To investigate a failing diagnostic, it's often helpful to introspect your cluster using the [Events Explorer](/warpstream/reference/events). Related agent-side activity (for example ACL denials or pipeline errors) appears in event types such as `acl_logs` and `pipeline_logs`. Diagnostic state-change events themselves are stored as `diagnostics_logs`. On Kafka clusters those control-plane events require [Events](/warpstream/reference/events#enabling-events) to be enabled and at least one Agent with the [`jobs` role](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) running **v829 or higher**. You can also access diagnostics and query events from your IDE using an AI assistant via the [MCP Server](/warpstream/reference/mcp-server). # Important Metrics and Logs On this page, we include a sample list of the most important logs and metrics emitted by the Agents. Before reading this documentation page, please familiarize yourself with [how logs and metrics are emitted from the Agents](/warpstream/agent-setup/monitor-the-warpstream-agents). {% hint style="warning" %} **Enable High-Cardinality Tags** Some metrics support the use of tags with potentially high cardinality, such as tags based on topics. **This feature is disabled by default**. **To enable high-cardinality tags:** * Use the command-line flag `-kafkaHighCardinalityMetrics`. * Alternatively, set the environment variable `WARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICS=true`. Tags that require enabling are clearly marked with "(requires enabling high-cardinality tags)" next to their name. Furthermore, per-topic distribution / histogram metrics have 10x to 20x higher cardinality than even the regular high cardinality metrics, so any high cardinality (per-topic) metrics with type `histogram` will not be emitted with per-topic tags unless the `-kafkaHighCardinalityDistributionMetrics` flag or `WARPSTREAM_KAFKA_HIGH_CARDINALITY_DISTRIBUTION_METRICS` environment variable is set to true. {% endhint %} {% hint style="info" %} **Datadog metrics** Starting from Warpstream Agent `v679` all metrics on Datadog will start with `warpstream.` and no longer `warpstream_` . All references to metrics in our doc will keep mentioning metrics starting with `warpstream_` so you have to do the conversion when you are using Datadog and a Warpstream Agent recent enough. You can fall back to the previous behavior by setting the `WARPSTREAM_DATADOG_NORMALIZER_PREFIX_WITH_DOT` environment variable to `false`. This change comes along the official release of our Datadog integration, making all the Warpstream Agent metrics free if you install the integration (and use the new naming convention). {% endhint %} ## Overview System performance metrics and logs, focusing on error detection and resource consumption. **\[logs]**** Error Logs** * query: `status:error` * metric: \* * note that some amount of error logs can be normal depending on the situation. Please contact the WarpStream team if you think a particular error log is too noisy! **\[metrics]**** Memory usage by host** * metric: `container.memory.usage` * group\_by: `host` **\[metrics]**** Used Cores** * metric: `container.cpu.usage` * group\_by: **\[metrics]**** Used Cores By Host** * metric: `container.cpu.usage` * group\_by: `host` ## Kafka Metrics and logs associated with the Kafka protocol provide insights into message handling, latency, and throughput. **\[metrics]**** Produce Throughput (uncompressed bytes)** * metric: `warpstream_agent_kafka_produce_uncompressed_bytes_counter` * group\_by: `topic` (requires enabling high-cardinality tags) * type: counter * unit: bytes * description: number of uncompressed bytes that were produced. **\[metrics]**** Produce Throughput (compressed bytes)** * metric: `warpstream_agent_kafka_produce_compressed_bytes_counter` **\[metrics]**** Produce Throughput (records)** * metric: `warpstream_agent_segment_batcher_flush_num_records_counter` * group\_by: * type: counter * unit: bytes * description: number of records that were produced. **\[metrics]**** Fetch Throughput (uncompressed bytes)** * metric: `warpstream_agent_kafka_fetch_uncompressed_bytes_counter` * group\_by: `topic` (requires enabling high-cardinality tags) * type: counter * unit: bytes * description: number of uncompressed bytes that were fetched. **\[metrics]**** Fetch Throughput (compressed bytes)** * metric: `warpstream_agent_kafka_fetch_compressed_bytes_counter` * group\_by: `topic` (requires enabling high-cardinality tags) * type: counter * unit: bytes * description: number of compressed bytes that were fetched. **\[metrics]**** Consumer Groups Lag** * metric: `warpstream_consumer_group_lag` * group\_by: `virtual_cluster_id`, `topic`, `consumer_group`, `partition` * tags: `virtual_cluster_id`, `topic`, `consumer_group`, `partition` * Partition is disabled by default. Set `-disableConsumerGroupsMetricsTags ""` flag or `WARPSTREAM_DISABLE_CONSUMER_GROUPS_METRICS_TAGS=""` environment variable on the Agents to enable the partition tag. * See these docs for more details: * type: gauge * unit: Kafka offsets * description: consumer group lag measured in *offsets*. **\[metrics]**** Kafka Inflight Connections** * metric: `warpstream_agent_kafka_inflight_connections` * group\_by: * type: gauge * unit: n/a * description: number of currently inflight / active connections. **\[metrics]**** Kafka Inflight Requests (per Connection)** * metric: `warpstream_agent_kafka_inflight_request_per_connection` * group\_by: * type: histogram * unit: n/a * description: number of currently in-flight Kafka protocol requests for individual connections. **\[metrics]**** Kafka Request Outcome** * metric: `warpstream_agent_kafka_request_outcome` * group\_by: `kafka_key,outcome` * type: counter * unit: n/a * description: outcome (success, error, etc) for each Kafka protocol request. **\[metrics]**** Kafka Request Latency** * metric: `warpstream_agent_kafka_request_latency` * group\_by: `kafka_key` * type: histogram * unit: seconds * description: latency for processing each Kafka protocol request. **\[metrics]**** Max Offset** * metric: `warpstream_max_offset` * group\_by: `virtual_cluster_id`, `topic`, `partition` * tags: `virtual_cluster_id`, `topic`, `partition` * Partition is disabled by default. Set `-disableConsumerGroupsMetricsTags ""` flag or `WARPSTREAM_DISABLE_CONSUMER_GROUPS_METRICS_TAGS=""` environment variable on the Agents to enable the partition tag. * See these docs for more details: * type: gauge * unit: Kafka offset * description: max offset for every topic-partition. Can be useful for monitoring lag for applications that don't use consumer groups and manage offsets externally. **\[metrics]**** Kafka Topic Count** * metric: `warpstream_topics_count` * group\_by: `virtual_cluster_id` * type: gauge * unit: n/a * description: how many topics are currently in your cluster **\[metrics]**** Kafka Topic Limit** * metric: `warpstream_topics_count_limit` * group\_by: `virtual_cluster_id` * type: gauge * unit: n/a * description: how many topics are allowed in your cluster. Upgrade your cluster tier or contact the WarpStream team if more are needed **\[metrics]**** Kafka Partition Count** * metric: `warpstream_partitions_count` * group\_by: `virtual_cluster_id` * type: gauge * unit: n/a * description: how many partitions are currently in your cluster **\[metrics]**** Kafka Partition Limit** * metric: `warpstream_partitions_count_limit` * group\_by: `virtual_cluster_id` * type: gauge * unit: n/a * description: how many partitions are allowed in your cluster. Upgrade your cluster tier or contact the WarpStream team if more are needed **\[metrics]**** Kafka Records Count** * metric: `warpstream_num_records` * group\_by: `topic`, `virtual_cluster_id` * type: gauge * unit: n/a * description: how many records are currently in a given topic or cluster * note this number might not match the number of active keys if you are using compacted topics ## Control Plane Visualizing WarpStream control plane latency and error rates can be useful for debugging. **\[metrics]**** Operations Outcome** * metric: `warpstream_agent_control_plane_operation_counter` * group\_by: `virtual_cluster_id`, `outcome`, `operation` * tags: `virtual_cluster_id`, `outcome`, `operation` * type: counter * unit: request * description: count of RPCs between Agents and control plane, groupable by outcome (success/error) and operation (RPC type). **\[metrics]**** Operations Latency** * metric: `warpstream_agent_control_plane_operation_latency` * group\_by: `virtual_cluster_id`, `outcome`, `operation` * tags: `virtual_cluster_id`, `outcome`, `operation` * type: histogram * unit: seconds * description: latency of RPCs between Agents and control plane, groupable by outcome (success/error) and operation (RPC type). ## Auto Migration **\[metrics]**** Proxy Throughput (compressed bytes)** * metric: `warpstream_agent_kafka_produce_forwarded_compressed_bytes_counter` * group\_by: `topic` (requires enabling high-cardinality tags), `outcome` * type: counter * unit: bytes * description: number of compressed bytes that were proxied to the source cluster. **\[metrics] Proxy Throughput (records)** * metric: `warpstream_agent_kafka_produce_forwarded_records_counter` * group\_by: `topic` (requires enabling high-cardinality tags), `outcome` * type: counter * unit: records * description: number of records that were proxied to the source cluster. **\[metrics] Unproxied Proxy Throughput (records)** * metric: `warpstream_orbit_auto_migration_unproxied_source_writes_num_records` * group\_by: `topic` * type: counter * unit: records * description: the number of detected unproxied records to the source cluster that bypassed the WarpStream proxy. ## Schema Registry Metrics and logs associated with the hosted schema registry provide insights into request handling, latency, and throughput. {% hint style="warning" %} **Enable Schema Registry Request Logs** Logging schema registry requests is **disabled by default**. **To enable request logs:** * Use the command-line flag `-schemaRegistryEnableLogRequest`. * Alternatively, set the environment variable `WARPSTREAM_SCHEMA_REGISTRY_ENABLE_LOG_REQUEST=true`. {% endhint %} **\[metrics]**** Schema Registry Requests** * metric: `warpstream_agent_schema_registry_outcome` * group\_by: `schema_registry_operation,outcome` * type: counter * unit: n/a * description: outcome (success, error, etc) for each Schema Registry request. **\[metrics]**** Schema Registry Latency** * metric: `warpstream_agent_schema_registry_request_latency` * group\_by: `schema_registry_operation,outcome` * type: histogram * unit: seconds * description: latency for processing each Schema protocol request. **\[metrics]**** Schema Registry Inflight Connections** * metric: `warpstream_agent_schema_registry_inflight_connections` * group\_by: * type: gauge * unit: n/a * description: number of currently inflight / active connections. **\[logs]**** Schema Registry Request Logs** * query: `service:sr-agent schema_registry_request` * group\_by: `outcome`, `request_type`, `schema_id`, `subject`, `version` * description: every schema registry request will emit a log with the following attributes: `request_type` and `outcome`. Some requests will also emit additional attributes such as `schema_id`, `subject`, etc if applicable. This is disabled by default and requires setting the command line flag `-schemaRegistryEnableLogRequest` or setting the environment variable `WARPSTREAM_SCHEMA_REGISTRY_ENABLE_LOG_REQUEST=true` **\[metrics]**** Num Invalid Records** * metric: `warpstream_schema_validation_invalid_record_count` * group\_by: `topic`, `reason` * type: counter * unit: n/a * description: counter of the number of invalid records that the agent detects when schema validation is enabled. * note that the topic would only be a tag if high cardinality is enabled. **\[metrics]**** Schema Linking Number of Source subject versions** * metric: `warpstream_schema_linking_source_subject_versions_count` * group\_by: `sync_id`, `config_id` * type: gauge * unit: n/a * description: number of source subject versions that Schema Linking is currently managing **\[metrics]**** Schema Linking Number of Newly Migrated Subject Versions** * metric: `warpstream_schema_linking_newly_migrated_subject_versions` * group\_by: `sync_id`, `config_id` * type: gauge * unit: n/a * description: number of newly migrated subject versions performed by the latest sync, this should usually be zero unless new schemas are found. Schemas are only new for one sync and the frequency is configurable with a default of 5m **\[metrics]**** Schema Versions Count** * metric: `warpstream_schema_versions_count` * group\_by: * type: gauge * unit: n/a * description: number of schemas currently in your registry. **\[metrics]**** Schema Versions Limit** * metric: `warpstream_schema_versions_limit` * group\_by: * type: gauge * unit: n/a * description: number of schemas allowed in your registry. ## Background Jobs The control plane assigns agents background jobs for things like compaction or retention. These are the metrics and logs on the efficiency and status of background operations, with a focus on compaction processes and the scanning of obsolete files. **\[logs]**** Compactions by Status and Level** * query: `service:warp-agent @stream_job_input.type:COMPACTION_JOB_TYPE status:info` * metric: \* * group\_by: `status`,`@stream_job_input.compaction.compaction_level` * description: number of compactions by compaction level and status (success, error). Occasional compaction errors are normal and expected, but most compactions should succeed. **\[metrics]**** Executed Jobs** * metric: `warpstream_agent_run_and_ack_job_outcome` * group\_by: `job_type` * tags: `job_type` `outcome` * type: counter * unit: n/a * description: outcome (and successful acknowledgement back to the control plane) of all jobs, groupable by job type and outcome. Jobs may fail intermittently, but most jobs should complete successfully. **\[metrics]**** Compaction Files per Level (Indicator of Compaction Lag)** * metric: `warpstream_files_count` * group\_by: `compaction_level` * type: gauge * unit: n/a * description: number of files in the LSM for each compaction level. The number of L2 files may be high for high volume / long retention workloads, but the number of files at L0 and L1 should always be low (< 1000 each). **\[metrics]**** Dead Files Scanner: Checked vs Deleted Files** * metric: `warpstream_agent_deadscanner_outcomes` * group\_by: `outcome` * tags: `outcome` * type: counter * unit: n/a * description: the "deadscanner" is a job type in WarpStream that instructs the Agents to scan the object storage bucket for files that are "dead". A file is considered dead if it exists in the object store, but the WarpStream control plane / metadata store has no record of it, indicating it failed to be committed or was deleted by data expiration / compaction. **\[logs]**** P99 Compaction Duration by Level** * query: `service:warp-agent @stream_job_input.type:COMPACTION_JOB_TYPE status:info` * metric: `@duration_ms` * group\_by: `status`,`@stream_job_input.compaction.compaction_level` * description: duration of compactions by compaction level. L1 and L2 compaction duration will vary based on workload, but L0 compactions should always be fast (<20 seconds). **\[logs]**** Compaction File Output Size** * query: `status:info` * metric: `@stream_job_output.compaction.file_metadatas.index_offset` * group\_by: `source`,`@stream_job_input.compaction.compaction_level` * description: compressed size of files generated by compaction. Useful for understanding the size of different files at different levels, but not something that needs to be closely monitored or paid attention to. ## Object Storage Metrics and logs on object storage operations' performance and usage patterns, offering insights into data retrieval, storage efficiency, and caching mechanisms. **\[metrics]**** S3 Operations (PUT)** * metric: `warpstream_blob_store_operation_latency` * filter\_tag: `operation:put_bytes`, `operation:put_stream` * group\_by: operation * type: histogram * unit: seconds * description: latency to perform PUT requests. Spikes of this value can indicate issues with the underlying object store. * note this metric is a histogram, so even if it emits latency, you can count the number of items emitted and get the number of operations. **\[metrics]**** S3 Operations (GET)** * metric: `warpstream_blob_store_operation_latency` * filter\_tag: `operation:get_stream`, `operation:get_stream_range` * group\_by: operation * type: histogram * unit: seconds * description: latency for time to first byte for GET requests. Spikes of this value can indicate issues with the underlying object store. * note this metric is a histogram, so even if it emits latency, you can count the number of items emitted and get the number of operations. # Hosted Prometheus Endpoint This page describes how to use WarpStream's hosted prometheus endpoint for collecting control plane metrics. ## Overview Almost all WarpStream metrics are exposed [directly in the Agents](/warpstream/agent-setup/monitor-the-warpstream-agents), including "control plane" metrics that correspond to control plane metadata and not any particular Agent. This is accomplished via a background job that the control plane schedules to "push" control plane metrics to an individual Agent that will then emit it as a regular metric. Exposing control plane metrics this way is convenient, but it can sometimes be problematic due to the resulting cardinality. For example, consumer group lag metrics are most useful when they're tagged by partition, but emitting consumer group metrics tagged by partition in the Agents makes the time series cardinality very high: `O(m * n)` where `m` is the number of topic-partitions and `n` is the number of the Agents. As a result, WarpStream offers a hosted Prometheus endpoint that captures WarpStream's control plane metrics. This endpoint is authenticated and can be scraped by your monitoring system to collect some control plane metrics without incurring the additional cardinality of the Agent host / pod names. ## Available Metrics ### All Clusters * `warpstream_control_plane_utilization` * `warpstream_diagnostic_failure` * `warpstream_files_count` * `warpstream_topics_count` * `warpstream_topics_count_limit` * `warpstream_partitions_count` * `warpstream_partitions_count_limit` * `warpstream_agent_heartbeat` * `warpstream_agent_cpu` * `warpstream_agent_num_vcpus` ### Kafka Clusters * `warpstream_consumer_group_state` * `warpstream_consumer_group_generation_id` * `warpstream_consumer_group_num_members` * `warpstream_consumer_group_num_topics` * `warpstream_consumer_group_num_partitions` * `warpstream_consumer_group_max_offset` * `warpstream_consumer_group_lag` * `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` * `warpstream_consumer_group_commit_ts` * `warpstream_produced_records` * `warpstream_max_offset` * `warpstream_min_offset` * `warpstream_num_records` * `warpstream_partition_size_uncompressed_bytes` * `warpstream_partition_size_estimated_compressed_bytes` ### Tableflow Clusters * `warpstream_tableflow_ingestion_lag_seconds` * `warpstream_tableflow_partition_offset_lag` * `warpstream_tableflow_query_lag_seconds` * `warpstream_tableflow_tables_count` * `warpstream_tableflow_files_count` * `warpstream_tableflow_snapshots_count` * `warpstream_tableflow_partitions_count` * `warpstream_tableflow_tables_limit` * `warpstream_tableflow_files_limit` * `warpstream_tableflow_snapshots_limit` * `warpstream_tableflow_partitions_limit` ### Schema Registry Clusters * `warpstream_schema_versions_count` * `warpstream_schema_versions_limit` ## Sample Prometheus Scraping Configuration ```yaml scrape_configs: - job_name: "warpstream" static_configs: - targets: ["api.warpstream.com"] metrics_path: "api/v1/monitoring/prometheus/virtual_clusters/$VIRTUAL_CLUSTER_ID" scheme: "https" basic_auth: username: prometheus password: $API_KEY ``` ## CURLing Manually {% code overflow="wrap" %} ```bash curl -u prometheus:$API_KEY "https://api.warpstream.com/api/v1/monitoring/prometheus/virtual_clusters/$VIRTUAL_CLUSTER_ID" ``` {% endcode %} An API Key can be obtained from the "API Keys" tab in the WarpStream console. For more details, see our [API Keys reference documentation](/warpstream/reference/api-reference/api-keys). ## Disabling Metrics Publishing Job in the Agents If you're scraping our Hosted Prometheus Endpoint, then you can disable the control plane metrics publishing job in the Agent as it will just be emitting duplicate metrics but with much higher cardinality due to the agent / pod level tags. To disable this job, set the `-disablePublishMetricsJob` flag or `WARPSTREAM_DISABLE_PUBLISH_METRICS_JOB=true` environment variable on your Agent deployment. ## Dedicated Metrics Agent Configuring a Prometheus scraping endpoint is not always convenient, especially if you're using our Datadog integration. As a result, we also support running the Agents in a dedicated `metrics` mode where the Agent binary will do nothing but scrape the Hosted Prometheus Endpoint and publish those metrics itself so you can ingest them alongside all your other Agent metrics. {% hint style="info" %} Agents running in `metrics` mode cannot process Kafka protocol messages, they will emit control plane metrics and do nothing else. {% endhint %} Running an Agent in metrics mode is as simple: {% code overflow="wrap" %} ```bash warpstream metrics -agentKey "$WARPSTREAM_AGENT_KEY" -defaultVirtualClusterID "$WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID" ``` {% endcode %} Optionally, you can enable the Datadog / statsd integration: {% code overflow="wrap" %} ```bash warpstream metrics -agentKey "$WARPSTREAM_AGENT_KEY" -defaultVirtualClusterID "$WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID" -enableDatadogMetrics ``` {% endcode %} Like the regular Agent binary, you can also use environment variables instead: {% code overflow="wrap" %} ```bash WARPSTREAM_AGENT_KEY=aks_XXXXX WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID=vci=XXXXX WARPSTREAM_ENABLE_DATADOG_METRICS=true warpstream metrics ``` {% endcode %} If you deploy WarpStream on Kubernetes with our Helm chart, all you have to do to enable this feature is set the value of `dedicatedMetricsPod.enabled` in your `values.yaml` to `true`. This will deployed a single dedicated pod that scrapes the WarpStream Hosted Prometheus Endpoint and publishes those metrics itself. Note that the chart will automatically take care of [disabling the metrics publishing job](#disabling-metrics-publishing-job-in-the-agents) in the Agents when you do this. ## API vs Agent Keys You can use an account-level or workspace-level API key to scrape Prometheus metrics for any cluster which that key is authorized for, or you can use an Agent Key to scrape prometheus metrics for a single cluster. In general, we recommend using a read-only Agent Key for scraping Prometheus metrics to provide the minimal required level of access to your monitoring tools. For more details, see our [Secrets Overview](/warpstream/reference/secrets-overview) reference documentation. # Recommended List of Alerts On this page we list the key metrics you should create alerts on. WarpStream was designed to minimize operational burden as much as possible. Therefore, the Agents are completely stateless and only depend on the underlying object store and the WarpStream control plane. The cloud provider manages and monitors the object store, and the WarpStream team manages and monitors the control plane. For this reason, the WarpStream Agents have very little to alert on. However, if you want to configure additional alerts, you can review our "Important Metrics and Logs" section for a list of key metrics/logs that are good candidates for monitors. In general, instrumentation is more useful for debugging than alerting. That said, we do recommend configuring a few alerts for resource usage. In addition to the ones below, we provide alerting through our [Diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) feature. ## Resource Usage Metrics It's important that the WarpStream Agents have sufficient capacity available to handle your workload and any potential spikes. For that reason, the most important thing to be alerted about with the WarpStream Agents is CPU and memory utilization. We usually recommend keeping memory/cpu under 50%. Note that since the WarpStream Agents are stateless, it's safe to auto-scale them based on CPU usage. * **CPU Usage** * Metric: `container.cpu.usage` * Alert condition: >70 * **Memory Usage** * Metric: `container.memory.usage` * Alert condition: >70 ## Application Metrics In addition to monitoring the resource utilization of the Agents, we also recommend monitoring your workload from your application. For example, tracking errors for producing and fetching data and monitoring your consumer group lag. You can find more information in [Monitoring Consumer Groups](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups). In addition, error rate and latencies for Kafka API operations can be monitored using these metrics emitted by the WarpStream Agents: * **Error Rate on Kafka API** * Metric: `warpstream_agent_kafka_request_outcome` * Filter by: `outcome:error` * Group by: `kafka_key` * **Latency on Kafka API** * Metric: `warpstream_agent_kafka_request_latency` * Group by: `kafka_key` But generally, **it's better to monitor them from your application** than from the Agents. # Monitoring Consumer Groups How to monitor your consumer groups. Most open source Kafka deployments use external tooling to monitor consumer group lag. Some of this tooling is compatible with WarpStream because it uses the public Kafka API, and others like Burrow are incompatible because they rely on internal implementation details of Kafka like reading the internal consumer group offset topics. Luckily, WarpStream has support for monitoring consumer groups built in, so no external tooling is required. In addition, WarpStream reports consumer group lag measured **in time** as well as measured **in offsets**. See [our blog post about measuring consumer lag in time](https://www.warpstream.com/blog/the-kafka-metric-youre-not-using-stop-counting-messages-start-measuring-time) for more details about why this is valuable. Consumer group metadata and lag is available in a variety of locations with WarpStream. ## UI The WarpStream UI exposes consumer group metadata and lag. This is not useful for alerting purposes, but can be helpful when debugging consumers.

Click on an individual consumer group to see more details.

## API Consumer group lag is available through dedicated [our HTTP/JSON API](/warpstream/reference/api-reference/monitoring/describe-all-consumer-groups). ## Metrics {% hint style="warning" %} **We recommend using the** [**hosted prometheus endpoint**](/warpstream/agent-setup/monitor-the-warpstream-agents/hosted-prometheus-endpoint) **for consumer group metrics rather than directly going through the Agents.**\ This can be helpful for workloads with a high number of topics / partitions where the the time series cardinality is already high and multiplying it by the unique Agent pod names would make it even higher. {% endhint %} The Agents expose [built-in metrics](/warpstream/agent-setup/monitor-the-warpstream-agents) that you can scrape within your own environment. Included in these metrics are all the metrics you need to monitor your applications for consumer group lag. Some of the metrics, particularly the consumer group metrics, can become very high cardinality if the cluster contains a lot of topics or partitions. To reduce the cardinality of the consumer group lag metrics, you can either disable them entirely using the `disableConsumerGroupMetrics` flag or setting `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS=true` as an environment variable. The most important metrics are `warpstream_consumer_group_lag` (lag in offsets per tuple of ``) and `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` which is a mouthful but can be used to configure alerts based on time instead of offset count. {% hint style="info" %} The `partition` tag is disabled by default to reduce cardinality. If you want to enable it, set the `disableConsumerGroupsMetricsTags` flag or `WARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGS` environment variable to an empty string (the default value is "partition").\ \ When the `partition` tag is disabled, the `consumer_group_lag` metric will be the sum of the consumer group lag across the topic's partitions. The `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` metric will be the max of the estimated lag across the topic's partitions. {% endhint %}
NameDescriptionTags
warpstream_consumer_group_lagDifference (in offsets) between the max offset and the committed offset for every active consumer group.virtual_cluster_id, topic, consumer_group and partition
warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds

Gives a rough estimate of how far behind (in seconds) a consumer group is from the latest messages.

Note: This is NOT for precise measurement; it's a coarse estimate.

virtual_cluster_id, topic, consumer_group and partition
warpstream_consumer_group_generation_idA unique identifier that increases with every consumer group rebalance. This allows you to easily track the number and frequency of rebalances.virtual_cluster_id and consumer_group
warpstream_consumer_group_max_offsetMax offset of a given topic-partition for every topic-partition in every consumer group.virtual_cluster_id, topic, consumer_group and partition
warpstream_consumer_group_stateState of each consumer group (stable, rebalancing, empty, etc)consumer_group, group_state
warpstream_consumer_group_num_membersNumber of members in each consumer group.consumer_group
warpstream_consumer_group_num_topicsNumber of topics in each consumer group.consumer_group
warpstream_consumer_group_num_partitionsNumber of partitions in each consumer group.consumer_group
## Measuring E2E Latency More Accurately As suggested by its name, the `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` metric is coarse. For example, if the actual end-to-end (E2E) latency of an application is 800ms, this metric may report the E2E latency as high as 5-8s. For most applications, this is sufficiently accurate for monitoring and alerting purposes, but some applications may require more fine-grained observability. In that case, the best approach is to monitor the E2E latency manually in your application. This can be accomplished by emitting a metric for the delta between the current timestamp and the timestamp of each individual record in your application. There are three different ways that you can assign a timestamp to individual records when they're produced so that they're available to your consumer application: 1. Every Kafka record has a built-in timestamp. If your application doesn't specifically override this value, then it will automatically be set to the current time by the producer client when the record was produced, or to the current time of the broker when the record was written to disk. Which value is used depends on the configured value of [`message.timestamp.type`](https://docs.warpstream.com/warpstream/agent-setup/monitor-the-warpstream-agents/pages/qC7LyPpt9ZsJygEz4tXF#message.timestamp.type) on your cluster / topic. 2. You can add a custom header to your Kafka records with the current timestamp when producing records. 3. You can add a custom field in the payload of your Kafka records with the current timestamp when producing records. {% hint style="info" %} Note that it's impossible for WarpStream to automate this measurement because the WarpStream Agents have no way to accurately measure at what time the consumer application actually received and processed the records. As a result, the `estimated_lag_very_coarse` metric has to wait for the records to be **committed** (which may happen many seconds after the records are processed) before it can consider them "processed" from an E2E latency perspective. That's why the built-in metric tends to over-estimate E2E latency by a non-trivial amount. The `estimated_lag_very_coarse` metric also has to rely on some amount of linear interpolation for efficiency reasons which also makes it less accurate than the approach described in this section. {% endhint %} # Datadog Integration {% hint style="info" %} **Datadog metrics** Starting from Warpstream Agent `v679` all metrics on Datadog will start with `warpstream.` and no longer `warpstream_` . All references to metrics in our doc will keep mentioning metrics starting with `warpstream_` so you have to do the conversion when you are using Datadog and a Warpstream Agent recent enough. You can fall back to the previous behavior by setting the `WARPSTREAM_DATADOG_NORMALIZER_PREFIX_WITH_DOT` environment variable to `false`. This change comes along the official release of our Datadog integration, making all the Warpstream Agent metrics free if you install the integration (and use the new naming convention) {% endhint %} ## Integration & Statsd Client (Push) We recommend [installing the official Datadog integration](https://docs.datadoghq.com/integrations/warpstream/). We also have a [pre-made Datadog Dashboard](/warpstream/agent-setup/monitor-the-warpstream-agents/premade-datadog-dashboard) that you can easily import to get started. The WarpStream Agents embed the Datadog statsd metrics client and can push directly to the Datadog agent by setting the `-enableDatadogMetrics` flag or adding `WARPSTREAM_ENABLE_DATADOG_METRICS=true` as an environment variable. Additionally, to configure the Datadog client properly, the `DD_AGENT_HOST` environment variable needs to be set to the host IP. For Agents running in AWS with IMDS enabled this step can be skipped. {% hint style="info" %} The integration may complain in Datadog that it's missing some integration metric. You can disregard this, the python integration steps are optional and do not provide any additional insights on the WarpStream Agent health. {% endhint %} ## Prometheus Exporter (pull) An alternative is to follow the [Datadog instructions](https://docs.datadoghq.com/integrations/openmetrics/) for scraping Prometheus/OpenTelemetry metrics using the Datadog Agent. Configuration will vary from environment to environment, but you should end up with something like the following configuration (Kubernetes example): ``` spec: template: metadata: annotations: ad.datadoghq.com/warpstream-agent.checks: | { "openmetrics": { "init_config": {}, "instances": [ { "openmetrics_endpoint": "http://%%host%%:8080/metrics", "metrics": [".*"], "send_distribution_buckets": true, "collect_counters_with_distributions": true, "max_returned_metrics": 2000 } ] } } ``` Which specifies that the Datadog Agent should scrape the WarpStream agent at port `8080` for metrics, and that it should scrape all the custom metrics that the WarpStream agent exposes. {% hint style="info" %} The Datadog Agent will scrape only 2,000 metrics by default. This limit may be too low for WarpStream if you have many topics and consumer groups and/or have high cardinality metrics enabled. If you observe dropped or missing metrics, consider increasing this value. {% endhint %} # Pre-made Datadog Dashboard Use the [Datadog dashboard JSON import feature](https://docs.datadoghq.com/dashboards/#copy-import-or-export-dashboard-json) to automatically import the following WarpStream agent dashboard into your Datadog account: [WarpStream Agent Datadog dashboard](https://raw.githubusercontent.com/warpstreamlabs/pre-made-dashboards/refs/heads/main/datadog-dashboard-warpstream-agent.json). # Pre-made Grafana Dashboard You can import this existing dashboard to your Grafana account: Example of a `prometheus.yml` file that you can use as a datasource with an agent running locally: ``` global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: ['localhost:9090'] - job_name: warpstream static_configs: - targets: ['localhost:8080'] labels: cluster: 'warpstream' namespace: 'warpstream' container: 'agent' job: 'warpstream/warpstream-agent' ``` # Infrastructure as Code This section contains information on how to configure WarpStream using "Infrastructure as Code" tools like Terraform and Helm. WarpStream provides a number of tools to manage clusters using "infrastructure as code": 1. The WarpStream terraform provider. 1. [Github](https://github.com/warpstreamlabs/terraform-provider-warpstream) 2. [Docs](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs) 3. [Example for managing BYOC clusters along with their topics and configuration](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/main/examples/byoc-with-topics/main.tf) 2. The [WarpStream Helm Chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent) for clusters deployed in Kubernetes. 3. [Hosted metadata endpoint for BYOC clusters](/warpstream/kafka/configure-kafka-client/administrate-your-byoc-clusters-with-serverless). # Terraform Provider Links to WarpStream Terraform provider resources. 1. [Github](https://github.com/warpstreamlabs/terraform-provider-warpstream) 2. [Docs](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs) 3. [Example for managing BYOC clusters along with their topics and configuration](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/main/examples/byoc-with-topics/main.tf) 4. [Example for managing Bento pipelines](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/main/examples/resources/warpstream_pipeline/resource.tf#L17) 5. [Example for managing Orbit pipelines](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/main/examples/resources/warpstream_pipeline/resource.tf#L38) # Helm charts WarpStream Helm charts. WarpStream's [Helm charts](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent) can be used to deploy the Agents in Kubernetes. # Terraform Modules To make deployment of WarpStream easier we have published Terraform modules and examples for various services. ### AWS - Elastic Container Service Source: This module creates an ECS cluster, a S3 storage bucket, IAM Roles, an ECS Task, and an ECS Service running the WarpStream Agent. This module assumes you already have a VPC with a NAT Gateway to the internet. Internet access is needed to be able to pull the WarpStream Agent container image. A fully working example can been seen here: ### AWS - Elastic Kubernetes Service Source: This module creates an EKS cluster, a S3 storage bucket, IAM Roles, and deploys the WarpStream Agent helm chart. This module assumes you already have a VPC with a NAT Gateway to the internet. Internet access is needed to be able to pull the WarpStream Agent container image. A fully working example can been seen here: # Protecting important resources from accidental deletion ## Soft deletion By default, when you delete a Virtual Cluster or a topic, Warpstream does not delete the associated data. The data is retained and you can restore the Virtual Cluster or topic. Warpstream offers two levels of protection against accidental deletion for both Virtual Clusters and topics. First, both Virtual Clusters and topics are soft-deleted by default. Second, WarpStream offers deletion protection, which protects resources from being deleted. ### Restoring a Virtual Cluster When you delete a virtual cluster, either through the API or through the console, it will appear in the "Recently Deleted Clusters" section on the [Virtual clusters page](https://console.warpstream.com/virtual_clusters). It will stay there for **30 days** before WarpStream deletes the data associated with it. You can restore the cluster at any time by clicking the `Restore Cluster` button.
### Restoring a Topic When you delete a topic from a Kafka client, the WarpStream API, or the console, you will no longer be able to produce to or consume from the topic. However, WarpStream will retain the data and allow you to restore the topic. Once a topic is restored, you will be able to resume producing and consuming. The default retention period for soft-deleted topics is 24 hours. You can restore the topic by clicking `Edit > Restore Topic` in the "Recently Deleted Topics" view in the WarpStream Console.
It is also possible to restore it by using the `api/v1/undelete_topic` API. To configure how WarpStream interacts with topic deletion with these parameters, please use your usual Kafka client to set these cluster-level parameters (in Kafka terminology, these are broker-level configurations):
configuration parameter
warpstream.soft.delete.topic.enable

if true, topic deletion will be a soft deletion, and it will be possible to restore the topics.

If false, deleting a topic will cause the immediate deletion of all of the associated data, with no way to recover it.

Defaults to true.

warpstream.soft.delete.topic.ttl.hoursIf warpstream.soft.delete.topic.enable is true, a deleted topic's data will be kept for this many hours before being irrecoverably deleted.

Defaults to 24.
## Deletion protection Virtual Clusters and topics in WarpStream can be protected from deletion. When deletion protection is enabled, you will be unable to delete resources until deletion protection is turned off. This prevents accidental destructive operations, which can occur when using infrastructure-as-code tools. ### Through terraform To set the deletion protection flag using Terraform, please refer to our Terraform provider documentation for [Virtual Clusters](https://registry.terraform.io/providers/warpstreamlabs/warpstream/2.1.2/docs/resources/virtual_cluster) and for [Topics](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs/resources/topic). Enabling deletion protection will look like this for virtual clusters: ```hcl resource "warpstream_virtual_cluster" "test_configuration" { name = "vcn_test_configuration" tier = "pro" configuration = { enable_deletion_protection = true } } ``` And like this for topics: ```hcl resource "warpstream_topic" "topic" { topic_name = "logs" partition_count = 1 virtual_cluster_id = warpstream_virtual_cluster.test.id enable_deletion_protection = true } ``` ### Through the API * To set the deletion protection flag on a Virtual Cluster, use the [UpdateConfiguration](/warpstream/reference/api-reference/virtual-clusters/updateconfiguration) API. * To set the deletion protection flag on a topic, set the topic-level configuration `warpstream.deletion.protection.enabled` to `true`. # Configure Clients This pages explains how to configure your Apache Kafka client with WarpStream. {% hint style="warning" %} Don't forget to review our documentation for [tuning your Kafka client for maximum performance with WarpStream](/warpstream/kafka/configure-kafka-client/tuning-for-performance) once you're done. A few small changes in client configuration can result in 10-20x higher throughput when using WarpStream. {% endhint %} WarpStream provides API-compatibility with Apache Kafka, so you can just connect your clients to the WarpStream agents by setting the WarpStream Application Bootstrap URL (obtained from the [WarpStream console](https://console.warpstream.com/)) as the value in the Kafka bootstrap settings. For example, using the `librdkafka` client in Go: ```go var ( // Not explicitly required, but will eliminate inter-zone networking // in multi-zone deployments. availabilityZone = lookupAZ() // Not explicitly required, but may help with debugging to isolate // individual clients in logs and telemetry. sessionID = uuid.New().String() // The string you would normally use as your client ID with regular // Apache Kafka. applicationID = "application-foo" // WarpStream Client ID configurations should be key value pairs and comma seperated. clientID = fmt.Sprintf("%s,ws_si=%s,ws_az=%s", applicationID, sessionID, availabilityZone) bootstrapServer = "api-80ba097c-d4ef-4e0b-8e86-d05b80fee6ed.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092" ) producerConfig := map[string]kafka.ConfigValue{ // Not developing locally? In K8s you should use the Agent // service from Agent chart. If you're not using K8s, then // you can use our convenience hosted bootstrap URL which // you can find in the "Connect" tab of the virtual cluster // view in the WarpStream console. "bootstrap.servers": "localhost:9092", "broker.address.family": "v4", "log.connection.close": "false", "client.id": clientID, } config := kafka.ConfigMap(producerConfig) producer, err := kafka.NewProducer(&config) if err != nil { return fmt.Errorf("error initialiazing kafka producer: %w", err) } ``` ### Client IDs {% hint style="warning" %} If you just want to get up and running quickly, you can use a regular client ID and skip encoding your application's availability zone in the client ID. However, beware that you may incur higher costs due to inter-zone networking. {% endhint %} You can read more about the WarpStream service discovery system in our [Service Discovery reference documentation](/warpstream/overview/architecture/service-discovery), but if you want to take advantage of WarpStream's zone-aware service discovery system and achieve good load balancing, you must encode your application's availability zone in your Kafka client's client ID using the format in the code sample above. To learn more about the WarpStream-specific client ID features (like ws\_si and ws\_az) check out [our documentation on configuring Kafka Client ID features](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features). Follow [this documentation](/warpstream/kafka/configure-kafka-client/configure-clients-to-eliminate-az-networking-costs) to learn how to determine your application's availability zone in all major cloud environments and properly template your client ID. # Tuning for Performance Instructions on how to tune various Kafka clients for performance with WarpStream. ## Write / Produce Throughput WarpStream Agents have no local disks and write data directly to object storage. Creating files in object storage is a relatively high latency operation compared to writing to a local disk. For example, in our experience, the P99 latency for writing a 4 MiB file to S3 is \~ 400ms. As a result of this property, WarpStream Agents will only reach their maximum write throughput with highly concurrent workloads or workloads that write large batches. This is a practical consideration that derives mathematically from the ratio of system throughput to system latency (known in queuing theory as [Little’s Law](http://en.wikipedia.org/wiki/Little's_law)). For example, an individual WarpStream cluster can easily sustain > 1GiB/s of write throughput, but to accomplish that with write latencies of 400ms, there must be a high degree of concurrency. For example, a write rate of 1GiB/s can be achieved with 1MiB batch sizes if there are 400 concurrent produce requests at any given moment. *Not having enough outstanding requests is the single biggest reason for low write throughput when using WarpStream.* ## Read / Fetch Throughput When Kafka clients issue Fetch requests, they specify a limit on the amount of data to be returned (in aggregate, and per-partition) by the Broker. In Apache Kafka, the brokers interpret the limit in terms of *compressed* bytes, but in WarpStream the Agents interpret the limit in terms of *uncompressed* bytes. For more details on why this decision was made, check out our more [detailed documentation about compression in WarpStream](/warpstream/kafka/reference/compression#difference-with-kafka-for-fetch-requests). That said, the practical implication of this decision is that for some workloads, you will need to tune your Kafka consumer clients to request more data per individual Fetch request to achieve the same throughput with WarpStream. {% hint style="warning" %} **Note on `fetch.min.bytes` and `fetch.max.wait.ms`:** WarpStream does not currently support the `fetch.min.bytes` configuration. As a result, `fetch.max.wait.ms` behaves differently from standard Apache Kafka: in WarpStream, it only controls how long the server will retry when there is no data available at all. If any data is available, WarpStream returns it immediately, regardless of the `fetch.max.wait.ms` value. In standard Kafka, `fetch.max.wait.ms` controls how long the broker waits when there isn't enough data to satisfy `fetch.min.bytes` -- since WarpStream doesn't support `fetch.min.bytes`, this interaction doesn't apply. To increase batch sizes and reduce consumer CPU overhead, we recommend increasing the poll interval in your application logic. {% endhint %} ## Client settings We currently have documentation and recommended settings on how to achieve high write/read throughput with the following clients: * [Librdkafka / confluent-kafka](#librdkafka) * [Java Client](#java-client) * [Franz-go](#franz-go) * [Segment kafka-go](#segment-kafka-go) * [Sarama](#sarama) * [KafkaJS](#kafkajs) * [Kafka Connect](#kafka-connect) {% hint style="info" %} The documentation on this page focuses on achieving as much write throughput as possible using a single "instance" of each Kafka client. However, if you're still struggling to drive the amount of write throughput you want from your application and the WarpStream Agents CPU utilization isn't very high, then you can always create more "instances" of the Kafka client in your application as well. {% endhint %} *** ## Librdkafka / confluent-kafka / confluent-kafka-javascript {% hint style="info" %} Note that the values of `socket.receive.buffer.bytes` and `socket.send.buffer.bytes` must be set to 0 if you want good performance in environments where the latency between the WarpStream Agents and the Kafka clients is high, like when they're deployed in different regions. For that reason, its best to always just set them to 0 so the kernel can auto-tune them based on the observed network latency and performance. {% endhint %} | Consumer Settings | Recommended Value | | ------------------------------------ | ----------------- | | `topic.metadata.refresh.interval.ms` | `60000` | | `fetch.max.bytes` | `50242880` | | `max.partition.fetch.bytes` | `50242880` | | `fetch.wait.max.ms` | `10000` | | `socket.send.buffer.bytes` | `0` | | `socket.receive.buffer.bytes` | `0` | | Producer Settings | Recommended Value | | --------------------------------------- | --------------------- | | `topic.metadata.refresh.interval.ms` | `60000` | | `queue.buffering.max.kbytes` | `1048576` | | `queue.buffering.max.messages` | `1000000` | | `message.max.bytes` | `64000000` | | `batch.size` | `16000000` | | `batch.num.messages` | `100000` | | `linger.ms` | `100` | | `sticky.partitioning.linger.ms` | `25` | | `enable.idempotence` | `false` | | `max.in.flight.requests.per.connection` | `1000000` | | `partitioner` | `consistent_random` | | `retry.backoff.max.ms` | `60000` | | `socket.send.buffer.bytes` | `0` | | `socket.receive.buffer.bytes` | `0` | | `request.timeout.ms` | `30000` (the default) | For both consumers and producers, the metadata refresh interval in librdkafka is configured to a more frequent 1 minute, compared to its default setting of 5 minutes. This adjustment enhances the client's responsiveness to changes in the cluster, ensuring efficient load balancing. We also configure `retry.backoff.max.ms` to `60000` to prevent Metadata retry storms in clusters with a very high number of Kafka clients (thousands or tens of thousands). The producer settings above should result in good write throughput performance, regardless of your key distribution or partitioning scheme. However, performance will generally be improved when using `NULL` keys and not specifying which partition individual records should be written to. This enables the `consistent_random` partitioner to use the "sticky partitioning linger" functionality to produce optimally sized batches which improves compression ratios over the wire and reduces overhead both in the Kafka client and the agent. ```go msg := &kafka.Message{ TopicPartition: kafka.TopicPartition{Topic: &topicName}, Key: nil, Value: record.Value, } err := producer.Produce(msg, events) if err != nil { return nil, fmt.Errorf("error producing record: %w", err) } ``` However, as long as the idempotent producer functionality is disabled, this is not strictly required for achieving good throughput. Another thing to keep in mind when using Librdkafka is that it is one of the few Kafka libraries that never combines batches for multiple partitions owned by the same broker into a single Produce request: That's why we recommend a high value for `max.in.flight.requests.per.connection`, especially when writing to a high number of partitions. ### A note on idempotence Enabling the idempotent producer functionality in the librdkafka client library in conjunction with WarpStream can result in extremely poor producer throughput and very high latency. This is the result of four conspiring factors: 1. WarpStream has higher produce latency than traditional Apache Kafka 2. [WarpStream's service discovery mechanism](https://www.warpstream.com/blog/hacking-the-kafka-protocol) is implemented such that each client believes a single Agent is the leader for all topic-partitions at any given moment 3. Librdkafka only allows 5 concurrent produce requests *per connection* when the idempotent producer functionality is enabled instead of *per partition* 4. Librdkafka never combines batches from multiple partitions owned by the same broker into a single Produce request: As a result of all this, the only way to achieve high write throughput with the Librdkafka library in conjunction with WarpStream is to avoid specifying any keys or specific partitions for each record as demonstrated above and allow the `consistent_random` partitioner and "sticky partitioning linger" functionality to generate produce requests containing very large batches for a single partition at a time. See our [librdkafka known issues documentation](/warpstream/kafka/configure-kafka-client/known-issues#idempotence-performance-in-librdkafka) for more suggestions on how to mitigate this performance problem. *** ## Java Client For both producers and consumers we recommend setting `metadata.max.age.ms` to `60000` to enable faster load-balancing and allow the clients to react to Agent auto-scaling faster. `metadata.recovery.strategy` should always be set to `rebootstrap` otherwise your clients may end up disconnected from the cluster and unable to reconnect if a sufficient number of the Agent containers are rescheduled which is common in highly dynamic environments like Kubernetes. Finally, `send.buffer.bytes` and `receive.buffer.bytes` should be set to `-1` to allow the Kernel to auto-tune them. {% hint style="info" %} The default values of `send.buffer.bytes` (100KB) and `receive.buffer.bytes` (64KB) will work in environments where latency between the WarpStream Agents and the Kafka client is low, like when they're both deployed in the same region. Howevewr, throughput will be very poor in environments where the latency between them the Agents and the clients is high, like when they're deployed in regions that are far apart. For this reason, its best to just set these configuration values to `-1` so the kernel can automatically tune them based on the observed latency and network performance. {% endhint %} The remainder of the recommended configuration settings are either Produce or Consumer specific so we've broken them out into dedicated sections. ### Producer Settings The Java client does not suffer from the same limitations that Librdkafka does and is able to create Produce requests that combine batches for many different partitions. That said, there are still a few settings that should be tweaked to enable producing data at a high rate. The easiest way to achieve high throughput with the Java producer client is with idempotency disabled. #### Idempotence Disabled (Recommended) | Producer Settings | Recommended Value | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable.idempotence` | `false` | | `max.in.flight.requests.per.connection` | `1000` | | `linger.ms` | `100` | | `metadata.max.age.ms` | `60000` | | `batch.size` | `100000` | | `buffer.memory` | `128000000` (for extremely high volume applications or scenarios where a single Kafka producer is being shared by many different cores, consider using a larger value like `512000000` here) | | `max.request.size` | `64000000` | | `compression.type` | `lz4` | | `metadata.recovery.strategy` | `rebootstrap` | | `send.buffer.bytes` | `-1` | | `receive.buffer.bytes` | `-1` | | `request.timeout.ms` | `30000` (the default) | This should allow your producer to achieve high throughput regardless of how many topic-partitions are in the output topic. #### Idempotence Enabled The Java Kafka producer client can still achieve high throughput with idempotence enabled, however, throughput will be limited if the number of partitions being actively produced to is low. For example, if the output topic only has one partition **or** the record key's result in all of the records getting written to a small subset of partitions. Start with the settings recommended below. However, if you suspect that your throughput is limited because your workload is only producing to a small number of topic-partitions, then slowly increase the value of `batch.size` until you're able to achieve the desired throughput. | Producer Settings | Recommended Value | | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable.idempotence` | `true` | | `max.in.flight.requests.per.connection` | `5` | | `linger.ms` | `100` | | `metadata.max.age.ms` | `60000` | | `batch.size` | `100000` | | `buffer.memory` | `128000000` (for extremely high volume applications or scenarios where a single Kafka producer is being shared by many different cores, consider using a larger value like `512000000` here) | | `max.request.size` | `64000000` | | `compression.type` | `lz4` | | `metadata.recovery.strategy` | `rebootstrap` | | `send.buffer.bytes` | `-1` | | `receive.buffer.bytes` | `-1` | | `request.timeout.ms` | `30000` (the default) | ### Consumer Settings {% hint style="warning" %} WarpStream's zone-aware service discovery system is incompatible with the Java client's rack-aware consumer strategy. When using WarpStream, do **not** configure the rack field on your Kafka clients or enable the rack-aware consumer strategy. If you enable those setting, your Kafka consumers will experience a large amount of unnecessary rebalances. Instead, unset those fields and follow [our documentation on how to eliminate inter-AZ networking costs](/warpstream/kafka/configure-kafka-client/configure-clients-to-eliminate-az-networking-costs). {% endhint %}
Consumer SettingsRecommended Value
metadata.max.age.ms60000
fetch.max.bytes50242880
max.partition.fetch.bytes50242880
metadata.recovery.strategyrebootstrap
fetch.max.wait.ms10000
send.buffer.bytes-1
receive.buffer.bytes-1
*** ## Franz-go ### Producer Configuration The Franz-go library is one of the most performant Kafka libraries. With the following configuration, it can achieve high write throughput for almost any workload. #### Idempotency Disabled (recommended) | Producer Settings | Recommended Value | | ------------------------------------- | ----------------------------------------------------------- | | `MetadataMaxAge` | `60 * time.Second` | | `MaxBufferedRecords` | `1_000_000` | | `ProducerBatchMaxBytes` | `16_000_000` | | `RecordPartitioner` | `kgo.UniformBytesPartitioner(1_000_000, false, false, nil)` | | `ProduceRequestTimeout` | `10 * time.Second` (the default) | | `DisableIdempotentWrite` | `true` | | `MaxProduceRequestsInflightPerBroker` | `100` | #### Idempotency Enabled | Producer Settings | Recommended Value | | ----------------------- | ----------------------------------------------------------- | | `MetadataMaxAge` | `60 * time.Second` | | `MaxBufferedRecords` | `1_000_000` | | `ProducerBatchMaxBytes` | `16_000_000` | | `RecordPartitioner` | `kgo.UniformBytesPartitioner(1_000_000, false, false, nil)` | | `ProduceRequestTimeout` | `10 * time.Second` (the default) | ### Consumer Configuration | Consumer Settings | Recommended Value | | ------------------------ | ------------------ | | `MetadataMaxAge` | `60 * time.Second` | | `FetchMaxBytes` | `50_000_000` | | `FetchMaxPartitionBytes` | `50_000_000` | | `FetchMaxWait` | `10 * time.Second` | {% hint style="warning" %} If you are using a version **older than v1.17.1**, in FranzGo, metadata doesn't auto-refresh on errors, unlike other libraries. Users need to manually call `ForceMetadataRefresh()` or shorten the refresh interval to 10 seconds. This approach boosts performance by quickly identifying and recovering from failures. {% endhint %} *** ## Segment kafka-go {% hint style="warning" %} If you're writing a new application in Go, we highly recommend using the franz-go library instead, as it is significantly more feature-rich, performant, and less buggy. That said, we do support the Segment kafka-go library since many existing applications already use it. {% endhint %} Similar to librdkafka, the Segment kafka-go library will never combine produce requests for multiple different partitions into a single request. Even worse, the Segment library will never issue concurrent Produce requests for a single partition, either! Those two combinations of things can be problematic for achieving high write throughput with a wide variety of workloads. However, in many cases, reasonable throughput can be achieved with the following settings: | Producer Settings | Recommended Value | | ----------------- | ---------------------------------------------------- | | `BatchTimeout` | `1 * time.Second` | | `BatchSize` | `100_000` (or `10_000` if this uses too much memory) | | `BatchBytes` | `16_000_000` | | `WriteTimeout` | `10 * time.Second` (the default) | Note: Unlike other libraries segmentio does not require tuning the metadata refresh interval. Producers default to a 6-second interval, while consumers automatically reconnect using the bootstrap server upon detecting a connection failure. Caution: Unexpected behavior, such as abrupt halts in metadata refreshing, has been observed in Segmentio clients. ### Idempotence This library does not support idempotency. ## **Sarama** {% hint style="warning" %} If you're writing a new application in Go, we highly recommend using the franz-go library. That said, we do support the Sarama library since many existing applications already use it. {% endhint %} ### Producer ```go config := sarama.NewConfig() config.Net.ReadTimeout = 60 * time.Second config.Metadata.RefreshFrequency = 60 * time.Second config.Producer.MaxMessageBytes = 16_000_000 config.Producer.Flush.Bytes = 16_000_000 config.Producer.Flush.MaxMessages = 1_000_000 config.Producer.Flush.Frequency = 25 * time.Millisecond config.Producer.Compression = sarama.CompressionLZ4 config.Producer.Timeout = 10 * time.Second ``` #### A note on idempotence and data ordering The latest version of the Sarama library has significant liveness and correctness issues. It fails to maintain strict ordering of produced records, even when `Net.MaxOpenRequest` is set to `1`, and it does not implement the idempotent producer protocol correctly which can lead to messages failing to be delivered and ultimately, data loss. More details about these issues are described in [this P.R](https://github.com/IBM/sarama/pull/2943). If you have an existing application that uses Sarama and doesn't enable idempotency, it will work fine with WarpStream. However, if you care about idempotency and/or strict data ordering of produced records, we highly recommend using the franz-go library instead. ### Consumer ```go config := sarama.NewConfig() config.Metadata.RefreshFrequency = 60 * time.Second config.Net.ReadTimeout = 90 * time.Second config.Consumer.Group.Rebalance.Timeout = 60 * time.Second config.Consumer.Group.Session.Timeout = 45 * time.Second config.Consumer.Group.Heartbeat.Interval = 5 * time.Second config.Consumer.MaxProcessingTime = 20 * time.Second config.Consumer.Fetch.Default = 50_000_000 config.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategyRange config.Consumer.MaxWaitTime = 10 * time.Second ``` ## KafkaJS ```javascript client = new Kafka({ clientId: '$CLIENT_ID', brokers: ['$BOOTSTRAP_BROKER'], connectionTimeout: 10000, requestTimeout: 30000 }); ``` ### Producer Unlike most Kafka clients, KafkaJS expects the user of the application to handle batching on their own. To achieve high write throughput, issue produce requests with large batches (many records in each call to `send()` and high concurrency. ```javascript const producer = kafka.producer({ metadataMaxAge: 60000, maxInFlightRequests: null, compression: CompressionTypes.LZ4 }); ``` ### Consumer ```javascript const consumer = kafka.consumer({ sessionTimeout: 60000, rebalanceTimeout: 60000, heartbeatInterval: 3000, metadataMaxAge: 60000, maxBytesPerPartition: 50000000 maxBytes: 50000000, maxInFlightRequests: null, maxWaitTimeInMs: 10000 }); ``` ## Python client We recommend using [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) for Python applications. As it is a wrapper around librdkafka, all [the recommended settings](#librdkafka-confluent-kafka) from the librdkafka section apply. ## Kafka Connect {% hint style="info" %} Kafka Connect uses the Java client under the hood, so we recommend familiarizing yourself with our [Java Client Setting Recommendations](#java-client) before tuning Kafka Connect. {% endhint %} Kafka Connect internally uses Kafka topics to manage its configuration, status, and offsets. A critical and performance-sensitive operation is task reconfiguration. During this process, Connect has a hardcoded 30-second timeout to perform two key actions: 1. Read the *entire* connector configuration topic from its beginning. 2. Write a new configuration for each task (as defined by `tasks.max`) within a single transaction. The default Kafka Connect settings are inefficient for this workflow. They generate a high volume of small requests: one request per-record write and numerous small fetches to read the topic. You must tune the Kafka Connect worker's configuration to batch requests. Increasing batch sizes, buffers and timeouts for both reads and writes will reduce the number of requests, preventing timeouts. For that you need to set the following environment variables: ```yaml CONNECT_CLIENT_ID: my_application_name,ws_az=$AZ CONNECT_FETCH_MAX_BYTES: '50242880' CONNECT_MAX_PARTITION_FETCH_BYTES: '26214400' CONNECT_FETCH_MAX_WAIT_MS: '10000' CONNECT_ENABLE_IDEMPOTENCE: 'false' CONNECT_MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION: '1000' CONNECT_LINGER_MS: '100' CONNECT_BATCH_SIZE: '100000' CONNECT_BUFFER_MEMORY: '128000000' CONNECT_MAX_REQUEST_SIZE: '64000000' CONNECT_COMPRESSION_TYPE: 'lz4' CONNECT_METADATA_MAX_AGE_MS: '60000' CONNECT_METADATA_RECOVERY_STRATEGY: 'rebootstrap' CONNECT_CONSUMER_FETCH_MAX_BYTES: '50242880' CONNECT_CONSUMER_MAX_PARTITION_FETCH_BYTES: '50242880' CONNECT_CONSUMER_FETCH_MAX_WAIT_MS: '10000' CONNECT_CONSUMER_METADATA_MAX_AGE_MS: '60000' CONNECT_CONSUMER_METADATA_RECOVERY_STRATEGY: 'rebootstrap' CONNECT_PRODUCER_ENABLE_IDEMPOTENCE: 'false' CONNECT_PRODUCER_MAX_INFLIGHT_REQUESTS_PER_CONNECTION: '1000' CONNECT_PRODUCER_LINGER_MS: '100' CONNECT_PRODUCER_BATCH_SIZE: '100000' CONNECT_PRODUCER_BUFFER_MEMORY: '128000000' CONNECT_PRODUCER_MAX_REQUEST_SIZE: '64000000' CONNECT_PRODUCER_COMPRESSION_TYPE: 'lz4' CONNECT_PRODUCER_METADATA_MAX_AGE_MS: '60000' CONNECT_PRODUCER_METADATA_RECOVERY_STRATEGY: 'rebootstrap' ``` ## Spark ### Producer Use the same exact settings as recommended for the [Java client](#java-client). ### Consumer Use the same exact settings as recommended for the [Java client](#java-client), but with the addition of the `max.poll.records` field.
Consumer SettingsRecommended Value
metadata.max.age.ms60000
fetch.max.bytes50242880
max.partition.fetch.bytes50242880
metadata.recovery.strategyrebootstrap
fetch.max.wait.ms10000
max.poll.records100000
# Automatic Fetch Size Auto-tuning How to allow the Agents to auto-tune settings ## Auto-tuning consumer settings to optimize fetch throughput WarpStream differs from tradition Kafka in a couple ways 1. WarpStream has higher latency. 2. Agents interpret fetch limits in terms of [uncompressed bytes](/warpstream/kafka/reference/compression#difference-with-kafka-for-fetch-requests) instead of compressed bytes like Kafka does. As such, sometimes customers need to tune their consumer setting to achieve better performance when migrating to WarpStream. To reduce the amount of manual tuning, the Agents automatically adjust the settings related to the amount of data to be returned in a fetch request on your behalf. In most cases, this is fine, but in some cases it can lead to excessive memory usage in the consumer clients. To disable this feature at the Agent level, use the `-autoTuneFetchLimits=false` flag or set the environment variable `WARPSTREAM_AUTO_TUNE_FETCH_LIMITS=FALSE` . Separately, you can also disable this feature on a per-client basis by adding the `ws_dfat=true` [client ID feature](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_disable_fetch_auto_tune) to your clients. # Configure Clients to Eliminate AZ Networking Costs How to configure your Kafka clients to keep all traffic zone-local. With WarpStream, there are no Availability Zone (AZ) networking costs between Agents. This means you can produce and consume data from different AZs without incurring additional networking expenses. The same applies to Agent <--> client communication: WarpStream eliminates AZ networking costs, allowing you to connect clients only to agents within the same AZ. ## **Requirements for Zonal Alignment of Kafka clients** {% hint style="warning" %} WarpStream's zone-aware service discovery system is incompatible with standard Kafka client's rack-aware consumer strategy. When using WarpStream, do **not** configure the rack field on your Kafka clients or enable the rack-aware consumer strategy. If you enable those setting, your Kafka consumers may experience a large amount of unnecessary rebalances. Instead, unset those fields and follow the instructions on this page. {% endhint %} To ensure your Kafka clients connect to Agents within the same availability zone, you need to ensure there is at least one Agent in the same availability zone as your clients and provide the Kafka client’s availability zone. There are two ways to provide the availability zone information: 1. Specifying the availability zone in your client ID 2. Mapping subnets to availability zones in the Agent configuration ### Specifying the availability zone in your client ID Append the following value to your Kafka client's `ClientID`: `ws_az=`. This flag indicates the AZ in which the client is operating.
#### **Example** Here are examples using various kafka libraries of how to set up the `clientID` with the AZ flag
librdkafka/confluent-kafka Generic Configuration ```ini client.id=application-foo,ws_az=us-east-1a ``` Python ```python availability_zone = lookup_az() client_id = "application-foo" p = Producer({ 'client.id': f"{client_id},ws_az={availability_zone}", }) ``` Golang ```go availabilityZone := lookupAZ() clientID = "application-foo" p, err := kafka.NewProducer(&kafka.ConfigMap{ "client.id": fmt.Sprintf("%s,ws_az=%s", clientID, availabilityZone) }) if err != nil { panic(err) } ``` Javascript ```javascript availabilityZone = lookupAZ() clientID = "application-foo" const producer = new Kafka().producer({ 'client.id': `${clientID},ws_az=${availabilityZone}` }); ```
Java ```java String availabilityZone = lookupAZ() String clientID = "application-foo" Properties properties = new Properties(); properties.setProperty(ConsumerConfig.CLIENT_ID_CONFIG, String.format("%s,ws_az=%s", clientID, availabilityZone)); KafkaConsumer consumer = new KafkaConsumer<>(properties); ```
Franz-go ```go availabilityZone := lookupAZ() clientID = "application-foo" cl, err := kgo.NewClient( kgo.SeedBrokers("localhost:9092"), kgo.ClientID(fmt.Sprintf("%s,ws_az=%s", clientID, availabilityZone)), ) ```
Sarama ```go availabilityZone := lookupAZ() clientID = "application-foo" config := sarama.NewConfig() config.ClientID = fmt.Sprintf("%s,ws_az=%s", clientID, availabilityZone) client, err := sarama.NewConsumerGroup(brokers, groupID, config) ```
KafkaJS ```javascript availabilityZone = lookupAZ() clientID = "application-foo" const kafka = new Kafka({ clientId: `${clientID},ws_az=${availabilityZone}` }); ```
To find the AZ that your application is in it is recommended to use your Cloud Provider's Metadata API, for example in AWS querying the following URL: `http://169.254.169.254/latest/meta-data/placement/availability-zone` will give you the availability zone. Alternatively, if you're [using availability zone IDs](#mapping-availability-zones-across-aws-accounts) then you would query: `http://169.254.169.254/latest/meta-data/placement/availability-zone-id` Our [warpstream-go library](https://github.com/warpstreamlabs/warpstream-go/blob/main/pkg/cloudmetadata/cloud_metadata.go) has sample code that demonstrates how to query for your application's availability zone in every major cloud. To learn more about the WarpStream-specific client ID features (like `ws_si` and `ws_az`) check out [our documentation on configuring Kafka Client ID features](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features). ### Mapping subnets to availability zones in the Agent configuration Pass a subnet mapping to the agents via the `-zonedCIDRBlocks`command-line flag or the `WARPSTREAM_ZONED_CIDR_BLOCKS` environment variable. This mapping allows the agents to determine which zone the Kafka client is sending traffic from. The value needs to be a `<>` delimited list of availability zone to CIDR range pairs, where each pair starts with an AZ, a `@`, and a comma separated list of CIDR blocks representing the range of IPs used by the Kafka clients in that given AZ. For example, `us-east-1a@10.0.0.0/19,10.0.32.0/19<>us-east-1b@10.0.64.0/19<>us-east-1c@10.0.96.0/19` indicates that the Kafka clients with IPs that match `10.0.0.0/19` and `10.0.32.0/19` belong to `us-east-1a`, those with IPs that match `10.0.64.0/19` belong to `us-east-1b`, and those with IPs that match `10.0.96.0/19` belong to `us-east-1c`. Note that if an AZ is appended to the Kafka client's `ClientID` and a subnet mapping is also provided to the agents but the values conflict with each other, the AZ from the `ClientID` will be used as the source of truth. ### Mapping availability zones across AWS accounts using IDs Availability zone names are not consistent between different AWS accounts. If you're deploying a WarpStream cluster where Agents and Clients will be deployed in different AWS accounts, you'll want the Agents to advertise their **availability zone ID** instead of their **availability zone name**. This can be accomplished by setting the `WARPSTREAM_LOOKUP_AVAILABILITY_ZONE_ID=true` (Agent v706+) environment variable on the Agents. Similarly, if you're using the client ID strategy, your Kafka clients will need to advertise their availability zone ID instead of availability zone name. Alternatively, if you're using the subnet-mapping strategy, your mapping should refer to avialability zone IDs instead of names. ## Dealing with Zonal Load Imbalances WarpStream will always route client connections to an Agent in the same zone as the client (when zone-alignment is properly configured) as long as there is at **least one Agent** in the desired zone. WarpStream will only allow the client to establish a cross-zone connection if there are **zero Agents** in the desired zone. This means that enabling zone alignment can result in load-imbalances between availability zones if your client traffic is not equally balanced across all zones. This is not a problem in of itself, but it can lead to problems if your Agent auto-scaler is not configured to scale each zone independently. For example, imagine a scenario where the WarpStream Agents are deployed across zones A, B, and C, with an equal number of Agents in each zone, but zone A has three times as much client traffic as zones B and C. In this scenario, the *average* CPU utilization of all the Agents across all the zones may not be enough for Kubernetes to trigger an up-scale, but the Agents in zone A may be overloaded resulting in degradation of client performance in that zone. This problem is easily resolved by configuring your auto-scaler to scale each zone independently. For more details on this, see the [zone-specific scaling section](/warpstream/agent-setup/deploy#zone-specific-scaling) of the HPA documentation. ## Monitoring that Zonal Alignment is Correctly Configured WarpStream's [automated diagnostics system](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) will automatically detect if it thinks Kafka client zonal alignment is misconfigured warn you via a cost diagnostic in the UI, as well as emit a metric for the diagnostic itself. The diagnostic will include a sample client ID for the misconfigured client. In addition, WarpStream Agents emit two metrics that are tagged by the detected availability zone of the Kafka client: 1. `warpstream_agent_kafka_produce_compressed_bytes_counter` 2. `warpstream_agent_kafka_fetch_compressed_bytes_counter` If either of these metrics are being emitted where the `client_az` value doesnt not match the availability zone of the Agent itself, cross-az traffic is occurring. # Force Interzone Load Balancing This page described how to force inter-zone load balancing for clients that don't regularly refresh their Metadata. {% hint style="warning" %} This is advanced documentation for users who care strongly about inter-zone networking fees *and* are required to use the segmentio Kafka library and can't migrate to a better client like franz-go for some reason. If that scenario does not apply to you, skip this page. {% endhint %} Some Kafka clients don't regularly refresh metadata, causing them not to discover agents within the same Availability Zones (AZs). This can lead to inter-zone bandwidth usage. Fortunately, Warpstream has introduced a straightforward solution to address this problem using the ClientID features. ### **Problematic Libraries** The `segmentio` library, to our knowledge, does not refresh metadata automatically. This behavior is primarily observed in their consumers. However, there have been instances where the producers too stop querying metadata for extended periods. If you're utilizing the `segmentio` library, it's recommended to activate this feature to minimize interzone network bandwidth consumption. ### **How to Enable Interzone Load Balancing** To activate the warpstream interzone load balancing in such scenarios, append the following flags to the clientID: `warpstream_az=,warpstream_interzone_lb=true`. * `warpstream_az=`: This flag indicates the AZ in which the client is operating. * `warpstream_interzone_lb=true`: This flag activates the load balancing mechanism in the agent specifically for this client. ```go availabilityZone := lookupAZ() sessionID := uuid.New().String() clientID := fmt.Sprintf( "warpstream_session_id=%s,warpstream_az=%s,warpstream_interzone_lb=true, sessionID, availabilityZone) ``` ### **How the Load Balancing Mechanism Works** When a client includes the aforementioned flag in the client ID: 1. The agent periodically assesses if the connection between itself and the client exists within the same AZ. If it does, no action is taken. 2. If they are in different AZs, the agent checks if there are other agents within the same AZ as the client. 3. If such agents are found, the agent closes the connection, forcing the client to restart the service discovery process from the beginning, ensuring it identifies agents within its own Availability Zone (AZ). ### **Tuning the Load Balancing Check Interval** You can adjust the frequency at which the agent verifies this (applicable only to clients who activate the flag in the ClientID) using: * Flag: `-kafkaInterzoneLoadBalancingInterval` * Environment Variable: `WARPSTREAM_KAFKA_INTERZONE_LOAD_BALANCING_INTERVAL` ### Error Handling in Interzone Load Balancing Interzone load balancing is activated under specific conditions: * When an agent is deployed to a new Availability Zone (AZ) for the first time. * When the only agent in an AZ is removed. While these occurrences are rare, clients might encounter the following errors: #### Specific Errors to Look For: * `io.ErrUnexpectedEOF` * `io.EOF` * `net.ErrClosed` In case you encounter any of these errors, you should simply retry if it's a `Produce` call, and log and continue if it's a `Consume` call. # Configuring Kafka Client ID Features WarpStream uses the client id setting that you set on Apache Kafka clients to control how certain features are activated {% hint style="info" %} Note that configuring the client id is optional, and that if you set the client id to a string that WarpStream does not recognize, no optional feature will be enabled. {% endhint %} WarpStream parses the Apache Kafka client id for key value pairs that are separated with commas. Each key value pair is defined as `key=value` This means that WarpStream parses the following Apache Kafka client id: `a=b,c=d,rest_of_the_client_id` into two key/value pairs: `key: a, value: b` and `key:c, value: d` These keys/values pair are referred to as "client id features" and are used to implement WarpStream-specific functionality while still remaining within the confines of the Kafka protocol. ## Available Features ### warpstream\_az ``` kgo.NewClient(..., kgo.ClientID("warpstream_az=us-east-1a"), ) ``` The `warpstream_az` feature enables zone-aware routing to eliminate inter-zone networking costs. The example above creates a new Apache Kafka client that will tell the WarpStream service discovery system that it's running in the `us-east-1a` availability zone. The service discovery system will use this information to route the client to Agents in the same zone to eliminate inter-zone networking fees. Note however that the service discovery system favors availability, so if it cannot find Agents in specified availability zone, it will direct the clients to Agents that are in other zones. Accepted aliases: * `warpstream_az` * `ws_az` For more information, check out [our documentation about configuring Kafka clients to eliminate inter-AZ networking costs](/warpstream/kafka/configure-kafka-client/configure-clients-to-eliminate-az-networking-costs). ### warpstream\_cluster\_id ``` kgo.NewClient(..., kgo.ClientID("warpstream_cluster_id=vci_077f2fed"), ) ``` The `warpstream_client_id` makes sure that the agent your are connecting to belongs to the correct Virtual Cluster ID. After a client resolved the DNS of your virtual cluster, it might hit an agent belonging to another virtual cluster if the previous agent stopped and the IP got reused. If you always send the virtual cluster in the client ID, you have the guarantee that the agent will reject the connection if they do not match. The ID passed should be a prefix of the full virtual cluster ID containing at least 9 characters overall (including the `vci_` prefix). For instance if the full ID is `vci_38fb01d8_e846_4d8b_be8d_dbcc0d23fdbd` the client can set `warpstream_cluster_id=vci_38fb0`. Accepted aliases: * `warpstream_client_id` * `ws_cluster_id` * `ws_vci` ### warpstream\_proxy\_target ``` kgo.NewClient(..., kgo.ClientID("warpstream_proxy_target=proxy-consume"), ) ``` The `warpstream_proxy_target` feature is used in conjunction with WarpStream's [Agent Roles](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) functionality to route individual Kafka clients to Agents that are running specific roles. If you have split your Agents into distinct roles (as described [here](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles)), you should configure your Apache Kafka Client by setting the `warpstream_proxy_target` feature to either `proxy-produce` (this client will connect to the nodes with the `proxy-produce` role) or to `proxy-consume` (and this client will connect to the nodes with the `proxy-consume` role). Accepted aliases: * `warpstream_proxy_target` * `ws_proxy_target` * `ws_pt` ### warpstream\_agent\_group ``` kgo.NewClient(..., kgo.ClientID("warpstream_agent_group=internal"), ) ``` The `warpstream_agent_group` feature is used in conjunction with WarpStream's [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) functionality to route individual Kafka clients to Agents that belong to a specific group. If you have split your Agents into multiple groups (as described [here](/warpstream/kafka/advanced-agent-deployment-options/agent-groups)), you can configure your Apache Kafka client by setting the `warpstream_agent_group` feature to the name of the Agent group you want target. If your clients use this client ID setting, then they have to set as bootstrap brokers something that exactly matches the same agent group (for instance [our bootstrap URL for the same group](/warpstream/kafka/advanced-agent-deployment-options/agent-groups#non-kubernetes)): if an Agent in group A receives a request from a client that indicated its intended target is Agents in group b, then the Agent in group A will reject the request with an error before closing the connection. This prevents issues that can occur where clients end up connected Agents in the wrong group due to IP reuse in high-churn environments like Kubernetes. Accepted aliases: * `warpstream_agent_group` * `ws_agent_group` * `ws_ag` ### warpstream\_partition\_assignment\_strategy ``` kgo.NewClient(..., kgo.ClientID("warpstream_partition_assignment_strategy=single_agent"), ) ``` WarpStream Agents are stateless, therefore it is possible to direct your Kafka clients writes or reads to any Agent in the cluster. That said, the WarpStream service discovery system still does have to pick which Agents to route individual clients to. WarpStream supports a number of different strategies for handling this routing. Accepted aliases: * `warpstream_partition_assignment_strategy` * `ws_partition_assignment_strategy` * `ws_pas` The possible values for this configuration are: * `consistent_random_jump` * `single_agent` See our [partition assignment strategy documentation](/warpstream/kafka/reference/partition-assignment-strategies) for more details on how each strategy works. ### **ws\_host\_override** ``` kgo.NewClient(..., kgo.ClientID("ws_host_override=agent-lb.yourcompany.com"), ) ``` `ws_host_override` instructs the Kafka client to connect to the specified hostname instead of the Agent's advertised address. This is particularly useful when agents are: * Running behind load balancers **or** * Deployed within containers where the advertised address is not routable from the outside * For example when [port-forwarding](/warpstream/kafka/configure-kafka-client/port-forwarding-k8s) for local development. For example, if your Warpstream agent is accessible via a load balancer with the DNS name `agent-lb.yourcompany.com`, you would set `ws_host_override` in your Kafka client configuration to this value. Accepted aliases: * `warpstream_partition_assignment_strategy` * `warpstream_hostname_override` * `ws_host_override` * `ws_ho` ### **warpstream\_disable\_fetch\_auto\_tune** ``` kgo.NewClient(..., kgo.ClientID("warpstream_disable_fetch_auto_tune=true"), ) ``` `warpstream_disable_fetch_auto_tune` instructs the WarpStream Agents to explicitly disable [fetch size auto-tuning](/warpstream/kafka/configure-kafka-client/client-configuration-auto-tuning), but only for this particular client. This is particularly useful when you want to keep fetch size auto-tuning enable at the cluster level, but one or more applications are sensitive to it (using excessive amounts of memory) and you want to disable it for just those applications. Accepted aliases: * `ws_dfat` ### warpstream\_fetch\_quick\_retry ``` kgo.NewClient(..., kgo.ClientID("warpstream_fetch_quick_retry=true"), ) ``` `warpstream_fetch_quick_retry` instructs the WarpStream Agents to opt-in to the [lower-latency control plane fetch polling mechanism](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters#lower-fetch-latency-for-low-volume-topics), but only for this particular consumer client. This is particularly useful when you want to keep the default fetch polling mechanism enabled at the cluster level, but have one or more applications that are particularly sensitive to end-to-end latency on low volume topics / partitions Accepted aliases: * `ws_fqr` ## Configuring Multiple Features It is possible to combine these, for example {% code overflow="wrap" fullWidth="true" %} ``` kgo.NewClient(..., kgo.ClientID("warpstream_proxy_target=proxy-produce,warpstream_partition_assignment_strategy=equal_spread,warpstream_az=us-east-1a,my_client_id"), ) ``` {% endcode %} will configure an Apache Kafka client that * indicates it is in the `us-east-1a` zone. * will write different partitions to different agents to spread its load to multiple Agents * will connect only to the `proxy-produce` agents * also has a custom string of your choosing, unused by WarpStream, in its id. # Client Metrics (KIP-714) Configure your Kafka clients to push runtime metrics to WarpStream using KIP-714 client metrics subscriptions, then query them from the Events Explorer. **This feature requires your Virtual Cluster's Agents to run on v809 or higher.** [KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) lets a Kafka cluster ask connected clients to ship their internal producer/consumer metrics to the broker on a regular interval, with no application code changes. WarpStream implements the broker side of KIP-714: you create one or more named **client metrics subscriptions** on a Virtual Cluster, and any matching client will start pushing OTLP-encoded metrics to the Agents on the schedule you configured. The Agents decode each push and write one event per data point to the cluster's events stream, where you can search and aggregate them from the [Events Explorer](/warpstream/reference/events) or the [MCP Server](/warpstream/reference/mcp-server). {% hint style="info" %} KIP-714 is a Kafka protocol feature, so it works with any KIP-714-aware client (modern Java client, librdkafka, etc.). You do not need to install any agent or sidecar in your application. {% endhint %} ## Quick start The fastest way to get telemetry flowing is to create a single subscription that targets every client at a 1-minute interval and asks for all metrics: {% code overflow="wrap" %} ```bash kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --alter \ --entity-type client-metrics \ --entity-name all-clients \ --add-config 'interval.ms=60000,metrics=,match=' ``` {% endcode %} An empty `metrics=` value is the KIP-714 wildcard meaning "subscribe to every metric the client exposes". An empty `match=` matches every client. After a minute or two, open the **Events** tab on your Virtual Cluster in the [WarpStream Console](https://console.warpstream.com), pick the `client_metrics` event type, and you should start seeing data points roll in. See [Viewing client metrics](#viewing-client-metrics) below for how to drill in. Once you are comfortable, replace this catch-all with one or more narrower subscriptions targeting specific applications and metric prefixes. ## Configuring subscriptions A subscription is a named record with three configurable fields: * `interval.ms` — how often matched clients push, in milliseconds. * Default: 300000 (5 minutes). Min: 100. Max: 3600000 (1 hour). * `metrics` — a comma-separated list of metric-name prefixes the client should send. An entry is a prefix match (`org.apache.kafka.producer.` matches every producer metric). An empty list means "all metrics". * `match` — a comma-separated list of `key=regex` selectors that pick which clients this subscription applies to. An empty list matches every client. Up to 100 subscriptions are allowed per Virtual Cluster. There are three interchangeable ways to manage subscriptions: the web console, the public HTTP API via WarpStream's Terraform provider, and the standard Kafka AdminClient. ### Using `kafka-configs.sh` (Kafka AdminClient) WarpStream supports the KIP-714 admin surface: `IncrementalAlterConfigs`, `AlterConfigs`, `DescribeConfigs`, and `ListConfigResources` all accept `client-metrics` as the entity type. Create or update a subscription: {% code overflow="wrap" %} ```bash kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --alter \ --entity-type client-metrics \ --entity-name producers \ --add-config 'interval.ms=60000,metrics=org.apache.kafka.producer.,match=client_id=app-.*' ``` {% endcode %} If a value contains a comma, wrap it in `[...]` so `kafka-configs.sh` does not treat the inner commas as separators between configs: {% code overflow="wrap" %} ```bash kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --alter \ --entity-type client-metrics \ --entity-name app-and-staging \ --add-config 'interval.ms=30000,metrics=[org.apache.kafka.producer.,org.apache.kafka.consumer.],match=[client_id=app-.*,client_software_name=apache-kafka-java]' ``` {% endcode %} List existing subscriptions: ```bash kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --describe \ --entity-type client-metrics ``` Describe one by name: ```bash kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --describe \ --entity-type client-metrics \ --entity-name producers ``` Delete a subscription by clearing all of its fields: {% code overflow="wrap" %} ```bash kafka-configs.sh --bootstrap-server $BOOTSTRAP \ --alter \ --entity-type client-metrics \ --entity-name producers \ --delete-config 'interval.ms,metrics,match' ``` {% endcode %} ### Using the WarpStream HTTP API Four endpoints under `/api/v1/` manage subscriptions. See the [Client Metrics API reference](/warpstream/reference/api-reference/client-metrics) for full request and response schemas. | Operation | Endpoint | | ------------------------------------------------------------------------------------ | -------------------------------------- | | [List Subscriptions](/warpstream/reference/api-reference/client-metrics/list) | `list_client_metrics_subscriptions` | | [Describe Subscription](/warpstream/reference/api-reference/client-metrics/describe) | `describe_client_metrics_subscription` | | [Update Subscriptions](/warpstream/reference/api-reference/client-metrics/update) | `update_client_metrics_subscriptions` | | [Delete Subscriptions](/warpstream/reference/api-reference/client-metrics/delete) | `delete_client_metrics_subscriptions` | The read endpoints are also exposed as MCP tools, so an AI assistant connected to your cluster can describe what subscriptions exist and how they are configured. Note that the HTTP field is `interval_ms`, while the Kafka admin config name is `interval.ms`. Both refer to the same value. ### Match selectors Match selectors filter which clients a subscription applies to. The supported keys mirror the KIP-714 specification: | Key | What the broker matches against | | ------------------------- | ------------------------------------------------------------------------------------------------- | | `client_instance_id` | The CONNECT-time UUID assigned to a single client instance. | | `client_id` | The Kafka `client.id` configured on the producer/consumer. | | `client_software_name` | The client library name reported via ApiVersions (for example `apache-kafka-java`, `librdkafka`). | | `client_software_version` | The client library version reported via ApiVersions. | | `client_source_address` | The client's source IP as the Agent observes it. | | `client_source_port` | The client's source TCP port as the Agent observes it. | A few things to keep in mind: * **Patterns are full-string regex.** Each pattern is anchored as `^(?:pattern)$`. `client_id=app` will not match `app-1`; write `client_id=app.*` (or the equivalent `^app.*$`). * **Duplicate keys: the last selector wins.** Inside one `match` value, `client_id=^app$,client_id=^never$` matches nothing, because the second `client_id` selector overrides the first. * **`client_software_*` selectors require ApiVersions v3+.** Older clients do not advertise their software name and version, so those selectors never match them. Use `client_id` instead if you need to target legacy clients. * **Multiple matching subscriptions are merged.** If a client matches more than one subscription, the Agent unions the metric prefixes and uses the smallest `push_interval_ms` across them. There is no priority ordering. ### Subscription examples Push every producer metric, from every Java client, every minute: ``` interval.ms = 60000 metrics = org.apache.kafka.producer. match = client_software_name=apache-kafka-java ``` Capture detailed latency from one specific application at 5-second resolution: ``` interval.ms = 5000 metrics = org.apache.kafka.producer.node.request.latency,org.apache.kafka.consumer.node.request.latency match = client_id=checkout-service-.* ``` Sample everything from staging at 1-minute resolution, while leaving production untouched: ``` interval.ms = 60000 metrics = match = client_id=staging-.* ``` ### Event payload Each data point in a `PushTelemetry` request becomes one event. The `data` payload looks like this: ```json { "metric_schema_version": 1, "metric_name": "org.apache.kafka.producer.request.total", "metric_type": "gauge", "metric_unit": "1", "labels": { "client_id": "prod-app", "node_id": "1" }, "time_unix_nano": 1737059620000000000, "point": { "value": 42 }, "source": { "kind": "client", "client_instance_id": "T48WSCxIRkZPj6r5...", "client_instance_id_uuid": "4f8f1648-...", "subscription_id": -123456789, "matched_subscription_names": ["prod-apps"], "terminating": false, "ingested_at_unix_nano": 1737059623000000000 } } ``` Useful fields to filter and group by: * `data.metric_name` — the OTLP metric name (typically the Kafka client metric, e.g. `org.apache.kafka.producer.request.total`). * `data.metric_type` — `gauge`, `sum`, `histogram`, `exponential_histogram`, or `summary`. * `data.labels` — OTLP resource, scope, and data-point attributes flattened into a string map. This is where `client_id` lives, plus any per-data-point dimensions the client emits (such as `node_id`, `topic`, or `partition`). * `data.point.value` — for gauges and sums; histogram/summary types nest their fields here too. * `data.source.client_instance_id` — the per-client UUID, useful for following one client over time. * `data.source.matched_subscription_names` — which subscription(s) caused this push, useful for attributing data to its owner. ## How it works When a client first connects, it sends `GetTelemetrySubscriptions` (Kafka API key 71). The Agent looks at all configured subscriptions on that Virtual Cluster, finds the ones whose `match` selectors apply to this client, and replies with: * a `subscription_id` (a hash of the resolved configuration; it changes whenever a subscription is added, removed, or modified), * the smallest `push_interval_ms` across matching subscriptions, * the union of subscribed metric prefixes, * the broker's accepted compression codecs. The client then sends `PushTelemetry` (API key 72) every `push_interval_ms`, carrying an OTLP `MetricsData` payload (compressed if both sides agree on a codec). The Agent decodes the payload and emits one CloudEvent per data point to the cluster's `client_metrics` events stream. ## Defaults and limits * **Default push interval (no subscription matches):** 5 minutes. The client will poll on this schedule but send no data points. * **Push interval bounds:** 100ms minimum, 1 hour maximum. * **Maximum subscriptions per cluster:** 100. [Contact us](https://www.warpstream.com/contact-us) if you need a higher limit. * **Maximum push payload (after decompression):** 1 MiB per request. Clients that have more data than this in one push will skip metrics until the next interval. * **Supported compression codecs (broker preference order):** `zstd`, `lz4`, `gzip`, `snappy`. Clients pick whichever they prefer from this list; uncompressed pushes are also accepted. * **Temporality:** delta only. Cumulative-temporality metrics are rejected. ## Notes and limitations * **Throttling state is per-Agent.** Each Agent independently enforces the configured push interval against the clients connected to it; that state is not shared. A client that reconnects to a different Agent may briefly bypass the per-interval throttle. This matches the KIP-714 specification and is rarely visible in practice because the events are still emitted with their `time_unix_nano`. * **OTLP attribute keys with `.` are normalized to `_`.** For example, an OTLP attribute named `kafka.client.id` becomes the event label `kafka_client_id`. * **The broker-derived `client_id` always wins.** If a client emits a `client_id` data-point attribute that disagrees with the `client.id` it set on the connection, the Agent overwrites it with the connection-level value and logs a one-shot warning per request. * **Empty `metrics=` means all metrics.** This is the KIP-714 wildcard, useful for exploration but expensive on busy clusters; switch to a list of prefixes once you know what you care about. ## See also * [Events Explorer](/warpstream/reference/events) * [Important Metrics and Logs](/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs) * [Diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) * [Apache Kafka KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) # Hosted Metadata Endpoint This page explains how you can leverage WarpStream's hosted metadata endpoint to administer your WarpStream BYOC clusters from anywhere. You can use WarpStream's hosted `serverless.warpstream.com:9092` endpoint to easily run any Kafka administrative tasks for your BYOC clusters, like creating topics. The hosted endpoint can handle almost all Kafka protocol requests except for Produce and Fetch because WarpStream does not have **any** access to your data with our BYOC product. ### **Instructions** First, identify your cluster's region and ID from the "Overview" tab (e.g., `ap-southeast-1` and `vci_xxx`).
Next, create a new dedicated agent key in the "Agent Keys" tab (e.g., `aks_xxx`).
Finally, arrange the credentials in this format and configure your Kafka client to connect with TLS enabled using SASL PLAIN: ```xml SASL_USERNAME=:: SASL_PASSWORD= ``` Concretely that would look like: ``` SASL_USERNAME=ap-southeast-1::vci_02e1aa27_5024_695d_819b_dc2d1719959d SASL_PASSWORD=aks_29c445a02ae375e31c316eab7c69e2f1709b16bdcc044ab3a2489da19ae9239b ``` #### **Example: franz-go** ```go var ( virtualCluster = "vci_xxx" agentKey = "aks_xxx" region = "xxx" // e.g. "us-east-1" ) opts := []kgo.Opt{ kgo.SeedBrokers("serverless.warpstream.com:9092"), kgo.DialTLS(), kgo.SASL(plain.Auth{ User: fmt.Sprintf("%s::%s", region, virtualCluster), Pass: agentKey, }.AsMechanism()), } adm, err := kadm.NewOptClient(opts...) ``` #### Example: terraform [See our terraform example on Github.](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/main/examples/byoc-with-topics/main.tf) # Port Forwarding (K8s) This page provides documentation on how to solve a common problem with WarpStream: connecting to Agents deployed in K8s by port-forwarding. If you're running WarpStream in Kubernetes, you may have noticed that port-forwarding to the Agents does not work immediately. For example, if you run the follow comand in one terminal: ``` kubectl port-forward $POD_NAME 9092 ``` and then run the following command in another terminal, you'll get an error: ``` warpstream cli diagnose-connection ``` You'll get an error like this: {% code overflow="wrap" fullWidth="true" %} ``` running diagnose-connection sub-command with bootstrap-host: localhost and bootstrap-port: 9092 Broker Details --------------- 10.212.3.4:9092 (NodeID: 206702995) [asia-southeast1-a] failed to communicate with Agent returned as part of Kafka Metadata response, err: , this usually means that the provided bootstrap host: localhost:9092 is accessible on the current network, but the URL that the Agent is advertising as its broker host/ip: 10.212.3.4:9092 is not accessible on this network. If this is occurring during local development whilst running the Agent in a docker container, consider adding the following flag to the docker run command: --env "WARPSTREAM_PRIVATE_IP_OVERRIDE=127.0.0.1" which will force the Agent to advertise its hostname/IP address as localhost for development purposes. ``` {% endcode %} The reason for this is that your initial client connection will reach the Agent successfully, but then the client will perform a Metadata request for service discovery and the WarpStream control plane will instruct the client to connect to other Agents using their internal IP addresses (the ones reachable in K8s, but not from your laptop). [You can read more about WarpStream service discovery in this documentation for more details](/warpstream/overview/architecture/service-discovery). To work around this issue, WarpStream has a feature to instruct the service discovery system to report all the other Agents hostnames as `localhost` so that when your client performs service discovery via the Metadata request and gets redirected, it will just re-use the existing port-forward. This works because all the WarpStream Agents are stateless, none are special, and any Agent can successfully handle any Kafka protocol request. To enable this feature from your laptop, simply add the following string to your Kafka client ID: `ws_host_override=localhost` . Using the WarpStream CLI, that would look like this: ``` warpstream cli diagnose-connection -client-id ws_host_override=localhost ``` This time it should just work, enjoy your local development experience! {% code overflow="wrap" fullWidth="true" %} ``` running diagnose-connection sub-command with bootstrap-host: localhost and bootstrap-port: 9092 Broker Details --------------- LoCALHOst:9092 (NodeID: 206702995) [asia-southeast1-a] ACCESSIBLE ✅ LOcALhOST:9092 (NodeID: 576636702) [asia-southeast1-b] ACCESSIBLE ✅ LoCAlhost:9092 (NodeID: 1053567971) [asia-southeast1-c] ACCESSIBLE ✅ ``` {% endcode %} # Known Issues ## Sarama Client ### Invalid Array Length when consuming #### Symptom Client logs with errors that look like the following {% code overflow="wrap" %} ``` kafka: error while consuming my-topic/3: kafka: error decoding packet: invalid array length ``` {% endcode %} #### Context When trying to consume records from a topic where the message size is very small, i.e 300 bytes or less the Kafka Fetch response from WarpStream can contain a lot more records then would normally be returned by Apache Kafka. This happens because WarpStream re-writes record batches to more efficiently store and retrieve them from object storage. #### Problem Before Sarama v1.4.5 it had a hard coded limit of `2*math.MaxUint16` (131070) number of records that could be processed in a single Fetch response. This caused any Fetch responses that contained more then 131070 records to be rejected by Sarama with the above error message. #### Solution Update to Sarama v1.4.5 which contains this fix to drastically increase the number of records that Sarama can process in a single Fetch response to 104857600.\ \ If needed you can increase this limit even more by setting the `MaxResponseSize` variable found [here](https://github.com/IBM/sarama/blob/d4acbec36f8e76dc731d12b108a84ed0cd3b1e3b/sarama.go#L113) before initializing your Sarama client. In our experience this limit will not need to be increased from it's current value. ### Occasional Read/Write TCP or EOF errors #### Symptom Client logs occasional errors that look like the following ``` read tcp 192.168.26.193:51587->192.168.26.193:9094: read: connection reset by peer write tcp 192.168.26.193:53585->192.168.26.193:9093: write: broken pipe got error from broker 1359066768 while fetching metadata: EOF ``` #### Context Like Apache Kafka, WarpStream closes idle connections to free up resources so they can be used for other connections or processes. When Sarama sets up it's Kafka client it keeps track of it's connection load on agents, when sending metadata requests Sarama chooses the least loaded agent. When WarpStream responds to metadata requests it will return all the available agents but only make a single agent the leader, see [Service Discovery](/warpstream/overview/architecture/service-discovery#partition-assignment) for details. In Sarama's view this causes all but one agent to have the least load. This means all connections except one will be idle a majority of the time. #### Problem The configuration to close idle connections on the WarpStream agents defaults to 1 hour. After 1 hour Sarama may try and use a closed idle connection which will log these errors. However the presence of these errors should not impact the functionality of your application. These errors are usually log spam due to how Sarama chooses the least loaded agent. #### Solution Increase the idle connection timeout on the WarpStream agents. For example setting this environment variable and value `WARPSTREAM_KAFKA_CLOSE_IDLE_CONN_AFTER=24h`will increase the idle connection timeout to 24 hours. ## **librdkafka Client** ### **Tries to connect to old agents** #### Symptom During and after a rolling restart of the WarpStream agents you will see error logs in your application that will look like the following: ``` 10.42.0.144:9092/356356332: Failed to connect to broker at 10.42.0.144:9092: Operation timed out 10.244.1.36:9092/393450102: Connect to ipv4#10.244.1.36:9092 failed: No route to host (after 14302ms in state CONNECT) 10.0.232.176:9092/249600149: Connection setup timed out in state CONNECT (after 30022ms in state CONNECT, 1 identical error(s) suppressed) ``` #### Context When running in a container platform like Kubernetes or ECS performing a rolling restart of a deployment will change the IP Addresses that the WarpStream Agents are using. #### Problem Client does not remove old agent IP Addresses from its internal configuration which leads to error logs about not able to connect to IP Addresses of old agents. The client should continue to function as normal but will emit errors about not being able to connect to old WarpStream agents. #### Solution Upgrade to version [2.10.0](https://github.com/confluentinc/librdkafka/releases/tag/v2.10.0) or newer. This issue has been fixed in the following pull request . ### **Bad leader epoch handling** #### Symptom Client does not try to open a connection with a new agent, especially during rolling restarts. #### Context Leader epoch is a monotonically increasing number representing a continuous period of leadership for a single partition in Kafka. Changes in leader epoch signals leader transition. In WarpStream the concept of a partition leader does not exist since any WarpStream agent can handle produce and consume requests for any topic and partition. As such WarpStream returns a leader epoch of 0 in all the responses that require a leader epoch. #### Problem `librdkafka` 2.4 introduced a stricter check such that metadata update is only considered if leader epoch is monotonically increasing ([PR](https://github.com/confluentinc/librdkafka/pull/4680)). `if (rktp->rktp_leader_epoch == -1 || leader_epoch > rktp->rktp_leader_epoch)` This means that if there is an agent with IP `ip1` and we would like to replace it with an agent with `ip2`, `librdkafka` will not open a connection against `ip2`. #### Solution There are a couple options: * Upgrade to `librdkafka` 2.8 * Downgrade to `librdkafka` 2.3 * Set `warpstream_strict_leader_epoch=true` or `ws_sle=true` in your Kafka client ID. * Note that WarpStream Kafka client ID features are expected to be comma-delimited, so if your existing client ID is `some_client_id` or `some_client_id,ws_az=us-east-1a` then the client ID should be changed to `some_client_id,ws_sle=true` or `some_client_id,ws_az=us-east-1a,ws_sle=true` respectively. * Contact us to enable a patch in the WarpStream control plane #### Relevant material Related `librdkafka` issues * * `librdkafka` fix * Leader epoch KIPs * * ### Idempotence Performance Enabling the idempotent producer functionality in the librdkafka client library can result in extremely poor producer throughput and very high latency even when using our [recommended settings](/warpstream/kafka/configure-kafka-client/tuning-for-performance#librdkafka). This is the result of four conspiring factors: * WarpStream has higher produce latency than traditional Apache Kafka * Librdkafka only allows 5 concurrent produce requests per connection when the idempotent producer functionality is enabled instead of per partition * Librdkafka never combines batches from multiple partitions owned by the same broker into a single Produce request:[ https://github.com/confluentinc/librdkafka/issues/1700](https://github.com/confluentinc/librdkafka/issues/1700) The easiest ways to mitigate this problem are to: 1. [Disable idempotence](#disable-idempotence) in the librdkafka client or 2. [Use NULL record keys](#use-null-record-keys) when producing However if neither of those are valid options, then we'd recommend trying to [tune WarpStream for Lower Latency](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters) as that will have the most minor trade-off (slightly increased cost, no impact on application correctness or behavior). #### Disable Idempotence This is the easiest solution, but may not be viable if your application depends on idempotence. #### Use NULL record keys This approach is not always viable if you need records with the same key to always be routed to the same topic-partition. However, if your application doesn't depend on that, then the easiest way to achieve high throughput with idempotent Produce requests in librdkafka is to avoid specifying keys for your records at all. If you do this while also using our [recommended settings](/warpstream/kafka/configure-kafka-client/tuning-for-performance#librdkafka) for librdkafka, then you should be able to achieve high throughput even with idempotence enabled. Note that you may want to consider increasing the value of `sticky.partitioning.linger.ms` to a higher value like `100ms` if you take this approach. #### Tune WarpStream for Lower Latency One of the reasons that the idempotent producer functionality in librdkafka is slow is because WarpStream has higher Produce latency than traditional Apache Kafka. Since Librdkafka will only allow 5 concurrent idempotent producer requests at a time, ensuring that each of those Produce requests completes faster will enable higher total throughput. Follow [these docs for tuning WarpStream for lower latency](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters). #### Use Multiple Clients {% hint style="warning" %} This is the most difficult approach and should only be considered once all other options have been exhausted. {% endhint %} The librdkafka limitations are all specific to running a single librdkafka instance. However, you can run multiple instances of the librdkafka client in your application and load-balance records between them. For example, you could create 4 instances of the librdkafka library in your application, spread records amongst them, and then the total number of concurrent Produce requests your application could make would increase by 4. However, Librdkafka Produce requests can only contain data for a single topic-partition so to make this approach effective you would have to distribute records to the different clients such that each client was only producing to a subset of the overall partitions. For example, something like this: ```python num_clients = 4 for records range records: expected_partition = determine_expected_partition(record, topic) client = clients[expected_partition % num_clients].produceAsync(record) ``` The key function is `determine_expected_partition` which would need to be constructed to match exactly the behavior of whichever librdkafka partitioning strategy you're currently using. This would involve making a Kafka Metadata request to determine the number of partitions in a topic (and caching that result for some period of time), and then hashing / partitioning the record keys using the same algorithm that librdkafka is configured to use. ## Java Client ### Got error produce response with correlation id XXXX on topic-partition YYYY, retrying. Error: Kafka Storage Error. In some instances you may observe your producer clients log error messages like the following: {% code overflow="wrap" %} ```log [Producer clientId=warpstream_az=us-east-1a] Got error produce response with correlation id 46572 on topic-partition test-topic-0000000-K6WhR0c-0, retrying (2147483646 attempts left). Error: KAFKA_STORAGE_ERROR ``` {% endcode %} The producer error logs may not correlate with any error logs in the Agents. This is normal and is caused by the interaction between the idempotent producer feature and WarpStream's load balancing system. The error logs are harmless and can be safely ignored as WarpStream returns a retriable error in this case and the Java client will automatically produce the records again with no data loss. That said, if this is happening a lot, it may lead to increased producer latency. There are two ways to resolve the issue besides just ignoring the error logs: 1. Disable idempotent producer by setting `enable.idempotence=false` on your producer client. 2. Change the partitioning strategy on the WarpStream Agents from the default of `single_agent` to `consistent_random_jump` by setting the environment variable `WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY=consistent_random_jump`. 1. This approach will not eliminate the error logs entirely, but it should make them significantly more rare as WarpStream will try to keep topic-partitions assigned to the same Agents as much as possible, and will only shift them from one Agent to another when absolutely necessary for load-balancing purposes. #### Why it happens The idempotent producer functionality in Kafka enables a Kafka client to have up to 5 outstanding concurrent produce requests per broker while still maintaining strict total ordering. Unlike Apache Kafka, any WarpStream Agent process produce requests for any topic-partition. As a result of this, WarpStream's partition assignment strategies will shift topic-partitions "ownership" (from the clients perspective) around the cluster quite often to keep the WarpStream Agents evenly balanced. This means that from the client's perspective, the leader of any individual topic-partition shifts much more frequently in WarpStream than it does in Apache Kafka, sometimes as frequently as once a minute. When the topic-partition leader changes, the client will stop producing to one Agent and start producing to another Agent. When this happens, there is a small chance that the idempotent producer functionality in WarpStream will detect that the Agents tried to commit some of the client's batches in the wrong order because batch 2 was sent to Agent A and batch 3 was sent to Agent B, but Agent B committed a file before Agent A did. When this happens, WarpStream returns a KAFKA\_STORAGE\_ERROR (retriable error code) back to the client so that the client knows to resend batch 3 again now that batch 2 has been committed. That's why disabling the idempotent producer functionality **or** changing WarpStream's partition assignment strategy to a more "sticky" algorithm like `consistent_random_jump` will resolve the issue. # Manage Security This page describes how to manage the various security aspects of WarpStream's Kafka product. ## ACLs WarpStream supports standard Kafka ACL functionality. More details are available in the [ACLs documentation](/warpstream/kafka/manage-security/configure-acls). ## TLS WarpStream Agents have native support for TLS termination. More details are available in the [TLS documentation](#tls). ## Authentication By default, WarpStream Agents are configured without authentication. WarpStream supports the following authentication mechanisms and protocols for WarpStream Agents. ### SASL SASL (Simple Authentication Security Layer) is a framework that provides developers of applications and shared libraries with mechanisms for authentication, data integrity-checking, and encryption. The following topic explains how to configure SASL in WarpStream. * [SASL Authentication](/warpstream/kafka/manage-security/sasl-authentication) * [SASL/OAUTHBEARER Authentication](/warpstream/kafka/manage-security/sasl-oauthbearer-authentication) ### Mutual TLS (mTLS) With mTLS (mutual TLS) authentication, both Kafka clients and servers use TLS certificates to verify each other’s identities to ensure that traffic is secure and trusted in both directions. The following topic explains how to configure mTLS in WarpStream. * [Mutual TLS](/warpstream/kafka/manage-security/mutual-tls-mtls) # ACLs The following page provides a management guide for ACLs (Access Control Lists). This document provides instructions on how to enable, disable, create, and delete ACLs for your WarpStream clusters. ## Prerequisites Before you begin, ensure you have administrative access to the WarpStream console and are familiar with the basic concepts of ACLs. The current version of ACLs supports SASL and mTLS authentication method. Review [Authentication](https://github.com/warpstreamlabs/docs/blob/master/kafka/manage-security/broken-reference/README.md) for more information. {% hint style="danger" %} **Important:** ACLs are only compatible with **Agent version 526 and above**. Make sure you use a compatible agent version, if your cluster has self-hosted agents. {% endhint %} ## Important considerations There are a few things to keep in mind when using ACLs, which are covered below. ### ACL Principal ACLs operate using Principals, which are entities capable of being authenticated by the Authorizer. In the context of WarpStream agents, clients authenticate as a specific principal by using either SASL or mTLS authentication. **With SASL/PLAIN or SASL/SCRAM-SHA-512**, the principal is identified by one of the three following: 1. The assigned **username** (prefixed with `ccun_`) 2. Chosen **name** during [credential creation](#generating-credentials), (prefixed with `ccn_`) 3. Chosen **name** during [credential creation](#generating-credentials), without prefix `ccn_` In the screenshot below, valid principal examples include: `user_1234`, `ccn_user_1234`, and `ccun_3cc6c05a56b396fc084ef1e113150970cd148c87b5f64babfd3b8cf67d4d8840`.
**With mTLS**, the principal is either the Distinguished Name(DN) from the client TLS certificate, or extracted from the DN according to the `-tlsPrincipalMappingRule` flag set in the Agents as described in the [mutual tls authentication](/warpstream/kafka/manage-security/mutual-tls-mtls) page. **With SASL/OAUTHBEARER** the principal is identified by the `subject` of the OAuth token. All principals should be prefixed with `User:` when creating ACL rules. ### Super users In the context of ACLs, a 'superuser' is a specially designated user who possesses elevated privileges. Superusers are granted overarching access rights, allowing them to bypass standard ACL restrictions. This includes unrestricted abilities to create, modify, or delete resources, and to manage access controls for other users. **For SASL/PLAIN or SASL/SCRAM-SHA-512**, you can authorize as super user by setting your SASL username as either the generated cluster credentials username(`ccun_`) or by using the "Name" which you set while generating credentials. You can create super users through the credentials, as explained at [SASL Generating Credentials.](/warpstream/kafka/manage-security/sasl-authentication#creating-credentials) **For mTLS**, you can create super users by clicking the "Create mTLS Super User" button in the Credentials page and entering the certificate DN or TLS principal. **For SASL/OAUTHBEARER**, you can create super users by clicking the "Create mTLS Super User" button in the Credentials page and entering the OAuth subject you want to be a super user. ### Caching of ACLs For performance reasons, ACLs are cached, so changes to ACLs may take between 30 seconds to 1 minute to take effect. Plan accordingly when updating ACLs in a production environment to ensure a smooth transition. ### allow\.everyone.if.no.acl.found Warpstream does not implement the `allow.everyone.if.no.acl.found` Kafka configuration, because: 1. It's not safe for production environments. 2. You can get the equivalent (and safer) behaviour by simply using [#super-users](#super-users "mention"), who are authorized to access everything. ## ACL Shadowing ACL Shadowing allows a WarpStream cluster to evaluate Kafka ACLs against live traffic without enforcing authorization decisions. This mode is designed to help operators validate ACL behavior before enabling full ACL enforcement. When ACL Shadowing is enabled: * All incoming requests are evaluated against the cluster’s configured ACLs. * Authorization results are not enforced (no requests are blocked). * Would-be denials are [logged](#denied-authorization-logs) and surfaced through [Diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics). ACL Shadowing can be enabled from the WarpStream Console in the ACLs tab for your cluster or via [Terraform](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs/resources/virtual_cluster).
acl_shadowing
Note that ACL Shadowing will be automatically disabled when you enable ACLs for your cluster. ## Enabling and Disabling ACLs {% hint style="danger" %} **Warning** Before enabling ACLs, be aware that improper configurations may lead to denied access for producers and consumers. Make sure to configure ACLs correctly to avoid any disruption of service. {% endhint %} ### To Enable ACLs 1. Navigate to the [WarpStream console](https://console.warpstream.com/virtual_clusters). 2. Select the desired Virtual Cluster, for example `vcn_default`. 3. Click on the `ACLs` tab. 4. You will see the `ACLs: Disabled` status label if ACLs are currently disabled.
5. Click on the `Enable ACLs` button to change its status to `Enabled`.
### To Disable ACLs 1. Follow the same steps to navigate to the `ACLs` tab of your Virtual Cluster. 2. If ACLs are enabled, you will see the `ACLs: Enabled` status label. 3. Click on the `Disable ACLs` button to turn off ACLs for the cluster. ### Enable/Disable Using the HTTP API Alternatively, you can switch ACLs on and off using the HTTP APIs, which is a convenient alternative to doing it through the console UI. Refer to [DescribeConfiguration](/warpstream/reference/api-reference/virtual-clusters/describeconfiguration) and [UpdateConfiguration](/warpstream/reference/api-reference/virtual-clusters/updateconfiguration) for details. ## Generating Credentials To interact with WarpsSream clusters, you need to generate credentials: 1. Within the WarpStream console, navigate to the `Credentials` tab of your Virtual Cluster. 2. Click on `Generate Credentials`. 3. Provide a name for the credentials. 4. If necessary, enable `Super User` for broader permissions. * **Important:** If a resource has no associated ACLs, then only superusers can access that resource. 5. Click `Generate Credentials`. The credentials will be displayed for you to use with the CLI.
{% hint style="info" %} **Important**: Credentials are shown only once. Store them securely. {% endhint %} ## Managing ACLs via WarpStream Console 1. Navigate to the [WarpStream console](https://console.warpstream.com/virtual_clusters). 2. Select the desired Virtual Cluster, for example `vcn_default`. 3. Click on the `ACLs` tab. 4. Click "Add ACL Rule"
5. Fill in the rule information. In this example we are creating a `TOPIC` ACL for topics that match the name `foo` exactly for the User `ccn_user_1234` and allowing that user to `WRITE` to the topic.
For information on ACL resource types and operations see the [Confluent Operation Documentation](https://docs.confluent.io/platform/current/security/authorization/acls/overview.html#operations). ## Managing ACLs via Kafka API To manage ACLs via the Kafka API, use the Kafka Admin API with appropriate Kafka credentials. ## Managing ACLs via HTTP API You can create, list, and delete ACLs using the WarpStream HTTP API. See the [ACLs API Reference](/warpstream/reference/api-reference/acls) for full endpoint documentation. ## Managing ACLs via Terraform ACLs can be managed using the [WarpStream Terraform provider](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs/resources/acl). ### Create an ACL To create an ACL, use the `createAcls` method with the required ACL details: ```go _, err = admin.CreateACLs( context.Background(), kafka.ACLBindings{{ Type: kafka.ResourceTopic, Name: "topic_name", ResourcePatternType: kafka.ResourcePatternTypeLiteral, // See above the above ACL Principal section for information. Principal: "User:", Host: "*", // Request can come from any host. Operation: kafka.ACLOperationWrite, PermissionType: kafka.ACLPermissionTypeAllow, }}, ) if err != nil { fmt.Printf("Failed to create ACLs: %s\n", err.Error()) os.Exit(1) } ``` ### Delete an ACL To delete an ACL, use the `deleteAcls` method with a filter matching the ACL you wish to remove: ```go _, err = admin.DeleteACLs( context.Background(), kafka.ACLBindingFilters{{ Type: kafka.ResourceTopic, Name: "topic_name", ResourcePatternType: kafka.ResourcePatternTypeLiteral, // See above the above ACL Principal section for information. Principal: "User:", Host: "*", Operation: kafka.ACLOperationWrite, PermissionType: kafka.ACLPermissionTypeAllow, }}, ) if err != nil { fmt.Printf("Failed to delete ACLs: %s\n", err) os.Exit(1) } ``` The examples above use the `librdkafka` package from Go. You can also try the `bin/kafka-acls.sh` script from Apache Kafka®'s official binary release for more hands-on experience: ``` bin/kafka-acls.sh --bootstrap-server localhost:9092 --remove --allow-principal User: --allow-principal User:Alice --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic ``` ``` bin/kafka-acls.sh --bootstrap-server localhost:9092 --add --allow-principal User:Bob --allow-principal User: --allow-host 198.51.100.0 --allow-host 198.51.100.1 --operation Read --operation Write --topic Test-topic ``` #### Using the `ANY` Resource Type in ACLs Our system supports creating ACLs with the resource type `ANY` through the console, API, and [Terraform](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs/resources/acl). This differs from standard Apache Kafka, where clients typically cannot create ACLs with resource type `ANY` and may reject them. {% hint style="warning" %} **Warning** Using the `ANY` resource type is **not recommended**. It grants permissions across all resources of that type, broadening the attack surface, and can cause compatibility issues with standard Kafka clients, which expect ACLs for specific resource types. Whenever possible, use particular resource types to ensure stricter access control and interoperability. {% endhint %} ## Denied Authorization Logs Logs for denied authorizations are enabled by default and can be enable/disabled by setting the `-enableACLLogs` flag or `WARPSTREAM_ENABLE_ACL_LOGS` environment variable to `true` or `false` on the Agents. Any actions that are denied due to a missing ACL or an explicit `DENY` ACL will show up in these logs. An example log is below: {% code overflow="wrap" %} ```json {"time":"2025-03-13T13:02:05.176353-05:00","level":"WARN","msg":"ACL Denied Operation on resource","principal":"User:ccun_9c5d...","operation":"CREATE","host":"127.0.0.1:56216","resource_type":"TOPIC","resource_name":"foo"} ``` {% endcode %} In the example above we can see the user `User:ccun_9c5d...` tried to `CREATE` a `TOPIC` called `foo` and was denied. # TLS This page described how to configure the Agents to terminate TLS. ## TLS Encryption Overview By default Kafka/Schema Registry clients communicating with WarpStream Agents use `PLAINTEXT`, meaning that all data is sent in plain text (unencrypted). To encrypt data in motion (or data in transit) between your clients and your WarpStream Agents, you should configure them to use TLS Encryption. WarpStream supports [Transport Layer Security (TLS)](https://en.wikipedia.org/wiki/Transport_Layer_Security) encryption based on [OpenSSL](https://www.openssl.org/), an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS). Enabling TLS encryption might have a performance impact due to overhead of encrypting and decrypting data. This performance impact can vary depending on the operating system, linux kernel version, and CPU used. We recommend using the newest and best possible versions of your Operating system, Kernel, and CPU to minimize any possible impacts. TLS uses private-key/certificate pairs, which are used during the TLS handshake process. * Each WarpStream Agent needs a private-key/certificate pair, and the Kafka client uses the certificate to authenticate to the WarpStream Agent. * Each logical client needs a private-key/certificate pair if client authentication is enabled, and the WarpStream Agent uses the certificate to authenticate the Kafka client. ### Mutual (mTLS) Authentication If you configure TLS encryption, you can optionally configure [mutual (mTLS) authentication](/warpstream/kafka/manage-security/mutual-tls-mtls). You can configure just TLS encryption (by default, TLS encryption includes certificate authentication of the server) and use a separate mechanism for client authentication (for example, mTLS or SASL). By default, TLS encryption enables one-way authentication in which the client authenticates the server certificate. For bidirectional authentication, where the broker also authenticates the client certificate, you can use mTLS. When you use [mTLS Authentication](/warpstream/kafka/manage-security/mutual-tls-mtls), the WarpStream Agent authenticates the Kafka client and the Kafka client also authenticates the WarpStream Agent. This bidirectional, or mutual, authentication provides an additional layer of security for your WarpStream cluster. ## Configure TLS Encryption for a WarpStream Cluster Configuring TLS for a WarpStream Cluster can be done in one of two ways. Which option is the best depends on your requirements and deployment environment. ### Option 1: Configuring the WarpStream Agents to terminate TLS This option is the recommended configuration when using TLS. This ensures that there is full end-to-end Kafka client to WarpStream Agent encryption. This comes at the disadvantage of needing to configure each WarpStream Agent with a TLS certificate. 1. Create x509 encoded TLS certificates * Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization. * If running WarpStream in Kubernetes using [cert-manager](https://cert-manager.io/) can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities. * We recommend that either the certificate is made with `SANS` and using hostnames to connect to your WarpStream Agents. Alternatively you can use `IP SANS` however that may be impractical in some deployments like Kubernetes. 2. Configure the WarpStream Agents to load the certificates * Once your certificates are generated to must set the `WARPSTREAM_TLS_SERVER_CERT_FILE` environment variable to the public key of the certificate and set `WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE` to the private key of the certificate. 3. Configure the WarpStream Agents to enable TLS encryption * To enable TLS encryption for your Kafka agent, set the `-kafkaTLS`agent flag. Alternatively, you can set the `WARPSTREAM_TLS_ENABLED`environment variable to true. * To enable TLS encryption for your Schema Registry agent, set the `-schemaRegistryTLS`agent flag. Alternatively, you can set the `WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLED`environment variable to true. By default WarpStream Agents will load the TLS certificate and key from the agent's local filesystem. The agents can be configured to instead load the TLS certificate and key from blob storage by using the `-tlsBlobURL` flag or `WARPSTREAM_TLS_BLOB_URL` environment variable. For example `-tlsBlobURL=s3://my-tls-bucket`. {% hint style="info" %} WarpStream uses modern x509 Certificate parsing algorithms which requires certificates to have both the Common Name and Subject Alternative Names (SAN) list to be set. The Common Name must also be in the SAN list. {% endhint %} #### TLS Certificate Configuration with the Helm Chart When configuring certificates using our [Helm Chart](/warpstream/agent-setup/infrastructure-as-code/helm-charts) it is recommended to use the following values ```yaml certificate: # Set to true to enable TLS termination on the WarpStream agent # see TLS documentation for details https://docs.warpstream.com/warpstream/byoc/advanced-agent-deployment-options/protect-data-in-motion-with-tls-encryption#tls-encryption-overview enableTLS: true # The Kubernetes TLS secret that contains a certificate and private key # see https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets secretName: "warpstream-agents-cert" deploymentKind: StatefulSet ``` The certificate should have the following `SANS`. ``` 'warpstream-agent.default.svc.cluster.local' '*.warpstream-agent-headless.default.svc.cluster.local' ``` {% hint style="info" %} Replace `default` with the namespace that the helm chart is installed into. If you are setting `nameOverride` or `fullNameOverride` fields then `warpstream-agent` may also need to be modified. {% endhint %} For example if you are using [cert-manager](https://cert-manager.io/) to generate the certificates, the certificate resource should look like the following. ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: warpstream-agents-cert namespace: default spec: dnsNames: - 'warpstream-agent.default.svc.cluster.local' - '*.warpstream-agent-headless.default.svc.cluster.local' secretName: warpstream-agents-cert issuerRef: name: my-ca-issuer kind: Issuer group: cert-manager.io ``` If you are deploying WarpStream with a load balancer infront, i.e [WarpStream behind a TCP Load Balancer Without Direct Connectivity](/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy#warpstream-behind-a-tcp-load-balancer-without-direct-connectivity) then `deploymentKind` does not need to be set to `StatefulSet` and the certificate `SANS` should only contain the hostname of the load balancer. ### Option 2: Configuring a Load Balancer to terminate TLS This option can be simpler to implement for WarpStream clusters behind a load balancer. This option requires that you [Advertise the WarpStream Agents behind a traditional load balancer](/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy#a-dvertise-the-warpstream-agents-behind-a-traditional-load-balancer) for it to work as expected. The disadvantage of this option is that the communication between the Load Balancer and WarpStream Agents are not encrypted. If you have a requirement for full end-to-end encryption we do not recommend using this option. This option also cannot use mTLS as an authentication mechanism due to TLS termination on Load Balancers not being able to pass-through the client certificate. Every Load Balancer is configured differently for TLS termination, bellow is information from the 3 major Cloud Providers for how to configure TLS termination on their Load Balancers. * AWS * [EKS NLB](https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.4/guide/use_cases/nlb_tls_termination/) * [Network Load Balancer](https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/) * GCP * GKE - As of 2024 GKE does not support TLS termination on Kubernetes Services with Load Balancer types * [Network Load Balancer SSL Proxy](https://cloud.google.com/load-balancing/docs/tcp/set-up-global-ext-proxy-ssl) * Azure * As of 2024 Azure Layer 4 load balancers do not support TLS termination {% hint style="info" %} WarpStream agents natively load balance clients across the cluster based off of agent load. Using a TCP load balancer in front of WarpStream in conjunction with setting a hostname override prevents WarpStream from performing its native load balancing which can negatively impact performance. This typically not a large impact but is noticeable in high performance scenarios. We recommend to only use a TCP load balancer when client applications cannot directly communicate with the agents.\ \ In some cases using [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) and deploying agents near the client application is preferred to maintain expected performance. {% endhint %} ## Configure TLS Encryption for Kafka clients For configuring TLS encryption in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures TLS differently and those configurations may change version to version. We recommend using the [Confluent Platform documentation](https://docs.confluent.io/platform/current/security/protect-data/encrypt-tls.html#configure-tls-encryption-for-ak-clients) to learn how to configure Java-based clients for TLS encryption. ### TLS Profiles By default the WarpStream Agent uses the default Golang TLS settings when serving the Kafka Protocol over TLS. These defaults change over time but and may not be suitable for all environments. The WarpStream agent supports setting different TLS Profiles based on recommendations from [Mozilla SSL](https://ssl-config.mozilla.org/). The ability to configure TLS Profiles was added in Agent Version [v657](/warpstream/overview/change-log#release-v657). These profiles can be set via the `-tlsProfile` flag or `WARPSTREAM_TLS_PROFILE` environment variable. All profiles use the following TLS Curves ```go tls.X25519 tls.CurveP256 tls.CurveP384 tls.CurveP521 tls.X25519MLKEM768 ``` All TLS 1.3 connections use the following Cipher Suites: ``` tls.TLS_AES_128_GCM_SHA256 tls.TLS_AES_256_GCM_SHA384 tls.TLS_CHACHA20_POLY1305_SHA256 ``` #### `golang-default` Use the Golang defaults for TLS Version, Curves and Cipher Suites. These will change over time. See [Golang TLS](https://pkg.go.dev/crypto/tls) documentation for details. #### `old` This profile supports older TLS clients with the minimum TLS version set to 1.0 with support for the following Cipher Suites for TLS 1.0, 1.1 and 1.2: ``` tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA tls.TLS_RSA_WITH_AES_128_GCM_SHA256 tls.TLS_RSA_WITH_AES_256_GCM_SHA384 tls.TLS_RSA_WITH_AES_128_CBC_SHA256 tls.TLS_RSA_WITH_AES_128_CBC_SHA tls.TLS_RSA_WITH_AES_256_CBC_SHA tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA ``` `intermediate` This profile supports intermediate TLS clients with the minimum TLS version set to 1.2 with support for the following Cipher Suites: ```go tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 ``` `modern` This profile supports modern TLS clients and TLS 1.3 only. ### Transparent HTTPS Proxies Some organizations use transparent HTTPS Proxies to decrypt and inspect HTTPS traffic. This may cause the WarpStream Agent to be unable to communicate with the WarpStream Control Plane. You may see the following error if that is the case {% code overflow="wrap" %} ``` Post "https://metadata.default.${region}$.${cloud}$.warpstream.com/api/v1/agent/agentpool": tls: failed to verify certificate: x509: certificate signed by unknown authority ``` {% endcode %} Starting in version [v657](/warpstream/overview/change-log#release-v657) the WarpStream agent as the ability to provide an HTTPS Proxy Certificate Authority for the Agent to trust. This can be done with the `-httpsProxyCACertFile` flag or `WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE` environment variable. An example `values.yaml` of this configuration via our `Kubernetes Helm Chart` is as follows: ```yaml extraEnv: - name: WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE # The path to where the configmap is mounted. # In this example the item inside the configmap is # called `ca.crt` so it get's created as that file. value: /etc/https-proxy/ca.crt volumeMounts: - name: https-proxy-ca # The path to where the confimap should be mounted. mountPath: /etc/https-proxy volumes: - name: https-proxy-ca configMap: # Name of the configmap in Kubenretes that contains # the x509 PEM file for the HTTPS Proxy. name: https-proxy-ca ``` # SASL Authentication This page describe how to configure the WarpStream Agents with SASL authentication. SASL Authentication uses usernames and passwords to authenticate your Kafka clients. By default Kafka clients communicating with WarpStream Agents use `PLAINTEXT`, meaning that all data is sent in plain text (unencrypted), this includes the SASL usernames and passwords. When using SASL it is recommended to [Configure TLS Encrpytion for your WarpStream Cluster.](/warpstream/kafka/manage-security/protect-data-in-motion-with-tls-encryption#configure-tls-encryption-for-a-warpstream-cluster) The WarpStream Agents support both `SASL/PLAIN` and `SASL/SCRAM-SHA-512` for communication. ## Configure WarpStream Agents Set the `requireSASLAuthentication` flag or `WARPSTREAM_REQUIRE_SASL_AUTHENTICATION=true` environment variable on the Agents. Once authentication is enabled on the Agents, they will enforce that all Apache Kafka clients that connect to them authenticate themselves via SASL. Improperly authenticated clients will be unable to connect. ## Creating Credentials In order to connect an Apache Kafka client to the authenticated WarpStream Agent, you'll need to create a set of credentials. You can do that by navigating to the ["Clusters" section of the WarpStream Console](https://console.warpstream.com/virtual_clusters) and then clicking "Credentials" within the Virtual Cluster that you want to create a set of credentials for.

Click "Credentials"

Once you're on the credentials view, you can create a new set of SASL credentials by clicking the "Create Credentials" button.

Click "Create Credentials"

Insert the name that you want the credential to have, check super user if desired, click Create Credentials

Enter a name and click Create Credentials

Once you're done creating the credentials, the admin console will show you the username and password one time. Store these values somewhere safe, as you'll never be able to view them again. WarpStream does not store them in plaintext, so we cannot retrieve them for you.

Save your credentials somewhere safe!

In the case that you lose your credentials, you can create a new set of credentials in the admin console following the same steps as above, up to a limit of 100 credentials. This limit can be increased to 1000 credentials upon request. ## Configure Kafka clients For configuring SASL in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures SASL differently and those configurations may change version to version. We recommend using the [Confluent Platform documentation](https://docs.confluent.io/platform/current/security/authentication/sasl/plain/overview.html#configure-ak-clients) to learn how to configure Java-based clients for SASL. ## Limiting Allowed SASL Mechanisms Configure the `-enabledSASLMechanisms` flag or `WARPSTREAM_ENABLED_SASL_MECHANISMS` environment variable to a comma-delimited list of allowed SASL mechanisms. If this flag / environment variable is not set, all SASL mechanisms are allowed. If it is set, only the specified mechanisms are allowed. For example: `WARPSTREAM_ENABLED_SASL_MECHANISMS=PLAIN,SCRAM-SHA-512` means that both `SASL/PLAIN` and `SASL/SCRAM-SHA-512` are allowed, but `WARPSTREAM_ENABLED_SASL_MECHANISMS=PLAIN` means that only `PLAIN` is allowed. Supported values: `PLAIN`, `SCRAM-SHA-512`. # Mutual TLS (mTLS) This page describes how to configure the Agents to enforce mTLS. Mututal TLS (mTLS) authentication requires TLS encryption, this page shows you how to configure both at the same time and is a superset of configurations required just for [Configuring the WarpStream Agents to terminate TLS](/warpstream/kafka/manage-security/protect-data-in-motion-with-tls-encryption#option-1-configuring-the-warpstream-agents-to-terminate-tls). ## Creating TLS keys and Certificates for the WarpStream Agents Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization. If running WarpStream in Kubernetes using [cert-manager](https://cert-manager.io/) can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities. We recommend that either the certificate is made with `IP SANs` if using IP addresses or `SANS` if using hostnames to connect to your WarpStream Agents. ## Creating TLS keys and Certificate for the Kafka/Schema Registry Clients When using mutual TLS, it is highly recommended to use a private certificate authority for certificates. Most large organizations typically already have their own internal private certificate authority that you can use to generate client certificates. For alternatives if your organization does not have an existing private certificate authority, every cloud provider offers a managed solution, for example [AWS Private CA](https://docs.aws.amazon.com/privateca/latest/userguide/PcaWelcome.html). You can also create a certificate authority manually using the `openssl` command line however that can be complex to manage in production environments. When using mutual TLS is it recommended to give each application their own unique certificate with their own unique Distinguished Name(DN). If using [ACLs](/warpstream/kafka/manage-security/configure-acls) having unique certificates with unique DN's will allow each application to have it's own set of permissions. ## Configure WarpStream Agents for Kafka Clients Once your certificates are generated, you must set the `WARPSTREAM_TLS_SERVER_CERT_FILE` environment variable to the public key of the certificate and set `WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE` to the private key of the certificate. First, to enable TLS you must set the `WARPSTREAM_TLS_ENABLED` environment variable to `true`, or set the `kafkaTLS` flag. Then, to enable mutual TLS you must set the `WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION` environment variable to `true`, or set the `requireMTLSAuthentication` flag. Once authentication is enabled on the Agent, it will enforce that all Apache Kafka clients that connect to them authenticate themselves via mTLS. Otherwise, it will refuse the connection. It is also highly recommended to set the environment variable `WARPSTREAM_TLS_CLIENT_CA_CERT_FILE` to the public keys of the certificate authorities that sign your client certificates. If this environment variable is not set WarpStream defaults to trusting all client certificates from your Operating System's root certificate store. This store typically contains all the public keys for all the publicly accepted certificate authorities. If this environment variable is not set any certificate from any public certificate authority will be valid and authenticated against your WarpStream agent which could cause external data leaks. By default, the Agent will use the Distinguished Name(DN) from the client TLS certificate as the principal for ACLs. A custom TLS mapping rule regex can be provided using the `-tlsPrincipalMappingRule` flag to extract a name from the DN. For example, the rule `CN=([^,]+)` will extract the Common Name(CN) from the DN, and use that as the ACL principal. For example, given a certificate with the DN of `CN=test_principal,O=ACME Corp`, if `-tlsPrincipalMappingRule` is set to `CN=([^,]+)` then the name `test_principal` will be used as the mTLS ACL principal. #### SPIFFE Support Requires Agent Version: v735 WarpStream supports decoding [SPIFFE](https://spiffe.io/) identity documents in X.509 format. SPIFFE IDs are a Uniform Resource Identifier (URI) which takes the following format: `spiffe://trust domain/workload identifier` for example `spiffe://acme.com/billing/payments`.\ \ To enable SPIFFE decoding set the flag `-spiffeMTLSAuthentication` or environment variable `WARPSTREAM_SPIFFE_MTLS_AUTHENTICATION` to true. Once enabled this will require all mTLS client certificates to have a URI SAN set on their certificate in the spiffe URI format. ACLs can then be made with the following principal format `User:spiffe://trust domain/workload identifier` for example `User:spiffe://acme.com/billing/payments`. An ACL principal can be defined to allow any workload ID under a trust domain by using a `*` symbol, for example `User:spiffe://acme.com/*`. ## Configure WarpStream Agents for Internal Communication Requires Agent Version: v693 WarpStream agents can be configured to encrypt and authenticate internal agent to agent communication.\ \ First, to enable TLS you must set the `WARPSTREAM_INTERNAL_TLS_ENABLED` environment variable to `true`, or set the `internalTLS` flag.\ \ Then set the following flags or environment variables to their respective public and private keys: * `internalTLSServerCertFile` or `WARPSTREAM_INTERNAL_TLS_SERVER_CERT_FILE` * The path to the public key of the certificate * `internalTLSServerPrivateKeyFile` or `WARPSTREAM_INTERNAL_TLS_SERVER_PRIVATE_KEY_FILE` * The path to the private key of the certificate * `internalTLSClientCACertFile` or `WARPSTREAM_INTERNAL_TLS_CLIENT_CA_CERT_FILE` * The path to the public key of the certificate authority that signed the above certificate WarpStream agents use the provided certificate for both the TLS server certificate and the mTLS client certificate.\ \ By default WarpStream agents do not perform validation on the contents of the certificate, they only validate that the certificate is signed by the provided certificate authority. It is therefore recommended to use a unique certificate authority for internal communicate that is different from Kafka Clients. If certificate validation is desired or a unique certificate authority cannot be used it is recommended to use SPIFFE as described bellow. This can be done by creating a certificate with a URI SAN in the SPIFFE URL format, for example `openssl req -new -key agent_key.pem -out agent.csr -subj "/CN=warpstream-agent" -addext 'subjectAltName = URI:spiffe://example.com/agent'`. For details on how to do this with your specific certificate authority see it's documentation. #### SPIFFE Support Requires Agent Version: v734 WarpStream Agents can be configured to validate [SPIFFE](https://spiffe.io/) identity documents for internal agent to agent communication. SPIFFE IDs are a Uniform Resource Identifier (URI) which takes the following format: `spiffe://trust domain/workload identifier` for example `spiffe://acme.com/billing/payments`.\ \ The Trust domain and workload ID can be specified via flags (`-internalSpiffeTrustDomain` and `-internalSpiffeWorkloadID`) or environment variables (`WARPSTREAM_INTERNAL_SPIFFE_TRUST_DOMAIN` and `WARPSTREAM_INTERNAL_SPIFFE_WORKLOAD_ID`).\ \ When set certificates without the specified trust domain and workload identity will be rejected. ## Configure Kafka and Schema Registry clients For configuring mTLS in your Kafka/Schema Registry clients it is recommended to review the documentation for your client. Every client configures mTLS differently and those configurations may change version to version. We recommend using the [Confluent Platform documentation](https://docs.confluent.io/platform/current/security/authentication/mutual-tls/overview.html#clients) to learn how to configure Java-based clients for mTLS. # SASL/OAUTHBEARER Authentication Requires Agent Version: v745 SASL/OAUTHBEARER Authentication uses OAuth tokens based on the Java Web Token (JWT) standard to authenticate your Kafka clients. By default Kafka clients communicating with WarpStream Agents use `PLAINTEXT`, meaning that all data is sent in plain text (unencrypted), this includes the SASL credentials. When using SASL it is recommended to [Configure TLS Encrpytion for your WarpStream Cluster.](/warpstream/kafka/manage-security/protect-data-in-motion-with-tls-encryption#configure-tls-encryption-for-a-warpstream-cluster) ## Configure WarpStream Agents Set the `requireSASLAuthentication` flag or `WARPSTREAM_REQUIRE_SASL_AUTHENTICATION=true` environment variable on the Agents. If limiting SASL authentication methods via the `enabledSASLMechanisms` flag or `WARPSTREAM_ENABLED_SASL_MECHANISMS` environment variable make sure `OAUTHBEARER` is added. \ Set the `saslOauthIssuerURL` flag or `WARPSTREAM_SASL_OAUTH_ISSUER_URL` environment variable to the URL of your OAuth provider for example `https://example.okta.com/oauth2/default`.\ \ Set the `saslOauthAudience` flag or `WARPSTREAM_SASL_OAUTH_AUDIENCE` to the OAuth provider audience for example `api://default`. Once configured is enabled on the Agents, they will enforce that all Apache Kafka clients that connect to them authenticate themselves via SASL. Improperly authenticated clients will be unable to connect. ### Configure Kafka clients For configuring SASL/OAUTHBEARER in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures SASL/OAUTHBEARER differently and those configurations may change version to version. We recommend using the [Confluent Platform documentation](https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/configure-clients.html) to learn how to configure Java-based clients for SASL/OAUTHBEARER. # Manage Connectors This page describes the various options for connector products in WarpStream. WarpStream offers a built-in product called "Managed Data Pipelines" that runs embedded inside the Agents. Managed Data Pipelines is built on-top of the [open source library Bento](https://github.com/warpstreamlabs/bento). For more information on what connectors are available, check out the [Bento docs](https://bento.dev/) as well as the [WarpStream Managed Data Pipelines docs](/warpstream/kafka/manage-connectors/bento). In addition, since WarpStream is Kafka protocol compatible, any other Kafka-compatible connector framework like open-source [Apache Kafka Connect](https://kafka.apache.org/documentation/) or [Confluent Platform Connectors](https://docs.confluent.io/platform/current/connect/kafka_connectors.html) will work as expected, although keep in mind you'll need to [tune their settings](/warpstream/kafka/configure-kafka-client/tuning-for-performance#kafka-connect) to achieve high throughput with WarpStream. Some self-managed Confluent Platform components are licensed for use with Confluent Cloud and verify the broker type before starting. See [Connect Confluent Platform Components Licensed for Confluent Cloud](/warpstream/kafka/manage-connectors/confluent-cloud-components) to configure these components for WarpStream. # Confluent Platform Components Connect eligible self-managed Confluent Platform components to WarpStream. Some self-managed Confluent Platform components use broker detection to verify that they are connected to Confluent Cloud. WarpStream Agents can publish the broker information these components expect, allowing eligible components to connect to a WarpStream cluster. {% hint style="warning" %} This setting does not provide a Confluent license or change its terms. You must have a valid license that permits the component to connect to your WarpStream deployment. Contact your Confluent account team if you are unsure whether your license applies. {% endhint %} ## Supported deployments This feature is available for WarpStream BYOC deployments running Agent v724 or later. This compatibility mode is intended for eligible self-managed Confluent Platform components. It does not establish compatibility with every Confluent Platform component, make a WarpStream cluster available in the Confluent Cloud Console, or provision fully-managed Confluent Cloud connectors. ## Before you begin You need: * A WarpStream BYOC deployment running Agent v724 or later. * A valid license for the Confluent Platform component. * The bootstrap URL and credentials for your WarpStream virtual cluster. You can copy them from the **Connect** tab in the [WarpStream Console](https://console.warpstream.com/). * Network connectivity from the component to the WarpStream Agents. ## Enable Confluent component compatibility Set the following environment variable on every WarpStream Agent in the deployment: ```bash WARPSTREAM_ENABLE_CONFLUENT_COMPONENTS=true ``` Alternatively, pass the corresponding Agent flag: ```bash -enableConfluentComponents ``` Restart the Agents after changing their configuration. A rolling restart is sufficient. When enabled, the Agents create and populate the internal broker-information topic used by eligible Confluent Platform components. Topic creation is idempotent, so keep the setting enabled on every Agent. This does not change your bootstrap URL. Continue using the WarpStream bootstrap URL; a `*.confluent.cloud` hostname is not required. ## Configure the component Configure the component to use the same bootstrap URL, authentication mechanism, and credentials as any other Kafka client connecting to the virtual cluster. For a typical SASL/PLAIN connection over TLS, use: ```properties bootstrap.servers= security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="" \ password=""; ``` Add the Confluent license using the property required by the component, commonly: ```properties confluent.license= ``` Follow the component's documentation for any additional properties. In particular, a commercial Kafka connector may require a separate connection for its license topic: ```properties confluent.topic.bootstrap.servers= confluent.topic.security.protocol=SASL_SSL confluent.topic.sasl.mechanism=PLAIN confluent.topic.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="" \ password=""; ``` Connectors that use Reporter or a separate history topic also require those producers and consumers to use the WarpStream bootstrap URL and credentials. See Confluent's [self-managed Kafka Connect configuration](https://docs.confluent.io/cloud/current/cp-component/connect-cloud-config.html) for the component-specific property prefixes. If you use a different authentication method, replace the SASL/PLAIN properties with the settings described in [Client Configuration](/warpstream/kafka/configure-kafka-client). ## Configure ACLs When ACL enforcement is enabled, grant the component's principal the permissions required by the component. These commonly include: * Read and write access to the application's source and destination topics. * Read, write, create, and describe access to Kafka Connect's configuration, offset, and status topics. * Read, write, create, and describe access to `_confluent-command` or another configured Confluent license topic. * Read and describe access to `__internal_confluent_only_broker_info`. * Access to Reporter, dead-letter queue, and schema-history topics when the connector uses them. For instructions, see [ACLs](/warpstream/kafka/manage-security/configure-acls). ## Verify the connection Start the component and check its license or status endpoint. The component should report a valid license and should not reject the cluster as an unsupported broker type. If the component does not start: 1. Confirm that every Kafka Agent is running v724 or later with `WARPSTREAM_ENABLE_CONFLUENT_COMPONENTS=true`. 2. Confirm that the component can reach the WarpStream bootstrap URL. 3. Test the credentials with one of the client examples from the virtual cluster's **Connect** tab. 4. If ACL enforcement is enabled, check the component logs for authorization failures on its internal topics. 5. Confirm that the license is valid for both the component and the deployment. For eligible licenses and minimum component versions, see [Manage Confluent Platform licenses](https://docs.confluent.io/platform/current/installation/license.html). # Managed Data Pipelines This page explains how to use the WarpStream Managed Data Pipelines functionality. {% hint style="info" %} Running managed data pipelines can significantly increase the load on the WarpStream Agents. This can disrupt production workloads if the same set of Agents are also used to serve Produce and Fetch requests. For large or high volume data pipelines, consider using WarpStream's [Agent Roles](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) feature to isolate the managed data pipelines onto dedicated Agent infrastructure. This can be accomplished by creating a separate deployment of the Agents with the only the "pipelines" role enabled, and then ensuring that the Agent deployment handling your Produce / Fetch requests does not have the "pipelines" role enabled. {% endhint %} ## Overview WarpStream managed data pipelines combine the power of WarpStream's control plane with [Bento](https://warpstreamlabs.github.io/bento/), a stream processing tool that makes "fancy stream processing operationally mundane". Bento is a lightweight stream processing framework that offers much of the functionality of Kafka Connect, as well as additional stream processing functionality like single message transforms, aggregations, multiplexing, enrichments, and more. It also has native support for WebAssembly (WASM) for more advanced processing. Like WarpStream, Bento is written in Go, so we embedded it natively into the WarpStream Agents to make it possible to perform basic ETL and stream processing tasks with a single, stateless binary. WarpStream managed data pipelines take Bento's capabilities one step further: by embedding Bento *inside* the WarpStream Agents, data pipelines can be written, edited, and fully controlled from WarpStream's managed control plane. This makes it possible to perform basic ETL and stream processing tasks with a single, stateless binary, and with no additional infrastructure beyond the WarpStream Agents. This approach brings all the benefits of WarpStream's BYOC Kafka replacement to Bento: managed data pipelines run in *your* cloud account, on *your* VMs, and process data in *your* buckets. WarpStream has no access to any of the data processed by your managed pipelines. Instead, WarpStream's control plane enhances Bento by adding: 1. A helpful UI for creating and editing the pipelines. 2. The ability to pause and resume pipelines dynamically. 3. A full version control system so pipeline configuration is versioned and can easily be rolled forward / backwards. 4. Much more! Managed data pipelines are (currently) only supported in WarpStream's BYOC product. We highly recommend watching the overview video below, which provides a comprehensive overview of Managed Data Pipeline features. {% embed url="" %} ## Getting Started Getting started with WarpStream managed data pipelines is easy. The WarpStream Agents embed Bento natively as a library, but by default managed data pipelines are disabled. To enable them, add the `-enableManagedPipelines` flag to your Agent binary, or set the `WARPSTREAM_ENABLE_MANAGED_PIPELINES=true` environment variable. For example: {% code overflow="wrap" %} ```bash warpstream agent -virtualClusterID XXXXXXXX -apiKey XXXXXXXX -bucketURL s3://my-warpstream-bucket -enableManagedPipelines ``` {% endcode %} However, before deploying your Bento configuration to production, let's get started with a small stream processing task using an ephemeral WarpStream Playground cluster. Start one now by running the command below (Note: the *playground* will set `-enableManagedPipelines` by default): ``` warpstream playground ``` The command will print a URL to a temporary WarpStream Playground account. Open the URL in your browser, then navigate to the "Pipelines" view for your cluster:
Click "Create Pipeline" and give provide a name to create your first managed data pipeline:
After creating your pipeline, you'll be dropped into the WarpStream pipeline editor UI:
From here, you can edit the configuration of your pipeline, pause it, resume it, and roll your configuration forwards and backwards. The first thing we need to do is save an initial configuration for the pipeline. The easiest way to learn how to do that is with the [official Bento documentation](https://warpstreamlabs.github.io/bento/). For now though, copy and paste the sample configuration below: ```yaml input: generate: count: 1000 interval: 1s mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "xyz" } } else { { "type": "bar", "bar": "abc" } } output: kafka_franz_warpstream: topic: bento_test ``` This configuration instructs Bento to generate some semi-random sample data, and then write that data to a WarpStream topic called: "bento\_test". Note that whenever we want to use the current WarpStream cluster as a Kafka source or sink, we use a `kafka_franz_warpstream` block instead of a generic `kafka_franz` block. This instructs the data pipeline to transparently take care of configuring the client for performance by setting an appropriate client ID to leverage WarpStream's zone-aware service discover system, automatically configuring the client with appropriate SASL credentials if the cluster has authentication or ACLs enabled, and much more. Once you've copied and pasted the configuration above, click "Save" in the pipeline editor.
The configuration is now saved and "deployed", but it's not running yet. Click the toggle button next to where it says "PAUSED" to start running the pipeline.
Once the pipeline is running, you can monitor its progress by clicking on the "Topics" tab.
Now lets try modifying a running pipeline! Navigate back to the pipelines view, click on the test pipeline, and then click "Edit" and change the output topic to "bento\_test\_2" :
Now click "Save".
Note that even though we edited the pipeline configuration, the old version is still running, that's why there is a green target symbol next to "Version 0". This indicates that even though "Version 1" is the latest version, the version that is *currently deployed* is "Version 0". This keeps the process of editing configuration separate from the process of deploying configuration. To deploy the new version of the pipeline, select "Version 1" from the menu on the left and then click "Deploy".
Navigate back to the topics tab, and you should see the new "bento\_test\_2" topic now!
That's it! If you have any questions or requests, hop in our [community Slack](https://console.warpstream.com/socials/slack)! ## WarpStream Specific YAML Configuration ### kafka\_franz\_warpstream blocks Anywhere in a Bento configuration file that a `kafka_franz` block can exist, it can be replaced with a `kafka_franz_warpstream` block which will be transparently transformed by WarpStream to refer to the WarpStream cluster that is running the managed pipeline. For example, the following configuration: ```yaml input: kafka_franz: seed_brokers: ["localhost:9092"] topics: ["test_topic"] processors: - mapping: "root = content().capitalize()" output: kafka_franz: seed_brokers: ["localhost:9092"] topic: "test_topic_capitalized" ``` Could instead be expressed as: ```yaml input: kafka_franz_warpstream: topics: ["test_topic"] processors: - mapping: "root = content().capitalize()" output: kafka_franz_warpstream: topic: "test_topic_capitalized" ``` While that may seem like a minor difference, the `kafka_franz_warpstream` block does more than just insert an appropriate value for `seed_broker`: 1. It automatically enables appropriate batching to improve performance with WarpStream. 2. It configures a zone-aware client ID that leverages WarpStream's zone-aware discovery system so that your data pipelines which write to and read from WarpStream will not incur any inter-az networking costs. 3. It detects which port the Agent is serving Kafka on, and configures that as the seed\_broker in-case a port other than `9092` is being used. 4. It automatically detects if authentication / authorization is required, and if so, configures the client with appropriate SASL credentials so your pipelines will work even when authentication is required and ACLs are enabled. Remember, these blocks works *anywhere* that a `kafka_franz` block could be used. For example, you can write more advanced configurations like this, and it will still work: ```yaml input: generate: count: 1000 interval: 1s mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "xyz" } } else { { "type": "bar", "bar": "abc" } } output: broker: outputs: - file: codec: lines path: /tmp/data.txt type: file - kafka_franz_warpstream: topic: bento_test type: kafka_franz pattern: fan_out ``` Whenever communicating with the WarpStream cluster via the Kafka API, we recommend using these blocks instead of standard `kafka_franz` blocks. {% hint style="info" %} The `kafka_franz_warpstream` blocks are a wrapped around standard Bento `kafka_franz` blocks, so anything additional fields specified in the [Bento documentation](https://warpstreamlabs.github.io/bento/docs/components/inputs/kafka_franz) (like `tls.enabled` ) can be used in `kafka_franz_warpstream` blocks as well {% endhint %} ## WarpStream Block The `warpstream` block supports additional WarpStream-specific functionality not available in Bento. ### Concurrency Management {% hint style="info" %} Increasing pipeline concurrency will increase the load on the WarpStream Agents. For large or high volume data pipelines, consider using WarpStream's [Agent Roles](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) or [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) features to isolate the data pipelines onto dedicated Agent deployments. {% endhint %} In addition to the standard Bento configuration, WarpStream data pipelines also support a special top level `warpstream` block. Today there are two additional configurations that this enables. The first is `cluster_concurrency_target` ```yaml input: generate: count: 1000 interval: 1s mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "xyz" } } else { { "type": "bar", "bar": "abc" } } output: kafka_franz_warpstream: topic: bento_test warpstream: cluster_concurrency_target: 1 ``` `cluster_concurrency_target` controls the "target concurrency" for this pipeline. For example, if its set to 1, then that means that ideally you would prefer that only 1 "instance" of this pipeline run in the cluster at any given moment. However, for balancing and simplicity reasons, this value will always round to a multiple of the number of currently running Agents. The table below demonstrates how this behavior plays out in practice.
Cluster Concurrency Target# of AgentsActual Concurrency
133
233
333
436
Alternatively, you can express the concurrency more directly in terms of how many instances of the pipeline each Agent should run using the `agent_concurrency_target` config. For example, 
input:
  generate:
    count: 1000
    interval: 1s
    mapping: |
      root = if random_int() % 2 == 0 {
        {
          "type": "foo",
          "foo": "xyz"
        }
      } else {
        {
          "type": "bar",
          "bar": "abc"
        }
      }
output:
  kafka_franz_warpstream:
    topic: bento_test
warpstream:
  agent_concurrency_target: 1
Agent Concurrency Target# of AgentsActual Concurrency
133
236
339
### Logging {% hint style="info" %} log\_level requires Agent version v797 or higher. {% endhint %} Bento pipelines automatically inherit the default log level of the Agent they're running on. However, the log level of an individual Bento pipeline can be overridden by setting the `log_level` field in the `warpstream` block. For example, the following configuration: {% code overflow="wrap" %} ```yaml warpstream: log_level: debug ``` {% endcode %} will enable debug logs for just that pipeline regardless of what the configured log level at the Agent level is. This log level impacts both logs emitted directly by the Agent, as well as pipeline log emitted to the `pipeline_logs` [events topic](/warpstream/reference/events). Supported values: `none`, `error`, `warn`, `info`, `analytics`, `debug` . ### Error handling By default, WarpStream automatically runs Bento pipelines in [strict mode](https://warpstreamlabs.github.io/bento/docs/components/processors/about#error-handling-1). This is achieved by automatically adding an implicit: {% code overflow="wrap" %} ```yaml error_handling: strategy: reject ``` {% endcode %} block on every running pipeline. In strict mode, Bento will reject all batches containing messages with errors, propagating a `nack` to the input layer instead of attempting to send message batches that contain messages with errors to the configured sink. ### Scheduling The scheduling block supports running a pipeline at regular intervals. This is particularly useful for Bento pipelines where the input source has a natural termination point (as opposed to a Kafka input where will poll for new records forever). For example, when using GCP BigQuery as an input, the pipeline will run until all the records returned by the query have been processed. ```yaml input: generate: count: 1 interval: 1s mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "xyz" } } else { { "type": "bar", "bar": "abc" } } output: kafka_franz_warpstream: topic: bento_test warpstream: scheduling: run_every: 1s ``` Without the `run_every` configuration, this pipeline would run once and generate a single event and then terminate, but with the `run_every` block it will generate one record every second forever. Note that the pipeline will still run on every Agent, so in practice it will be one record/s \* NUM\_RUNNING\_AGENTS. See the [Concurrency Management](#concurrency-management) section below for more details on how to control concurrency. ### Pipeline Groups Pipeline groups are similar to [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups), but for managed data pipelines. They can be used to isolate different managed pipelines onto different groups of Agents. For example, if you had a mixture of small and very large managed pipelines, you can isolate the larger pipelines onto a dedicated group of Agents so that they cant interfere with the smaller pipelines. Agents with managed pipelines enabled (running the `pipelines` role) will run any pipeline that is assigned to the same group as them. If no pipeline group is specified, the Agents will run any pipeline that also doesn't have an assigned group. | Agent Pipeline Group Name | Pipeline Group Name | Will Run? | | ------------------------- | ------------------- | --------- | | `N/A` | `N/A` | Yes | | `small` | `N/A` | No | | `large` | `N/A` | No | | `N/A` | `small` | No | | `small` | `small` | Yes | | `large` | `small` | No | | `N/A` | `large` | No | | `small` | `large` | No | | `large` | `large` | Yes | #### Configuring the Agents The pipeline group name can be any valid string. Configuring the pipeline group name for an Agent deployment is done by setting the `managedPipelinesGroupName` flag or `WARPSTREAM_MANAGED_PIPELINES_GROUP_NAME` environment variable. #### Configuring the Pipeline ```yaml input: generate: count: 1000 interval: 1s mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "xyz" } } else { { "type": "bar", "bar": "abc" } } output: kafka_franz_warpstream: topic: bento_test warpstream: agent_concurrency_target: 1 pipeline_group: $PIPELINE_GROUP_NAME ``` ## Monitoring / Observability Managed data pipelines expose [all native Bento metrics](https://warpstreamlabs.github.io/bento/docs/components/metrics/about/) with the `warpstream_bento_` prefix. In addition, we provide a `warpstream_active_pipeline_instances` metric which tracks the number of instances currently running on each agent for a given managed pipeline. Finally, most WarpStream managed data pipelines run as simple Kafka consumers (if the primary input is a `kafka_franz_warpstream` block, so monitoring the health and progress of the pipeline is best handled in the same way that any other Kafka consumer is monitored, by monitoring [consumer group lag](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups). ## Secrets Secrets are managed just as they are in Bento: using environment variables. This ensures that secrets never leave your environment and that the WarpStream control plane has no access to them. Provide the secret as an environment variable to your WarpStream Agents, and they'll be available for use in the managed data pipelines configuration files. For example: ```yaml input: kafka_franz_warpstream: consumer_group: "${SOME_SECRET}" topics: - test processors: - mapping: root = content().capitalize() output: stdout: codec: lines warpstream: cluster_concurrency_target: 1 ``` The configuration above will look for an environment variable called `SOME_SECRET` and use that as the consumer group name for the `kafka_franz_warpstream` input block. {% hint style="info" %} For more about propagating secrets as environment variables to Bento, check out the [Bento secrets documentation](https://warpstreamlabs.github.io/bento/docs/configuration/secrets). {% endhint %} ## Rate limiting To prevent pipelines from overloading agents, the total throughput processed by an agent across all its pipelines is rate-limited. By default, the rate limit is: * 5MB/vCPU/s if the agent has both the `pipelines` role and any other role. * 50MB/vCPU/s if the agent has only the `pipelines` role. See documentation about agent roles [here](https://github.com/warpstreamlabs/docs/blob/master/byoc/bento/byoc/advanced-agent-deployment-options/splitting-agent-roles.md). We highly recommend having dedicated agents with only the `pipelines` role. You can override the default rate limit using the flag `-overridePipelinesSharedRateLimitPerVCPU` or the environment variable `WARPSTREAM_OVERRIDE_PIPELINES_SHARED_RATE_LIMIT_PER_VCPU`. This sets the rate limit in bytes per vCPU per second. **Note:** Each agent enforces its own rate limit independently. The total cluster throughput scales with the number of agents and their vCPU counts. **Example:** If you have 3 agents, each with 4 vCPUs and only the `pipelines` role: * Each agent has a rate limit of 50MB/s × 4 vCPUs = 200MB/s * Total cluster throughput capacity = 3 agents × 200MB/s = 600MB/s * All pipelines running on the same agent share that agent's 200MB/s budget # Cookbooks This page contains a collection of WarpStream-specific Bento recipes to help you accomplish common tasks. You can always [read the docs](https://warpstreamlabs.github.io/bento/docs/about) for more information about how to use Bento, but this page contains snippets that you can copy-and-paste to accomplish common tasks. ## Stream to Parquet Files ```yaml input: kafka_franz_warpstream: topics: - logs output: aws_s3: batching: byte_size: 32000000 count: 0 period: 5s processors: - mutation: | root.value = content().string() root.key = @kafka_key root.kafka_topic = @kafka_topic root.kafka_partition = @kafka_partition root.kafka_offset = @kafka_offset - parquet_encode: default_compression: zstd default_encoding: PLAIN schema: - name: kafka_topic type: BYTE_ARRAY - name: kafka_partition type: INT64 - name: kafka_offset type: INT64 - name: key type: BYTE_ARRAY - name: value type: BYTE_ARRAY bucket: $YOUR_S3_BUCKET path: parquet_logs/${! timestamp_unix() }-${! uuid_v4() }.parquet region: $YOUR_S3_REGION warpstream: cluster_concurrency_target: 6 ``` ## Stream to GCP BigQuery ```yaml input: kafka_franz_warpstream: topics: - logs output: gcp_bigquery_write_api: dataset: $YOUR_BQ_DATASET project: $YOUR_GCP_PROJECT table: $YOUR_BQ_TABLE pipeline: processors: - mapping: | root.kafka_topic = @kafka_topic root.kafka_partition = @kafka_partition root.kafka_offset = @kafka_offset root.record = content().string() warpstream: cluster_concurrency_target: 6 ``` Note that GCP BigQuery has strict quotas on how many table modifications can be performed per day, so we recommend inserting data using large batches as shown above. ## Stream to GCP BigQuery Iceberg tables Follow the same instructions as [regular GCP BigQuery](#write-data-ingested-into-a-warpstream-kafka-topic-into-gcp-bigquery), except first create an Iceberg table in BigQuery.\ Note that an Iceberg table needs to be [created](https://cloud.google.com/bigquery/docs/iceberg-tables#create-iceberg-tables) in BigQuery first for this work. This bento component leverages BigQuery's [load jobs](https://cloud.google.com/bigquery/docs/iceberg-tables#sql) to import data into the IceBerg table and that API does not support creating Iceberg tables on the fly yet. ## Stream to Redshift ```yaml output: sql_insert: driver: postgres dsn: postgresql://username_from_secret:password_from_secret@"$REDSHIFT_ENDPOINT"/dev table: test columns: [age, name] args_mapping: | root = [ this.age, this.name, ] init_statement: | CREATE TABLE test (name varchar(255), age int); secret_name: "$REDSHIFT_SECRET_NAME" region: eu-west-1 credentials: id: "$AWS_ACCESS_KEY_ID" secret: "$AWS_SECRET_ACCESS_KEY" ``` [Read more details about connecting to Redshift in the Bento documentation.](/warpstream/overview/architecture) # Deploy (advanced) This section explains how to configure more advanced deployment options for the WarpStream BYOC Agents. WarpStream BYOC Agents support a number of advanced features: 1. [Agent Roles](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) can be used to split an Agent deployment into multiple where each split handles a subset of the functionality. This is helpful for isolating the write path, read path, and background jobs from each other. 2. [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) can be used to split a single logical WarpStream cluster at the service discovery level for isolation purposes, or so it can be flexed across multiple network boundaries (VPCs, cloud accounts, etc). 3. Agents can authenticate to the control plane with short-lived, federated OIDC tokens instead of a static Agent Key using [Workload Identity Federation](/warpstream/kafka/advanced-agent-deployment-options/workload-identity-federation). 4. Agents can be configured with [TLS, mTLS, and SASL authentication](https://github.com/warpstreamlabs/docs/blob/master/kafka/advanced-agent-deployment-options/broken-reference/README.md). 5. Agents can be configured to have [much lower latency](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters) in exchange for higher costs. 6. Agents can be [deployed behind a standard proxy / network load balancer](/warpstream/kafka/advanced-agent-deployment-options/configure-warpstream-agent-within-a-container-or-behind-a-proxy) because they're completely stateless and any Agent can serve requests for any topic-partition. # Agent Roles How to run different services on different sets of Agents {% hint style="warning" %} Configuring individual Agent roles is an advanced feature. We recommend familiarizing yourself with WarpStream in its default configuration first before considering splitting roles. Also keep in mind that at least some of your Agents **must** run the `jobs` role or your cluster will eventually cease to function until `jobs` Agents are created and they can finish processing the accumulated backlog of unprocessed jobs. {% endhint %} ### What are Agent Roles The WarpStream Agent has two primary roles: 1. Receiving data from Apache Kafka producers and serving data to Apache Kafka consumers 2. Running background jobs The first role is what we call the `proxy` role. It runs three main services. * The HTTP and TCP servers that respond to produce requests * The HTTP and TCP servers that respond to fetch requests * The file cache for reducing the number of object storage GET operations to respond to Fetch requests. The second role is called the `jobs` role. It runs all of the background tasks that are necessary for a WarpStream cluster to continue functioning. It runs a few different kinds of jobs: * Compaction jobs periodically rewrite and merge files in object storage. * Retention jobs delete files which contain out-of-retention data. * Publish metrics jobs that makes some WarpStream-internal metrics available over the Prometheus endpoint. By default, all Agents run both the `proxy`and `jobs` roles, but it is possible to configure your Agents to start only a subset of these roles. Optionally, there is one additional role that can be configured called `pipelines`. This role is used to run WarpStream Managed Data Pipelines. See [our documentation](/warpstream/kafka/manage-connectors/bento) for more details about this role. {% hint style="warning" %} All roles (except pipelines) are necessary in a deployment of WarpStream. You cannot run only one of the roles for example. However, you can have some Agents run a single role. {% endhint %} ### Configuring Agent Roles {% hint style="warning" %} If you're upgrading an existing WarpStream cluster with traffic to enable roles, follow the [Targeting Agent Groups](#targeting-agent-groups) instructions to configure your Kafka clients to target specific roles first, then redeploy the Agents to configure them into dedicated groups. This is especially important if you're splitting out the `proxy-produce` and `proxy-consume` roles. If you don't change your Kafka client configuration first, then some of your producers will end up connected to the `proxy-consume` Agents and their Produce requests will be rejected with an `INVALID_REQUEST` error message. Similarly, some of your consumers will end up connected to the `proxy-produce` Agents and their Fetch requests will be rejected as well. {% endhint %} Use the `-roles` command line flag or the `WARPSTREAM_AGENT_ROLES` environment variable to configure the roles that an Agent should run. Valid values are: * `"proxy"` to run only the `proxy`role, and to advertise that this Agent is able to process all Kafka protocol requests. You can connect your clients to this agent to produce records or to consume records. In the case of Schema Registry agents, using this role means the agent can handle all API requests. * `"proxy-produce"` to run only the `proxy-produce` role which is a subset of the `proxy` role and indicates that this Agent will process Produce requests, but not Fetch requests. This role is not applicable for Schema Registry clusters. * `"proxy-consume"` to run only the `proxy-consume` role which is a subset of the `proxy` role and indicates that this Agent will process Fetch requests, but not Produce requests. This role is not applicable for Schema Registry clusters. * `"jobs"` to run only the jobs role. * `"pipelines"` to run only the pipelines role (only applicable if using the [Managed Data Pipelines](/warpstream/kafka/manage-connectors/bento) product). You can also combine values, for example: * `"proxy-consume,jobs"` runs the `jobs` role and the `proxy-consume` role. * `"proxy,jobs,pipelines"` runs all roles. {% hint style="danger" %} Make sure at least some of your Agents are running the `jobs` role. This role performs compaction and clean up of deleted files in object storage, and without it your cluster will eventually cease to function entirely until `jobs` Agents are added and they can finish processing the accumulated backlog of unprocessed jobs. {% endhint %} A simple example using our Docker container: ``` docker run public.ecr.aws/warpstream-labs/warpstream_agent_linux_amd64:latest \ agent \ -bucketURL mem://mem_bucket \ -apiKey $YOUR_API_KEY \ -defaultVirtualClusterID $YOUR_VIRTUAL_CLUSTER_ID -roles "jobs" ``` ### Targeting Agent Roles {% hint style="warning" %} If you don't configure the `warpstream_proxy_target` feature on your Kafka client IDs, then your clients will connect to any Agent running a proxy role, regardless of whether its `proxy-consume` or `proxy-produce`. This may break your application as the `proxy-consume` Agents will reject Produce requests and the `proxy-produce` Agents will reject Fetch requests. {% endhint %} If some of your Agents are running with either the `proxy-consume` or the `proxy-produce` role then you will need to update your Apache Kafka client configuration to indicate which set of Agents you want to target based on whether your client is a Producer or Consumer. * For a Kafka consumer application that needs to target Agents running the `proxy-consume` role, add `,warpstream_proxy_target=proxy-consume` to the end of your client\_id. * For a Kafka producer application that needs to target Agents running the `proxy-produce` role, add `,warpstream_proxy_target=proxy-producer` to the end of your client\_id. For more information about how to configure your Apache Kafka client with additional WarpStream features like role target, see our [Configuring Kafka Client ID Features](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features) documentation. ### Filtering Job Types {% hint style="danger" %} Job type filtering is an advanced feature that should not be manually configured unless absolutely necessary. It is primarily designed to be used in pre-defined configurations constructed by the WarpStream team. Please reach out to the WarpStream team before using this feature in your own deployments. {% endhint %} Requires Agent Version: `v760+` When running agents with the `jobs` role, you can control which types of background jobs an agent handles using the `-jobSelector` command line flag or the `WARPSTREAM_JOB_SELECTOR` environment variable. This is useful for isolating different types of jobs onto dedicated Agent deployments so that they can be isolated and/or scaled independently. For example, running query jobs on separate agents from where compaction or orbit jobs run to keep interactive user queries isolated from your online workloads. {% hint style="warning" %} The `-jobSelector` flag requires the `jobs` role. If you specify a job selector without including `jobs` in `-roles`, the agent will fail to start. {% endhint %} The job selector accepts a comma-separated list of job categories: | Category | Description | | --------- | ------------------------------- | | `orbit` | Orbit jobs | | `query` | Query engine jobs | | `metrics` | Cluster metrics publishing jobs | Specify one or more categories to only run those job types. Prefix categories with `-` to exclude them instead: ``` # Only run orbit jobs warpstream agent -roles jobs -jobSelector "orbit" # Run all jobs except orbit warpstream agent -roles jobs -jobSelector "-orbit" # Only run orbit and query jobs warpstream agent -roles jobs -jobSelector "orbit,query" ``` You cannot mix include and exclude syntax in the same selector (e.g., `orbit,-query` is invalid). If no job selector is specified, the agent handles **all** job types. Existing deployments are unaffected. {% hint style="danger" %} Every job type must be covered by at least one running agent. If no agent is configured to handle a particular job type, those jobs will fail to run and break your cluster. When using job selectors, ensure that the union of all your agents' selectors covers all job categories. {% endhint %} A common deployment pattern is to run separate agent groups for different job types: ``` # Group 1: Dedicated orbit agents warpstream agent \ -roles jobs \ -jobSelector "orbit" \ -agentGroup orbit-jobs # Group 2: Dedicated query agents warpstream agent \ -roles jobs \ -jobSelector "query" \ -agentGroup query-jobs # Group 3: All other jobs (everything except orbit and query) warpstream agent \ -roles jobs \ -jobSelector "-orbit,-query" \ -agentGroup other-jobs ``` This allows you to independently scale and tune the resources for each workload. The control plane automatically routes jobs to agents that have registered for the appropriate job types. # Agent Groups How to split Agents for a cluster into different "groups". ### Agent Groups Agent Groups are distinct sets of Agents that all belong to the same logical cluster. Groups enable a single logical cluster to be split into many different "groups" that are isolated at the network / service discovery layer. For example, consider the scenario where a single logical WarpStream cluster is "flexed" across multiple VPCs, regions, or even cloud providers:
In the diagram above producer and consumer clients running in `vpc_1` will only ever connect to Agents running in `group_vpc_1`. Similarly, producers and consumer running in `vpc_2` will only ever connect to Agents running in `group_vpc_2`. However, since both Agent Groups belong to the same logical virtual cluster and have access to the same object storage bucket, clients in each VPC will be able to write and read data for all topics and partitions, even those that were created by clients / Agents running in a completely different VPC! Agent groups are a powerful abstraction that enable a variety of use-cases: 1. Isolating specific producers or consumers to dedicated Agent Groups to avoid noisy neighbors. 2. "Flexing" a single logical cluster across multiple VPCs, regions, or even clouds providers without resorting to complex VPC peering setups. ### Configuring Agent Goups Configuring Agent Groups is simple. Just add the `-agentGroup $GROUP_NAME` flag to your Agent deployment. For example, if you wanted to flex a single logical WarpStream cluster across two Kubernetes clusters running in different VPCs: ``` # In Kubernetes cluster 1 warpstream agent -virtualClusterID $CLUSTER_ID -agentGroup group-1 # In Kubernetes cluster 2 warpstream agent -virtualClusterID $CLUSTER_ID -agentGroup group-2 ``` Alternatively, you can set the `WARPSTREAM_AGENT_GROUP` environment variable instead. {% hint style="info" %} Agent group names may only contain lowercase letters, numbers, and dashes (-). {% endhint %} ### Targeting Agent Groups When your Kafka (or Schema Registry) client connects to an Agent in a specific group, the WarpStream service discovery system will ensure that your client only connects to other Agents in the same group. This means that in order to take advantage of the Agent group functionality, you need to ensure that the bootstrap URL you configure in your client will only ever resolve to Agents in the correct group. #### Kubernetes Targeting specific Agent Groups in Kubernetes is easy. Each Agent group will have its own helm deployment, and therefore its own Kubernetes service. Use the Kubernetes service name that corresponds to the Agent Group that you want to target as the bootstrap URL for your Kafka client. In addition, we highly recommend setting the `ws_ag` [client ID feature](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_agent_group) on your Kafka clients. This is important in dynamic environments like K8s where pods churn frequently and I.P addresses may be quickly cycled between pods running in different agent groups. In that scenario, the Kubernetes service for agent group A may temporarily return the I.P address of an Agent in group B which will result in your client connecting and getting "stuck" in the wrong group. {% hint style="info" %} See [this link](/warpstream/agent-setup/deploy/kubernetes-known-issues#when-an-ip-is-reused-by-another-agents-pod) for more details about how I.P address reuse can cause problems for WarpStream clusters deployed in Kubernetes. {% endhint %} To avoid this issue, configure the name of the agent group you want to target in your client ID. For example, if your existing client ID is: `foo`, or `foo,ws_az=us-east-1a` and you want to target an agent group called `bar` then you should change your client ID to `foo,ws_ag=bar` or `foo,ws_az=us-east-1a,ws_ag=bar` respectively. This way, even if K8s service discovery returns a stale I.P address, the Agent that the client ends up connected to will know which agent group the client intended to connect to and be able to re-route it there. #### Non-Kubernetes Most non-Kubernetes deployments use WarpStream's hosted convenience bootstrap URL for service discovery. These URLs are visible in the WarpStream UI under the "Connect" tab in the cluster view and generally look something like this: `api-XXXX-XXXX-XXXX-XXXX-XXXX.group$GROUP_NAME.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092`\ \ Note that if you haven't explicitly configured any groups for your Agents yet, they will be part of the `default` group so the standard bootstrap URL for a WarpStream cluster that has not explicitly opted into specific groups yet would look like: `api-XXXX-XXXX-XXXX-XXXX-XXXX.groupdefault.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092` If you want to connect to specific Agent groups just replace `default` with the name of your group, for example, if the agent group name was: `test-group-foo`, then the bootstrap URL would become: `api-XXXX-XXXX-XXXX-XXXX-XXXX.grouptest-group-foo.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092` # Workload Identity Federation Authenticate WarpStream Agents to the control plane with short-lived, federated OIDC tokens instead of a long-lived static Agent Key. By default, a WarpStream Agent authenticates to the control plane with a long-lived Agent Key (`aks_...`) that you provision and set via `WARPSTREAM_AGENT_KEY`. Long-lived secrets have to be distributed, rotated, and revoked out of band, and a leaked key is valid until you notice and rotate it. **Workload Identity Federation** removes the static secret. Instead, the Agent proves its identity using a token minted by its own cloud platform (AWS or GCP) and exchanges it with the control plane for a **short-lived, cluster-scoped credential**. There is no static secret to store on the Agent, and access is tied to the cloud IAM identity the Agent already runs as. ### How it works 1. On startup, the Agent asks its cloud platform for a signed OIDC token identifying the workload: * **AWS** — the Agent calls STS [`GetWebIdentityToken`](https://docs.aws.amazon.com/STS/latest/APIReference/API_GetWebIdentityToken.html) using its ambient IAM role. The role's ARN ends up in the token's `sub` claim. * **GCP** — the Agent fetches an identity token from the [instance metadata server](https://cloud.google.com/compute/docs/instances/verifying-instance-identity). The workload's service account email ends up in the token's `email` claim. 2. The Agent exchanges that OIDC token with the WarpStream control plane. The requested audience is always your **Virtual Cluster ID** (`vci_...`). 3. The control plane looks up the **federation binding(s)** you configured on that Virtual Cluster and, for the first binding whose issuer and claim rules match the token, mints a short-lived Agent credential (a `wsa_` token) scoped to that cluster. 4. The Agent uses the `wsa_` token exactly like an Agent Key and refreshes it automatically in the background before it expires. A binding never grants access on a valid signature alone — every configured claim rule must match, so only the specific IAM role or service account you name can authenticate. ### Supported identity providers Workload Identity Federation currently supports the following token sources, selected with the `WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE` Agent environment variable: | Provider | `WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE` | OIDC issuer | | -------- | ------------------------------------------------- | ------------------------------------------ | | AWS | `aws` | `https://.tokens.sts.global.api.aws` | | GCP | `gcp` | `https://accounts.google.com` | Support for additional providers will be added over time. [Reach out](https://www.warpstream.com/contact-us) if you need one that isn't listed yet. ### Before you begin You will need: * A Virtual Cluster (note its `vci_...` ID). * Permission to create bindings on that cluster: a WarpStream **Application Key** for the API/Terraform, generated in the admin console. See [Secrets Overview](/warpstream/reference/secrets-overview). * Agents running in a cloud environment with an identity you control — an AWS IAM role (for example an EKS IRSA / Pod Identity role) or a GCP service account. Setup is three steps: prepare the cloud identity, create the federation binding, then point the Agent at Workload Identity. ## Step 1 — Prepare the cloud identity {% tabs %} {% tab title="AWS" %} **1. Enable outbound identity federation on your AWS account.** This is a one-time, account-wide action that provisions your account's OIDC issuer URL. See [Getting started with outbound identity federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_outbound_getting_started.html). {% code overflow="wrap" %} ```bash aws iam enable-outbound-web-identity-federation ``` {% endcode %} The response contains your account-specific **issuer URL**, which looks like `https://.tokens.sts.global.api.aws`. You'll need it in Step 2. If you enabled the feature previously, retrieve the URL again at any time: {% code overflow="wrap" %} ```bash aws iam get-outbound-web-identity-federation-info ``` {% endcode %} You can also find it in the IAM console under **Access management → Account settings → Outbound identity federation**. **2. Grant the Agent's IAM role permission to mint tokens.** Attach a policy allowing `sts:GetWebIdentityToken` to the role your Agents run as. Restrict the audience to your Virtual Cluster ID so the role can only mint tokens for WarpStream: {% code overflow="wrap" %} ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "MintWarpStreamWorkloadIdentityToken", "Effect": "Allow", "Action": "sts:GetWebIdentityToken", "Resource": "*", "Condition": { "ForAllValues:StringEquals": { "sts:IdentityTokenAudience": "vci_XXXXXXXXXX" }, "NumericLessThanEquals": { "sts:DurationSeconds": "3600" } } } ] } ``` {% endcode %} To let one role authenticate to several WarpStream clusters, list each Virtual Cluster ID as an allowed audience: {% code overflow="wrap" %} ```json "ForAllValues:StringEquals": { "sts:IdentityTokenAudience": ["vci_YYYYYYYYYY", "vci_ZZZZZZZZZZ"] } ``` {% endcode %} {% endtab %} {% tab title="GCP" %} No extra IAM permissions are required — any workload can fetch its own identity token from the GCP metadata server. Just make sure your Agents run **as the service account you intend to authorize**, and note that service account's email (for example `warp-agent@my-project.iam.gserviceaccount.com`). You'll use it in the binding's claim rule in Step 2. * On GKE, associate the Agent's Kubernetes service account with the Google service account using [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity). * On Compute Engine, this is the VM's attached service account. {% endtab %} {% endtabs %} ## Step 2 — Create the federation binding A binding tells the control plane which OIDC issuer and claims to accept for a given Virtual Cluster. Create it with Terraform (recommended) or the API. ### Terraform Use the [`warpstream_workload_identity_federation`](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs/resources/workload_identity_federation) resource (provider `v2.7.10`+): {% tabs %} {% tab title="AWS" %} {% code overflow="wrap" %} ```hcl resource "warpstream_workload_identity_federation" "aws_agents" { virtual_cluster_id = "vci_XXXXXXXXXX" name = "aws-agents" issuer_url = "https://.tokens.sts.global.api.aws" read_only = false max_credential_ttl_seconds = 3600 claim_match_rules = [ { claim_path = "sub" # Replace with the ARN of the AWS IAM role your Agents are deployed with. expected_value = "arn:aws:iam::123456789012:role/warp-agent" }, ] } ``` {% endcode %} {% endtab %} {% tab title="GCP" %} {% code overflow="wrap" %} ```hcl resource "warpstream_workload_identity_federation" "gcp_agents" { virtual_cluster_id = "vci_XXXXXXXXXX" name = "gcp-agents" issuer_url = "https://accounts.google.com" read_only = false max_credential_ttl_seconds = 3600 claim_match_rules = [ { claim_path = "email" # Replace with the email of the GCP service account your Agents are deployed with. expected_value = "warp-agent@my-project.iam.gserviceaccount.com" }, ] } ``` {% endcode %} {% endtab %} {% endtabs %} ### API Send a `POST` to `create_workload_identity_federation` with your Application Key in the `warpstream-api-key` header: {% tabs %} {% tab title="AWS" %} {% code overflow="wrap" %} ```bash curl https://api.warpstream.com/api/v1/create_workload_identity_federation \ -H 'warpstream-api-key: XXXXXXXXXX' \ -H 'Content-Type: application/json' \ -d '{ "virtual_cluster_id": "vci_XXXXXXXXXX", "name": "aws-agents", "issuer_url": "https://.tokens.sts.global.api.aws", "read_only": false, "max_credential_ttl_seconds": 3600, "claim_match_rules": [ { "claim_path": "sub", "expected_value": "arn:aws:iam::123456789012:role/warp-agent" } ] }' ``` {% endcode %} {% endtab %} {% tab title="GCP" %} {% code overflow="wrap" %} ```bash curl https://api.warpstream.com/api/v1/create_workload_identity_federation \ -H 'warpstream-api-key: XXXXXXXXXX' \ -H 'Content-Type: application/json' \ -d '{ "virtual_cluster_id": "vci_XXXXXXXXXX", "name": "gcp-agents", "issuer_url": "https://accounts.google.com", "read_only": false, "max_credential_ttl_seconds": 3600, "claim_match_rules": [ { "claim_path": "email", "expected_value": "warp-agent@my-project.iam.gserviceaccount.com" } ] }' ``` {% endcode %} {% endtab %} {% endtabs %} List the bindings on a cluster: {% code overflow="wrap" %} ```bash curl https://api.warpstream.com/api/v1/list_workload_identity_federations \ -H 'warpstream-api-key: XXXXXXXXXX' \ -H 'Content-Type: application/json' \ -d '{"virtual_cluster_id": "vci_XXXXXXXXXX"}' ``` {% endcode %} Delete a binding by its `wif_...` ID (from the list response): {% code overflow="wrap" %} ```bash curl https://api.warpstream.com/api/v1/delete_workload_identity_federation \ -H 'warpstream-api-key: XXXXXXXXXX' \ -H 'Content-Type: application/json' \ -d '{"virtual_cluster_id": "vci_XXXXXXXXXX", "id": "wif_XXXXXXXXXX"}' ``` {% endcode %} {% hint style="info" %} You can create multiple bindings on one Virtual Cluster — for example, one per availability zone if each zone's Agents run as a different IAM role or service account. Binding names must be unique within a cluster. The control plane tries each binding and mints a credential for the first one whose claims match. {% endhint %} ## Step 3 — Configure the Agent Set the token source on the Agent and **remove the static Agent Key** — the two are mutually exclusive. {% tabs %} {% tab title="AWS" %} {% code overflow="wrap" %} ```bash WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE=aws ``` {% endcode %} {% endtab %} {% tab title="GCP" %} {% code overflow="wrap" %} ```bash WARPSTREAM_AGENT_WORKLOAD_IDENTITY_TOKEN_SOURCE=gcp ``` {% endcode %} {% endtab %} {% endtabs %} The equivalent command-line flag is `-workloadIdentityTokenSource aws` (or `gcp`). Do not also set `WARPSTREAM_AGENT_KEY`, `WARPSTREAM_AGENT_KEY_PATH`, or `WARPSTREAM_API_KEY`; the Agent will refuse to start if a static key is combined with a workload identity token source. On startup the Agent exchanges a token per region and begins refreshing it automatically. If authentication fails, the Agent logs the exchange failure; the most common causes are a missing IAM permission (AWS), the wrong role/service-account identity in the claim rule, or the account's issuer URL not matching the binding. ## Reference ### Binding fields | Field | Required | Description | | ---------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `virtual_cluster_id` | yes | The Virtual Cluster (`vci_...`) the binding grants access to. | | `name` | yes | Human-readable name, unique within the cluster (3–128 characters). | | `issuer_url` | yes | HTTPS URL of the OIDC issuer whose tokens this binding accepts (your AWS account issuer, or `https://accounts.google.com`). | | `claim_match_rules` | yes | One or more rules; **all** must match. See below. | | `read_only` | no | If `true`, the minted credential is read-only. Defaults to `false`. | | `max_credential_ttl_seconds` | no | Maximum lifetime of a minted credential, between `60` (1 minute) and `86400` (24 hours). Defaults to `3600`. | | `audience` | — | Read-only. Always the Virtual Cluster ID; derived by the control plane and not configurable. This is the audience the Agent's OIDC token must request. | Bindings are immutable — to change a binding, delete it and create a new one. ### Claim matching Each rule is a `claim_path` / `expected_value` pair: * **`claim_path`** is a dot-separated path into the token's claims (e.g. `sub`, `email`, or a nested `context.namespace`). If a single path segment itself contains a literal dot, wrap that segment in double quotes, e.g. `"kubernetes.io".namespace`. * **`expected_value`** is matched **case-sensitively**. A single trailing `*` acts as a prefix wildcard (`arn:aws:iam::123456789012:role/warp-*`); a `*` anywhere else is treated literally. All rules in a binding are ANDed together, and at least one rule is always required — a token is never accepted on a valid signature and audience alone. ### Common claims | Cloud | `issuer_url` | Claim to match | Example value | | ----- | ------------------------------------------ | -------------- | ----------------------------------------------- | | AWS | `https://.tokens.sts.global.api.aws` | `sub` | `arn:aws:iam::123456789012:role/warp-agent` | | GCP | `https://accounts.google.com` | `email` | `warp-agent@my-project.iam.gserviceaccount.com` | For the full list of claims available in an AWS token, see [Understanding token claims](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_outbound.html). # Low Latency Clusters Configure the WarpStream Agent with lightning topics and a S3 Express, DynamoDB or Spanner storage layer to reduce Produce latency. ## Overview By default, WarpStream is tuned for maximum throughput and minimal costs at the expense of higher latency. However, WarpStream clusters can be tuned to provide much lower Produce and End-to-End latency. The rest of this document outlines all of the different approaches that can be taken to reduce latency. Note that all of these approaches are cumulative and the lowest possible latency is achieved by combining all of them. The table below summarizes the different approaches and their trade-offs. | Approach | Reduces Produce Latency | Reduces E2E Latency | Full Consistency | Increases Costs | | :------------------------: | :---------------------: | :------------------: | :------------------: | :-------------------------------------: | | Reduce client linger | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: | :x: | | Reduce Agent batch timeout | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: | | Control Plane Cluster Tier | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: | | S3 Express | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: | :white\_check\_mark: (\~20% on average) | | Lightning Topics | :white\_check\_mark: | :x: | :x: | :x: | The table below shows achievable produce and E2E latencies for a variety of different setups. | Setup | Produce Latency | E2E Latency | | :--------------------------------------------------------------------------------------: | :--------------------------------: | :---------------------------------: | | 25ms linger, 250ms batch timeout (default), S3 Standard, Fundamental cluster tier |

p50: 250ms
p99: 500ms

|

p50: 500ms
p99: 900ms

| | 10ms linger, 50ms batch timeout, S3 Express, Fundamentals cluster tier |

p50: < 80ms
p99: < 150ms

|

p50: < 200ms
p99: < 400ms

| | 10ms linger, 25ms batch timeout, S3 Express, Fundamentals cluster tier, lightning topics |

p50: < 35ms
p99: < 50ms

|

p50: < 200ms
p99: < 400ms

| ## Configuration ### Client Linger Before tuning WarpStream itself, first check your client configuration. The WarpStream documentation [has recommendations on how to tune various Kafka clients for maximum performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) with WarpStream. You should still follow all those recommendations, however, if you want to minimize cluster latency then you should consider reducing the value of `linger` in your Kafka client from our default recommendation of 100ms to 25ms or 10ms. ### Agent Batch Timeout The WarpStream Agents accept a `-batchTimeout` (`WARPSTREAM_BATCH_TIMEOUT` environment variable) that controls how long the Agents will buffer data in-memory before flushing it to object storage. Produce requests are never acknowledged back to the client before data is durably persisted in object storage, so this option has no impact on durability or correctness, but it does directly impact the latency of Produce requests. The default `batchTimeout` in the Agents is `250ms` , but the value can be decreased as low as `25ms` to reduce Produce latency. Lowering this value will result in higher cloud infrastructure costs because the Agents will have to create more files in object storage and will incur higher PUT request API fees as a result. Note that [S3 Express](#s3-express) PUTs are \~ 1/5th the cost of a regular S3 PUT, so reducing your batch timeout from `250ms` to `50ms` while also switching to S3OZ would only increase your ingestion PUT request costs by 2x instead of 5x. $$ 250/50 \* (1/5) \* 2 (azs) = 2x $$ ### Control Plane Cluster Tier Similar to the Agents, the WarpStream control plane batches some virtual cluster operations, resulting in higher latency in exchange for reduced control plane costs. Higher [cluster tiers](/warpstream/reference/billing#cluster-tiers) like Fundamentals and Pro batch less and thus have lower control plane latency. Switching cluster tiers is a one-click operation in the WarpStream UI or terraform. ### Lightning topics Lightning topics are a special topic type in WarpStream where the Agents skip committing data to the control plane in the critical path of a produce request. Instead, they journal produce requests to object storage, and commit them to the control plane asynchronously. As a result, lightning topics have significantly lower Produce request latency than regular topics, especially if you have already lowered your [batch timeout](#batch-timeout) and switched to a low latency storage backend like [S3 Express](#s3-express). Lightning topics provide the exact same durability guarantees as regular topics (acknowled data is guaranteed to not be lost), but they do have a few caveats and relaxed consistency guarantees that you can learn more about in our [dedicated lightning topics](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/lightning-topics) documentation. ### Lower Fetch Latency for Low Volume Topics {% hint style="info" %} This feature requires v796+ of the Agent. {% endhint %} When a consumer is caught up to the tail end of a partition and there are no more records left for it to process, fetch requests for that partition need to poll the WarpStream control plane to detect when new records are available. This polling mechanism uses a custom exponential back-off mechanism that is designed to minimize load on the control plane even when tens of thousands of consumer clients are all polling simultaneously. This is the ideal configuration for the vast majority of workloads, but it can lead to a few hundred ms of additional end-to-end latency for low volume topics and partitions which is undesirable for some workloads. If you have such a workload, you can opt into a faster, but less scalable backoff mechanism using one of two methods. First, you can set the `-fetch-quick-retry` flag or `WARPSTREAM_FETCH_QUICK_RETRY=true` environment variable on your Agent deployment. This will automatically opt-in all consumers connected to this Agent deployment to the lower-latency polling mechanism. Alternatively, you can set the `ws_fqr=true` [client ID feature](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_fetch_quick_retry) if you only want to change the behavior of a subset of applications. {% hint style="warning" %} This feature increases the load on the WarpStream control plane which is a finite resource for each virtual cluster. That's fine for most use-cases, but can be problematic for workloads with thousands of tends of thousands of consumer clients. Check the value of the `warpstream_control_plane_utilization` metric before and after enabling this feature to prevent overloading the control plane. {% endhint %} ## Low Latency Storage Backends WarpStream supports low-latency storage backends in all three of the major cloud providers: AWS, GCP, and Azure. ### S3 Express (AWS) [S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/) is a tier of AWS S3 that provides much lower latency for writes and reads. The WarpStream Agents have native support for S3 Express and can use it to store newly written data. Combined with a reduced batch timeout and [lightning topics](#lightning-topics), S3 express can reduce the P99 latency of Produce requests to less than 100ms. Learn how to configure Warpstream agents to write to S3 Express One Zone [here](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/s3-express). ### Rapid Buckets (GCP) Like S3 Express One Zone, [Rapid Buckets](https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket) are a tier of GCP GCS that provides much lower latency for writes and reads. The WarpStream Agents have native support for Rapid Buckets and can use it to store newly written data. Combined with a reduced batch timeout, Rapid Buckets can reduce the P99 latency of produce requests to less than 150ms. Rapid Buckets are **not** compatible with lightning topics. Learn how to configure the WarpStream Agents to write to GCP Rapid Buckets [here](#rapid-buckets-gcp). ### Premium Blob Storage (Azure) [Azure Premium Blob Storage](https://azure.microsoft.com/en-us/blog/premium-block-blob-storage-a-new-level-of-performance/) is a tier of Azure Blob Storage that provides much lower latency for writes and reads, as well as much cheaper blob storage API calls as well. Combined with a reduced batch timeout and lightning topics, this Premium tier can reduce the P99 latency of Produce requests to less than 100ms. One downside of using Azure Premium Blob Storage is that the storage costs are 10x higher than regular blob storage buckets. However, WarpStream can mitigate this downside automatically by landing newly produced data in a premium blob storage bucket to reduce latency, and then subsequently compacting the data into a regular blob storage bucket for long term storage. This is a form of tiered storage where both the "hot" and "cold" storage happen to be blob storage and allows you to get the best of both worlds: low latency, low blob storage API call costs, and low storage costs. To configure this set the `-ingestionBucketURL` or `WARPSTREAM_INGESTION_BUCKET_URL` environment variable to the bucket URL for the premium blob storage bucket and then set the `-compactionBucketURL` or `WARPSTREAM_COMPACTION_BUCKET_URL` environment variable to the bucket url for the standard blob storage bucket and WarpStream will automatically take care of minimizing storage costs for you. ## Alternative Storage Backends In addition to S3 Express, we offer a few additional lower latency storage backends like AWS DynamoDB and Google Spanner. While useful for some applications, keep in mind tha these alternative storage backends are much more expensive than traditional object storage or S3 Express and are not suitable for high volume applications. ### AWS DynamoDB In addition to S3 Express One Zone, AWS developers have the option to deploy their WarpStream agents using [DynamoDB](https://aws.amazon.com/dynamodb/) as the storage layer. Using DynamoDB yields latencies similar to S3 Express One Zone and generally costs less if the workload's throughput is low enough. Higher volume workloads should always prefer S3 Express One Zone over DynamoDB for cost reasons. See the [Cost Estimates](#cost-estimates) section below for more details. Learn how to configure WarpStream agents to use AWS DynamoDB as the storage layer [here](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/aws-dynamo-db) ### Google Spanner (beta) {% hint style="warning" %} Google Spanner support for the data plane is available only for agents using version v709 and above {% endhint %} On GCP deployments, developers can choose to use [Spanner](https://cloud.google.com/spanner) as the storage layer. This is the only low-latency ingestion alternative in GCP, and offers similar tradeoffs to the DynamoDB option described above. It's also only recommended for low-throughput clusters for cost reasons. See the [Cost Estimates](#cost-estimates) section below for more details. Learn how to configure WarpStream agents to use Google Spanner as the storage layer [here](#cost-estimates) # S3 Express [S3 Express One Zone](https://aws.amazon.com/s3/storage-classes/express-one-zone/) is a tier of AWS S3 that provides much lower latency for writes and reads, but only stores the data in a single availability zone. The WarpStream Agents have native support for S3 Express and can use it to store newly written data. Combined with a reduced batch timeout, S3 express can reduce the P99 latency of Produce requests to less than 150ms. ### Tradeoffs and How We Mitigate Them This latency improvement comes with tradeoffs that WarpStream helps you mitigate so that you get the best of both worlds. S3 Express offers faster reads and writes, but charges more for storage. It also provides less resilience than S3 "classic", since by default it doesn't duplicate data across multiple zones. To mitigate S3 Express's increased storage costs, the WarpStream Agents can use different buckets for data ingestion and data compaction. This enables the Agents to ingest data into S3 Express to reduce Produce request latency, but then compact the data into a regular object storage bucket to keep storage costs low. Think of this as a form of tiered storage *within* the object store itself. This is the recommended way to leverage S3 Express with WarpStream because the storage cost of retaining data in in S3 Express is \~7x higher than regular object storage **before** taking replication into account. As the name implies, S3 Express One Zone only stores data in a single availability zone. Therefore to prevent availability zone failures from interrupting your cluster, WarpStream will automatically replicate your ingested data across a quorum of multiple S3 Express single-zone buckets. WarpStream's multi-bucket replication makes S3 Express as resilient as S3 "classic", but drives up your storage costs even further. By restricting S3 Express to data ingestion only, you limit the cost increase to network transfer (which is not free for S3 Express buckets like it is for "classic" S3 buckets) while saving on storage. For more details on S3 Express One Zone pricing, see [AWS's documentation](https://aws.amazon.com/s3/pricing/). ### Configuration The first step to using S3 Express is to create the buckets. This can be done in the AWS console, or by using infrastructure as code like Terraform. Below is a sample Terraform block: ```hcl locals { # S3 Express may not be available in every zone in a region. This # is fine though because we don't get billed for inter-zone networking # between EC2 and S3 Express buckets. You can see the list of available # zone IDs here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-express-Endpoints.html s3_express_zones_ids = ["use1-az4", "use1-az5", "use1-az6"] } resource "aws_s3_directory_bucket" "warpstream_s3_express_buckets" { count = length(local.s3_express_zones_ids) # AZ has to be encoded in this exact format, see docs: # https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_directory_bucket bucket = "warpstream_s3_express--${local.s3_express_zones_ids[count.index]}--x-s3" data_redundancy = "SingleAvailabilityZone" type = "Directory" location { name = local.s3_express_zones_ids[count.index] type = "AvailabilityZone" } } data "aws_region" "current" {} data "aws_caller_identity" "current" {} data "aws_iam_policy_document" "warpstream_s3_express_buckets" { statement { effect = "Allow" actions = [ "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3express:CreateSession" ] resources = concat([ for bucket in aws_s3_directory_bucket.warpstream_s3_express_buckets[*].bucket : "arn:aws:s3express:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:bucket/${bucket}" ], [ for bucket in aws_s3_directory_bucket.warpstream_s3_express_buckets[*].bucket : "arn:aws:s3express:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:bucket/${bucket}/*" ] ) } } resource "aws_iam_role_policy" "warpstream_s3_express_buckets" { name = "warpstream-s3express" role = "YOUR ROLE ID" policy = data.aws_iam_policy_document.warpstream_s3_express_buckets.json } ``` Note that in addition to creating the S3 express buckets, you'll also want to add an S3 express endpoint your VPC. This is free, and will ensure that you don't pay internet egress fees for your S3OZ traffic, as well as reduce latency. ```hcl resource "aws_vpc_endpoint" "s3_express" { vpc_id = $VPC_ID service_name = "com.amazonaws.$REGION.s3express" route_table_ids = $ROUTE_TABLE_IDS } ``` Note that we created *three* S3 Express directory buckets. The reason for this is that the WarpStream Agents will flush ingestion files to all S3 Express directory buckets, and then wait for a quorum of acknowledgements before considering the data durable. In the future we will allow more flexible configurations, but for now we require that at least 3 buckets are configured and all writes must succeed to at least 2 buckets before being considered successful. In addition to creating the buckets, you'll also need to grant your WarpStream Agents' IAM role one extra permission: `s3express:CreateSession`. Once you've created the buckets, and updated the WarpStream Agent IAM role, the final step is to change the Agent configuration to write newly ingested data to a quorum of the S3 Express directory buckets instead of the regular object storage bucket. This is done by deleting the `-bucketURL` flag (`WARPSTREAM_BUCKET_URL` environment variable) and replacing it with two new flags: 1. `-ingestionBucketURL` (`WARPSTREAM_INGESTION_BUCKET_URL`) 2. `-compactionBucketURL` (`WARPSTREAM_COMPACTION_BUCKET_URL`) The value of `compactionBucketURL` should point to a classic S3 bucket configured for Warpstream, i.e. the same value as `bucketURL` in [the default object store configuration](https://docs.warpstream.com/warpstream/configuration/different-object-stores). The value of `ingestionBucketURL` should be a `<>` delimited list of S3 Express bucket directory URLs with a `warpstream_multi://` prefix. For example: {% code overflow="wrap" %} ``` warpstream_multi://s3://warpstream_s3_express--us-east-1a--x-s3?region=us-east-1<>s3://warpstream_s3_express--us-east-1d--x-s3?region=us-east-1<>s3://warpstream_s3_express--us-east-1f--x-s3?region=us-east-1 ``` {% endcode %} That's it! The WarpStream Agents will automatically write newly ingested data to a quorum of the S3 Express directory buckets, and then asynchronously compact those files into the regular object storage bucket. The Agents will also automatically take care of deleting files whose data has completely expired from both the S3 Express directory buckets, and the regular object storage bucket. # GCP Rapid Buckets (beta) {% hint style="danger" %} The default storage quotas for GCP rapid buckets are very low, on the order of 1 TiB per GCP project. We attempt to minimize this limitation as much as possible by writing recently ingested data to Rapid Buckets to reduce latency and then quickly tiering it to a regular regional GCS bucket for long term storage, but high volume workloads may still hit the 1TiB limit in a short period of time before the tiering has kicked in. \ If you're planning to use GCP Rapid Buckets for a high volume use-case, reach out to your GCP account representative and request a quota increase before deploying to production. {% endhint %} {% hint style="danger" %} GCP Rapid Buckets are **not** compatible with [lightning topics](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/lightning-topics). Enabling lightning topics on Agents configured to use Rapid Buckets may actually increase latency instead of decreasing it. {% endhint %} [GCP Rapid Buckets](https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket) are a special type of GCS bucket that is backed [Rapid Storage](https://docs.cloud.google.com/storage/docs/storage-classes#rapid) which is a GCP storage class that provides much lower latency for writes and reads, but only stores the data in a single availability zone. The WarpStream Agents have native support for Rapid Buckets and can use them to store newly written data. Combined with a reduced batch timeout, GCP Rapid Buckets can reduce the P99 latency of Produce requests to less than 150ms. {% hint style="info" %} Rapid Buckets require v815+ of the Agents. {% endhint %} ### Tradeoffs and How We Mitigate Them This latency improvement comes with tradeoffs that WarpStream helps you mitigate so that you get the best of both worlds. Rapid Buckets offer faster reads and writes, but charges more for storage. It also provides less resilience than GCS "classic", since by default it doesn't duplicate data across multiple zones. To mitigate Rapid Buckets' increased storage costs, the WarpStream Agents can use different buckets for data ingestion and data compaction. This enables the Agents to ingest data into Rapid Buckets to reduce Produce request latency, but then compact the data into a regular GCS object storage bucket to keep storage costs low. Think of this as a form of tiered storage *within* the object store itself. This is the recommended way to leverage GCP Rapid Buckets with WarpStream because the storage cost of retaining data in in Rapid Buckets is \~5.5x higher than regular object storage **before** taking replication into account. GCP Rapid Buckets only store data in a single availability zone. Therefore to prevent availability zone failures from interrupting your cluster, WarpStream will automatically replicate your ingested data across a quorum of multiple GCS Rapid Buckets deployed in different availability zones. WarpStream's multi-bucket replication makes Rapid Buckets as resilient as GCS "classic", but drives up your storage costs even further. By restricting GCP Rapid Buckets to data ingestion only, you limit the cost increase to just paying for network transfer (which is not free for Rapid Buckets like it is for "classic" GCS buckets) while saving on storage. For more details on Rapid Buckets pricing, see [GCP's documentation](https://cloud.google.com/storage/pricing). ### Configuration The first step to using GCP Rapid Buckets is to create the buckets. This can be done in the GCP console, or by using infrastructure as code like Terraform. Below is a sample Terraform block: ```hcl locals { # GCP Rapid Storage buckets are zonal, so create one bucket per zone that # your WarpStream Agents run in. # # https://cloud.google.com/storage/docs/locations region = "us-east1" rapid_storage_zones = ["us-east1-b", "us-east1-c", "us-east1-d"] } resource "google_storage_bucket" "warpstream_rapid_storage_buckets" { count = length(local.rapid_storage_zones) name = "warpstream-rapid-${local.rapid_storage_zones[count.index]}" location = local.region soft_delete_policy { # Disable soft deletion, otherwise deleted data is retained (and billed) # for 7 days by default. retention_duration_seconds = 0 } versioning { # Make sure versioning is disabled or it will massively inflate your # storage costs. enabled = false } lifecycle_rule { condition { age = 7 } action { type = "AbortIncompleteMultipartUpload" } } # Everything below is required for Rapid Storage, do not remove. storage_class = "RAPID" uniform_bucket_level_access = true hierarchical_namespace { enabled = true } custom_placement_config { # This is what pins the bucket to a single zone (the equivalent of # gcloud's --placement flag). data_locations = [local.rapid_storage_zones[count.index]] } } # roles/storage.objectUser grants object read/write/delete/list, and is also # required for the folder operations (create/delete) that hierarchical # namespace buckets use. resource "google_storage_bucket_iam_member" "warpstream_rapid_storage_object_user" { count = length(local.rapid_storage_zones) bucket = google_storage_bucket.warpstream_rapid_storage_buckets[count.index].name role = "roles/storage.objectUser" member = "serviceAccount:YOUR_SERVICE_ACCOUNT_EMAIL" } resource "google_storage_bucket_iam_member" "warpstream_rapid_storage_bucket_viewer" { count = length(local.rapid_storage_zones) bucket = google_storage_bucket.warpstream_rapid_storage_buckets[count.index].name role = "roles/storage.bucketViewer" member = "serviceAccount:YOUR_SERVICE_ACCOUNT_EMAIL" } ``` Note that we created *three* Rapid Buckets. The reason for this is that the WarpStream Agents will flush ingestion files to all Rapid Buckets buckets, and then wait for a quorum of acknowledgements before considering the data durable. In the future we will allow more flexible configurations, but for now we require that at least 3 buckets are configured and all writes must succeed to at least 2 buckets before being considered successful. Once you've created the buckets, the final step is to change the Agent configuration to write newly ingested data to a quorum of the Rapid Buckets instead of the regular object storage bucket. This is done by deleting the `-bucketURL` flag (`WARPSTREAM_BUCKET_URL` environment variable) and replacing it with two new flags: 1. `-ingestionBucketURL` (`WARPSTREAM_INGESTION_BUCKET_URL`) 2. `-compactionBucketURL` (`WARPSTREAM_COMPACTION_BUCKET_URL`) The value of `compactionBucketURL` should point to a classic regional GCS bucket configured for WarpStream, i.e. the same value as `bucketURL` in [the default object store configuration](https://docs.warpstream.com/warpstream/configuration/different-object-stores). The value of `ingestionBucketURL` should be a `<>` delimited list of Rapid Bucket URLs with a `warpstream_multi://` prefix. For example: {% code overflow="wrap" %} ``` warpstream_multi://gs://warpstream_rapid_us_east1_a<>gs://warpstream_rapid_us_east1_b<>gs://warpstream_rapid_us_east1_c ``` {% endcode %} That's it! The WarpStream Agents will automatically write newly ingested data to a quorum of the GCP Rapid Buckets and then asynchronously compact those files into the regular regional GCS bucket. The Agents will also automatically take care of deleting files whose data has completely expired from both the Rapid Buckets, and the regular object storage bucket. # AWS DynamoDB ## Flags Pointing the agent to DynamoDB as the backing store is as simple as passing a bucket URL with the following schema. ```go dynamodb://$aws_region/$files_table<>$chunks_table ``` `$aws_region` is the agent's current region. If `$files_table` and `$chunks_table` are existing DynamoDB tables accessible from the same region, the agent will use those for storage. If the tables don't exist, the agent will create them. The need for two separate tables is an implementation detail that shouldn't otherwise affect developers. As with S3 Express, we recommend replacing the `-bucketURL` flag with separate `-ingestionBucketURL` and `-compactionBucketURL` flags. The former should point to DynamoDB and the latter to S3. See the last two paragraphs of [S3 Express](#s3-express) above for details. We also recommend setting the [`-batchTimeout` flag](#batch-timeout) to as low as 50 ms. When S3 is the backing store, lowering this value increases costs. Larger batching is advantageous with S3 because API usage is billed per request, regardless of payload sizes. DynamoDB charges per byte written and read, regardless of the number of API calls. Therefore a lower batch timeout reduces produce latency without affecting cost. Finally, WarpStream's own control plane batching can be tuned for lower latency. See [Control Plane Latency](#control-plane-latency) above. ## AWS IAM Permissions The process running the agent requires the following IAM permissions to use DynamoDB as the backing store. ``` "dynamodb:BatchWrite*", "dynamodb:CreateTable", "dynamodb:DeleteItem", "dynamodb:Update*", "dynamodb:PutItem", "dynamodb:TagResource", "dynamodb:BatchGet*", "dynamodb:DescribeStream", "dynamodb:DescribeTable", "dynamodb:Get*", "dynamodb:Query", "dynamodb:Scan" ``` ## Cost estimates The table below presents the rough cost of each AWS service that can be used as the agent's storage layer for a hypothetical workload of a hundred kilobytes, one megabyte, and ten megabytes per second. These estimates are based on various assumptions, for example that one agent is deployed in each of three availability zone and that the compression ratio is 1:4. In the case of DynamoDB with provisioned usage, the budget is over-provisioned by a factor of 2 for headroom. Most importantly, these numbers only reflect the storage cost of keeping the last five seconds of data at any time. Since we recommend storing compacted data in S3 regardless where it's first ingested, the table below excludes any storage costs incurred after compaction. See the last two paragraphs of [S3 Express](#s3-express) above for details. | Storage layer | 100 KB / s | 1 MB / s | 10 MB / s | | -------------------- | ---------- | -------- | --------- | | S3 | $ 159 | $ 159 | $ 159 | | S3 Express | $ 235 | $ 235 | $ 235 | | DynamoDB on-demand | $ 81 | $810 | $ 8100 | | DynamoDB provisioned | $ 7.5 | $ 75 | $ 750 | While these numbers are only estimates, they illustrate the advantage of using DynamoDB as the agent's storage layer for workloads with sufficiently low throughput. # Google Spanner (beta) ## Flags Pointing the agent to Spanner as the backing store is as simple as passing a bucket URL with the following schema. ```go spanner://projects/$PROJECT/instances/$INSTANCE/databases/$DATABASE ``` This is the most common way to address individual Spanner databases in GCP, just replace `$PROJECT`, `$INSTANCE` and `$DATABASE` with your GCP Project Name, Spanner Instance ID and database name. The database is expected to be provisioned by the user already. We recommend not sharing this database with other applications to prevent accidental deletions or other incidents. On startup, WarpStream agents will create the necessary tables for the data plane inside this database if they are not present yet. These are two simple tables: `warpstream_files` and `warpstream_chunks`. Tampering with these tables through means other than the WarpStream agent itself will result in undefined behavior and most probably a broken cluster. As with S3 Express and DynamoDB, we recommend replacing the `-bucketURL` flag with separate `-ingestionBucketURL` and `-compactionBucketURL` flags. The former should point to Spanner and the latter to GCS. See the last two paragraphs of [S3 Express](#s3-express) above for details. We also recommend setting the [`-batchTimeout` flag](#batch-timeout) to as low as 50 ms. When S3 is the backing store, lowering this value increases costs. Larger batching is advantageous with S3 because API usage is billed per request, regardless of payload sizes. Spanner charges for compute and storage, regardless of the number of API calls. Therefore a lower batch timeout reduces produce latency without affecting cost. Finally, WarpStream's own control plane batching can be tuned for lower latency. See [Control Plane Latency](#control-plane-latency) above. ## Google IAM Permissions We recommend running the agents with the IAM role `roles/spanner.databaseUser` assigned to them for the relevant database. This role gives agents all the permissions they need to run the data plane. # Lightning topics This page explains how to configure topics as Lightning Topics and what impact that will have. ## Overview Lightning Topics are a special topic type in WarpStream where the Agents skip committing metadata to the control plane in the critical path of a `Produce()` request. Instead, they journal `Produce()` requests to object storage, and then commit them to the control plane asynchronously. As a result, Lightning Topics have dramatically lower `Produce()` request latency than regular topics. E2E latency is not impacted. Lightning Topics provide the exact same durability guarantees as regular topics: any acknowledged `Produce()` request is guaranteed to be durable and (eventually) consumable. However, acknowledged data is not immediately visible to consumers due to the async commit which has a few broader implications. See the [caveats sections](#caveats) for more details. {% hint style="danger" %} Lightning Topics are not a substitute for [Ripcord](/warpstream/kafka/advanced-agent-deployment-options/ripcord). If your goal is to configure your Agents to continue processing `Produce()` requests even when the control plane is unavailable, you'll need to configure Ripcord. Lightning Topics are tool for reducing the `Produce()` request latency for specific topics, but they don't guarantee that the workload will continue functioning in the face of control plane unavailability. {% endhint %} {% hint style="info" %} You need at least v743 of the WarpStream agent to produce records to a lightning topic. {% endhint %} ## Configuration Converting an existing topic to a Lightning Topic can be accomplished by adding the Kafka topic configuration property: `warpstream.topic.type` and setting its value to `lightning` . This can be done via the Kafka API using standard Kafka tooling, or via the WarpStream UI, or via our Terraform provider. ```hcl resource "warpstream_topic" "topic" { topic_name = "logs" partition_count = 1 virtual_cluster_id = warpstream_virtual_cluster.example.id config { name = "retention.ms" value = "604800000" } config { name = "warpstream.topic.type" value = "lightning" } } ``` It is also possible to configure the default topic type of all newly created topics in a cluster. For this you can modify the broker configuration `warpstream.default.topic.type` and set the value to `lightning` using most standard Kafka tooling (note that some Kafka tools don't allow setting configs which aren't available in Kafka), or change the default topic type in the Cluster Settings page of the WarpStream console. You can also set this at the virtual cluster level in terraform. ```hcl resource "warpstream_virtual_cluster" "my_cluster" { name = "vcn_my_cluster" tier = "pro" configuration = { default_topic_type = "lightning" } } ``` ## Caveats 1. Lightning topics are **not** compatible with GCP Rapid Buckets. Enabling lightning topics on Agents configured to use Rapid Buckets may actually increase latency. 2. Offsets returned from `Produce()` requests will always be 0. Applications cannot rely on the returned offsets in the ProduceResponse. Almost no applications rely on this, but it's good to be aware of it. 3. The idempotent producer Kafka feature will not work (it must be disabled on all clients producing to Lightning Topics).\ (e.g. if your producer is configured with `enable.idempotence` , its produce requests will be rejected) 4. Kafka Transactions will not work (Transactions must be disabled on all clients producing to the Lightning Topics).\ (e.g. if your producer is configured with `transactional.id` , its produce requests will be rejected) 5. External consistency is no longer guaranteed. For example, if batch A is produced at time T0 and then a successful acknowledgement is received at T1, a batch B that is produced at T2 **is not guaranteed to show up in the log after batch A.** ## Differences with Ripcord [Ripcord](/warpstream/kafka/advanced-agent-deployment-options/ripcord) and Lightning Topics are implemented in a very similar way. Agents running in Ripcord mode are effectively treating all Kafka topics as Lightning Topics, although Ripcord mode also makes a few other changes to make the Agents more resilient to control plane unavailability. If your objective is to reduce the latency of `Produce()` requests, use Lightning Topics. If your objective is to ensure that your WarpStream cluster can still accept `Produce()` requests if the WarpStream control plane is unavailable, configure Ripcord. # Multi-Region Clusters By default, the WarpStream control plane is a single-region service. Since July 2025, we offer a control plane option that is backed by multiple regions at the same time. Paired with a multi-region data plane setup, this allows your workload to sustain the loss of a full cloud provider region while keeping operations running with minimal disruption and no data loss, achieving a Recovery Point Objective of 0. See our [blog post](https://www.warpstream.com/blog/the-hitchhikers-guide-to-disaster-recovery-and-multi-region-kafka) about multi-region Kafka deployments for our detailed recommendations when it comes to streaming data across regions and making your workloads resilient to region-wide cloud provider outages. ### How it works In multi-region mode, your clusters are backed by two or three control plane regions rather than one. You can optionally also choose to back your data plane with multiple buckets in different regions at once, but these are two separate choices. This diagram shows what full WarpStream deployment with multi-region on both the data plane (the agents) and the control plane would look like:

Fully multi-region architecture

For details on the internals of multi-region control planes, see the related [blog post](https://www.warpstream.com/blog/no-record-left-behind-how-warpstream-can-withstand-cloud-provider-regional-outages). The main idea is that the control plane will replicate your metadata across regions, so that if one fails there is always a copy remaining. The agents that form your data plane will talk to one of these regional control planes at any given time, falling back to the other regions if one of them is degraded. Your agents will only talk to a single region at a given time to avoid writing conflicts on the metadata storage, which would impact throughput and latency. If you choose to also spread your data plane across multiple regions, your agents will write all of your actual data to a quorum of object storage buckets rather than a single bucket to ensure that losing one region's worth of object storage doesn't cause data loss for the cluster. ### Multi-region control plane #### Creating a multi-region cluster {% hint style="warning" %} Multi-region support can only be enabled at creation time, and is exclusive to Enterprise-tier clusters. Please select the "Enterprise" cluster tier during cluster creation and enable multi-region support in the same dialog. {% endhint %} To create a multi-region control plane, simply click on the "*Enable multi-region support*" checkbox on the cluster creation dialog and choose one of the multi-region configurations available. These configurations are spread across different sets of regions, as detailed in this table: | Configuration | Provider | Region 1 | Region 2 | Region 3 (if applicable) | | ---------------- | -------- | -------------------- | -------------------- | ------------------------ | | multiregion\_us1 | AWS | us-east-1 | us-west-2 | N/A | | multiregion\_au1 | GCP | australia-southeast1 | australia-southeast2 | N/A | Selecting one of them will tell the control plane to start storing the metadata for your cluster in the corresponding multi-region storage. This will also give you access to the "Multi-Region" tab on the cluster's detail page, which will allow you to control multi-region specific settings. #### Setting up the agents for a multi-region control plane To get the agents to talk to a multi-region cluster, it's just a matter of setting the right agent flags (or environment variables). Agents use the `-multiregion ,,` flag to know which regions form part of their multi-region control plane. These can be in any order but must be the exact set of regions that appears in the table above. Agents talking to a cluster with the `multiregion_us1` configuration should be deployed with the `-multiregion` flag or `WARPSTREAM_MULTIREGION` environment variable set to `"us-east-1,us-west-2"`. The agents also support an additional flag, `-multiregionMetadataURLs` which is mutually exclusive with `-multiregion`. The purpose of this flag is so you can use custom metadata endpoints instead of the default metadata endpoint for a given region name. For example, if you want to talk to your regions using Static IPs, you'll need to use this flag with a comma-separated list of the endpoints that we will provide for you. Please reach out to us on Slack if you want to know the static IP metadata URL set for a specific multi region configuration. #### Spreading your data plane across multiple regions The other half of a multi-region deployment is to not only use a multi-region control plane, but also write your data to several object storage buckets in several regions. You can do this simply by setting a `warpstream_multi://` destination on your agents startup with the `-bucketURL` flag, instead of a single `s3://` (or other object storage equivalent) destination. The format is `warpstream_multi://$BUCKET_1_URL<>$BUCKET_2_URL<>$BUCKET_3_URL` . See the [Multi Buckets](/warpstream/agent-setup/different-object-stores/multi-buckets) page for details, and the [Object Storage Configuration](/warpstream/agent-setup/different-object-stores) page for how to construct each sub-bucket URL. Here's an example of a multi-bucket destination with three buckets spread across three AWS regions: 
-bucketURL "warpstream_multi://s3://bucket-a?region=us-east-1<>s3://bucket-b?region=us-west-2<>s3://bucket-c?region=us-east-2"
### Leader election To avoid writing conflicts, agents only talk to a single region at a given time. To do this, we run a leader election internally that chooses one of the regions as the current leader. This is transparent to you and you should get a similar experience regardless of the current leader, with minimal impact other than a brief (seconds) spike in latency in case of a leadership transition. #### Automatically choosing the fastest region available {% hint style="warning" %} We recommend all our users to run their clusters on **Auto Mode** (the default setting). This the default setting and the one that gives you the best experience when using multi-region clusters. Manual preference selection is meant to be used only in case the system fails to detect a soft failure / regional degradation, so that you can swiftly move leadership away from an "unhealthy" region. {% endhint %} Some of the control planes that conform a multi-region configuration have less latency than others, due to the nature of multi-region storage. By default, the election process runs on what we call "**Auto Mode**", which will automatically choose one of the regions for you. You also have the option to not run on Auto Mode, so you can choose a specific region as "preferred". This will tell the control planes to prefer that region as leader if it is healthy. You can do this from the "Multi-Region" section of the control plane. You'll also be able to see which region currently holds leadership. To learn more about Multi-Region Clusters, [contact us](https://www.warpstream.com/contact-us). # Migrating Control Plane Regions {% hint style="info" %} This feature is only available for WarpStream Kafka clusters. If you need this for Schema Registry or TableFlow, [contact us](https://www.warpstream.com/contact-us). {% endhint %} WarpStream Cluster's Control Plane (the part that WarpStream owns and hosts) lives in a specific (or multiple, if you're using[ Multi-Region Clusters](/warpstream/kafka/advanced-agent-deployment-options/multi-region)) cloud provider region. If you have a cluster for which you want to move the Control Plane from one region to another (to reduce latency or switch cloud providers), or to convert it from a single-region control plane to a multi-region control plane, or any combination thereof, WarpStream offers a way to do so with virtually no downtime. This will not affect the location of your actual data, of your agents, or anything else. It's mostly useful when moving live workloads around or upgrading a production cluster from single-region to multi-region for improved availability guarantees. ### Changing a Cluster's Region To change a cluster's region, there are four simple steps: * Initiate the process by migrating the cluster metadata from the console * Rolling restart your consumers with new config * Rolling restart your agents with new config * Click the finish migration button in the console After this, the cluster will behave as if it had lived in the destination region(s) all along. {% hint style="warning" %} The only important thing to remember during this whole process, is that each step must be fully completed before proceeding onto the next. It's also important that you actually go through with all the steps, as the cluster will technically be more sensitive (outages on either region can affect it) during the migration. {% endhint %} #### Initiate the migration Go to the cluster list page, click on "Operations" and open up the "Migrate Cluster" modal. Select your destination region(s) and hit "Migrate". This will put the cluster in a state where agents configured to talk to the old region(s) will still work normally, but the Control Plane will be migrated to the new region almost instantly (within a few seconds). Agents will automatically start rerouting their traffic to the new control plane region without any manual intervention. Through this process, the only step where you might see some errors/backpressure from the Agents is this one. The cluster might appear unavailable for a few seconds right after you click the button (less than 10s).
After you click this button and until the end of the migration, a new tab will appear under this cluster's detail page, "Regional Migration", which will track your progress through the rest of the steps and tell you exactly what you need to do to finish it. #### Update your client configurations When you migrate the cluster, it gets a new Bootstrap URL for your kafka clients to connect to. Don't worry, until you finish the migration, the old ones will keep working! Head over to the "Connect" tab (or the "Regional Migration" tab) to grab the new URL and roll all your clients talking to this cluster to use it. It's very important that you do this before continuing to avoid client downtime, as the next steps will make the old Bootstrap URLs stop working. If your Kafka clients were not using this kind of Bootstrap URL previously, and instead were communicating through a Kubernetes DNS or some other kind of resolution mechanism, you can ignore this step entirely and there is no need to update them. #### Update your agent configurations Update the `-region` or `-multiregion` flag on your WarpStream agents to the new region. You can see exactly what value you must set in the "Deploy" tab of the cluster detail page, or on the "Regional Migration" tab. Once you fully roll the agents out with the new flag, and no agent is trying to talk to the old region, you'll see that the Regional Migration wizard marks this step as done. It dynamically watches for agent messages to the old region to keep track of this. You won't be able to click on the finish button until this happens. #### Finish the migration Once you are done with the above steps, simply click the finish migration button on the wizard to finish the process. You can now migrate to any other region again if you wish, or even back to the original one. ### A note on Terraform We don't allow regional migrations through Terraform. Changing the region field in Terraform will try to delete the cluster (which should hopefully be deletion protected). To prevent disasters, if your migrating cluster is managed through terraform, remember to: * Update the cluster's terraform code to the new region right after the first step * Run a plan to make sure it's clean # Network Architecture Considerations This page describes a variety of different approaches that can be used to deploy WarpStream with more advanced network setups. ## Normal Network Architecture Setup In most WarpStream deployments, client applications must connect directly to the WarpStream agents. This requires direct layer 3 network connectivity between the client applications and agents with no proxies, load balancers, NATing, etc. in the middle. The below architecture is a normal network architecture where all the applications can directly communicate with all the WarpStream agents.
This is the recommended architecture for most WarpStream deployments because it is the easiest, most-effective, and most performant way to run a WarpStream cluster. ## Approaches for Connectivity Between Unconnected Networks In some situations, the direct connectivity described in the previous section is not always possible or desired. One example situation would be when the WarpStream agents are deployed in a Kubernetes cluster, but the client applications are deployed outside of the Kubernetes cluster in a completely different VPC. There are two different approaches for enabling connectivity between Kafka Clients and the WarpStream Agents when the clients and agents are running in different networks with no direct connectivity between them: 1. [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) 2. [TCP Load Balancer](#tcp-load-balancer) In general, Agent Groups are the preferred solution. They're easier to set up, (generally) more cost-effective, and they don't suffer from any of the [performance penalties](#why-tcp-load-balancers-can-cause-performance-problems) that are associated with using a TCP load balancer. ### Agent Groups (recommended) {% hint style="info" %} Agent Groups are the recommended approach for solving lack of direct connectivity between Kafka clients and WarpStream agents. The only scenario where we don't recommend this approach is if it will require a very high number of Agent groups. {% endhint %} WarpStream's diskless architecture means that any Agent can write or read data for any topic-partition. As a result, WarpStream clusters can be split into distinct "groups" that are completely isolated from each other at the networking / service discovery layer.
This feature is called Agent Groups and is very useful for enabling a single WarpStream cluster to be flexed across multiple disparate networks with no inter-connectivity without incurring the cost and [performance penalties](#why-tcp-load-balancers-can-cause-performance-problems) of using a TCP load balancer. For more details, read the [Agent Groups documentation](/warpstream/kafka/advanced-agent-deployment-options/agent-groups). ### TCP Load Balancer {% hint style="danger" %} Make sure you've at least considered using the [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) approach before deploying a TCP load balancer for WarpStream. While you **can** run the WarpStream Agents behind a load balancer, keep in mind that it may result in [reduced performance](#why-tcp-load-balancers-can-cause-performance-problems). Whenever possible, direct connectivity between Kafka clients and the WarpStream Agents is preferred, especially for high volume workloads. {% endhint %} If the Agent Group approach is not viable for some reason, you'll have to setup a TCP load balancer instead.
{% hint style="info" %} When exposing WarpStream to external networks (I.E over the internet) it is highly recommended to configure TLS and Authentication. See [TLS](/warpstream/kafka/manage-security/protect-data-in-motion-with-tls-encryption), [SASL Authentication](/warpstream/kafka/manage-security/sasl-authentication), [Mutual TLS (mTLS)](/warpstream/kafka/manage-security/mutual-tls-mtls) for configuration details. {% endhint %} Agent configuration: * `WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID=$VIRTUAL_CLUSTER_ID` * `WARPSTREAM_REQUIRE_SASL_AUTHENTICATION=true` * `WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=$LOAD_BALANCER_HOSTNAME` In some cases, the load balancer may be listening on a port that's different from the port the Agents are listening on (defaut `9092` for TCP/Kafka protocol traffic). In that scenario, you'll need to add one additional environment variable to the Agent configuration: ``` WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE=$EXTERNAL_NLB_PORT ``` This instructs the Agents to advertise the load balancer's port within the Kafka protocol instead of the port that the Agents are listening on. {% hint style="info" %} Note that this change will make the Agents **advertise** a different port within the Kafka protocol, but they'll continue **listening** on the same port (default `9092`) so traffic between the load balancer and the Agents will not be impacted by this change. It's just required due to a quirk of how service discovery within the Kafka protocol works. {% endhint %} ### Kubernetes {% hint style="danger" %} Keep in mind that introducing TCP load balancers between Kafka clients and the WarpStream Agents can lead to [performance issues](#why-tcp-load-balancers-can-cause-performance-problems). Whenever possible, try to solve connectivity problems with [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) instead. {% endhint %}
Running WarpStream within Kubernetes can be simple and straightforward with our [Helm charts](/warpstream/agent-setup/infrastructure-as-code/helm-charts). However, when applications that are running outside of the Kubernetes cluster / VPC need to connect to WarpStream additional configuration may be required. In this example setup we will have at least 3 helm deployments for 3 different agent groups. See [Agent Groups](/warpstream/kafka/advanced-agent-deployment-options/agent-groups) for information about groups. Agent Group One will handle applications running in the same Kubernetes cluster as the agents via direct connectivity within Kubernetes. Agent Group Two will handle applications running in the same VPC as the Kubernetes cluster but not running in the Kubernetes cluster itself. {% hint style="info" %} In some setups this group isn't needed due to pod IPs being routable on the VPC, consult your cloud provider's Kubernetes documentation for details about routable pod IPs. {% endhint %} Agent Group Three will handle applications running outside of the VPC, for example connecting over the internet. In all three cases the bootstrap server will be printed out in the `NOTES` section during the `helm install`. Bellow are the recommended helm values to set for the various groups. {% code title="one-values.yaml" lineNumbers="true" %} ```yaml config: agentGroup: one bucketURL: apiKey: virtualClusterID: region: ``` {% endcode %} {% code title="two-values.yaml" lineNumbers="true" %} ```yaml config: agentAroup: two bucketURL: apiKey: virtualClusterID: region: kafkaService: enabled: true annotations: # Uncomment one of the following annotations depending on your Cloud Provider # networking.gke.io/load-balancer-type: "Internal" # service.beta.kubernetes.io/azure-load-balancer-internal: "true" # service.beta.kubernetes.io/aws-load-balancer-scheme: "internal" type: LoadBalancer port: 9092 # Override the hostname to be the hostname of the internal TCP Load Balancer # In some environments this isn't needed if your Kubernetes pod IPs are routable. # See your Kubernetes provider network documentation for details. extraEnv: - name: WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE # Replace this with the hostname of your internal TCP load balancer value: nlb-internal.xxx ``` {% endcode %} {% code title="three-values.yaml" lineNumbers="true" %} ```yaml config: agentGroup: three bucketURL: apiKey: virtualClusterID: region: kafkaService: enabled: true annotations: # If using AWS EKS uncomment the following annotation # service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" type: LoadBalancer port: 9092 # Set a certificate since this load balancer is exposed to the internet certificate: enableTLS: true # The Kubernetes TLS secret that contains a certificate and private key # see https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets secretName: warpstream-external-tls # If using mtls uncomment the following # mtls: # enabled: true # # # The secret key reference for the certificate authority public key # certificateAuthoritySecretKeyRef: # name: "warpstream-external-tls" # key: "ca.crt" # Override the hostname to be the hostname of the external TCP Load Balancer extraEnv: - name: WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE # Replace this with the hostname of your external TCP load balancer value: nlb-external.xxx # If using SASL authentication uncomment the following # - name: WARPSTREAM_REQUIRE_SASL_AUTHENTICATION # value: "true" # # If using mTLS authentication uncomment the following # - name: WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION # value: "true" ``` {% endcode %} You can then install all three agent groups by running the following commands: ```bash helm upgrade --install warpstream-agent-one warpstream/warpstream-agent \ --namespace $YOUR_NAMESPACE \ --values one-values.yaml helm upgrade --install warpstream-agent-two warpstream/warpstream-agent \ --namespace $YOUR_NAMESPACE \ --values two-values.yaml helm upgrade --install warpstream-agent-three warpstream/warpstream-agent \ --namespace $YOUR_NAMESPACE \ --values three-values.yaml ``` When using AWS EKS it is recommended to use the [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html), the old in-tree or out-of-tree cloud provider for EKS is considered [Legacy](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) by AWS. While it is still possible to use the legacy provider for Load Balancers there is little available public documentation and the required annotations may be different. ## Additional Configuration ### Client Specific Override It is sometimes useful to override the hostname on a client level. This is typically needed when using `kubectl port-forward`. Set the `ws_host_override` parameter **within the client's** ID when creating the Kafka client (check [Configuring Kafka Client ID Features](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features) for more details): ```go kgo.NewClient(..., kgo.ClientID("ws_host_override=127.0.0.1"), ) ``` Our recommendation is to only use the above configuration in debugging situations and not long-term deployments. ### Internal Listener Override WarpStream agents must be able to directly communicate with each other. [They need to communicate to efficiently share files and data](https://www.warpstream.com/blog/minimizing-s3-api-costs-with-distributed-mmap). In rare situations it may be necessary to override the internal agent to agent hostname. This can be done by setting the `-advertiseHostnameStrategy` flag or the `WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY` environment variable to `custom`. Then, provide the custom hostname by setting either the `-advertiseHostnameCustom` flag or the `WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM` environment variable. However, our recommendation is to always allow agents to directly communicate with each other and not adjust the above mentioned configurations. ## FAQ ### Why TCP Load Balancers Can Cause Performance Problems There are two reasons that introducing a load balancer between Kafka clients and the WarpStream Agent can result in performance problems: 1. WarpStream has a built-in load balancing mechanism that keeps the WarpStream Agents evenly utilized. 2. While any Agent can handle writes or reads for any partition, WarpStream will generally try to align writes/reads for the same topic-partition from different clients on the same Agent. This improves data locality which dramatically improves performance in a variety of different dimensions (latency, utilization, compression, etc). Both of these mechanisms rely on WarpStream controlling (via the Kafka protocol) which clients connect to which Agents. As a result, these mechanisms degrade when a load balancer is inserted between the Kafka clients and the WarpStream Agents. For well behaved workloads, WarpStream can still work well when running behind a load balancer, but direct connectivity between Kafka clients and the WarpStream Agents is always recommended for the most demanding workloads. {% hint style="info" %} Whenever possible, try to solve connectivity problems with [Agent Groups](#agent-groups-recommended) instead. {% endhint %} ### Typical issues when hostname is not overridden correctly WarpStream agents utilize their private IP and ports for ongoing connections after the initial bootstrap. Without the correct configurations, clients might connect to bootstrap successfully yet experience issues when progressing beyond the initial phase. For example you may receive the following errors when hostname override is incorrectly set: {% code overflow="wrap" %} ``` % warpstream cli diagnose-connection -bootstrap-host my-kafka.example.com running diagnose-connection sub-command with bootstrap-host: my-kafka.exampl.com and bootstrap-port: 9092 Broker Details --------------- 10.212.2.26:9092 (NodeID: 1195648645) failed to communicate with Agent returned as part of Kafka Metadata response, err: , this usually means that the provided bootstrap host: my-kafka.exampl.com:9092 is accessible on the current network, but the URL that the Agent is advertising as its broker host/ip: 10.212.2.26:9092 is not accessible on this network. If this is occurring during local development whilst running the Agent in a docker container, consider adding the following flag to the docker run command: --env "WARPSTREAM_PRIVATE_IP_OVERRIDE=127.0.0.1" which will force the Agent to advertise its hostname/IP address as localhost for development purposes. ``` {% endcode %} {% code overflow="wrap" %} ``` % kafka-topics --bootstrap-server my-kafka.example.com --list [2025-02-03 15:08:19,631] WARN [AdminClient clientId=adminclient-1] Connection to node 1195648645 (10.212.2.26:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient) ``` {% endcode %} In these examples we are trying to connect to `my-kafka.example.com`. However, the `WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE` environment variable is not set on the agent to that hostname. We can see that the Kafka clients are trying to connect to `10.212.2.26:9092`which is the private IP of the agent. Our Kafka clients cannot connect to the IP so they fail with connection errors. # Agent Configuration Reference Reference documentation for WarpStream Agent flags. ## Required Command Line Flags and Environment Variables All WarpStream Agent configurations can be set via command-line flags or environment variables. Command-line flags take precedence over environment variables.
FlagEnvironment VariableDescription
bucketURLWARPSTREAM_BUCKET_URLSee the dedicated documentation section
agentKeyWARPSTREAM_AGENT_KEYWarpStream Agent Key obtained from the WarpStream admin console.
defaultVirtualClusterIDWARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_IDWarpStream Virtual Cluster ID obtained from the WarpStream admin console.
regionWARPSTREAM_REGIONWarpStream virtual cluster's control plane region. Can be obtained from the WarpStream admin console.
## Optional Command Line Flags and Environment Variables All WarpStream Agent configuration can be set either via command line flags, or environment variables. Command line flags take precedence over environment variables. To connect eligible self-managed Confluent Platform components, enable `enableConfluentComponents` or set `WARPSTREAM_ENABLE_CONFLUENT_COMPONENTS=true`. See [Connect Confluent Platform Components Licensed for Confluent Cloud](/warpstream/kafka/manage-connectors/confluent-cloud-components) for prerequisites, licensing considerations, and client configuration.
FlagEnvironment VariableDescription
apiKeyWARPSTREAM_API_KEYBackward-compatible alias of agentKey
agentGroupWARPSTREAM_AGENT_GROUPName of the 'group' that the Agent belongs to. This feature is used to isolate groups of Agents that belong to the same logical cluster, but should not communicate with each other because they're deployed in separate cloud accounts, vpcs, or regions. Leave blank to indicate the Agent belongs to the default group.
heartbeatEveryWARPSTREAM_HEARTBEAT_EVERYHow often the agent should heartbeat the WarpStream backend. Recommended to not modify this.
httpPortWARPSTREAM_HTTP_PORTThe port the Agent will use for serving HTTP requests (Kinesis API requests, distributed file cache requests, exposing Prometheus metrics, etc) (default 8080).
enableKafkaWARPSTREAM_ENABLE_KAFKAEnable kafka server (default true).
kafkaPortWARPSTREAM_KAFKA_PORTThe port the Agent will listen on for Kafka client TCP connections (default 9092).
kafkaFetchCompressionWARPSTREAM_KAFKA_FETCH_COMPRESSIONCompression type to use for Fetch responses: none, gzip, snappy, lz4 (by default), zstd. This is only used if no compression is set explicitly, or if 'agent' type compress.
kafkaMetadataRefreshIntervalWARPSTREAM_KAFKA_METADATA_REFRESH_INTERVALPeriod of time at which topic metadata is refreshed. Unlike Kafka, this metadata cache refresh also affects the timestamp type associated with a stream (default 1m0s).
kafkaHandleConsumerGroupsInBackendWARPSTREAM_KAFKA_HANDLE_CONSUMER_GROUPS_IN_BACKENDHandle consumer group 'JoinGroup' and 'SyncGroup' requests in the backend instead of in the agent. When handled in the backend, the 'Rebalance Timeout' is always set to 10 seconds, whereas in the agent, it will be determined by client specifications. Enabling this option offers the advantage of reduced error potential and seamless integration of backend improvements and bug fixes. However, exercise caution when enabling it for large consumer groups, as a 10-second rebalance timeout may lead to extended rebalancing times and consequently, prolonged consumption pauses. Warning: Ensure uniformity within your agent pool regarding this setting. Having a mix of enabled and disabled settings may lead to rebalancing issues and potential disruptions.
kafkaHighCardinalityMetricsWARPSTREAM_KAFKA_HIGH_CARDINALITY_METRICSWhether to emit metrics with high cardinality tags. When set to true, it enables detailed tracking at a granular level, such as metrics for individual fetch and produce operations on a per-topic basis. Use with caution as high cardinality can significantly increase the amount of data collected, potentially impacting performance.
kafkaCloseIdeConnAfterWARPSTREAM_KAFKA_CLOSE_IDLE_CONN_AFTERClose idle connections after the number of duration specified by this config (default 10m0s).
kafkaMaxFetchRequestBytesUncompressedOverrideWARPSTREAM_KAFKA_MAX_FETCH_REQUEST_BYTES_UNCOMPRESSED_OVERRIDEMaximum number of uncompressed bytes that can be fetched in a single fetch request (default 128MiB).
kafkaMaxFetchPartitionBytesUncompressedOverrideWARPSTREAM_KAFKA_MAX_FETCH_PARTITION_BYTES_UNCOMPRESSED_OVERRIDEMaximum number of uncompressed bytes that can be fetched in a single fetch request for a single topic-partition (default 128MiB).
fileCacheSizeBytesWARPSTREAM_FILE_CACHE_SIZE_BYTESSize of the Agent file cache size in bytes. This cache is used to reduce the number of object storage GET requests that required to serve consumers.

Defaults to 0.5GiB/vCPU if omitted.
fileCacheExtraReplicasWARPSTREAM_FILE_CACHE_EXTRA_REPLICASNumber of extra replicas for the distributed file cache. Helps improve availability and reduce errors when Agents shutdown ungracefully. You can override this to 0, but do not increase this value above 1 unless you know what you're doing.
gracefulShutdownDurationWARPSTREAM_GRACEFUL_SHUTDOWN_DURATIONAmount of time to wait after receiving SIGTERM before exiting to allow graceful removal from service discovery (default 1m0s).
maxConcurrentRequestPerCPUWARPSTREAM_MAX_CONCURRENT_REQUEST_PER_CPUMaximum number of concurrent requests (per CPU) allowed by the Kafka server.
enableClusterWideEnvironmentWARPSTREAM_ENABLE_CLUSTER_WIDE_ENVIRONMENTWhether the cluster wide environment should be enabled.
clusterWideEnvironmentPortWARPSTREAM_CLUSTER_WIDE_ENVIRONMENT_PORTThe default port to use for the cluster wide environment (default 9999).
ingestionBucketURLWARPSTREAM_INGESTION_BUCKET_URLObject storage URL to use for data ingestion (produce requests).
compactionBucketURLWARPSTREAM_COMPACTION_BUCKET_URLObject storage URL to use for files created by compactions.
batchTimeoutWARPSTREAM_BATCH_TIMEOUTControls the maximum amount of time the WarpStream Agents will allow a produced record to remain buffered in batch before flushing it to object storage. Increasing this value reduces object storage API costs, but increases latency, and vice versa.

Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency.

Defaults to 250ms, minimum is 50ms.
batchMaxSizeBytesWARPSTREAM_BATCH_MAX_SIZE_BYTESControls the maximum number of bytes that will be buffered by the WarpStream Agents before flushing it to object storage. Increasing this value reduces object storage API costs for workloads that write more than uncompressed 16MiB/s/Agent, but increases latency, and vice versa.

Note the WarpStream agents never acknowledge data until it has been flushed to object storage so this value has no impact on correctness or durability guarantees, only latency.

Defaults to 4MiB, minimum is 1MiB, maximum is 64MiB.
batcherMaxInflightBytesPerCPUWARPSTREAM_BATCHER_MAX_INFLIGHT_BYTES_PER_CPUMaximum number of inflight bytes per CPU from Produce requests that have not yet been flushed to object storage that can be in memory at once before the Agent will begin backpressuring.
batcherMaxInflightFilesPerCPUWARPSTREAM_BATCHER_MAX_INFLIGHT_FILES_PER_CPUMaximum number of inflight files per CPU from Produce requests that have not yet been flushed to object storage that can be in memory at once before the Agent will begin backpressuring.
metadataURLWARPSTREAM_METADATA_URLAddress for WarpStream metadata backend (favor using the -regionflag instead).
schemaRegistryURLWARPSTREAM_SCHEMA_REGISTRY_URLAddress for WarpStream schema registry backend.
schemaRegistryPortWARPSTREAM_SCHEMA_REGISTRY_PORTPort to run the schema registry server on (default 9094).
regionWARPSTREAM_REGIONRegion that the WarpStream control plane is running in. Value for your cluster can be obtained from the WarpStream console. Optional if your control plane is in us-east-1, otherwise must be provided.
kafkaLoadBalancingIntervalWARPSTREAM_KAFKA_LOAD_BALANCING_INTERVALTime after which the Kafka connection will be closed. This mechanism helps load balance the clients by forcing them to query the magic URL again. By resetting the connection periodically, clients are evenly distributed across available Kafka connections. (default 8760h0m0s).
kafkaInterzoneLoadBalancingIntervalWARPSTREAM_KAFKA_INTERZONE_LOAD_BALANCING_INTERVALInterval at which the Kafka connection assesses if the client-agent connection resides in the same Availability Zone (AZ). If they are not in the same AZ and there are agents available within the client's AZ, the connection is terminated. This approach encourages load balancing by prompting clients to re-query the magic URL and, consequently, connect to agents within their respective AZ. For this mechanism to function, clients should include 'waprstream_az=X' and 'warpstream_interzone_lb=true' in their clientID. (default 1m0s).
kafkaLoadBalancingDrainingTimeWARPSTREAM_KAFKA_LOAD_BALANCING_DRAINING_TIMETime given to gracefully close the Kafka connection after the reconnect interval is reached.
advertiseHostnameStrategyWARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY

Which hostname strategy should be used the agent should advertise itself on. Accepted values: auto-ip4/auto-ip6/local/custom.

auto-ip4 means that it will try to automatically find an IP v4 that makes sense

auto-ip6 will do the same with an IPv6.

local will use localhost

If you select custom them you have to also define advertiseHostnameCustom.

advertiseHostnameCustomWARPSTREAM_ADVERTISE_HOSTNAME_CUSTOMCustom hostname value to advertise to service discovery for clustering purposes if the custom advertise strategy is selected.
requireSASLAuthenticationWARPSTREAM_REQUIRE_SASL_AUTHENTICATIONIf set to true, the Agents will require that all Kafka clients authenticate themselves with proper SASL credentials.
enabledSASLMechanismsWARPSTREAM_SASL_ENABLED_MECHANISMSIf you provide it with a comma-separated list of SASL mechanisms, only those will be enabled. Valid values are "PLAIN" and "SCRAM-SHA-512". For example, if you set WARPSTREAM_SASL_ENABLED_MECHANISMS="SCRAM-SHA-512", you will not be able to use the PLAIN mechanism to connect, only SCRAM-SHA-512.
logIntervalWARPSTREAM_LOG_INTERVALInterval for logging service status (default 15s).
enableDatadogProfilingWARPSTREAM_ENABLE_DATADOG_PROFILINGEnable datadog profiling (default false).
enableDatadogTracingWARPSTREAM_ENABLE_DATADOG_TRACINGEnable datadog tracing (default false).
enablePrometheusMetricsWARPSTREAM_ENABLE_PROMETHEUS_METRICSEnable prometheus metrics (default true).
enableDatadogMetricsWARPSTREAM_ENABLE_DATADOG_METRICSEnable datadog metrics (default false).
disableConsumerGroupMetricsWARPSTREAM_DISABLE_CONSUMER_GROUP_METRICSDisable the consumer group offset metrics automatically published by default (warpstream_consumer_group_lag and warpstream_consumer_group_max_offset).
disableConsumerGroupsMetricsTagsWARPSTREAM_DISABLE_CONSUMER_GROUP_METRICS_TAGSComma-separated list of the tags to not expose in the consumer group offset metrics (warpstream_consumer_group_lag and warpstream_consumer_group_max_offset).
disableLogsCollectionWARPSTREAM_DISABLE_LOGS_COLLECTIONDisable the logs collection sent to warpstream backend (default false).
rolesWARPSTREAM_AGENT_ROLESRoles that the agent should start (comma-separated) (default "proxy, jobs").
bentoBucketURLWARPSTREAM_BENTO_BUCKET_URLBucket URL to use when fetching the bento configuration.
bentoConfigPathWARPSTREAM_BENTO_CONFIG_PATHPath in the bucket to fetch the bento configuration.
enableManagedPipelinesWARPSTREAM_ENABLE_MANAGED_PIPELINESWhether data pipelines can be managed by the control plane.
availabilityZoneRequiredWARPSTREAM_AVAILABILITY_ZONE_REQUIREDWhen enabled, the agent will synchronously try to resolve its az during startup for 1min, and will not start serving its /v1/status health check until it succeeds. The process will exit early if it did not manage to resolve the availability zone. Only used in agent mode.
kafkaTLSWARPSTREAM_TLS_ENABLEDEnable TLS for Kafka client/Agent connections. Must also specify tlsServerCertFile and tlsServerPrivateKeyFile.
schemaRegistryTLSWARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLEDEnable TLS encryption over the schema registry port.
tlsServerCertFileWARPSTREAM_TLS_SERVER_CERT_FILEPath to the X.509 certificate file in PEM format for the server.
tlsServerPrivateKeyFileWARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILEPath to the X.509 private key file in PEM format for the server.
tlsClientCACertFileWARPSTREAM_TLS_CLIENT_CA_CERT_FILEPath to the X.509 certificate file in PEM format for the client certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification.
requireMTLSAuthenticationWARPSTREAM_REQUIRE_MTLS_AUTHENTICATIONIf set to true, the Agents will require that all Kafka clients authenticate themselves with mTLS. enableTLS must be set to true.
tlsPrincipalMappingRuleWARPSTREAM_TLS_PRINCIPAL_MAPPING_RULERegular expression to extract the ACL principal from the client certificate distinguished name. requireMTLSAuthentication must be set to true.
storageCompressionWARPSTREAM_STORAGE_COMPRESSIONCompression used in object store, either zstd or lz4.
externalSchemaRegistryBasicAuthUsernameWARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_USERNAMEUsername for the external schema registry.
externalSchemaRegistryBasicAuthPasswordWARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_PASSWORDPassword for the external schema registry.
externalSchemaRegistryTlsServerCACertFileWARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_SERVER_CA_CERT_FILEPath to the X.509 certificate file in PEM format for the schema registry server's certificate authority. If not specified, the host's root certificate pool will be used for client certificate verification.
externalSchemaRegistryTlsClientCertFileWARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_CERT_FILEPath to the X.509 certificate file in PEM format for the schema registry client.
externalSchemaRegistryTlsClientPrivateKeyFileWARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_PRIVATE_KEY_FILEPath to the X.509 private key file in PEM format for the schema registry client.
enableSetMutexProfileFractionWARPSTREAM_ENABLE_SET_MUTEX_PROFILE_FRACTIONEnable this flag to call runtime.SetMutexProfileFraction with the value passed along -mutexProfileFraction.
mutexProfileFractionWARPSTREAM_MUTEX_PROFILE_FRACTIONTune the value passed to call runtime.SetMutexProfileFraction (default 10).
enableSetBlockProfileRateWARPSTREAM_ENABLE_SET_BLOCK_PROFILE_RATEEnable this flag to call runtime.SetBlockProfileRate with the value passed along -blockProfileRate.
blockProfileRateWARPSTREAM_BLOCK_PROFILE_RATETune the value passed to call runtime.SetBlockProfileRate (default 100000000).
maxProduceRecordSizeBytesWARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTESMaximum size of a record that can be produced. Value needs to be between 4MiB and 256 MiB (default 32 MB).
disableProfileForwardingWARPSTREAM_DISABLE_PROFILE_FORWARDINGDisable profile forwarding to warpstream backend. Note that if both Datadog profiling and profile forwarding are on, profile forwarding will automatically be turned off (default false).
maxProfileSizeWARPSTREAM_MAX_PROFILE_SIZEMaximum number of bytes for buffering profiles in memory. Value needs to be smaller than 1 MiB (default 500 KiB). This is only used if -disableProfileForwarding is false.
zonedCIDRBlocksWARPSTREAM_ZONED_CIDR_BLOCKSA mapping of availability zones to Kafka client IPs. The mapping should be a <> delimited list of AZ to CIDR range pairs, where each pair starts with an AZ, a @, and a comma separated list of CIDR blocks for that given AZ. For example, us-east-1a@10.0.0.0/19,10.0.32.0/19<>us-east-1b@10.0.64.0/19<>us-east-1c@10.0.96.0/19.
autoTuneFetchLimitsWARPSTREAM_AUTO_TUNE_FETCH_LIMITSAllow consumer fetch limits to be auto-adjusted (default true).
N/AWARPSTREAM_AVAILABILITY_ZONE

Override the Availability Zone name which is discovered by the WarpStream Agent automatically using Cloud Instance Metadata (see section below).

We do not recommend overriding this in the general case.

N/AWARPSTREAM_LOG_LEVEL

Override the log level of the WarpStream Agent from the default value of info. Acceptable values are debug, info, warn, and error.

Defaults to info.

N/AWARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDEOverrides the hostname that the WarpStream Agents will report to the WarpStream discovery system (instead of the default of reporting their private IP4 address).

This is useful when running the Agents behind a network load balancer which requires that the Agents report their hostname as the hostname of the network load balancer instead of their private IP.
# Reducing Infrastructure Costs How to reduce infrastructure costs for WarpStream BYOC clusters. ## Reducing Infrastructure Costs WarpStream infrastructure costs can originate from four different sources: 1. Networking 2. Storage 3. Compute 4. Object Storage API Fees ### Networking With WarpStream, you can avoid 100% of inter-AZ networking fees by properly configuring your Kafka clients. Unlike Apache Kafka, WarpStream Agents will *never* manually replicate data across availability zones, but Kafka producer/consumer clients can still connect cross-zone, resulting in inter-zone networking fees.

Kafka producer/consumer clients incurring inter-zone networking fees.

This happens because by default WarpStream has no way of knowing which availability zone the client is connecting from. To avoid this issue, configure your Kafka clients to announce what availability zone they're running in using a [client ID feature](/warpstream/kafka/configure-kafka-client/configure-clients-to-eliminate-az-networking-costs), and WarpStream will take care of zonally aligning your Kafka clients (for both Produce and Fetch requests) resulting in almost zero inter-zone networking fees.

Kafka producer/consumer clients using WarpStream's zonal-alignment functionality to eliminate inter-zone networking fees entirely.

### Storage WarpStream uses object storage as the primary and only storage in the system. As a result, storage costs in WarpStream tend to be [more than an order of magnitude lower](https://www.warpstream.com/blog/cloud-disks-are-expensive#cloud-disks-are-expensive) in WarpStream than they are in Apache Kafka. Storage costs can be reduced even further by configuring the WarpStream Agents to store data compressed using ZSTD instead of LZ4. Check out our [compression documentation](/warpstream/kafka/reference/compression) for more details. In addition, just like with Apache Kafka, storage costs can be reduced by reducing the retention of your largest topics as well. ### Compute The easiest way to reduce WarpStream Agent compute costs is to auto-scale the Agents based on CPU usage. This feature is built-in to our [Helm Chart for Kubernetes](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent). ### Object Storage API Fees WarpStream's entire [storage engine](/warpstream/overview/architecture) is designed around minimizing object storage API fees as much as possible. This is accomplished with a file format that can store data for many different topic-partitions, as well as heavy usage of buffering, batching, and caching in the Agents. The most expensive source of object storage API fees in WarpStream are the PUT requests required to create files as a result of Produce requests. By default, the WarpStream Agents will buffer data in-memory until one of the following two events occur: * The batch timeout elapses * The Agent estimates that the file it will create with the accumulated data reaches a certain size at which point the Agent will flush a file to the object store and then acknowledge the Produce request as a success back to the client. #### Batch timeout The default value for the batch timeout in the agent is 250ms. It can be changed with the `-batchTimeout` Agent flag or the `WARPSTREAM_BATCH_TIMEOUT` environment variable. If you decrease this to 100ms for example, you will force a file to be created every 100ms even if it is small, increasing the number of Object Storage PUTs the Agent makes, but lowering latency. If you increase it to 400ms for example, you will allow more data to accumulate, but probably increase the Produce latency. #### Batch size There are two different ways to control the batch size the Agent uses. You can tune either the *compressed* or the *uncompressed* batch size. The Agent is configured by default with a maximum compressed batch size of 1MiB and a maximum uncompressed batch size of 64MiB. This means that, by default, the files the Agent creates will be less than 1MiB compressed. The 64MiB is mostly a safeguard, it's hard to write 64MiB of uncompressed data in 1MiB (your compression needs to be very high). {% hint style="warning" %} Note that before v750 of the Agent, only the uncompressed batch size flag existed. The default uncompressed batch size was 4MB. {% endhint %} That being said, you can override them. 1. If you want to control the size in terms of *uncompressed bytes* then change the `-batchMaxSizeBytes` flag or the `WARPSTREAM_BATCH_MAX_SIZE_BYTES` environment variable. This disables the default compressed batch size. 2. If you want to control the size in terms of *compressed bytes* then change the `-batchMaxCompressedSizeBytes` flag or the `WARPSTREAM_BATCH_MAX_COMPRESSED_SIZE_BYTES` . 3. Optionally, you can also set both flags and the Agent will create a file whenever the file goes above any of the two limits. {% hint style="info" %} Note that `-batchMaxCompressedSizeBytes` is only enforced approximatively: the Agent does not know exactly how big a file is going to be before it actually writes it. {% endhint %} #### Choosing the batch size to minimize costs Follow this guide to tune when the Agent creates files: 1. First, understand what parameter causes files to be created. Graph the sum of the `warpstream_agent_segment_batcher_flush_outcome` metric grouped by `flush_cause`. This will tell you if most of the time the Agent creates a new file because of the `timeout` (e.g. the batch timeout was reached) or because it was `buffer_full` (e.g. the size limit was reached). 2. If you mostly hit the timeout, it means that you are creating files that are smaller than the limit. 1. If you want to minimize costs, you can reduce the number of agents (and use bigger instances) so that each agent receives more data in the interval. You can also split your Agents using [Agent Roles](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) so that the Produce traffic targets only a subset of the Agents. 2. You can also increase the Batch Timeout, increasing latency but creating less, bigger files. 3. If you mostly hit the `buffer_full` , it means the files you create reach their limit. You can increase either the compressed or the uncompressed batch size to make bigger files. These files will take a little longer to upload, but your PUT costs will decrease. To monitor the size of the files you create, you can plot the average of the `warpstream_agent_segment_batcher_flush_file_size_uncompressed_bytes` metric (for the uncompressed size) or the `warpstream_agent_segment_batcher_flush_file_size_compressed_bytes` metric (for the compressed size). You can repeat those steps multiple times: 1. Increase the batch timeout until it's batch size that is the limit 2. Increase the batch size so that it's again the timeout that becomes the limit to further reduce the PUT request costs. # Ripcord Ripcord enables WarpStream Agents to continue ingesting data and accepting Produce requests even when the control plane is unavailable. ## Overview Ripcord is a special mode that can be enabled on the Agents that makes them resilient to control plane unavailability for ingesting data and processing Produce requests. This means that when the WarpStream Agents are running in Ripcord mode, they can continue ingesting data and processing Produce requests without interruption. Consumers will still become unavailable until control plane availability is restored. Ripcord behaves very similarly to [lightning topics](/warpstream/kafka/advanced-agent-deployment-options/low-latency-clusters/lightning-topics): the Agents skip committing data to the control plane in the critical path of a produce request. Instead, they journal produce requests to object storage, and then commit them to the control plane asynchronously. This reduces Produce request latency as a side-effect, but the primary purpose is to eliminate the control plane as a critical path dependency for Producing. While Ripcord mode is very similar to just converting all of the topics in a cluster to lightning topics, it also makes a few additional tweaks in the Agent to make them more resilient to control plane unavailability. For example, in Ripcord mode, the Agents will configure their topic metadata caches to more strongly favor availability over consistency. This means that newly created topics may take longer to become available for Produce requests. Ripcord Agents provide the exact same durability guarantees as regular Agents: any acknowledged Produce request is guaranteed to be durable and (eventually) consumable. However, acknowledged data is not immediately visible to consumers due to the async commit which has a few broader implications. See the [caveats sections](#caveats) for more details. ## Request Flow ### Producing records without Ripcord Kafka clients connect to the WarpStream Agents in order to produce data to specific topics. When a client produces records, three steps are normally required. 1. The agent writes the record data to a new file in object storage. 2. The agent registers the address of the new file with the control plane and the control plane responds with the offsets that were assigned to the new records. 3. The agent forwards this information to the client in response so the client's Produce request. With this the client can acknowledge that the records have been safely persisted to the topic and at which offset. For #2 the agent must be able to reach the WarpSteam control plane over the Internet. If that connection goes down, e.g. because of a sustained network failure or due to a potential control plane incident, the agent will reject the Produce request by default. ### Producing records with Ripcord Ripcord is a resilience feature enabling agents to process Produce requests without a working connection to the control plane. In Ripcord mode, the agents continuously journals files to your object storage bucket, and data is being replayed asynchronously from this journal and placed inside the topics. It works as follows: 1. The agent writes the record data to a new file in object storage 2. In response to the Produce request, the agent notifies the Kafka client that the new records have safely been journaled to object storage, but that it does not know at which offset they will be inserted yet. The unknown offset is represented as offset 0. 3. Asynchronously, the agent notifies the control plane about the journaled files so that the records can be assigned their respective offsets. In the case where the control plane is momentarily unreachable, the journal grows bigger and is replayed when the control plane is reachable again. ## The Agent "Ripcord mode" {% hint style="info" %} You need at least v748 of the WarpStream agent to enable ripcord. {% endhint %} If you set `-enableRipcord` in your agent startup flags (or if you set the `WARPSTREAM_ENABLE_RIPCORD` environment variable to `"true"`) the control plane will not be in the critical path of your produce requests, for all topics, as explained above. Note that you if you split your agent with [agent roles](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles), you can enable ripcord only on the produce agents. However, it's very important that you upgrade all agents in the virtual cluster to at least v748, not only the produce agents. ## Caveats There are a few caveats that you should be aware of when enabling Ripcord on your agents: 1. You should not use the application bootstrap URL (the one that looks like `api-305fe61f-4074-4e5b-3c791-33d17d4be89a.groupdefault.kafka.discoveryv2.prod-z.us-east-1.warpstream.com:9092` or something similar. This DNS address is resolved by our control plane, and if you lose connectivity to the control plane, your clients will not be able to resolve the agents anymore. If you're running in Kubernetes, the easiest solution is just use the Kubernetes service that is automatically created by the WarpStream chart as the bootstrap URL. 2. New agents cannot start as long as the control plane is unreachable, nor can existing agents restart. Only agents that are already running can keep serving Produce requests, provided that they have Ripcord enabled. 3. Without connectivity to the control plane, Ripcord agents can keep producing records but cannot process Fetch requests. Consumers will not be able to make progress until control plane availability is restored.Other than Produce, any request that depends on the control plane will fail. For example topics cannot be created and topic retention periods cannot be modified. 4. Offsets returned from Produce requests will always be 0. Applications cannot rely on the returned offsets in the ProduceResponse. Almost no applications rely on this, but it's good to be aware of it. 5. The idempotent producer Kafka feature will not work (it must be disabled on all clients producing to ripcord agents). 6. Kafka transactions will not work (it must be disabled on all clients producing to ripcord agents). 7. External consistency is no longer guaranteed. I.E if batch A is produced at time T0 and then a successful acknowledgement is received at T1, a batch B that is produced at T2 **is not guaranteed to show up in the log after batch A.** ## Monitoring Ripcord files are written to the object storage bucket and processed asynchronously. Developers can monitor the size of the unprocessed files backlog. First, the `warpstream.agent_ripcord_replayed_file` metric indicates how many files are currently being asynchronously ingested. Second, `warpstream.agent_ripcord_oldest_replay_age` and `warpstream.agent_ripcord_outstanding_replays_count` report the size of the backlog of replays waiting to be ingested. Since the process that registers files after ingestion is slow, the oldest replay age can typically range from a few minutes to 15 minutes, and it's OK to have 500 outstanding replays, but more probably means your agents are not processing the records fast enough. Also note that these last two metrics are not emitted when the connectivity to the control plane is lost, so you should monitor the number of replayed file too. If this falls to 0 for agents that have `-enableRipcord`, and that are receiving traffic, something is wrong. ## Testing To see Ripcord in action, visit your cluster's `Cluster Settings` page in the admin console and click the `Reject Ripcord Agent Connections` button. Ripcord agents will not be able to connect to the control plane until the setting is disabled. Use with caution! ## Impact on latency Enabling ripcord on your agents has no noticeable impact on end to end latency. The reason for this is that, although the records are journaled to S3, the agent starts replaying the journal **immediately** after finishing to write the file. However, enabling ripcord lowers the produce latency because the agent responds to the client without needing to wait for the control plane to respond. # Reference This section contains various reference information about the WarpStream Kafka product. The following topics contain various reference information about the WarpStream Kafka product, like: 1. How [compression](/warpstream/kafka/reference/compression) works. 2. Which [Kafka features and protocol messages](/warpstream/kafka/reference/protocol-and-feature-support) are supported. 3. Additional features that are WarpStream-specific and not available in Kafka, like the [Partitions Auto Scaler](/warpstream/kafka/reference/partitions-auto-scaler-beta). 4. The [MCP Server](/warpstream/reference/mcp-server) for querying events and diagnostics from AI assistants and IDEs. # Compression Overview of how compression works in WarpStream and how to configure it. ## How It Works in Kafka Compression in open source Apache Kafka is (mostly) controlled by the producer clients. Producers batch records together to create compressed batches, send those compressed batches to the Kafka brokers, and then the Kafka brokers write them to disk unchanged. Similarly, when a Consumer client fetches records from the Kafka Broker, the broker will transmit the records "as is" over the wire without modification (using the SendFile syscall). This means that in practice with Apache Kafka, the compression of batches both at rest and over the wire is almost completely controlled by the Producer clients. ## How It Works in WarpStream Unlike Apache Kafka, WarpStream's storage engine is not tightly coupled with the record-batch format. As a result, when batches are sent to the WarpStream Agents by producer clients, they're decompressed, encoded into WarpStream's file format, and then recompressed. {% hint style="info" %} While this may sound expensive, in practice it's quite cheap, especially when you consider the fact that in Kafka every record-batch has to be processed by at least three different Kafka brokers (due to replication), whereas in WarpStream it will only be processed by a single Agent. This approach has a number of other benefits as well, but we won't delve into them right now. {% endhint %} Similarly, when a Consumer client fetches records from a WarpStream Agent, the records are decoded/decompressed out of WarpStream's file format, and then re-encoded/compressed into Kafka's record-batch format. The practical implication of this is that users can control which compression algorithm is used to compress the record-batches that are sent to the consumer clients independently from the compression that was used by the storage engine and producers. For example, in WarpStream it's possible to configure your producer clients to compress records using LZ4 over the wire, then have the WarpStream Agents store them using ZSTD in the object storage bucket, and finally transmit them as GZIP from the Agents to the consumer. The table below compares Open Source Kafka with WarpStream in terms of who is in control of the compression algorithm for every operation. | Operation | Kafka | WarpStream | | ------------------------- | -------------------------------------- | ------------------- | | Producer --> Broker/Agent | Producer Client | Producer Client | | Storage (at rest) | Producer Client or Topic Configuration | Agent Configuration | | Broker/Agent --> Consumer | Producer Client or Topic Configuration | Agent Configuration | ### Configuring Compression in WarpStream #### Agent Level Configuration There are two Agent level flags that control compression: * `-storageCompression` (`WARPSTREAM_STORAGE_COMPRESSION`) * Default: `zstd` (v671+, previous `lz4`) * Valid options: `lz4`, `zstd` * Topic level configuration override: N/A * `-kafkaFetchCompression` (`WARPSTREAM_KAFKA_FETCH_COMPRESSION`) * Default: `lz4` * Valid options: `lz4`, `snappy`, `gzip`, `zstd`, `none` * **Note**: We highly recommend sticking with the default of `lz4` as it strikes the best balance between compression ratio and performance. `zstd` is also a good choice for clients that support it. * Topic level configuration override: `warpstream.compression.type.fetch` * Setting this value on the topic's configuration will override which compression algorithm is used to return compressed batches to consumers for this topic, regardless of which setting is configured in the Agents. * For example, setting `warpstream.compression.type.fetch=zstd` on topic `logs` would cause all consumers of the `logs` topic to receive record-batches compressed using `zstd`, even if the value of `kafkaFetchCompression` was set to `lz4`.
### Why Compression Still Matters in WarpStream WarpStream eliminates all inter-zone networking fees, and replaces all local disks / EBS volumes with *only* object storage. As a result, it would be easy to conclude that compression is not nearly as important for WarpStream clusters as it is for Apache Kafka clusters. After all, networking is free and object storage is [\~24x cheaper than EBS per GiB-stored](https://www.warpstream.com/blog/cloud-disks-are-expensive). However, compression is still important for WarpStream clusters for two reasons: First, workloads with long retention will still see significant cost savings, even when using object storage as the only storage tier, with improved compression ratios. Second, even though networking is free, the networking *capacity* of the cloud VMs that the Agents are running on is not unlimited. Between producers, consumers, and background compactions, the WarpStream Agents are very network intensive. In general, we recommend running on network-optimized instances like `m6in` in AWS, but even so, the best way to prevent the Agents from exceeding the network capacity of their VMs is to use a strong compression algorithm like ZSTD. ### Difference with Kafka for Fetch Requests There is one additional difference between WarpStream and Kafka with regards to compression and Fetch requests. When Kafka clients issue Fetch requests, they specify a limit on the amount of data to be returned (in aggregate, and per-partition) by the Broker. In Apache Kafka, the brokers interpret the limit in terms of *compressed* bytes, but in WarpStream the Agents interpret the limit in terms of *uncompressed* bytes. WarpStream intentionally breaks from the behavior of Apache Kafka here primarily for safety reasons. A common issue with Apache Kafka is that something changes in the workload which dramatically improves the compression ratio and then causes the downstream consumers to begin OOMing. This is particularly problematic because the change could be something as benign as the upstream consumers changing compression algorithms, or sending a burst of highly repetitive and easily compressible data. This problem would have been exacerbated even further in WarpStream because the background compactions that the Agents perform improve data locality, and as a result, data compression. WarpStream avoids all of these issues altogether by interpreting the limits as uncompressed bytes so that the amount of raw data returned in each Fetch remains the same *regardless of fluctuations in the underlying compression ratio*. One side effect of this decision is that for some workloads, you will need to tune your Kafka consumer clients to request more data per individual Fetch request to achieve the same throughput with WarpStream. We have detailed recommended settings for a variety of different clients in our [tuning for performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) documentation. # Protocol and Feature Support The current implementation of the Apache Kafka protocol in WarpStream supports the basic ability to create topics, delete topics, produce data, consume data, and use consumer groups to load balance consumers and track offsets. Specifically, the following Apache Kafka messages are currently supported: 1. `Produce` 2. `InitProducerID` 3. `Fetch` 4. `ListOffsets` 5. `Metadata` 6. `OffsetCommit` 7. `OffsetFetch` 8. `FindCoordinator` 9. `JoinGroup` 10. `Heartbeat` 11. `SyncGroup` 12. `OffsetDelete` 13. `ApiVersions` 14. `CreateTopics` 15. `DeleteTopics` 16. `ListGroups` 17. `AlterConfigs` 18. `DescribeConfigs` 19. `DescribeCluster` 20. `DescribeGroups` 21. `DeleteGroup` 22. `LeaveGroup` 23. `CreateACLs` 24. `DescribeACLs` 25. `DeleteACLs` 26. `CreatePartitions` 27. `AddPartitionsToTxnResponse` 28. `AddOffsetsToTxnRequest` 29. `EndTxnRequest` 30. `TxnOffsetCommitRequest` 31. `DescribeTransactionsRequest` 32. `ListTransactionsRequest` 33. `GetTelemetrySubscriptions` 34. `PushTelemetry` 35. `ListConfigResources` We're continuously adding support for more Apache Kafka features and message types. Please [contact us](https://www.warpstream.com/contact-us) for specific feature requests or if you notice any discrepancies. Note that because of WarpStream's stateless architecture, many of the Apache Kafka protocol messages are irrelevant. For example, messages like: 1. `AlterReplicaLogDirs` 2. `ElectLeaders` 3. `ListPartitionReassignments` 4. `AlterPartitionReassignments` 5. `DescribeQuorum` 6. `UnregisterBroker` 7. `ControlledShutdown` 8. `StopReplica` 9. `LeaderAndIsr` have no meaning or value when using WarpStream because data durability and replication is managed by the underlying object store. Partitions do not have assigned "leaders", and clean shutdown is automated by virtue of the Agents being stateless. ### Transactions / Exactly Once Semantics WarpStream supports Apache Kafka transactions and Exactly Once Semantics. To make use of it, please set `enable.idempotence` to `true` and add a non-empty `transactional.id` in your client configuration. ### Client Metrics (KIP-714) WarpStream implements [KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) so connected Kafka clients can push their internal producer/consumer metrics to the cluster on a configurable schedule. See [Client Metrics (KIP-714)](/warpstream/kafka/configure-kafka-client/client-metrics-kip-714) for how to create subscriptions and view the data. ### Schema Registry WarpStream has a BYOC Schema Registry built into the Agent binary. Check out the [documentation](https://docs.warpstream.com/warpstream/byoc/schema-registry/warpstream-byoc-schema-registry) for how it works. WarpStream BYOC Schema Registry supports most APIs listed in Confluent Schema Registry's[ API documentation](https://docs.confluent.io/platform/current/schema-registry/develop/api.html). Here are the list of features it doesn't support: * it doesn't support data contracts, so the `metadata` and `ruleSet` fields in the `schemas` are ignored. * since `metadata`isn't supported, it doesn't support the `/subjects/{subject}/metadata`endpoint * it doesn't support `/mode` endpoints. * it doesn't support `/exporters` endpoints. * it doesn't support subject aliases. * it doesn't support compatibility groups ### Schema Validation WarpStream supports server-side schema validation which not only validate that the record contains a valid schema ID, but that the record actually conforms to the corresponding schema. Currently, WarpStream supports two types of schema registries: * Kafka-compatible Schema Registry * AWS Glue Schema Registry It supports the following data formats: Avro, JSON Schema. For limitations and features not supported, check out the [Enforce Schemas](/warpstream/schema-registry/schema-validation#limitations) page. ### Record Retention Based on Custom Timestamps Kafka implements record retention using the timestamps within the records themselves. If the client sets the timestamp using the `CREATE_TIME` timestamp type, it can send a record with a timestamp far in the future or the past. This will result in the record being deleted based on the timestamp rather than real-time passing. However, WarpStream differs in this aspect. In Warpstream, retention is based solely on the real-time when the record was created. Although you can set a custom timestamp for the record, it will not be used to calculate retention. The retention mechanism in Warpstream strictly adheres to the actual creation time of the record. ### Supported Clients WarpStream should work with any correctly implemented Kafka client. Officially we support [librdkafka](#transactions-atomicity), [franz-go](https://github.com/twmb/franz-go), as well as the standard Java client. Please check [our documentation on tuning your clients for maximum performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) with WarpStream. ### Known Incompatibilities 1. The current implementation any of the \_\_tagged\_\_ fields in the protocol and ignores them entirely. 2. The current implementation does not enforce throttling and ignores all throttling-related fields/settings. 3. All Kafka protocol requests have a maximum timeout of 15s (except for JoinGroup and SyncGroup). # Kafka vs WarpStream Configuration Reference The purpose of this page is to compare configuration options in Kafka and WarpStream 1:1 as applicable. The primary goal of this page is to reassure experienced Kafka operators migrating to WarpStream that most of the configuration and settings they're accustomed to tuning and managing in Apache Kafka have no equivalent in WarpStream and don't need to be managed at all due to WarpStream's architecture.
Kafka ConfigWarpStream
group.max.session.timeout.msNot supported.
auto.create.topics.enableSame. Managed via Kafka API or WarpStream console.
auto.leader.rebalance.enableN/A. WarpStream Agents do not have leaders.
controlled.shutdown.enableWarpStream Agents accept a "gracefulShutdownDuration" flag that defaults to 80 seconds.
controlled.shutdown.max.retriesN/A. Nothing can go wrong during an Agent clean shutdown because they're stateless.
default.replication.factorN/A. Doesn't apply to WarpStream because the Agents are stateless and data is stored durably in object storage at all times. Response is hard-coded to 3 to signify high durability.
delete.topic.enableSame, set as a cluster level configuration via the Kafka API.
kafka.http.metrics.hostNot supported.
kafka.http.metrics.portAgent flag "httpPort" to expose Prometheus metrics.
kafka.log4j.dirNot supported.
kerberos.auth.enableN/A
leader.imbalance.check.interval.secondsN/A. WarpStream Agents do not have leaders.
leader.imbalance.per.broker.percentageN/A. WarpStream Agents do not have leaders.
log.cleaner.dedupe.buffer.sizeNot supported. WarpStream Agents have an equivalent to this setting internally, but its not configurable by default.
log.cleaner.delete.retention.msTopic-level config not supported. Broker-level config, "delete.retention.ms", supported.
log.cleaner.enableN/A. Compacted topics are always compacted.
log.cleaner.min.cleanable.ratioN/A. WarpStream uses different heuristics to determine when to compact the topic. This is handled automatically.
log.cleaner.threadsN/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.
log.retention.bytesNot supported. Response hard-coded to -1 to signify "no limit".
log.retention.check.interval.msN/A. Retention is enforced using a different mechanism called the "dead scanner" in WarpStream, and that process is managed automatically.
log.retention.hoursSame as Kafka.
Logcleaner ratioN/A. WarpStream uses different heuristics to determine when to compact the topic. This is handled automatically.
log.roll.hoursN/A. WarpStream uses object storage and doesn't have physical log files that need to be rolled.
log.segment.bytesN/A. Object storage file size is an implementation detail of WarpStream compaction, and WarpStream files contain data from many different topic-partitions.
message.max.bytesAgent environment variable WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES
min.insync.replicasN/A. Response hard-coded to 1 as object storage handles replication / durability.
num.io.threadsN/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.
num.partitionsSame as Kafka.
num.replica.fetchersN/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.
offsets.topic.num.partitionsN/A. WarpStream virtual clusters do not publish offset topics. Offsets are stored in WarpStream control plane / metadata store.
offsets.topic.replication.factorN/A. WarpStream virtual clusters do not publish offset topics. Offsets are stored in WarpStream control plane / metadata store.
portAgent environment variable "WARPSTREAM_KAFKA_PORT".
replica.fetch.max.bytesN/A. WarpStream Agents are stateless and therefore have no replicas.
replica.lag.max.messagesN/A. WarpStream Agents are stateless and therefore have no replicas.
replica.lag.time.max.msN/A. WarpStream Agents are stateless and therefore have no replicas.
unclean.leader.election.enableN/A. WarpStream has no leaders, and therefore unclean leader elections are impossible.
zookeeper.session.timeout.msN/A. WarpStream does not depend on Zookeeper.
zookeeper.connection.timeout.msN/A. WarpStream does not depend on Zookeeper.
security.inter.broker.protocolNot supported.
ssl.client.authAgent environment variable "WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION"
ssl.truststore.locationAgent environment variable "WARPSTREAM_TLS_CLIENT_CA_CERT_FILE"
ssl.truststore.passwordNot supported.
ssl.keystore.locationAgent environment variables "WARPSTREAM_TLS_SERVER_CERT_FILE", "WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE"
ssl.keystore.passwordN/A
ssl.key.passwordN/A
ssl.protocolAgent environment variable "WARPSTREAM_TLS_ENABLED".
ssl.enabled.protocolsN/A
ssl.keystore.typeN/A
ssl.truststore.typeN/A
broker.id.generation.enableN/A. Broker IDs in WarpStream are virtual and generated automatically.
sasl.kerberos.service.nameN/A
num.network.threadsN/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.
num.recovery.threads.per.data.dirN/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.

Also WarpStream Agents are stateless and have no concept of a recovery.
socket.send.buffer.bytesN/A
socket.receive.buffer.bytesN/A
socket.request.max.bytesN/A
replica.fetch.wait.max.msN/A. WarpStream Agents are stateless and therefore have no replicas.
replica.socket.timeout.msN/A. WarpStream Agents are stateless and therefore have no replicas.
replica.socket.receive.buffer.bytesN/A. WarpStream Agents are stateless and therefore have no replicas.
replica.high.watermark.checkpoint.interval.msN/A. WarpStream Agents are stateless and therefore have no replicas.
controller.socket.timeout.msN/A
controller.message.queue.sizeN/A
zookeeper.sync.time.msN/A. WarpStream does not depend on Zookeeper.
queued.max.requestsN/A
fetch.purgatory.purge.interval.requestsN/A
producer.purgatory.purge.interval.requestsN/A
authorizer.class.nameN/A. There is only one ACL Authorized implementation in WarpStream.
allow.everyone.if.no.acl.foundSame as Kafka.
broker.idN/A. WarpStream Broker IDs are generated automatically.
log.dirsN/A. WarpStream Agents are stateless and store data exclusively in object storage.
zookeeper.connectN/A. WarpStream does not depend on Zookeeper.
listenersAgent environment variables: "WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE" and "WARPSTREAM_KAFKA_PORT"
# Compacted topics Information about compacted topics in WarpStream. As described in [our blogpost](https://www.warpstream.com/blog/kafka-kv-store), the way WarpStream implements compacted topics is very different from how Apache Kafka does it, although at its core, the algorithm is very similar. One key difference is that WarpStream compacts the data for many different compacted topics in a virtual cluster at the same time. This is due to the fact that [WarpStream's storage engine](https://docs.warpstream.com/warpstream/overview/architecture/write-path) stores data for many different topic-partitions in the same files to minimize object storage costs. The rest of this document will focus on explaining the heuristic used by WarpStream on when to schedule a compact for compacted topics to provide readers with intuition on how compacted topics will perform in WarpStream. {% hint style="info" %} Note that in this document we will use two words that are very similar but mean two very different things: 1. We will use compaction when we want to talk about the system, specific to WarpStream, that takes small files and makes bigger files to replace them. The compaction system is a critical piece of how a WarpStream Virtual Cluster operates. 2. We will use compacted topics when we want to talk about a topic where the user has configured `cleanup.policy = compact` or `cleanup.policy = compact,delete`. Much like in Apache Kafka, this is a topic where WarpStream proactively removes records that share the same key as more recent records to reclaim space and reduce the amount of work performed by consumers. {% endhint %} ### L0 - L1 compaction When processing Produce requests from Kafka producers the agents create ingestion files that we'll henceforth refer to as “L0 files”. When the compaction system detects that there are a large number (currently 32) of L0 files it compacts them into “L1 files” that are bigger. During this first compaction, we do not deduplicate keys, we just copy the data from L0 files to L1 files, without applying any special logic for compacted topics. ## Higher level compactions Any compactions beyond L0->L1 will deduplicate records. The highest compaction level a cluster may get to is L4. The compaction scheduler kicks in whenever it detects a large number of files in that level. In general, the more continuous throughput a cluster has, the more compactions will run. For instance, the compaction scheduler will schedule a L1 to l2 compaction when there are 32 L1 files. A single agent receiving low but continuous throughput writes 4 L0 files per second (one file every 250ms), which means that a virtual cluster with only this agent will take about 256 seconds (32 \* 32 / 4) to create enough files to trigger a L1 to L2 compaction, which will trigger record deduplication. If your virtual cluster has more traffic, or more agents, it will be faster to accumulate enough files to trigger record deduplication. However, it’s never instantaneous. For small clusters, file needs to be compacted twice before the record reaches the highest level, first from L0 to L1, then second from L1 to L2 for this to happen. For big clusters, a file needs to be compacted even more times before reaching L4. ## Exact guarantees There are two main limitations to WarpStream's compaction system: * If there are more than 3 million distinct keys in a topic-partition within a file, not all records will be de-duplicated. This is covered in more detail in the next section. * If you have more than 128GiB of uncompressed data in a single partition, there could be duplicate records. That is because WarpStream never compacts files that contain more than 128GiB of uncompressed records. Furthermore, unless `max.compaction.lag.ms` is set, WarpStream doesn't guarantee exactly when compactions will run. The heuristic that WarpStream uses to choose when to compact files together after the first L0 -> L1 and L1 -> L2 compaction is designed to minimize write amplification as well as the number of files. As data is added to the Virtual Cluster, and compacted into L2 (or L4 for big clusters), this will create a continuous sequence of files. WarpStream scans these files, and when it finds 10 or more files that have a similar size, it will compact them together. WarpStream does not choose to compact files together when their size is very different because rewriting a 30GB file together with a 2MB file is very expensive and results in very little gain. In addition, roughly once a day, WarpStream will run a compaction even on files which have very different sizes, to minimize the number of duplicate keys in the data set. ## Distinct key limitation In compacted topics, when you have two records with the same key, the first one can be deleted to reclaim space, because the second record with the same key represents a newer version of the same “resource”. WarpStream does not guarantee that all of the records which could be deleted are indeed deleted. The reason for this is explained in detail in [the blogpost](https://www.warpstream.com/blog/kafka-kv-store). The way WarpStream deduplicates records in compacted topics is as follows: For every topic-partition participating in a compaction job, WarpStream allocates a 128MiB buffer where it will store the hash of keys of the records it encounters. When the compaction engine encounters a new record, it hashes its key into a 32-byte number (using SHA512/256) and checks whether it has already seen this key or not. If the key has already been seen the old record will be deleted. Each key takes 41 bytes in the buffer (the 32 byte hash + 2 4-byte integers + one byte) which means that there is room for 3,273, 603 keys (128Mib / 41) in the buffer. If there are enough distinct keys to fill the buffer (e.g. when, during a single compaction, WarpStream has read more than 3 million distinct keys for a single topic-partition), it will clear the buffer and start afresh. This means that you can expect to have one record per key in each file if you have less than 3 million distinct keys in a topic-partition. If there are more than 3 million distinct keys, you can expect to have one record per key per file post-compaction for each set of 3 million distinct keys encountered within a single compaction job. This is similar to Apache Kafka in terms of configuring the buffer size for the log cleaning modules. {% hint style="info" %} WarpStream uses a variation on the streaming k-way merge algorithm to perform compactions for compacted topics. This means that the entire 3 million keys buffer is available to every single topic-partition participating in a compaction, but the buffer is only allocated once and re-used between topic-partitions which keeps memory usage low. {% endhint %} ## max.compaction.lag.ms {% hint style="warning" %} Note that the lower the `max.compaction.lag.ms`, the higher the write amplification and therefore more costly to keep records compacted. In general, if a topic's `max.compaction.lag.ms` is set to N hours, all files that contain data for that topic will be rewritten at least once every N hours. {% endhint %} If you want stronger guarantees for when files are compacted, you could set `max.compaction.lag.ms`. The topic's `max.compaction.lag.ms` is the target maximum delay between when a record is produced and when that record gets deduplicated with previous records. We call a record "dirty" if it exceeds that maximum delay, or more precisely if the record's timestamp is older than `now - max.compaction.lag.ms`. Once a record becomes "dirty", the compaction system will compact the dirty record within 30 minutes. Note that to compact a record, the compaction system would need to rewrite the entire file containing that record. Therefore, the lower the `max.compaction.lag.ms`, the higher the write amplification and the more costly it is to keep records compacted. For example, if you set the `max.compaction.lag.ms` for a topic to 2 hours, then a record will be compacted at most 2.5 hours after its creation (it can be before too). The reason for the additional delay is because compacting files isn't instantaneous. Compacting larger files may take up to 10 minutes. In addition, WarpStream's compaction system will not compact as soon as it detects when a record is eligible for compaction. Instead, it buffers for a short period of time (roughly 15 \~ 30 minutes) before actually compacting the files containing the dirty records. This reduces write amplification drastically and prevents the system from continuously recompacting files every time there is a new dirty record. The smallest `max.compaction.lag.ms` supported in WarpStream is 2 hours. If you set the max lag to a value below 2 hours, the system will silently override it to 2 hours. ## min.compaction.lag.ms If `min.compaction.lag.ms` is set, WarpStream guarantees that a record will *not* be deduplicated with previous records (records with lower offsets) as long as the record's timestamp is more recent than `now - min.compaction.lag.ms`. This is useful for scenarios where you want to ensure that consumers are guaranteed to see all keys before they're deduplicated as long as they don't fall more than [`min.compaction.lag.ms`](http://min.compaction.lag.ms/) behind. ## Supported Configuration WarpStream supports the following topic-level configurations that are relevant to Kafka compacted topics: * `retention.ms` * `delete.retention.ms` * `min.compaction.lag.ms` * `max.compaction.lag.ms` * `cleanup.policy` ### Additional Caveats Unlike in Apache Kafka, a topic created with `cleanup.policy = delete` cannot be converted to a topic with `cleanup.policy = compact` and vice versa. The reason for this is that data for compacted and non-compacted topics are maintained in completely separate files in WarpStream. # Broker Configuration Reference This page lists the supported broker configuration for WarpStream. Configuration not listed here is either ignored (for non-WarpStream keys) or rejected (for unknown `warpstream.*` keys). Configuration items that start with `warpstream.` are WarpStream-specific. Some native Kafka tooling may not handle these configs as expected. Broker config in WarpStream is cluster-scoped: `DescribeConfigs` using an empty broker name or a concrete broker ID reflects the same effective cluster configuration. #### auto.create.topics.enable Whether topics can be auto-created when clients reference unknown topic names. | Type: | string (boolean) | | ------------- | ---------------- | | Default: | true | | Valid Values: | \[true, false] | #### delete.topic.enable Whether topic deletion is allowed. | Type: | string (boolean) | | ------------- | ---------------- | | Default: | true | | Valid Values: | \[true, false] | #### group.consumer.heartbeat.interval.ms Heartbeat interval for modern (KIP-848) consumer groups. | Type: | int | | ------------- | ------------------------------------------- | | Default: | 5000 | | Valid Values: | \[1,2147483647] with cross-validation rules | #### group.consumer.max.heartbeat.interval.ms Upper bound for modern-group heartbeat interval. | Type: | int | | ------------- | ------------------------------------------- | | Default: | 15000 | | Valid Values: | \[1,2147483647] with cross-validation rules | #### group.consumer.max.session.timeout.ms Upper bound for modern-group session timeout. | Type: | int | | ------------- | ------------------------------------------- | | Default: | 60000 | | Valid Values: | \[1,2147483647] with cross-validation rules | #### group.consumer.max.size Maximum allowed size for modern (KIP-848) consumer groups. | Type: | int | | ------------- | --------------- | | Default: | 32000 | | Valid Values: | \[1,2147483647] | #### group.consumer.min.heartbeat.interval.ms Lower bound for modern-group heartbeat interval. | Type: | int | | ------------- | ------------------------------------------- | | Default: | 5000 | | Valid Values: | \[1,2147483647] with cross-validation rules | #### group.consumer.min.session.timeout.ms Lower bound for modern-group session timeout. | Type: | int | | ------------- | ------------------------------------------- | | Default: | 45000 | | Valid Values: | \[1,2147483647] with cross-validation rules | #### group.consumer.session.timeout.ms Session timeout for modern (KIP-848) consumer groups. | Type: | int | | ------------- | ------------------------------------------- | | Default: | 45000 | | Valid Values: | \[1,2147483647] with cross-validation rules | #### log.retention.hours Same setting as `log.retention.ms`, but expressed in hours. | Type: | long | | ------------- | ---------- | | Default: | 24 (1 day) | | Valid Values: | \[-1,...] | #### log.retention.minutes Same setting as `log.retention.ms`, but expressed in minutes. | Type: | long | | ------------- | ------------ | | Default: | 1440 (1 day) | | Valid Values: | \[-1,...] | #### log.retention.ms Default retention for topics, in milliseconds. If set to a negative value, WarpStream treats retention as effectively infinite. If set to a positive value lower than 1 minute, WarpStream raises it to 1 minute. | Type: | long | | ------------- | ----------------------- | | Default: | 86400000 (1 day) | | Valid Values: | \[-1,...,3153600000000] | #### message.max.bytes Cluster-level record size cap. If this value is not explicitly set, WarpStream may omit it from `DescribeConfigs`, because the effective default depends on agent-side limits. | Type: | int | | ------------- | ----------------------------------------------- | | Default: | agent-dependent (unset at cluster config layer) | | Valid Values: | \[0,...] | #### num.partitions Default partition count used when topics are auto-created. | Type: | int | | ------------- | ---------------- | | Default: | 1 | | Valid Values: | \[0, 4294967295] | #### offsets.retention.minutes Retention for committed consumer group offsets. | Type: | long | | ------------- | -------------- | | Default: | 10080 (7 days) | | Valid Values: | \[1,...] | #### warpstream.default.partitions\_auto\_scaler.enabled Cluster-level default for topic config `warpstream.partitions_auto_scaler.enabled`. This default is applied to newly created topics (including auto-created topics). If a topic explicitly sets `warpstream.partitions_auto_scaler.enabled`, the topic value takes precedence. | Type: | string (boolean) | | ------------- | ---------------- | | Default: | false | | Valid Values: | \[true, false] | #### warpstream.default.partitions\_auto\_scaler.max\_partition\_count Cluster-level default for topic config `warpstream.partitions_auto_scaler.max_partition_count`. This default is applied to newly created topics. A value of 0 means unlimited. If a topic explicitly sets `warpstream.partitions_auto_scaler.max_partition_count`, the topic value takes precedence. | Type: | long | | ------------- | ------------- | | Default: | 0 (unlimited) | | Valid Values: | \[0,...] | #### warpstream.default.partitions\_auto\_scaler.per\_partition\_throughput\_uncompressed\_bytes\_per\_second Cluster-level default for topic config `warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second`. This default is applied to newly created topics. If a topic explicitly sets `warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second`, the topic value takes precedence. | Type: | long | | ------------- | -------- | | Default: | 2500000 | | Valid Values: | \[0,...] | #### warpstream.default.topic.type Default topic type for newly created topics. | Type: | string | | ------------- | --------------------- | | Default: | classic | | Valid Values: | \[classic, lightning] | #### warpstream.soft.delete.topic.enable Whether soft-delete behavior is enabled for inactive topics. | Type: | string (boolean) | | ------------- | ---------------- | | Default: | true | | Valid Values: | \[true, false] | #### warpstream.soft.delete.topic.ttl.hours Inactive topic TTL in hours when soft-delete is enabled. If set to a negative value, WarpStream treats TTL as effectively infinite. | Type: | long | | ------------- | ---------- | | Default: | 24 (1 day) | | Valid Values: | \[-1,...] | #### group.consumer.\* cross-validation rules WarpStream enforces the following invariants for KIP-848 timing configs: * `group.consumer.max.heartbeat.interval.ms >= group.consumer.min.heartbeat.interval.ms` * `group.consumer.heartbeat.interval.ms` must be within `[min.heartbeat, max.heartbeat]` * `group.consumer.max.session.timeout.ms >= group.consumer.min.session.timeout.ms` * `group.consumer.session.timeout.ms` must be within `[min.session, max.session]` * `group.consumer.heartbeat.interval.ms < group.consumer.session.timeout.ms` #### Alter behavior `AlterConfigs` uses classic replace semantics for known broker configs: * keys present in request are set * known keys omitted from request are reset/unset `IncrementalAlterConfigs` supports: * `SET` * `DELETE` `APPEND` and `SUBTRACT` are not supported for broker configs. #### Unknown config behavior * Unknown keys containing `warpstream` are rejected with `INVALID_CONFIG`. * Unknown non-WarpStream keys are ignored for compatibility. #### Kafka compatibility values returned by DescribeConfigs WarpStream also reports a set of read-only Kafka-style broker config values for compatibility: | Config key | Value | | ------------------------------ | ---------- | | compression.type | lz4 | | log.flush.interval.messages | 1 | | log.retention.bytes | -1 | | unclean.leader.election.enable | false | | broker.id.generation.enable | true | | default.replication.factor | 3 | | reserved.broker.max.id | 2147483647 | #### CLI examples Using WarpStream CLI commands: ```bash warpstream cli describe-broker-configs \ -bootstrap-host localhost \ -bootstrap-port 9092 ``` ```bash warpstream cli alter-broker-config \ -bootstrap-host localhost \ -bootstrap-port 9092 \ -config-name warpstream.default.topic.type \ -config-value lightning ```
# Topic Configuration Reference This page lists the supported configuration for WarpStream topics. Configuration not listed here will be silently ignored by the WarpStream Kafka API. Configuration items that start with `warpstream.` are WarpStream specific configurations, native Kafka tooling like the `kafka-configs` CLI may not be able to set these configurations. It is recommended to set these configurations using our REST API for topics, WarpStream terraform provider, or a non-Java library like franz-go to manage these configurations. ### message.timestamp.type Define whether the timestamp in the message is message create time or log append time. The value should be either *CreateTime* or *LogAppendTime* | Type: | string | | ------------- | ---------------------------- | | Default: | CreateTime | | Valid Values: | \[CreateTime, LogAppendTime] | ### cleanup.policy This config designates the retention policy to use on topics. The “delete” policy (which is the default) will discard old message when their retention time or size limit has been reached. The “compact” policy will enable topic compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. “delete,compact”). In this case, old messages will be discarded per the retention time and size configuration, while retained messages will be compacted. {% hint style="info" %} In WarpStream cleanup.policy can be changed after topic creation but with restrictions. Specifically, a non-compacted topic cannot be made compacted, and a compacted topic cannot be made non-compacted. For example, going from `compact` to `compact,delete` or `delete,compact` is allowed, so is going from `compact,delete` to `compact`. However, going from `compact,delete` to `delete` is not allowed, and neither is going from `delete` to `compact,delete`. {% endhint %} | Type: | list | | ------------- | ------------------ | | Default: | delete | | Valid Values: | \[compact, delete] | ### retention.ms This configuration controls the maximum time we will retain messages in a topic partition. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied. The minimum retention is 1 minute. If you set it to a positive value that's lower than 1 minute, WarpStream will set it to 1 minute. | Type: | long | | ------------- | -------------------------- | | Default: | 86400000 (1 day) | | Valid Values: | \[-1,60000…,3153600000000] | ### min.compaction.lag.ms The minimum time a message will remain uncompacted in the topic partition. Only applicable for topics that are being compact. | Type: | long | | ------------- | -------------------- | | Default: | 0 | | Valid Values: | \[0,…,3153600000000] | ### delete.retention.ms The amount of time to retain delete tombstone markers for compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan). | Type: | long | | ------------- | ----------------------------- | | Default: | 86400000 (1 day) | | Valid Values: | \[-1,3600000,…,3153600000000] | ### warpstream.topic.type The type of this topic. There are currently two types of warpstream topics: classic topics and lightning topics (see | Type: | long | | ------------- | --------------------- | | Default: | classic | | Valid Values: | \[classic, lightning] | ### warpstream.compression.type.fetch The compression algorithm that is used to return compressed batches to consumers for this topic. | Type: | string | | ------------- | ------------------- | | Default: | agent (lz4) | | Valid Values: | \[agent, lz4, zstd] | ### warpstream.schema.registry.type The schema registry type to use when schema validation is enabled. | Type: | string | | ------------- | ---------------------- | | Default: | STANDARD | | Valid Values: | \[STANDARD, AWS\_GLUE] | ### warpstream.schema.validation.warning.only Should schema validations only return warnings and not block producing of messages to the topic. | Type: | string | | ------------- | -------------- | | Default: | true | | Valid Values: | \[true, false] | ### warpstream.key.schema.validation Should message key schemas be validated when messages are being produced to the topic. | Type: | string | | ------------- | -------------- | | Default: | false | | Valid Values: | \[true, false] | ### warpstream.key.subject.name.strategy Specifies how to construct the subject name for message keys. This determines the subject name allowed during schema validation. | Type: | string | | ------------- | ----------------------------------------------------------------- | | Default: | TopicNameStrategy | | Valid Values: | \[TopicNameStrategy, RecordNameStrategy, TopicRecordNameStrategy] | ### warpstream.value.subject.name.strategy Specifies how to construct the subject name for message values. This determines the subject name allowed during schema validation. | Type: | string | | ------------- | ----------------------------------------------------------------- | | Default: | TopicNameStrategy | | Valid Values: | \[TopicNameStrategy, RecordNameStrategy, TopicRecordNameStrategy] | ### warpstream.value.schema.validation Should message value schemas be validated when messages are being produced to the topic. | Type: | string | | ------------- | -------------- | | Default: | false | | Valid Values: | \[true, false] | ### warpstream.partitions\_auto\_scaler.enabled Should the [partition autoscaler](/warpstream/kafka/reference/partitions-auto-scaler-beta) be enabled for the topic. | Type: | string | | ------------- | -------------- | | Default: | false | | Valid Values: | \[true, false] | ### warpstream.partitions\_auto\_scaler.per\_partition\_throughput\_uncompressed\_bytes\_per\_second The maximum thoughput per partition a topic should have before more partitions are added through autoscaling. | Type: | long | | ------------- | ------ | | Default: | 0 | | Valid Values: | \[0,…] | ### warpstream.partitions\_auto\_scaler.max\_partition\_count The maximum number of partitions that the topic can autoscale to. | Type: | long | | ------------- | -------------------- | | Default: | 0 | | Valid Values: | \[0,…,3153600000000] | ### warpstream.deletion.protection.enabled Should the topic have delete protection enabled. If enabled the topic cannot be deleted until disabled. | Type: | string | | ------------- | -------------- | | Default: | false | | Valid Values: | \[true, false] | # HTTP Endpoints This pages describes the HTTP endpoints available in the WarpStream Kafka product. ## Authentication and Authorization These rules apply to all HTTP endpoints on this page. Each API section below only documents what is specific to it, such as the required ACL permission or a different credential header. ### Authentication All endpoints support optional HTTP Basic Auth carrying your WarpStream credentials (the same ones used for Kafka SASL/PLAIN). When the Agent is configured to require SASL authentication, credentials are mandatory; otherwise they can be omitted. With curl, use the `-u` flag: ```bash curl -u 'ccun_YOUR_USERNAME:ccp_YOUR_PASSWORD' ... ``` This is equivalent to setting the `Authorization` header manually with the base64-encoded `username:password` pair: ```bash curl -H 'Authorization: Basic Y2N1bl9ZT1VSX1VTRVJOQU1FOmNjcF9ZT1VSX1BBU1NXT1JE' ... ``` The Datadog log intake endpoints are the one exception: they accept the same credentials via the `DD-API-KEY` header instead of Basic Auth. See [Datadog HTTP Log Intake](#datadog-http-log-intake) for details. mTLS authentication is not supported for any of the HTTP endpoints. ### Authorization (ACLs) All endpoints enforce the same Kafka ACLs as the native Kafka protocol: * The ACL principal is derived from the authenticated username as `User:`. For example, if you authenticate as `ccun_abc123`, the ACL principal will be `User:ccun_abc123`. * If authentication is optional and no credentials are provided, the request is evaluated as the anonymous principal. * The required permission depends on the API: the fetch endpoints require `READ` on each topic being fetched, while the produce and Datadog log intake endpoints require `WRITE` on each destination topic. ## HTTP Fetch APIs {% hint style="info" %} Requires Agent version v755+. {% endhint %} The WarpStream Agent exposes HTTP/JSON endpoints for fetching records from Kafka topics without a native Kafka client. All endpoints are served on the Agent's HTTP port (8080 by default). ### Authentication and Authorization See [Authentication and Authorization](#authentication-and-authorization). Specific to the fetch endpoints: * They require `READ` permission on each topic being fetched. * If the authenticated user does not have `READ` access to a topic, the partition will be returned with a `TOPIC_AUTHORIZATION_FAILED` error code (for `/v1/kafka/fetch`) or a `400` error response (for the single-record endpoints). ### Common Headers | Header | Description | Default | | ----------------- | ------------------------------------------------- | ------------------------------------------------------- | | `kafka-client-id` | Identifies the client for logging and diagnostics | `http-fetch-client` / `http-fetch-single-record-client` | ### Endpoints #### `GET` or `POST /v1/kafka/fetch` Full-featured fetch endpoint that mirrors the Kafka Fetch protocol. Supports fetching from multiple topics and partitions in a single request. **Request Body (JSON)** | Field | Type | Required | Description | | ------------------------------------------- | ------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- | | `topics` | array | **yes** | List of topics to fetch from | | `topics[].topic` | string | **yes** | Topic name | | `topics[].partitions` | array | **yes** | List of partitions to fetch from | | `topics[].partitions[].partition` | integer | **yes** | Partition index (≥ 0) | | `topics[].partitions[].fetch_offset` | integer | **yes** | Offset to start fetching from (≥ 0) | | `topics[].partitions[].partition_max_bytes` | integer | **yes** | Max bytes to fetch for this partition (> 0) | | `max_bytes` | integer | no | Max bytes for the entire fetch response (≥ 0). Default: `0` (agent-managed) | | `max_records` | integer | no | Max total records to return across all partitions (≥ 0). `0` means unlimited. Parsing stops early once the budget is reached. | | `isolation_level` | string | no | `"read_uncommitted"` (default) or `"read_committed"` | | `long_poll` | boolean | no | If `true`, the agent waits up to 30s for new data instead of returning immediately. Default: `false` | **Example Request** ```bash curl -X GET \ 'http://localhost:8080/v1/kafka/fetch' \ -H 'Content-Type: application/json' \ -u 'username:password' \ -d '{ "max_bytes": 1048576, "max_records": 10, "topics": [ { "topic": "my-topic", "partitions": [ { "partition": 0, "fetch_offset": 0, "partition_max_bytes": 1048576 } ] } ] }' ``` **Success Response (`200 OK`)** ```json { "throttle_time_ms": 0, "topics": [ { "topic": "my-topic", "partitions": [ { "partition": 0, "error_code": "NONE", "high_watermark": 150, "last_stable_offset": 150, "log_start_offset": 0, "records": [ { "offset": 0, "timestamp": 1707744000000, "key": "dGVzdC1rZXk=", "value": "dGVzdC12YWx1ZQ==", "headers": [ { "key": "header-name", "value": "aGVhZGVyLXZhbHVl" } ] } ] } ] } ] } ``` > **Note:** `key`, `value`, and header `value` fields are base64-encoded byte arrays. A JSON `null` indicates a nil key/value. **Error Response (`400 Bad Request` / `500 Internal Server Error`)** ```json { "code": "INVALID_REQUEST", "message": "invalid request: topics must not be empty" } ``` *** #### `GET /v1/kafka/fetch_single_record` Convenience endpoint for fetching a single record at a specific offset using query parameters. **Query Parameters** | Parameter | Type | Required | Description | | ----------- | ------- | -------- | --------------------------- | | `topic` | string | **yes** | Topic name | | `partition` | integer | **yes** | Partition index (≥ 0) | | `offset` | integer | **yes** | Exact offset to fetch (≥ 0) | **Example Request** ```bash curl -X GET \ 'http://localhost:8080/v1/kafka/fetch_single_record?topic=my-topic&partition=0&offset=42' \ -u 'username:password' ``` **Success Response (`200 OK`)** Returns the record directly as a JSON object (not wrapped in the topic/partition structure): ```json { "offset": 42, "timestamp": 1707744000000, "key": "dGVzdC1rZXk=", "value": "dGVzdC12YWx1ZQ==", "headers": [] } ``` **Not Found Response (`404 Not Found`)** Returned when no record exists at the requested offset (e.g., the offset is beyond the high watermark or below the low watermark): ```json { "code": "OFFSET_OUT_OF_RANGE", "message": "The requested offset is not within the range of offsets maintained by the server.: no record found at topic=my-topic partition=0 offset=999999" } ``` **Error Response (`400 Bad Request`)** ```json { "code": "INVALID_REQUEST", "message": "missing required query parameter: topic" } ``` *** #### `GET /v1/kafka/topics/{topic}/partitions/{partition}/records/{offset}` REST-style convenience endpoint for fetching a single record at a specific offset using path parameters. Functionally identical to `/v1/kafka/fetch_single_record`. **Path Parameters** | Parameter | Type | Required | Description | | ----------- | ------- | -------- | --------------------------- | | `topic` | string | **yes** | Topic name | | `partition` | integer | **yes** | Partition index (≥ 0) | | `offset` | integer | **yes** | Exact offset to fetch (≥ 0) | **Example Request** ```bash curl -X GET \ 'http://localhost:8080/v1/kafka/topics/my-topic/partitions/0/records/42' \ -u 'username:password' ``` **Responses** Identical to `/v1/kafka/fetch_single_record`: * **`200 OK`** — The record as a JSON object. * **`404 Not Found`** — No record at the requested offset (`OFFSET_OUT_OF_RANGE`). * **`400 Bad Request`** — Invalid path parameters. *** ### Error Codes The `code` field in error responses maps to Kafka protocol error codes: | HTTP Status | Code | Meaning | | ----------- | ---------------------------- | ------------------------------------------------------------ | | `400` | `INVALID_REQUEST` | Malformed request (bad JSON, missing fields, invalid values) | | `401` | `SASL_AUTHENTICATION_FAILED` | Missing or invalid credentials | | `403` | `TOPIC_AUTHORIZATION_FAILED` | ACL denied read access to the topic | | `404` | `OFFSET_OUT_OF_RANGE` | No record exists at the requested offset | | `500` | `KAFKA_STORAGE_ERROR` | Internal error fetching data | ## HTTP Produce APIs {% hint style="info" %} Requires Agent version v825+. {% endhint %} The WarpStream Agent exposes HTTP/JSON endpoints for producing records to Kafka topics without a native Kafka client. All endpoints are served on the Agent's HTTP port (`8080` by default). Two API styles are available: * **Confluent REST Proxy v2-compatible endpoints** (`POST /topics/{topic}` and `POST /topics/{topic}/partitions/{partition}`) — drop-in compatible with existing Confluent REST Proxy v2 tooling. Single topic per request, automatic partitioning. * **WarpStream-native endpoint** (`POST /v1/kafka/produce`) — mirrors the Kafka Produce protocol. Supports multiple topics and partitions per request with explicit partition assignment, per-record timestamps, and headers. ### Authentication and Authorization See [Authentication and Authorization](#authentication-and-authorization). Specific to the produce endpoints: * They require `WRITE` permission on each destination topic. * If the authenticated principal does not have write access, the affected records fail with an authorization error: `403 TOPIC_AUTHORIZATION_FAILED` for the native endpoint, or error code `40301` for the v2-compatible endpoints. ### Common Headers | Header | Description | Default | | ------------------ | ------------------------------------------------- | ------------------------------------------------ | | `kafka-client-id` | Identifies the client for logging and diagnostics | `http-produce-client` / `http-produce-v2-client` | | `Content-Encoding` | Optional request compression (`gzip` or `zstd`) | none | The request body limit is `64 MiB` after decompression. ### Confluent REST Proxy v2-Compatible Endpoints These endpoints implement the [Confluent REST Proxy v2 produce API](https://docs.confluent.io/platform/current/kafka-rest/api.html) for the `binary` and `json` embedded formats. Schema-based formats (`avro`, `jsonschema`, `protobuf`) are not supported and return `415 Unsupported Media Type`. #### Content Types The `Content-Type` request header selects the embedded format: | Content-Type | Embedded format | | -------------------------------------- | -------------------------------------------------------------------------------- | | `application/vnd.kafka.binary.v2+json` | `binary`: `key` and `value` are base64-encoded strings | | `application/vnd.kafka.json.v2+json` | `json`: `key` and `value` are arbitrary JSON, stored as their JSON serialization | | `application/vnd.kafka.v2+json` | Treated as `binary` | | `application/json` | Treated as `binary` | | `application/octet-stream` | Treated as `binary` | Responses always use `Content-Type: application/vnd.kafka.v2+json`. #### `POST /topics/{topic}` Produce records to a topic, optionally specifying keys or partitions per record. **Request Body (JSON)** | Field | Type | Required | Description | | --------------------- | ------- | -------- | -------------------------------------------------------------------------------- | | `records` | array | **yes** | List of records to produce (must not be empty) | | `records[].key` | object | no | Record key, formatted according to the embedded format, or `null` to omit | | `records[].value` | object | no | Record value, formatted according to the embedded format | | `records[].partition` | integer | no | Partition to store the record in | | `records[].headers` | array | no | Record headers: `{"key": string, "value": base64 string}` (WarpStream extension) | **Partitioning** Each record's partition is chosen with the same semantics as Kafka's default partitioner: 1. If `records[].partition` is set, it is used directly. 2. Otherwise, if the record has a key, the partition is `murmur2(serialized key) % partition count` — identical to the Kafka Java client and Confluent REST Proxy, so records produced over HTTP land on the same partitions as records produced with a Kafka client. 3. Otherwise (keyless records), all keyless records in the request share one sticky partition, which rotates round-robin across requests. **Example Requests** Binary format (base64 key and value): {% code overflow="wrap" %} ```bash curl -X POST 'http://localhost:8080/topics/my-topic' \ -H 'Content-Type: application/vnd.kafka.binary.v2+json' \ -u 'username:password' \ -d '{"records": [{"key": "a2V5", "value": "dmFsdWU="}]}' ``` {% endcode %} JSON format (arbitrary JSON key and value): {% code overflow="wrap" %} ```bash curl -X POST 'http://localhost:8080/topics/my-topic' \ -H 'Content-Type: application/vnd.kafka.json.v2+json' \ -u 'username:password' \ -d '{"records": [{"value": {"name": "testUser"}}]}' ``` {% endcode %} **Success Response (`200 OK`)** `offsets` contains one entry per record, in the same order as the request: ```json { "offsets": [ { "partition": 0, "offset": 42, "error_code": null, "error": null } ], "key_schema_id": null, "value_schema_id": null } ``` If an individual record fails, its entry has `null` `partition` and `offset` and a non-null `error_code` and `error`: ```json { "offsets": [ { "partition": null, "offset": null, "error_code": 50003, "error": "..." } ], "key_schema_id": null, "value_schema_id": null } ``` **Request-Level Error Response** ```json { "error_code": 40401, "message": "topic: my-topic does not exist" } ``` *** #### `POST /topics/{topic}/partitions/{partition}` Produce records directly to a single partition. Identical to `POST /topics/{topic}` except that all records are written to the partition from the path, and per-record `partition` fields are ignored. **Path Parameters** | Parameter | Type | Required | Description | | ----------- | ------- | -------- | --------------------- | | `topic` | string | **yes** | Topic name | | `partition` | integer | **yes** | Partition index (≥ 0) | **Example Request** {% code overflow="wrap" %} ```bash curl -X POST 'http://localhost:8080/topics/my-topic/partitions/0' \ -H 'Content-Type: application/vnd.kafka.binary.v2+json' \ -u 'username:password' \ -d '{"records": [{"value": "dmFsdWU="}]}' ``` {% endcode %} If the partition does not exist, the request fails with `404` and error code `40402`. *** #### v2 Error Codes Error codes follow the Confluent REST Proxy v2 numeric scheme: | HTTP Status | `error_code` | Meaning | | ----------- | ------------ | ---------------------------------------------------------------------- | | `400` | `400` | Serialization failure (e.g. invalid base64 in `binary` format) | | `401` | `40101` | Missing or invalid credentials | | `403` | `40301` | ACL denied write access to the topic | | `404` | `40401` | Topic does not exist | | `404` | `40402` | Partition does not exist | | `413` | `413` | Request body exceeds `64 MiB` after decompression, or record too large | | `415` | `415` | Unsupported embedded format (e.g. `avro`, `protobuf`) | | `422` | `422` | Request validation failure (e.g. empty `records` list) | | `500` | `50002` | Non-retriable Kafka error | | `500` | `50003` | Retriable Kafka error; the produce might succeed if retried | Per-record failures inside a `200 OK` response use `50002`, `50003`, or `40301` in the `offsets[].error_code` field. If any record fails with an authorization error, the overall HTTP status is `403`. ### WarpStream-Native Endpoint #### `POST /v1/kafka/produce` Full-featured produce endpoint that mirrors the Kafka Produce protocol. Supports producing to multiple topics and partitions in a single request with explicit partition assignment. **Request Body (JSON)** | Field | Type | Required | Description | | ------------------------------------------- | ------- | -------- | ------------------------------------------------------------------------------------------------------------- | | `topics` | array | **yes** | List of topics to produce to | | `topics[].topic` | string | **yes** | Topic name | | `topics[].partitions` | array | **yes** | List of partitions to produce to | | `topics[].partitions[].partition` | integer | **yes** | Partition index (≥ 0) | | `topics[].partitions[].records` | array | **yes** | List of records for this partition | | `topics[].partitions[].records[].key` | string | no | Base64-encoded record key, or `null` for no key | | `topics[].partitions[].records[].value` | string | no | Base64-encoded record value, or `null` for a tombstone | | `topics[].partitions[].records[].headers` | array | no | Record headers: `{"key": string, "value": base64 string}` | | `topics[].partitions[].records[].timestamp` | integer | no | Record timestamp in milliseconds since the Unix epoch. If `0` or omitted, the Agent assigns the current time. | | `timeout_ms` | integer | no | Produce timeout in milliseconds. Default: `10000` | **Example Request** ```bash curl -X POST \ 'http://localhost:8080/v1/kafka/produce' \ -H 'Content-Type: application/json' \ -u 'username:password' \ -d '{ "topics": [ { "topic": "my-topic", "partitions": [ { "partition": 0, "records": [ { "key": "dGVzdC1rZXk=", "value": "dGVzdC12YWx1ZQ==", "headers": [ { "key": "header-name", "value": "aGVhZGVyLXZhbHVl" } ] } ] } ] } ] }' ``` **Success Response (`200 OK`)** The response mirrors the Kafka Produce response. Per-partition failures are reported in the body with a `200` status, so check each partition's `error_code`: ```json { "throttle_time_ms": 0, "topics": [ { "topic": "my-topic", "partitions": [ { "partition": 0, "error_code": "NONE", "base_offset": 42, "log_append_time": -1 } ] } ] } ``` `base_offset` is the offset assigned to the first record in the partition's batch; subsequent records occupy consecutive offsets. **Error Response (`400 Bad Request` / other statuses)** ```json { "code": "INVALID_REQUEST", "message": "invalid request: topics must not be empty" } ``` *** #### Native Endpoint Error Codes The `code` field in error responses maps to Kafka protocol error codes, in the same style as the HTTP fetch endpoints: | HTTP Status | Code | Meaning | | ----------- | ---------------------------- | ------------------------------------------------------------ | | `400` | `INVALID_REQUEST` | Malformed request (bad JSON, missing fields, invalid values) | | `401` | `SASL_AUTHENTICATION_FAILED` | Missing or invalid credentials | | `403` | `TOPIC_AUTHORIZATION_FAILED` | ACL denied write access to the topic | | `404` | `UNKNOWN_TOPIC_OR_PARTITION` | Topic or partition does not exist | | `413` | `MESSAGE_TOO_LARGE` | Request body exceeds `64 MiB` after decompression | | `500` | `KAFKA_STORAGE_ERROR` | Internal error producing data | ## Datadog HTTP Log Intake ### Datadog Log Intake APIs {% hint style="info" %} Request version v772+ of the Agent. {% endhint %} The WarpStream Agent exposes Datadog-compatible HTTP/JSON endpoints for accepting log batches from the Datadog Agent and producing them into Kafka topics. All endpoints are served on the Agent's HTTP port (`8080` by default). ### Authentication and Authorization See [Authentication and Authorization](#authentication-and-authorization). Specific to the log intake endpoints: * These endpoints use the `DD-API-KEY` header rather than HTTP Basic Auth. WarpStream interprets `DD-API-KEY` as Kafka SASL/PLAIN credentials encoded as `$SASL_USERNAME:$SASL_PASSWORD`. * If `DD-API-KEY` is present but malformed and SASL auth is optional, it is ignored. * If `DD-API-KEY` is missing or malformed and SASL auth is required, the request fails with `401 SASL_AUTHENTICATION_FAILED`. * They require `WRITE` permission on the destination topic. If the authenticated principal does not have write access to the topic, the request fails with `403 TOPIC_AUTHORIZATION_FAILED`. Example: {% code overflow="wrap" %} ```bash curl -X POST \'http://localhost:8080/dd/my-topic/api/v2/logs' \-H 'Content-Type: application/json' \-H 'DD-API-KEY: YOUR_USERNAME:YOUR_PASSWORD' \-d '[{"message":"hello from datadog","service":"my-service"}]' ``` {% endcode %} ### Common Headers | Header | Description | Default | | ------------------ | ---------------------------------------------------------------------- | -------------------------- | | `DD-API-KEY` | WarpStream Kafka SASL/PLAIN credentials encoded as `username:password` | omitted | | `kafka-client-id` | Identifies the client for logging and diagnostics | `http-datadog-logs-client` | | `Content-Type` | Must be `application/json` | none | | `Content-Encoding` | Optional request compression | none | ### Datadog Agent Configuration When configuring the Datadog Agent, point `logs_dd_url` at the path prefix only. Do not include `/api/v2/logs` or `/v1/input` in the configured URL, because the Datadog Agent appends the intake suffix itself. Example Datadog Agent configuration for the primary endpoint: {% code overflow="wrap" %} ```yaml logs_enabled: true logs_config: use_v2_api: true logs_dd_url: https://warpstream.example.com/dd/$TOPIC_NAME ``` {% endcode %} If you want Datadog to use the compatibility endpoint instead, set: {% code overflow="wrap" %} ```yaml logs_enabled: true logs_config: use_v2_api: false logs_dd_url: https://warpstream.example.com/dd/$TOPIC_NAME ``` {% endcode %} ### Common Behavior These endpoints share the following behavior: * The `$TOPIC_NAME` path parameter is the exact Kafka topic name to produce to. * The topic must already exist. * The topic name must be a valid Kafka topic name. * The request body limit is `64 MiB` after decompression. * Each JSON object in the submitted array becomes one Kafka record. * The JSON object is stored as the Kafka record value exactly as submitted. * No Kafka key is set. * No Datadog-specific fields are interpreted or transformed. * All records from a single HTTP request are written to the same partition. * Across requests, the Agent rotates partitions over time for rough byte-based balancing. ### Raw Endpoint Specs {% hint style="info" %} The raw endpoint specifications is not required to use this integration. Simply follow the instructions above to configure the Datadog Agent and you'll be good to go. That said, the endpoints are documented for posterity. {% endhint %} #### `POST /dd/$TOPIC_NAME/api/v2/logs` Primary Datadog-compatible log intake endpoint. This is the preferred route for Datadog Agents configured with `use_v2_api: true`. **Path Parameters** | Parameter | Type | Required | Description | | --------- | ------ | -------- | ---------------------------- | | `topic` | string | yes | Destination Kafka topic name | **Request Body (JSON)** The request body must be one of: | Shape | Description | | ---------------- | ------------------------------------------------------------------------------- | | `[{...}, {...}]` | A JSON array of log objects. Each array element becomes one Kafka record value. | | `{}` | Connectivity probe. Accepted but does not produce any records. | The Agent treats each array element as an opaque JSON object. Common Datadog fields like `message`, `service`, `ddsource`, `ddtags`, `hostname`, `status`, and `timestamp` are preserved exactly as sent. **Example Request** {% code overflow="wrap" %} ```bash curl -X POST \'http://localhost:8080/dd/my-topic/api/v2/logs' \-H 'Content-Type: application/json' \-H 'DD-API-KEY: YOUR_USERNAME:YOUR_PASSWORD' \-d '[{"message": "application started","service": "payments","ddsource": "kubernetes","ddtags": "env:staging,team:data"},{"message": "worker ready","service": "payments","ddsource": "kubernetes","ddtags": "env:staging,team:data"}]' ``` {% endcode %} **Success Response (`200 OK`)** {% code overflow="wrap" %} ``` {} ``` {% endcode %} #### `POST /dd/$TOPIC_NAME/v1/input` Compatibility alias for Datadog Agents configured with `use_v2_api: false`. This endpoint has the same authentication, authorization, request-body, partitioning, and response semantics as `/dd/$TOPIC_NAME/api/v2/logs`. **Example Request** {% code overflow="wrap" %} ```yaml curl -X POST \'http://localhost:8080/dd/my-topic/v1/input' \-H 'Content-Type: application/json' \-H 'DD-API-KEY: YOUR_USERNAME:YOUR_PASSWORD' \-d '[{"message": "legacy intake example","service": "payments"}]' ``` {% endcode %} **Success Response (`200 OK`)** {% code overflow="wrap" %} ``` {} ``` {% endcode %} ### Connectivity Probe The Datadog Agent may send an empty JSON object to check HTTP connectivity: {% code overflow="wrap" %} ``` {} ``` {% endcode %} This is accepted and returns `200 OK` with an empty JSON response body: {% code overflow="wrap" %} ``` {} ``` {% endcode %} No Kafka records are produced for connectivity probes. ### Request Compression These endpoints support compressed request bodies via `Content-Encoding`. Supported encodings for Datadog-style traffic: * `gzip` * `zstd` The `64 MiB` request-size limit is enforced after decompression. ### Error Codes The `code` field in error responses maps either to Kafka protocol error codes or endpoint-specific validation errors. | HTTP Status | Code | Meaning | | ----------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `400` | `INVALID_REQUEST` | Malformed JSON, unsupported content type, non-empty top-level object, trailing JSON tokens, or other request-shape validation failure | | `400` | `INVALID_TOPIC_EXCEPTION` | Invalid topic name or topic does not exist | | `401` | `SASL_AUTHENTICATION_FAILED` | Missing, malformed, or invalid `DD-API-KEY` when authentication is required | | `403` | `TOPIC_AUTHORIZATION_FAILED` | ACL denied write access to the topic | | `413` | `PAYLOAD_TOO_LARGE` | Request body exceeds `64 MiB` after decompression | | `413` | `MESSAGE_TOO_LARGE` | One or more produced records exceed the Agent's maximum record size | | `500` | `KAFKA_STORAGE_ERROR` | Internal metadata lookup, authorization, or produce failure | #### Example Error Response {% code overflow="wrap" %} ```json {"code": "INVALID_REQUEST","message": "payload must be either an empty JSON object or a JSON array"} ``` {% endcode %} ### Notes * These endpoints are intended for Datadog-compatible log ingestion. For general-purpose Kafka-over-HTTP produce, see the [HTTP Produce APIs](#http-produce-apis) above. * Partitioning is request-scoped: all records in one HTTP request go to one partition. * Partition selection starts from a random partition per topic and rotates round-robin over time based on accumulated request bytes. * If you need different routing, use a different `$TOPIC_NAME` path prefix for each Datadog sender configuration. # Partitions Auto-Scaler This page describes the functionality of WarpStream's Partitions Auto-Scaler. The partitions auto-scaler is a convenience feature that automatically scales the number of partitions in a topic to ensure that the average write throughput (uncompressed bytes per second) stays below a configured threshold. This enables the partition count to increase automatically with organic traffic growth so that operators don't have to take manual actions or perform capacity planning. For example, consider a topic called called "logs" with the following configuration values: | Configuration | Value | | ------------------------------------------------------------------------------------------ | --------- | | `warpstream.partitions_auto_scaler.enabled` | `true` | | `warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second` | `2500000` | | `warpstream.partitions_auto_scaler.max_partition_count` | `1024` | The logs topic has 50 partitions and an average total throughput of 100 uncompressed MB/s. If we do the math then we'll see that: $$ 100/50 == 2MB/s/partition $$ which is below the limit of 2.5MB/s/partition. In this scenario, the partitions auto-scaler will take no action. However, if the total throughput increases to 250 uncompressed MB/s, then the throughput per partition will increase to: $$ 250/50 == 5MB/s/partition $$ which is above the limit of 2.5MB/s. As a result, the partitions auto-scaler will detect this and begin adding additional partitions to the topic until the average throughput per partition falls back below 2.5MB/s (I.E once the topic reaches roughly 100 partitions). The partitions auto-scaler can add partitions to a topic, but it can never remove them. As a result, it does not take action on every temporary spike in write throughput, and instead waits for throughput to exceeded the configured target consistently for a contiguous window of time (currently 15 minutes) before taking action. The partitions auto-scaler is particularly convenient for workloads where WarpStream is being used as a highly scalable and cost-effective "pipe" where records with the same key don't always need to be written to the same partition. ## Caveats 1. This feature is *not* suitable for any workloads where records with a specific key must always be written and consumed from the same partition. The reason for this is that in the Kafka protocol, record partitioning happens *client side*, so when the number of partitions in a topic *increases*, the destination partition for each record key will change. 2. The partitions auto-scaler can *increase* the number of partitions in a topic, but it cannot *decrease* the number of partitions in a topic. The reason for this is that the Kafka protocol (and the domain model of Kafka itself) are such that it's impossible to delete partitions from a topic with active producers/consumers in a safe manner. However, in the future we will solve this problem with WarpStream by creating "Virtual Topics" that abstract over multiple individual topics. ## Configuration Options All of the configuration options below are topic-level configuration values. | Configuration | Description | | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `warpstream.partitions_auto_scaler.enabled` |

Boolean value.

Whether the partitions auto-scaler is enabled for this topic. Defaults to false.

| | `warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second` |

Integer value.

Target throughput (in uncompressed bytes per second) for each partition in the topic. If average throughput per partition is below this value, the partitions auto-scaler will continue to add partitions to the topic until throughput per partition falls below this value or the max partitions limit is reached.

For example, if this was set to 2500000 (2.5 MB/s/partition) then the partitions auto-scaler would begin adding additional partitions once the total throughput of the topic exceeded 10MB/s (10MB / 4 partitions = 2.5MB/s/partition).

| | `warpstream.partitions_auto_scaler.max_partitions` | Maximum number of partitions beyond which the partitions auto-scaler will not add any more partitions to the topic, regardless of how high the throughput is. | ## How to Edit Configuration There are two ways to enable / configure partitions auto-scaling on a topic: 1. The WarpStream UI. 2. The Kafka protocol's AlterConfigs API. ### WarpStream UI Click "Edit Configuration" for a topic, in the Topics view for your cluster, and then navigate to the "Partitions Auto scaler" section.
### AlterConfigs API Use a Kafka client / tool / UI of your choice that allows editing topic-level configuration values to update the configuration values described in the [configuration section](#configuration-options) for your topic. ### Enable Auto-Scaling By Default For New Topics You can also configure cluster-level defaults so that newly created topics automatically inherit partitions auto-scaler settings. Set the following broker configuration values: | Configuration | Description | | -------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | warpstream.default.partitions\_auto\_scaler.enabled | Boolean value. Enables topic-level partitions auto-scaling by default for newly created topics. | | warpstream.default.partitions\_auto\_scaler.per\_partition\_throughput\_uncompressed\_bytes\_per\_second | Integer value. Default per-partition throughput target applied to newly created topics. | | warpstream.default.partitions\_auto\_scaler.max\_partition\_count | Integer value. Default max partition count auto-scaler limit applied to newly created topics. | You can set these values either in the WarpStream UI (cluster configuration) or via broker configuration APIs / tooling. To set in the UI, please navigate to "Cluster Settings", and set **Enable Partition Auto Scaler by Default, Default Partition Auto Scaler Target Throughput,** and\ **Default Partition Auto Scaler Max Partitions** as desired.
With the WarpStream CLI, you can run the following commands for example: 1. View current broker configuration: ```bash warpstream cli describe-broker-configs -bootstrap-host HOST -bootstrap-port PORT ``` 2. Update defaults: ```bash warpstream cli alter-broker-config -bootstrap-host HOST -bootstrap-port PORT \ -config-name warpstream.default.partitions_auto_scaler.enabled \ -config-value true warpstream cli alter-broker-config -bootstrap-host HOST -bootstrap-port PORT \ -config-name warpstream.default.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second \ -config-value 2500000 warpstream cli alter-broker-config -bootstrap-host HOST -bootstrap-port PORT \ -config-name warpstream.default.partitions_auto_scaler.max_partition_count \ -config-value 1024 ``` If a topic explicitly sets its own partitions auto-scaler configuration, those topic-level values take precedence over these cluster defaults. # Serverless Clusters WarpStream Serverless clusters are entirely managed by WarpStream. In addition to the fully-managed control plane, which is used by all cluster types, the Agents for Serverless clusters run in WarpStream's environment, and use object storage in WarpStream's cloud account. While Serverless cluster operations are similar to BYOC clusters, there are some known differences you should be aware of. ### Dedicated Kafka Headers If you are using a Serverless cluster, then the Kafka headers starting with “\_*ws*” are reserved for WarpStream. Concretely this means that: * Produce requests containing records with at least one header starting with this prefix will be rejected. * Those headers are used for internal purposes only and will never be returned to users when records are fetched. ### Ratelimits Severless clusters are rate-limited as follows: #### Global ratelimits (per virtual cluster) * Produce Compressed bytes: 20Mib/s * Produce Uncompressed bytes: 20Mib/s * Fetch Uncompressed bytes: 80Mib/s * New Connections Rate: 1k/s * Produce Requests Rate: 6k/s * General Requests Rate: 1k/s * Fetch Parallel Requests: 360 * Consumer Rebalance Timeout: 2m #### Connection Ratelimits (per connection) * Connection throughput: 64 MiB/s. * Number of Connections: 2k #### Request Ratelimits * Max TCP Request Size: 32MiB * Max Produce Request Uncompressed Bytes: 20 MiB # Partition Assignment Strategies This page explains the various different partition assignment strategies in WarpStream. Unlike Apache Kafka, WarpStream Agents are not "leaders" for individual topic-partitions. In fact, any Agent can write or read any record for any topic-partition at any time. However, due to [how the Apache Kafka protocol works](/warpstream/overview/architecture/service-discovery#partition-assignment), the `Metadata` responses need to inform the client which "brokers" are the leader for all the topic-partitions it wants to write to and read from. In addition, WarpStream needs to balance the traffic of all the different Kafka clients evenly amongst all the Agents. This means that WarpStream needs some "strategy" for determining which Agent it should tell clients is the leader for each topic-partition. In essence, you can think of WarpStream's partition assignment strategies as different load balancing strategies with different trade-offs. The rest of this document outlines the various strategies that WarpStream supports and their respective trade-offs. ## Strategies ### consistent\_random\_jump (default, recommended) `consistent_random_jump` uses an algorithm called ["random jump consistent hashing"](https://arxiv.org/pdf/1908.08762) to assign topic-partitions to individual Agents. The result is that the vast majority of produce requests for a given topic-partition will be processed by the same Agent. This dramatically improves the performance of the cluster for two reasons: First, the control plane has to process significantly less metadata. See the [Control Plane Utilization documentation](/warpstream/kafka/reference/control-plane-utilization) for more details on this. Second, it concentrates data for each partition in a smaller number of files which reduces the need for compaction as well as the effort required by the Agents to process fetch requests for live consumers.
The downside is that this consistent hashing strategy can result in Agent hot spots if traffic for some topic-partitions is higher than others. However, the "random jump" aspect of the load balancing algorithm will detect hot-spots automatically and spread the hot topic-partitions across multiple Agents. This results in slightly less good balancing than `single_agent` for clusters with extreme skews, but is the right trade-off for the vast majority of workloads. ### consistent\_spread (recommended in some specific cases) `consistent_spread` is similar to `consistent_random_jump` in that it uses a consistent hashing ring to assign topic-partitions to individual agents. However, unlike `consistent_random_jump`, `consistent_spread` is not load-aware and will never route around perceived hot-spots. As a result, `consistent_spread` is not recommended for multi-tenant clusters running a large number of heterogenous workloads and generally speaking `consistent_random_jump` is a better generic default. That said, `consistent_spread` can be useful for particularly demanding workloads where topic-partitions are very balanced and there is one highly dominant workload in the cluster. In that scenario, `consistent_spread` can reduce the number of batches that need to be processed by the system compared to `consistent_random_jump` by 2-3x which can help with reducing [control plane utilization](/warpstream/kafka/reference/control-plane-utilization). ### single\_agent (not recommended) Every time a client performs a `Metadata` request, the WarpStream service discovery system returns a view of the cluster in which a *single agent* is the leader for *all topic-partitions*. This means that in the general case, each Kafka client is connected to only a single WarpStream Agent at a time. Load balancing is accomplished by balancing client *connections* instead of individual `Produce` / `Fetch` requests, and is handled automatically by the WarpStream control plane. The control plane uses a power of 2 random choices load balancing strategy to return a different WarpStream Agent every time it receives a `Metadata` request from a Kafka Client. The `single_agent` strategy has two primary benefits: First, its the strategy that results in the lowest outlier latency for Produce requests. The reason for this is simple: object storage tends to have higher outlier latency than more traditional SSD storage. As a result, `single_agent` minimizes each individual client's exposure to outlier latency because each client is only exposed to the outlier latency from a single WarpStream Agent and it's file flushes at any given moment. Second, it has the most powerful load-balancing capabilities. The `single_agent` load balancing strategy effectively just balances Kafka client connections based on the observed load of the Agents, regardless of the shape of the underlying Kafka workload in terms of topic-partitions. In other words, the `single_agent` strategy results in the most even load utilization in the cluster because load is the **only factor** that it takes into consideration when making decisions. The primary downside of this strategy (and why its not the default / recommended strategy) is that it spreads data for the same topic-partition across many different files created by many different Agents. This results in the cluster having to spend more resources performing compactions, as well as making it more difficult for the Agents to serve fetch requests for all of the live consumers. Also, this strategy can result in a huge number of batches being generated which can lead to extremely [high control plane utilization](/warpstream/kafka/reference/control-plane-utilization) and degraded performance. For all of these reasons, this strategy is almost never recommended.
## Configuration There are two ways to configure partition assignment strategies in WarpStream: 1. At the Agent level. 2. Using a client id feature. ### Agent Configuration Set the `-defaultPartitionAssignmentStrategy` flag or `WARPSTREAM_DEFAULT_PARTITION_ASSIGNMENT_STRATEGY` environment variable on the Agents to the name of the partition assignment strategy that you want to use, I.E `consistent_random_jump` or `single_agent` . Once set, this will apply to any Kafka clients that are connected to those Agents. ### Client ID Alternatively, the partition assignment strategy can be configured for just a single application without impacting the default behavior for other applications using a [Kafka client ID feature](/warpstream/kafka/configure-kafka-client/configuring-kafka-client-id-features#warpstream_partition_assignment_strategy). # Control Plane Utilization The purpose of this page is to provide an intuitive understanding of the factors that influence WarpStream Control Plane utilization, and how to reduce it to make your workloads more efficient. ## WarpStream Control Plane Utilization Every WarpStream cluster gets a dedicated control plane that is fully managed by the WarpStream team. Control planes are always provisioned for maximum potential capacity and cannot be scaled further even if they reach maximum utilization. When a control plane reaches maximum utilization its performance will begin to degrade and the cluster may become unavailable until utilization is reduced. Control plane utilization is exposed via a metric called `warpstream_control_plane_utilization` (Prometheus) or `warpstream.control_plane_utilization` (Datadog). The value is exposed as a fraction between 0 and 1 where 0 means 0% utilization and 1 means 100% utilization. Every control plane is fully independent. This means that if you have two different clusters in your account, and one of the clusters reaches 100% utilization, the other cluster will not be impacted and vice versa. The vast majority of workloads will never approach maximum utilization of a single WarpStream control plane. However, some extremely demanding or pathological workloads can saturate a WarpStream control plane resulting in performance degradation, or in the worst case, unavailability of the cluster. ## Processed Batches Virtually every operation performed in a WarpStream cluster contributes to control plane utilization in some way. That said, the vast majority of control plane utilization is driven by one factor: how many batches the cluster has to process. In WarpStream, a "batch" is a group of records that all belong to the same topic-partition. The raw data in your batches never leaves your VPC, but the amount of metadata that the control plane needs to process scales linearly with the number of batches processed by the cluster. This can be understood intuitively by considering the fact that one of the control plane's primary responsibilities is to assign offsets to records, and a batch is the minimum unit of work for which offsets can be assigned. When Kaka's idempotency feature is **disabled**, the number of batches that must be processed is primarily a function of **the number of partitions that are actively produced to in a given time interval**. The number of partitions that are actively produced to in a given time interval is primarily a function of: 1. Record key distribution. 2. [Partition assignment strategy](/warpstream/kafka/reference/partition-assignment-strategies) (more on this later). 3. The number of partitions in the topics that are actively being produced to. When Kafka's idempotency feature is **enabled**, the number of batches that must be processed is a function of all of the above **plus the number of producers that are actively producing in a given time interval**. In order words, enabling idempotency makes the batches/s problem much worse. The reason for this is that when idempotency is disabled, the Agents can merge together batches from different producers as long as those batches belong to the same topic-partition. For many workloads, this dramatically reduces the number of batches that need processing. When idempotency is enabled, the Agent's can't perform this merge operation and the number of batches that need to be processed may increase dramatically The number of batches processed by your cluster is exposed via a metric called `warpstream_agent_segment_batcher_flush_num_batches` (Prometheus) or `warpstream.agent_segment_batcher_flush_num_batches` (Datadog). The number of processed batches can also be visualized in the WarpStream UI as shown below.
If your workload starts to approach more than `80,000` batches processed per second, you should consider following the steps in the next section to reduce the number of batches. ## Reducing Processed Batches ### Configure Kafka Clients Make sure you've configured your Kafka clients according to [our recommendations](/warpstream/kafka/configure-kafka-client/tuning-for-performance). ### Disable Idempotency As discussed in the [processed batches](#processed-batches) section, Kafka's idempotency features disables the Agent's ability to merge batches of data together that belong to the same topic-partition, potentially resulting in a significantly higher number of batches to process. Also, several of the strategies discussed below are only effective when idempotency is disabled (this is called out in the documentation when relevant). For particularly high volume or demanding workloads, we **strongly** recommend disabling this feature. ### Reduce the Number of Active Partitions As discussed above, the number of batches that must be processed is primarily a function of **the number of partitions that are actively produced to in a given time interval**. As a result, anything that reduces the number of partitions actively produced to will also reduce the number of batches that need to be processed. There are four ways to reduce the number of active partitions in a given time interval: 1. [Use NULL record keys.](#use-null-record-keys) 1. Works with idempotency enabled. 2. [Reduce the partition count in the topics that are being produced to.](#reduce-the-partition-count) 1. Works with idempotency enabled. 3. [Increase the batch timeout in the Agent.](#reduce-the-agent-batch-timeout) 1. Requires idempotency to be disabled. 4. [Change the partition assignment strategy.](#change-the-partition-assignment-strategy) 1. Requires idempotency to be disabled. #### Use NULL Record Keys When using non-null record keys, the producer will map each record to the partition it belongs to based on the record's key. For example, consider a single producer that produces 1024 records to a topic with 256 partitions such that each partition ends up receiving 4 records on average. In a given interval, this producer will generate \~256 batches of data. Now consider the same producer, but each record has a NULL key. When a record doesn't have a key specified, the Kafka client is free to assign that record to any partition as it sees fit. In practice, what most Kafka clients do is pick one partition, write a bunch of records to it until some threshold (like 1MiB) is reached, and then pick another partition and repeat. In that scenario, the 1024 records could end up being assigned to just one or two partitions in a given time interval, resulting in the producer generating just 1-2 batches of data in total. The partitions will stay almost perfectly balanced in aggregate since each Kafka producer will rotate the partition they're producing to on a regular basis. The downside of this approach is that records will be spread across all of the partitions with no consideration for specific records ending up in any particular partition which may not be acceptable for your consumers. #### Reduce the Partition Count {% hint style="info" %} You can ignore this section if your workload is using NULL record keys as described in the section above. {% endhint %} If your workload is using non-NULL record keys then the number of batches it will generate in a given time interval is a function of: 1. The distribution of your workload's keys. 2. The number of partitions in the topic(s) being produced to. You probably have no control over the distribution of your workload's keys, the data is the data, but you may have control over the number of partitions in each topic. Reducing this value will usually result in a \~ linear decrease in the number of batches processed by the cluster. The downside of this approach is that it will decrease the maximum number of parallel consumers that can process a given topic because the number of consumers in a workload cannot be scaled higher than the number of topic-partitions available to distribute amongst the consumers. #### Increase the Agent Batch Timeout {% hint style="info" %} This approach only works if you [disable idempotency](#disable-idempotency) because it relies on the Agent's ability to merge together batches of data that belong to the same topic-partition. {% endhint %} The WarpStream Agents buffer produced records in memory for the configured batch timeout (default 250ms) and then once the timeout elapses they flush one or more files containing all of the batches they received. If idempotency is disabled, then all of the batches for a given topic-partition received by a single Agent in this time interval will be merged together and presented to the control plane as a single batch. As a result of this merge operation, increasing the Agent batch timeout gives the Agent more time to accumulate more batches from the producer clients and merge them together. Therefore, increasing the batch timeout reduces the number of batches processed by the control plane. For example, consider a workload with 1024 partitions actively being produced to and Agents configure with a 250ms batch timeout running in three different availability zones. The minimum possible number of batches/s to be processed by the cluster then is: `NUM_PARTITIONS * NUM_FLUSHES_PER_SECOND * NUM_AVAILABILITY_ZONES` To make that concrete for our example: `1024 * 1000/250 * 3 == 12,288 batches/s` However, if we increase the batch timeout from 250ms to 500ms, then the Agent has twice as long to merge together batches for the same topic-partition and the number of batches drops in half: `1024 * 1000/500 * 3 == 6,144 batches/s` . The Agent batch timeout can be modified via the `-batchTimeout` flag or environment variable `WARPSTREAM_BATCH_TIMEOUT` . The default value is `250ms` . The downside of this approach is that increasing the batch timeout will increase the latency of Produce requests. #### Change the Partition Assignment Strategy {% hint style="info" %} This approach only works if you [disable idempotency](#disable-idempotency) because it relies on the Agent's ability to merge together batches of data that belong to the same topic-partition. {% endhint %} The default [partition assignment strategy](/warpstream/kafka/reference/partition-assignment-strategies) in WarpStream is `consistent_random_jump` which strikes a good balance between load-balancing and reducing the number of batches that must be processed. However, in many cases `consistent_random_jump` will end up spreading the load for a single topic-partition between 2-3 Agents instead of just 1 which can increase the number of batches that need to be processed by 2-3x respectively. As a result, if you have a fairly homogenous workload where your topic-partitions are highly balanced, then switching the partition assignment strategy to [`consistent_spread`](/warpstream/kafka/reference/partition-assignment-strategies#consistent_spread-recommended-in-some-specific-cases) could reduce the number of batches that need to be processed significantly compared to `consistent_random_jump`. The downside of this approach is that `consistent_spread` is not load-aware, so if your workload has significant load skew then you may end up with hotspots in the Agents resulting in degraded performance. ### Factors that Won't Help In the past, we used to recommend several approaches for reducing the number of batches that need to be processed that are no longer relevant: 1. Increasing the size of files that the Agents are allowed to create. 2. Vertically scaling the Agents and running less Agents. #### Increasing File Sizes Increasing the size of files that the Agents are allowed to create used to reduce the number of batches that needed to be processed significantly, but in the latest versions of the Agent this no longer true. In the latest Agent versions (v800+) we've improved the logic such that even when the maximum allowed file size is small, the Agents split batches into files in an intelligent way that maximizes their ability to merge batches for the same topic-partition together. As a result, increasing the size of files that the Agents generate is no longer helpful for reducing the number of batches that need to be processed. Each file does some have associated control plane overhead, so increasing the file size can still help reduce control plane utilization just by virtue of reducing the number of files that need to be processed, but the latest version of the Agents will automatically increase the maximum allow file size if they detect that the number of files being created is putting significant load on the control plane. Increasing the maximum allowed file size is still useful for reducing object storage PUT costs. The downside of this approach is that it increases Produce request latency. #### Vertically Scaling and Running Less Agents Historically, we used to recommend that customers vertically scale their Agents to reduce control plane utilization. The reason for this is that in older versions of the Agent the [default partition assignment strategy](/warpstream/kafka/reference/partition-assignment-strategies) was `single_agent` where the number of batches that needed to be processed scaled almost linearly with the number of deployed Agents. In the latest Agent versions (v800+), the default partition assignment strategy is `consistent_random_jump` which does not suffer from this problem. That said, vertically scaling the Agents and not overprovisioning can still be useful for reducing object storage PUT costs, and generally spealing larger Agents are more resilient to traffic spike and load imbalances, it just won't help much with control plane utilization except to reduce the number of files processed by the control plane which does have an effect on utilization, but a much smaller effect than the number of processed batches. # Benchmark How to Benchmark WarpStream. {% hint style="info" %} If you prefer to skip benchmarking WarpStream yourself, you can read [our public benchmarking blog post](https://www.warpstream.com/blog/warpstream-benchmarks-and-tco) where we provide detailed WarpStream benchmarks and TCO analysis. {% endhint %} The most important thing to consider when benchmarking WarpStream is that because WarpStream is a higher latency system than Apache Kafka, your Kafka client settings must be tuned appropriately to work with WarpStream to achieve high throughput. Start by reading our "[Tuning Kafka Clients for Performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance)" documentation. Ideally, benchmarking is performed with a real application running in in a pre-prod environment, or by teeing traffic from a production workload to WarpStream. However, we also understand that many people like to begin the evaluation process with simple synthetic benchmarks so the rest of this document is focused on how to do that correctly. ## WarpStream Benchmark Tools WarpStream has built-in tools to run Producer and Consumer benchmarks against any compatible Kafka cluster. These tools were added in the [v651](/warpstream/overview/change-log#release-v651) and [v652](/warpstream/overview/change-log#release-v652) WarpStream releases. These benchmark tools can be ran against any Kafka API compatible product so you can easily compare performance against your existing Kafka infrastructure. These benchmark tools are tuned using our [Tuning for Performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) guide. While these tools are tuned for a WarpStream Cluster they will work without issue against other Kafka API compatible products. ## Producer Benchmark CLI ### Example This is an example of running the producer benchmark tool against a local playground WarpStream Cluster with a single client. ```bash $ warpstream cli benchmark-producer -topic ws-benchmark -num-clients 1 ``` ``` 46798 records sent (446.00 MiB), 9359.60 records/sec (89.26 MiB/sec), 186.476ms min latency, 300.85209ms avg latency, 557.667ms max latency, 2842 buffered records. 49441 records sent (471.00 MiB), 9888.20 records/sec (94.30 MiB/sec), 192.584ms min latency, 257.174298ms avg latency, 392.739ms max latency, 2926 buffered records. 49734 records sent (474.00 MiB), 9946.80 records/sec (94.86 MiB/sec), 189.176ms min latency, 250.716986ms avg latency, 316.025ms max latency, 2914 buffered records. 49613 records sent (473.00 MiB), 9922.60 records/sec (94.63 MiB/sec), 186.951ms min latency, 280.346658ms avg latency, 480.603ms max latency, 2890 buffered records. 49032 records sent (467.00 MiB), 9806.40 records/sec (93.52 MiB/sec), 182.87ms min latency, 265.414339ms avg latency, 403.456ms max latency, 2925 buffered records. 49597 records sent (472.00 MiB), 9919.40 records/sec (94.60 MiB/sec), 187.95ms min latency, 265.404274ms avg latency, 484.618ms max latency, 2892 buffered records. 49626 records sent (473.00 MiB), 9925.20 records/sec (94.65 MiB/sec), 185.409ms min latency, 250.42801ms avg latency, 311.784ms max latency, 2970 buffered records. ``` ### Technical Details The produce benchmark uses the [franz-go](https://github.com/twmb/franz-go) Kafka library with the following configuration: ```go opts = append(opts, kgo.DefaultProduceTopic(c.topic)) opts = append(opts, kgo.MetadataMaxAge(60*time.Second)) opts = append(opts, kgo.MaxBufferedRecords(1_000_000)) opts = append(opts, kgo.ProducerBatchMaxBytes(int32(c.producerMaxBytes))) opts = append(opts, kgo.RecordPartitioner(kgo.UniformBytesPartitioner(1_000_000, false, false, nil))) opts = append(opts, kgo.ProduceRequestTimeout(c.produceRecordTimeout)) if c.disableIdempotentWrite { opts = append(opts, kgo.DisableIdempotentWrite()) } ``` This configuration is similar configuration that we [recommend for the best performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance#producer-configuration). ### Usage ```bash $ warpstream cli benchmark-producer --help Usage of benchmark-producer: -bootstrap-host string kafka bootstrap host (default "localhost") -bootstrap-port int kafka bootstrap port (default 9092) -client-id string client-id to pass along to kafka (default "warpstream-cli") -disable-idempotent-write disables idempotent write, WARNING: this may cause poor performance as this sets the maximum in-flight requests to 1 -enable-tls dial with TLS or not -kafka-log-level string the log level to set on the kafka client, accepted values are DEBUG, INFO, WARN, ERROR (default "WARN") -max-records-per-second int maximum number of records per second to produce per kafka client (default 10000) -num-clients int number of kafka clients (default 3) -num-records int number of messages to produce, -1 for unlimited. (default 1000000) -produce-record-timeout duration maximum amount of time to wait for a record to be produced (default 10s) -producer-max-bytes int upper bounds the size of a record batch, this mirrors Kafka's max.message.bytes. (default 16000000) -prometheus-port int the port to serve promethes metrics on, -1 to disable (default 8081) -record-size int message size in bytes (default 10000) -sasl-password string password for SASL authentication -sasl-scram uses sasl scram authentication (sasl plain by default) -sasl-username string username for SASL authentication -tls-client-cert-file string path to the X.509 certificate file in PEM format for the client -tls-client-key-file string path to the X.509 private key file in PEM format for the client -tls-server-ca-cert-file string path to the X.509 certificate file in PEM format for the server certificate authority. If not specified, the host's root certificate pool will be used for server certificate verification. -topic string the topic to produce to ``` ## Consumer Benchmark CLI ### Example This is an example of running the consumer benchmark tool against a local playground WarpStream Cluster with a single client.\ \ The consumer is consuming data in real-time that is being produced from the producer benchmark tool. Note: End to End latency can only be calculated when consuming data in real-time that was produced using the WapStream producer benchmark tool. ```bash $ warpstream cli benchmark-consumer -topic ws-benchmark -num-clients 1 ``` ``` 45784 records consumed (436.00 MiB), 9156.80 records/sec (87.33 MiB/sec), 305.785ms min e2e latency, 423.749632ms avg e2e latency, 552.51ms max e2e latency. 49570 records consumed (472.00 MiB), 9914.00 records/sec (94.55 MiB/sec), 245.006ms min e2e latency, 437.278189ms avg e2e latency, 649.385ms max e2e latency. 49070 records consumed (467.00 MiB), 9814.00 records/sec (93.59 MiB/sec), 238.257ms min e2e latency, 428.520332ms avg e2e latency, 628.591ms max e2e latency. 49550 records consumed (472.00 MiB), 9910.00 records/sec (94.51 MiB/sec), 229.308ms min e2e latency, 445.467432ms avg e2e latency, 642.138ms max e2e latency. 49663 records consumed (473.00 MiB), 9932.60 records/sec (94.72 MiB/sec), 307.591ms min e2e latency, 422.433481ms avg e2e latency, 539.169ms max e2e latency. 49697 records consumed (473.00 MiB), 9939.40 records/sec (94.79 MiB/sec), 310.196ms min e2e latency, 425.985862ms avg e2e latency, 620.136ms max e2e latency. 49678 records consumed (473.00 MiB), 9935.60 records/sec (94.75 MiB/sec), 307.071ms min e2e latency, 435.650686ms avg e2e latency, 658.087ms max e2e latency. ``` ### Technical Details The produce benchmark uses the [franz-go](https://github.com/twmb/franz-go) Kafka library with the following configuration: ```go opts = append(opts, kgo.ConsumeTopics(c.topic)) opts = append(opts, kgo.MetadataMaxAge(60*time.Second)) opts = append(opts, kgo.FetchMaxBytes(int32(c.fetchMaxBytes))) if c.fetchMaxBytes > 50_000_000 { opts = append(opts, kgo.BrokerMaxReadBytes(int32(c.fetchMaxBytes*2))) } opts = append(opts, kgo.FetchMaxPartitionBytes(25_000_000)) opts = append(opts, kgo.FetchMaxWait(10*time.Second)) if c.fromBeginning { opts = append(opts, kgo.ConsumeResetOffset(kgo.NewOffset().AtStart())) } else { opts = append(opts, kgo.ConsumeResetOffset(kgo.NewOffset().AtEnd())) } if c.consumerGroup != "" { opts = append(opts, kgo.ConsumerGroup(c.consumerGroup)) } ``` This configuration is similar configuration that we [recommend for the best performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance#consumer-configuration). ### Usage ```bash $ warpstream cli benchmark-producer --help Usage of benchmark-consumer: -bootstrap-host string kafka bootstrap host (default "localhost") -bootstrap-port int kafka bootstrap port (default 9092) -client-id string client-id to pass along to kafka (default "warpstream-cli") -consumer-group string the consumer group to use to consume, if unset (default) no consumer group is used -enable-tls dial with TLS or not -fetch-max-bytes int the maximum amount of bytes a broker will try to send during a fetch, this corresponds to the java fetch.max.bytes setting (default 50000000) -fetch-max-partition-bytes int the maximum amount of bytes that will be consumed for a single partition in a fetch request, this corresponds to the java max.partition.fetch.bytes setting (default 25000000) -from-beginning start with the earliest message present in the topic partition rather than the latest message, when enabled e2e latency can't be calculated -kafka-log-level string the log level to set on the kafka client, accepted values are DEBUG, INFO, WARN, ERROR (default "WARN") -num-clients int number of kafka clients (default 3) -prometheus-port int the port to serve promethes metrics on, -1 to disable (default 8082) -sasl-password string password for SASL authentication -sasl-scram uses sasl scram authentication (sasl plain by default) -sasl-username string username for SASL authentication -tls-client-cert-file string path to the X.509 certificate file in PEM format for the client -tls-client-key-file string path to the X.509 private key file in PEM format for the client -tls-server-ca-cert-file string path to the X.509 certificate file in PEM format for the server certificate authority. If not specified, the host's root certificate pool will be used for server certificate verification. -topic string the topic to consume from ``` ## Helm Chart A Helm Chart is available [here](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-benchmark) to deploy these tools into Kubernetes. The benchmark configuration can be changed by modifying the following values ```yaml topicName: "ws-benchmark" bootstrapHost: "localhost" bootstrapPort: 9092 consumer: enabled: true replicaCount: 1 numClients: 3 fetchMaxBytes: 50000000 fetchMaxPartitionBytes: 25000000 producer: enabled: true replicaCount: 1 numClients: 3 recordSize: 10000 maxRecordsPerSecond: 10000 producerMaxBytes: 16000000 ``` ## Prometheus Metrics Both the producer and consumer benchmark tools expose prometheus metrics on port 8081 and 8082 respectively and expose various metrics. Some important metrics to monitor and example queries and graphs are bellow. Note: The example graphs are of benchmark results from running a WarpStream Playground on a laptop and running the benchmark tooling on the same machine. WarpStream Playgrounds are heavily rate limited and are not tuned for performance. When running benchmarks we recommend running them against a production like setup using real WarpStream Agents. ### Produce Throughput Measure the total amount of Bytes Produced. Metric: `franz_go_produce_bytes_total` Example Query: `sum(rate(franz_go_produce_bytes_total[1m]))` - Measure the per second throughput. The higher the number the more data the benchmark is producing with a higher number equaling better performance.
### Produce Buffered Records Measure the total number of records that are being buffered. Metric: `franz_go_buffered_produce_records_total` Example Query: `sum(rate(franz_go_buffered_produce_records_total[1m]))` - Measure the per second buffered rate. The higher the number the more records that are being buffered. If this is increasing over time your benchmark is producing faster then your Kafka cluster can handle.
### Produce Latency Measure the amount of latency to produce a record, measured from the time the record is added to a batch until an ACK is returned from the Kafka Cluster. Metric: `warpstream_produce_benchmark_produce_request_duration_seconds_bucket` Example Query: `warpstream_produce_benchmark_produce_request_duration_seconds_bucket` - Measure the P90 Latency of producing records. The higher the number the more latency there is.
### End to End Consume Latency Measure the amount of latency to produce a record and to consume the same record. Measured from the time the record is created in memory in the producer benchmark to the time a consumer fetches and starts processing the record from the Kafka cluster. Note: End to End latency can only be calculated when consuming data in real-time that was produced using the WapStream producer benchmark tool. Metric: `warpstream_consume_benchmark_e2e_consume_duration_seconds_bucket` Example Query: `histogram_quantile(0.90, sum by(le) (rate(warpstream_consume_benchmark_e2e_consume_duration_seconds_bucket[1m])))` - Measure the P90 End to End Latency of producing and consuming the same record. The higher the number the more End to End latency there is.
## Kafka Benchmark tools While we recommend using WarpStream benchmark tooling to perform your synthetic benchmarks you can use any benchmark tool including the native Kafka ones. Other benchmark tools may need to be tuned to get the best performance out of WarpStream. See out [Tuning for Performance](/warpstream/kafka/configure-kafka-client/tuning-for-performance) guide. Due to the nature of how the Java Kafka protocol is implemented, you'll most likely struggle to achieve more than 60-100MiB/s of producer traffic from a single instance of the kafka perf testing tooling. However, once you've found configuration that you're happy with, you can increase the total throughput of the benchmark by running multiple instances of kafka-producer-perf-test.sh concurrently. On the contrast, the WarpStream benchmark tooling can achieve multi-gigabyte per second producer traffic within a single instance if given enough cpu, memory, and network bandwidth. ### kafka-producer-perf-test.sh One of the most common utilities for performing synthetic benchmarks of Kafka clusters is the kafka-producer-perf-test.sh utility. This utility embeds a native Java Kafka client, so it should be tuned according to our recommend settings. For example: {% code overflow="wrap" %} ```bash kafka-producer-perf-test.sh --print-metrics --producer-props bootstrap.servers=$BROKERS enable.idempotence=false compression.type=lz4 linger.ms=25 batch.size=10000000 buffer.memory=128000000 max.request.size=64000000 metadata.max.age.ms=60000 --record-size 10000 --topic "test" --throughput 1000 --num-records 1000000 ``` {% endcode %} The settings above are just a starting point, you'll want to slowly increase the values of throughput and num-records as you perform your testing. More importantly, you'll have to consider how many partitions the test topic you're producing to has. If the topic you're producing to has many partitions, you may need to **reduce** the value of batch.size to prevent the producer utility from OOMing. If the topic you're producing to has less partitions, then you may need to **increase** the value of batch.size instead to achieve higher throughput. Running multiple instances of kafka-producer-perf-tesh.sh is highly recommended because load-balancing in WarpStream works differently than it does in Apache Kafka. Specifically, Apache Kafka **balances partitions across Brokers**, whereas WarpStream (due to its stateless nature) **balances client connections across Agents**. As a result, a single instance of kafka-producer-perf-test.sh will generally route all of its traffic to a single WarpStream Agent. However, if you run multiple instances of the benchmarking utility concurrently, you'll see the traffic begin to spread evenly amongst all your deployed Agents. # Orbit (Cluster Linking) Replicate and migrate Kafka clusters. ## Overview Orbit is a feature of the WarpStream BYOC product that can automatically replicate data *and metadata* from any *source* Kafka-compatible cluster to a *destination* WarpStream cluster. Orbit is built directly into the WarpStream Agents, and doesn't require any additional deployments or binaries, it runs directly in the destination WarpStream cluster. Orbit can perfectly mirror Kafka records (including headers and preserving offsets), topic configurations, cluster configurations, topic deletions, consumer group offsets, and more. Records copied from source to destination are *identical* down to the offsets of the records. This, combined with the fact that Orbit can also copy consumer groups offsets, means that Orbit can be used to reliably migrate Kafka clients to the destination cluster without any duplicate consumption of data. There are four primary use-cases for Orbit: 1. Replication from a source Kafka cluster into WarpStream for migrations. 2. Creating replicated copies of source Kafka clusters for disaster recovery. 3. Creating replicated copies of source Kafka clusters for scalable tiered storage. 4. Creating replicated copies of source Kafka clusters to provide additional read replicas, and isolate workloads. For example, analytical or batch consumers could read from the WarpStream replica cluster to avoid overloading the source Kafka cluster. We highly recommend watching the overview video below, which provides a comprehensive overview of Orbit features. {% embed url="" %} ## Orbit Deployment The WarpStream agent binary ships with Orbit built in. Agents launched with the [`jobs` role](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) will automatically pick up Orbit configurations and perform replication. ## Orbit Configuration Orbit is fully controllable from a single YAML file which can be edited through the WarpStream console or created programmatically through [Terraform](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/dcb0c7b9f7ae3ef69bea0c513807407808d2a0d7/examples/resources/warpstream_pipeline/resource.tf#L39). ### Overview ```yaml # WarpStream Orbit YAML configuration file. source_bootstrap_brokers: # These nodes should belong to the same source Kafka cluster. # Orbit does not (currently) support replicating from multiple # source Kafka clusters. - hostname: localhost # Kafka server hostname. port: 9092 - hostname: example.kafkaserver.com port: 9092 source_cluster_credentials: # Username, password, and mechanism are optional and should be # omitted if SASL is not required to connect to the source # cluster. # # Set username/password as environment variables in the Agents # with an ORBIT_ prefix. sasl_username_env: SASL_USERNAME_ENV_VAR sasl_password_env: SASL_PASSWORD_ENV_VAR sasl_mechanism: plain # Independent of whether SASL is enabled, set this to true or # false based on whether the source cluster has TLS enabled. use_tls: false # Whether TLS verification should be skipped. tls_insecure_skip_verify: false topic_mappings: - source_regex: topic.* # Exact match, not substring. # Optional prefix added to topic names when they're replicated. # Leave this empty if you want topics replicated to WarpStream # to have the exact same name as they did in the source cluster. destination_prefix: "" begin_fetch_at_latest_offset: false cluster_config: # Don't copy cluster configs from source (topic auto topic creation # policy, default partition count, etc). copy_source_cluster_configuration: false consumer_groups: # Optional prefix added to consumer group names when they're # replicated. Leave this empty if you want consumer groups replicated # to WarpStream to have the exact same name as they did in the source # cluster. destination_group_prefix: "" # Whether consumer groups and their corresponding offsets # should be copied or not. copy_offsets_enabled: true warpstream: # Used to control the rate at which Orbit will consume from # the source Kafka topic. Higher values will increase # throughput, but put more load on the source cluster. cluster_fetch_concurrency: 2 ``` Here's a quick summary of the YAML file above: * Orbit is set up to connect to a Kafka cluster with 2 source brokers. * It will connect to the source brokers using SASL PLAIN and will retrieve the username and password by reading the `ORBIT_SASL_USERNAME_ENV_VAR` and `ORBIT_SASL_PASSWORD_ENV_VAR` environment variables respectively in the Agent. * Topics which match the `topic.*` regex in the source will be mirrored to the destination with their name unchanged (no prefix will be added to their name). * Cluster configurations will not be copied over. * Consumer group names and offsets will be replicated into the destination cluster. The consumer group names will be mirrored to the destination with their name unchanged (no prefix will be added to their name). * At most 2 concurrent fetches (globally, regardless of the number of Agents) will be executed in the WarpStream Agents at any given time. ### Specify Source Brokers Host address and port of the the source cluster Kafka brokers or Agents. ```yaml source_bootstrap_brokers: - hostname: localhost port: 9092 - hostname: example.kafkaserver.com port: 9092 ``` * Source cluster brokers are listed under the `source_bootstrap_brokers` in the YAML. * Each broker is defined as a `hostname` and a `port`. * The brokers must be reachable from your WarpStream Agents. ### Specify Credentials Credentials used by the destination WarpStream agents to connect to the source cluster. * Credentials for the brokers are defined under the `source_cluster_credentials` section in the YAML. * Since the Agents runs in your VPC, WarpStream does not have access to your credentials. * To force the Agents to use TLS when connecting to your source clusters, use the `use_tls` flag in the `source_cluster_credentials` section. * If you have ACLs enabled in the source cluster, then you need to make sure the principal associated with Orbit credentials used to connect with the source has the following permissions: * `Describe` `DescribeConfigs` and `Read` operations for the Topic resources which you want mirrored by Orbit. * `Describe` and `DescribeConfigs` for the Cluster resource. * `Describe` and `Read` operations for the Group resource (consumer groups). Read is required to fetch offsets. #### SASL ```yaml source_cluster_credentials: sasl_username_env: SASL_USERNAME_ENV_VAR sasl_password_env: SASL_PASSWORD_ENV_VAR use_tls: false sasl_mechanism: plain ``` * Both the `SASL_USERNAME_ENV_VAR` and the `SASL_PASSWORD_ENV_VAR` fields refer to environment variables. The Agents will append a `ORBIT_` prefix to the values of the fields before using the environment variables, so the environment variables in the Agent should be configured as `ORBIT_SASL_USERNAME_ENV_VAR` and `ORBIT_SASL_PASSWORD_ENV_VAR` respectively. * By default Orbit uses the `plain` SASL mechanism. To specify the SASL mechanism add the `sasl_mechanism` field in the `source_cluster_credentials` section. Supported mechanisms include: * `plain` * `scram-256` * `scram-512` * `oauthbearer` * When using `oauthbearer` the following fields are available: ```yaml source_cluster_credentials: sasl_mechanism: oauthbearer sasl_oauth2_enabled: true sasl_oauth2_client_key_env: KAFKA_OAUTH_CLIENT_ID sasl_oauth2_client_secret_env: KAFKA_OAUTH_CLIENT_SECRET sasl_oauth2_token_url: http://... # optional: # sasl_oauth2_scopes: [] # sasl_oauth2_endpoint_params: {} # sasl_oauth2_extensions: {} # if sasl_oauth2_enabled is not set to true: # sasl_access_token_env: KAFKA_OAUTH_ACCESS_TOKEN # optional: # sasl_extensions: {} ``` #### Secrets Manager (Agent v811+) Instead of using environment variables, it is also possible to use a cloud provider Secrets Manager. The following syntax is expected: ```yaml source_cluster_credentials: sasl_username_as_secret: provider: "aws" id: arn:aws:secretsmanager:us-east-2:012345678901:secret:my-username-piC0u4 key: username #optional (Agent v818+) sasl_password_as_secret: provider: "aws" id: arn:aws:secretsmanager:us-east-2:012345678901:secret:my-password-ghOgvg key: password #optional (Agent v818+) use_tls: false sasl_mechanism: plain ``` More specifically, here is what is supported: * `provider` has to be `aws`, `gcp` or `azure` * the id depends on the cloud provider: * if it's `aws` then the full secret ARN is expected (including the 6 random characters at the end). See examples in the `yaml` above. * if it's `gcp` then a full path (with the version optional) is expected. Examples: * `projects/721600436206/secrets/my_orbit_username/versions/latest` * `projects/721600436206/secrets/my_orbit_username` (latest is implicitly used) * if it's `azure` then the full URL is expected (`Secret Identifier` in the Azure Portal UI). Examples: * \`[https://my-vault.vault.azure.net/secrets/my-orbit-username/2a506dff10694007bfd2d705f8bbe459](https://warpstream-staging-c.vault.azure.net/secrets/saasy-staging-c-metronome-api-key/2a506dee10694006bfd2e705f8bbe459)\` * `key` is optional: * If omitted, the entire secret value is used as the credential. This is the right choice when the secret is a plaintext value (e.g. just the password). * If set, the secret value must be a JSON object, and the value stored at that key is used instead. This matches the key/value secrets that the AWS Secrets Manager console creates (e.g. `{"username": "foo", "password": "..."}`), and also works for GCP and Azure secrets as long as the payload is a JSON object. * The same secret can be referenced by multiple fields with a different `key` for each, so a single secret can hold both the username and the password. * Only top-level keys are supported. Nested lookups (e.g. `key: credentials.username` or pointing at a key whose value is itself an object or array) are not supported: the value at the key must be a plain JSON string. Key matching is exact and case-sensitive. #### mTLS PEM encoded certs ```yaml source_cluster_credentials: mtls_client_cert_env: MTLS_CERT_PATH_ENV # or `mtls_client_cert_as_secret` (see syntax above) mtls_client_key_env: MTLS_KEY_PATH_ENV # or `mtls_client_key_as_secret` (see syntax above) mtls_server_ca_cert_env: MTLS_SERVER_CA_CERT_PATH_ENV # or `mtls_server_ca_cert_as_secret` (see syntax above) use_tls: true ``` * The `MTLS_CERT_PATH_ENV`, `MTLS_KEY_PATH_ENV`, and `MTLS_SERVER_CA_CERT_PATH_ENV` fields refer to environment variables. The Agents will append a `ORBIT_` prefix to the values of the fields before using the environment variables, so the environment variables in the Agent should be configured as `ORBIT_MTLS_CERT_PATH_ENV`, `ORBIT_MTLS_KEY_PATH_ENV`, and `ORBIT_MTLS_SERVER_CA_CERT_PATH_ENV` respectively. * Note that these environment variables should point to file paths for the respective PEM encoded certificate files and must not encode the certificates directly. * `mtls_server_ca_cert_env` is an optional field. However, it is highly recommended to set the environment variable to the public keys of the certificate authorities that sign your server certificates. If this environment variable is not set, WarpStream defaults to trusting all server certificates from your Operating System's root certificate store. #### mTLS JKS encoded certs ```yaml source_cluster_credentials: jks_key_store_file_path_env: JKS_KEYSTORE_PATH_ENV # or `jks_key_store_file_path_as_secret` (see syntax above) jks_key_store_password_env: JKS_KEYSTORE_PASSWORD_ENV # or `jks_key_store_password_as_secret` (see syntax above) jks_key_store_key_password_env: JKS_KEY_STORE_KEY_PASSWORD_ENV # or `jks_key_store_key_password_as_secret` (see syntax above) jks_trust_store_file_path_env: JKS_TRUST_STORE_FILE_PATH_ENV # or `jks_trust_store_file_path_as_secret` (see syntax above) jks_trust_store_password_env: JKS_TRUST_STORE_PASSWORD_ENV # or `jks_trust_store_password_as_secret` (see syntax above) use_tls: true ``` * `JKS_KEYSTORE_PATH_ENV`, `JKS_KEYSTORE_PASSWORD_ENV`, `JKS_KEY_STORE_KEY_PASSWORD_ENV`, `JKS_TRUST_STORE_FILE_PATH_ENV`, `JKS_TRUST_STORE_PASSWORD_ENV` fields refer to environment variables. The Agents will append a `ORBIT_` prefix to the values of the fields before using the environment variables, so the environment variables in the Agent should be configured as `ORBIT_JKS_KEYSTORE_PATH_ENV`, `ORBIT_JKS_KEYSTORE_PASSWORD_ENV`, etc. * The environment variable for `jks_key_store_file_path_env` should be assigned the file path to the jks keystore file. * The environment variable for `jks_key_store_password_env` should be assigned the key store password. * The environment variable for `jks_key_store_key_password_env` should be assigned the key password. * The environment variable for `jks_trust_store_file_path_env` should be assigned the file path to the trust store. * The environment variable for `jks_trust_store_password_env` should be assigned the trust store password. ### Topic Mappings #### Replicate Topics Topics can be replicated by adding mappings under the `topic_mappings` section of the YAML. 
topic_mappings:
    - source_regex: foo_topic.*
      # Topics whose name match the foo_topic.* regex will be
      # replicated with their name unmodified.
      destination_prefix: ""
    - source_regex: bar_topic.*
      # Topics whose name match the bar_topic.* regex will be
      # replicated, but their name will have baz_ added as a prefix.
      destination_prefix: baz_
      topic_config_overrides:
        - config_name: retention.ms
          config_value: "72000000"
        - config_name: cleanup.policy
          config_value: "delete,compact"
      begin_fetch_at_latest_offset: false
* To replicate a topic from the source cluster to the destination, add a topic mapping to the `topic_mappings` section with a `source_regex` which will match the source topic which should be replicated. * For example in the above regex, topics in the source cluster with prefixes `foo_topic` and `bar_topic` would be replicated. * Note that the mapping `*` will not match every sequence of characters. Since Orbit accepts regular expressions, use `.*` to match every sequence of characters. * Note that topics being actively replicated by Orbit **cannot receive writes from external Kafka clients** until the `irreversible_disable_orbit_management` field in the topic mapping is set to true. * See the [producer migration](#producers) section for details on how to use this field. * Leave `destination_prefix` empty if you want topic names to be perfectly preserved when topics are replicated, or set it to a non-empty string if you want Orbit to add a prefix to Orbit-replicated topics. * For example, if the source cluster has a topic `bar_topicA` , then it will be replicated in WarpStream as `baz_bar_topicA` , but a topic with name `foo_topicB` will be replicated with its name unmodified. * `topic_config_overrides` can be used to override the value of the specified topic configurations in the source cluster which will be copied to the destination. Currently, only the `retention.ms` and `cleanup.policy` topic configurations are supported for override. See the [topic mapping rules](#topic-mapping-rules) section for further details. * By default, Orbit will copy all records starting from the earliest topic partition offsets in the source cluster for a given topic. If `begin_fetch_at_latest_offset` is set to `true`, then Orbit will copy all records beginning at the latest topic partition offsets in the source cluster. See the [topic mapping rules](#topic-mapping-rules) section for further details. #### Pause Topic Replication ```yaml topic_mappings: - source_regex: foo_topic.* destination_prefix: "" - source_regex: bar_topic.* destination_prefix: baz_ ``` * To pause replication for a topic, remove the mapping which matches the topic from the YAML. * In some cases, it's possible that a topic can match multiple topic mappings, and you do not want to remove all the topic mappings. In such a case, you can explicitly mark a topic mapping as paused. In the following example, replication for the topic `test` will be paused since the first topic mapping which matches `test` has `pause` set to `true`. ```yaml topic_mappings: - source_regex: test pause: true - source_regex: .* ``` #### Topic Mapping Rules Detailed rules about the topic mappings which indicates which topics mappings match which topics. ```yaml topic_mappings: - source_regex: foo_topic.* destination_prefix: "" - source_regex: bar_topic.* destination_prefix: baz_ topic_config_overrides: - config_name: retention.ms config_value: "72000000" begin_fetch_at_latest_offset: false ``` * Each mapping consists of `source_regex`, `destination_prefix`, and `irreversible_disable_orbit_management` fields. * Orbit will create a new topic in the destination only if there exists a topic mapping with a `source_regex` which matches the source cluster topic. * If a topic already exists in the destination cluster but is not managed by Orbit, Orbit will automatically take ownership of it if **both** of the following conditions are met: * The topic is empty (has never been written to). * The topic's partition count and `cleanup.policy` match those of the corresponding source topic. Partition count must match because it cannot decrease once set. `cleanup.policy` must match because there are [restrictions on how it can be changed](https://docs.warpstream.com/warpstream/kafka/pages/qC7LyPpt9ZsJygEz4tXF#cleanup.policy) (e.g. a non-compacted topic cannot be made compacted). Other configurations like `retention.ms` are not compared and will be synced from the source after Orbit takes ownership. This is useful when pre-creating topics in WarpStream before setting up Orbit replication. Once Orbit takes ownership, it will manage the topic's configurations and replication like any other Orbit-managed topic. * To determine if a topic in the source cluster matches a topic mapping, it must match a `source_regex` in at least one topic mapping. For example, the topic `foo_topicA` matches the first topic mapping. * The value of the `destination_prefix` field is appended as a prefix to the topic name for the topic which will be created in the destination cluster. Empty string is a valid value for `destination_prefix` if you want the topics to have the same name in the destination WarpStream cluster as they did in the source cluster. * To determine if a topic which has been created by Orbit in the destination cluster matches a topic matching, it must match the `destination_prefix` + `source_regex` in at least one of the topic mappings. For example, `bar_topicB` is created as `baz_bar_topicB` in the destination, and `baz_bar_topicB` in the destination, matches the second topic mapping. * A topic in either the source or the destination cluster will match the mappings in the `topic_mappings` section in the order in which the topic mappings are listed, and only a single topic mapping will apply (the first one). * Orbit will keep the topic configurations in sync for a source/destination topic pair only if the destination topic matches a topic mapping. * Orbit will keep deleted topics in sync for a source/destination topic pair only if the destination topic matches a topic mapping. * Orbit will replicate records for a source/destination topic pair only if the destination topic matches a topic mapping. * The `irreversible_disable_orbit_management` will only be respected if a destination topic matches the topic mapping with the flag set to `true`. See more details on topic migration in the [producer migration section](#producers) where the meaning of this configuration option is explained. * `topic_config_overrides` overrides values of specified topic configurations for a given topic mapping. For example, if topics which match the `bar_topic.*` regex in the source cluster have a `retention.ms` value of `60000000`, the corresponding topics which will be created in the destination will have a `retention.ms`value of `72000000`. * `begin_fetch_at_latest_offset` can be set to `true` for a topic mapping to begin fetches at the latest offset of a topic partition rather than the earliest. * By default this field is set to `false` and records are copied starting from the earliest topic partition offsets in the source cluster. * When set to `true`, any topics created in the destination Orbit cluster which match that topic mapping will begin fetching records starting from the latest offsets in the source cluster. * As an example if some topic `topic_A` partition `0`, has records from offsets `100` to `1000` in the source cluster, then Orbit will start mirroring `topic_A` beginning at offset `1000` rather than `100`. Every records after offset `1000` will be mirrored identically. #### Topic Configurations Orbit will copy topic configurations from the source cluster for all of the topics which Orbit is managing in the destination cluster. Note that only the following configuration values are considered relevant to WarpStream and will be copied by Orbit: * `cleanup.policy` * `message.timestamp.type` * `retention.ms` * `delete.retention.ms` * `min.compaction.lag.ms` ### Sync Consumer Groups Orbit can sync offsets of consumer groups which exist in the source cluster to the destination cluster. ```yaml consumer_groups: copy_offsets_enabled: true destination_group_prefix: "" ``` * Set `copy_offsets_enabled` to `true` under the `consumer_groups` section and Orbit will start mirroring consumer groups from the source cluster to the destination. * Similar to topic names, consumer group replication can optionally be configured to add a destination group prefix to the replicated consumer group names. Leave `destination_group_prefix` empty if you want consumer group names to be replicated as-is. ### Cluster configurations #### Sync cluster configurations ```yaml cluster_config: copy_source_cluster_configuration: true ``` * Orbit will copy cluster configurations from the source cluster to the destination if the `copy_source_cluster_configuration` flag under the `cluster_config` section is set to `true`. * Note that Orbit only copies cluster configurations which are relevant to WarpStream. These include: * `num.partitions` which is used to determine the default number of partitions a new topic should be created with. This is referring to new topics created by a Kafka client, and not Orbit. * `auto.create.topics.enable` which indicates if new topics should be created automatically when the client performs a metadata request for a topic which does not exist. * `log.retention.ms` `log.retention.minutes` `log.retention.hours` which indicate the default retention used by the topics. * `offsets.retention.minutes` which indicates the expiration duration of the offsets committed by a consumer group. #### Disable copying of records ```yaml cluster_config: disable_copy_records: true ``` * You can prevent Orbit from copying any records from the source to the destination cluster for every single topic mapping by setting the above cluster config. ## Orbit UI You can edit the Orbit YAML, view the offset lag for topic partitions being mirrored, and view which topics Orbit is mirroring all from the WarpStream UI. #### Create Orbit YAML 1. Select your virtual cluster from the WarpStream console and navigate to the Orbit tab. You'll see a text editor which can be used to create and edit the Orbit YAML file.

Orbit YAML editor

2. Edit the YAML, and hit Save. Note that the pipeline is paused by default. You can start it by hitting the toggle next to the word PAUSED.

Version 0 of Orbit YAML is deployed, but pipeline is paused

3. Once the pipeline is running, you'll see the topics which are mirrored by Orbit show up under the Orbit tab. In the image below, Orbit created a topic called `test_topic` with a prefix `orbit_` as expected.

orbit_test_topic created by Orbit shows up under the Orbit tab

4. To modify the YAML, hit the edit button, edit the YAML, and hit Save again. Orbit will create a new configuration for the pipeline. In the image below, I've set `copy_offsets_enabled` to `true` so that consumer group offsets will be copied from the source cluster to the destination. **Note:** You will have to hit the Deploy button on the new version before it will take effect. You can also deploy older versions to "rollback".

copy_offsets_enabled is true so orbit will start syncing source/destination consumer group offsets

5. You can view the lag between the source and destination cluster topic partitions from the Consumers tab. The `orbit_source_cluster_offsets` consumer group will include every topic and partition that is being mirrored by Orbit. The offset lag indicates how far behind Orbit is compared to the source cluster. The time lag indicates time.Since(X), where X is the timestamp of the most recently copied source record.

Consumer group view showing offset lag for mirrored topic partitions, offset lag is 0 in this case.

## Programmatic Access: HTTP API and Terraform For programmatic access, Orbit can be configured via its [HTTP API](/warpstream/reference/api-reference/pipelines/create-pipeline#create-orbit-pipeline) or declaratively using our [Terraform provider](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/dcb0c7b9f7ae3ef69bea0c513807407808d2a0d7/examples/resources/warpstream_pipeline/resource.tf#L39). ### Orbit HTTP API: create, update, and deploy configurations This guide shows how to manage the Orbit pipeline via HTTP APIs using an API key. It covers creating the pipeline, creating/updating configurations (YAML), deploying a configuration, and pausing/resuming the pipeline. Prerequisites: * You have a Virtual Cluster ID (for example `vci_xxx...`). * You have a WarpStream API key with admin permissions for that virtual cluster. * You know your Console/API base URL (for example `https://console.yourcompany.com`). Authentication: * Include your API key in the `warpstream-api-key` header on every request. Notes: * All endpoints are `POST` and accept/return JSON. * When updating an Orbit configuration, you create a new configuration version and then deploy it. You do not mutate an existing configuration in place. *** #### 1) Create the Orbit pipeline (once) Endpoint: `/api/v1/create_pipeline` Request body: ```bash curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/create_pipeline" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'", "pipeline_name": "orbit_default", "pipeline_type": "orbit" }' ``` Response will include a `pipeline_id` you can save. If the pipeline already exists, you can obtain its ID using “Describe pipeline” below. *** #### 2) Create a new Orbit configuration (YAML) Endpoint: `/api/v1/create_pipeline_configuration` Send the YAML as a string in `configuration_yaml`: ```bash ORBIT_YAML=$(cat <<'YAML' source_bootstrap_brokers: - hostname: "kafka-source-1" port: 9092 topic_mappings: - source_regex: "^orders\\.v[0-9]+$" destination_prefix: "src_" cluster_config: copy_source_cluster_configuration: true disable_copy_records: false consumer_groups: destination_group_prefix: "src_" copy_offsets_enabled: true warpstream: cluster_fetch_concurrency: 8 YAML ) curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/create_pipeline_configuration" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'", "pipeline_id": "'$PIPELINE_ID'", "configuration_yaml": '"'$ORBIT_YAML'"' }' ``` The response includes a `configuration_id` for the created version. *** #### 3) Deploy a configuration and control state (run/pause) Endpoint: `/api/v1/change_pipeline_state` Deploy a specific configuration and set desired state to running: ```bash curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/change_pipeline_state" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'", "pipeline_id": "'$PIPELINE_ID'", "deployed_configuration_id": "'$CONFIGURATION_ID'", "desired_state": "running" }' ``` Pause the pipeline without changing the deployed configuration: ```bash curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/change_pipeline_state" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'", "pipeline_id": "'$PIPELINE_ID'", "desired_state": "paused" }' ``` To “update” Orbit’s configuration, create a new configuration version (step 2) and deploy it (step 3). *** #### 4) Discover the Orbit pipeline and configurations If you don’t have the pipeline ID, describe by type to fetch the Orbit pipeline and its configuration versions. Endpoint: `/api/v1/describe_pipeline` ```bash curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/describe_pipeline" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'", "pipeline_type": "orbit" }' ``` The response includes the pipeline overview and an array of `pipeline_configurations` with their `id` and `version`. We return up to 100 of your latest configs sorted from oldest to newest. You can find the active config with `deployed_configuration_id` . You can also list pipelines: Endpoint: `/api/v1/list_pipelines` ```bash curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/list_pipelines" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'" }' ``` *** #### 5) Delete the Orbit pipeline (optional) Endpoint: `/api/v1/delete_pipeline` ```bash curl -sS -X POST \ -H "Content-Type: application/json" \ -H "warpstream-api-key: $WARPSTREAM_API_KEY" \ "$CONSOLE_BASE_URL/api/v1/delete_pipeline" \ -d '{ "virtual_cluster_id": "'$VIRTUAL_CLUSTER_ID'", "pipeline_id": "'$PIPELINE_ID'" }' ``` *** ## Migration Orbit can be used to migrate existing Kafka compatible clusters to WarpStream. ### Replication First, follow the instructions above to configure WarpStream to replicate all of the topics and consumer groups from the source cluster. Monitor the lag of the `orbit_source_cluster_offsets` consumer group to wait for WarpStream to "catch up" on all the source Kafka topics. ### Consumers The following migration flow will ensure that your Kafka consumer clients can be migrated to WarpStream without duplicate processing of records. The Orbit YAML must have the following config. ```yaml consumer_groups: copy_offsets_enabled: true ``` 1. Orbit will periodically sync committed offsets for consumer groups to WarpStream. 2. Let's say you want to migrate some consumer group `X`. 3. First, shut down the consumers belonging to this group in the source cluster. 4. Wait 1 minute for Orbit to copy the consumer group offsets for group `X` to the destination cluster. You can use the `warpstream_consumer_group_max_offset` metric emitted from the agents to view the offsets copied by Orbit to the destination. See the [monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents#observability) page for more details on this metric. You can also see the currently committed offsets in the consumers tab in the WarpStream UI. 5. Restart consumers but point them to the destination WarpStream cluster instead by changing their bootstrap URL. The consumers will start reading from the offsets which were last committed in the source cluster, picking up where they left off. ### Producers The following migration flow will ensure that your Kafka producer clients can be cut over to WarpStream seamlessly. Using the `irreversible_disable_orbit_management` field in a topic mapping will disable Orbit management of the topic. The following example shows how and when to use the field. 1. Let's say you have the following topic mapping in the Orbit YAML, and you have decided that you want to move the topic `topic0` matched by this mapping from the source to the destination. To make the example more complicated, let's assume that a `topic1` also exists in the source cluster which is also being mirrored by Orbit. ```yaml topic_mappings: - source_regex: topic.* destination_prefix: test0_ ``` 2. First shut down the producer clients producing to `topic0` in the source. 3. Wait for the offset lag between the source topic `topic0` and destination topic `test0_topic0` to become 0. You can view the lag for the topic from the `orbit_source_cluster_offsets` group in the Consumer view in the Warpstream console. The lag for all of the partitions of the topic are also emitted as the `warpstream_consumer_group_lag` metric from the Agents. See the [monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents#observability) page for more details on this metric. You can also see the currently committed offsets in the consumers tab in the WarpStream UI. 4. Create a new topic mapping to migrate the `test0_topic` topic. Note the `irreversible_disable_orbit_management` field. This field indicates that Orbit should stop managing the topic which will allow Kafka Producer clients to write to the topic. This decision is irreversible. Note that this process is async, and it can take orbit up to **10s** until it allows Kafka clients to write to the topic. During this period, the clients will get a retriable error. ```yaml topic_mappings: - source_regex: topic0 destination_prefix: test0_ irreversible_disable_orbit_management: true - source_regex: topic.* destination_prefix: test0_ ``` 5. Since topic mappings match topics in order, the `test0_topic0` will now accept writes from regular Kafka clients, and Orbit will no longer copy records from source for this topic. Note that `test0_topic1` is unaffected because it does not match the topic mapping. 6. Restart your producer clients by pointing them to the destination WarpStream cluster. Since `topic0` is known as `test0_topic0` in the destination due to the prefix, you'll have to write to `test0_topic0` instead. ## Observability You can monitor Orbit using the metrics built into the WarpStream Agents. See our [monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents) documents for more details. Orbit will also emit some additional metrics: 1. You can use the `warpstream_agent_kafka_produce_with_offset_uncompressed_bytes_counter` metric to view the data successfully written by Orbit to the destination WarpStream cluster. 2. You can use the `warpstream_consumer_group_lag` and filter for the group `orbit_source_cluster_offsets` and the topics you care about to determine the lag between your topics in the source cluster and WarpStream. Specifically, the lag is defined as the difference (in offsets) between the max offset in the source cluster and the most recently copied offset. View the [monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents#observability) page for more details on this metric. 3. You can use the `warpstream_consumer_group_estimated_lag_very_coarse_do_not_use_to_measure_e2e_seconds` metric and filter for the consumer group `orbit_source_cluster_offsets` and `topic` to get the time since the latest record copied by Orbit was assigned a timestamp in the source. View the [monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups#metrics) page for more details on this metric. This metric is a rough measure of E2E lag from the data being timestamped by your producer Kafka client to it being queriable in WarpStream. However, the timestamp is processed asynchronously and it can take 10-15 seconds before they are used to compute time lag. This means the time lag is 10-15 seconds higher than the actual E2E lag from your Kafka producer client to the consumers reading from WarpStream, and also higher than just the orbit replication lag because it includes the time it takes for the data to be processed through the source Kafka cluster as well. 4. You can use the `warpstream_consumer_group_max_offset` metric emitted from the agents to view the offsets copied by Orbit to the consumer group in the destination. See the [monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents/monitoring-consumer-groups#metrics) page for more details on this metric. 5. `warpstream_agent_kafka_source_cluster_connections_counter` metric is a counter which is incremented for short lived Orbit connections made against the source cluster. It can be used to estimate the rate at which Orbit is creating connections against the source cluster. ## Tuning Orbit ### Concurrency The primary knob for controlling Orbit throughput is `cluster_fetch_concurrency` . This is a global knob that controls how many concurrent Orbit fetch jobs will run against the source Kafka cluster regardless of how many Agents are deployed. ```yaml warpstream: cluster_fetch_concurrency: 10 ``` Increasing this value will increase throughput, but put more load on the source Kafka cluster / WarpStream Agents, and vice versa. ### Latency By default Orbit uses a `fetch_max_wait_ms` of 1000(1 s) to fetch from the source cluster. The config can be reduced further to improve latency to fetch from the source. ```yaml warpstream: fetch_max_wait_ms: 250 ``` {% hint style="warning" %} Reducing the `fetch_max_wait_ms` config can impact performance for some Kafka vendors. {% endhint %} ### Memory Usage The default Orbit configuration is optimized for high throughput, but may lead to excessive memory usage or even OOMs depending on the nature of the workload / topics being replicated. If you experience memory issues while running Orbit, consider modifying your Orbit pipeline as follows: ```yaml warpstream: fetch_config: # Reduced from default of 104857600. fetch_max_bytes: 20857600 # Reduced from default of 52428800. fetch_max_partition_bytes: 10428800 ``` ## Unclean Leader Election Note that it is expected that the topics replicated by Orbit have `unclean.leader.election.enable` set to `false`. If this configuration is set to `true` , and if there can be data loss in the source cluster, Orbit won't retroactively delete data in the destination WarpStream cluster which has been replicated using Orbit, but is lost in the source cluster. ## FAQ ### What is the difference between "managed by Orbit" and matching a `topic_mappings` entry? A topic is "managed by Orbit" if and only if it was created by Orbit or Orbit took ownership of it, and it hasn't been migrated using `irreversible_disable_orbit_management: true`. This is a property of the destination topic itself, and is distinct from whether a source topic currently matches a `topic_mappings` regex. For example, removing a topic's `topic_mappings` entry does not stop it from being managed by Orbit. ### How does retention work for historical data replicated with `begin_fetch_at_latest_offset: false`? Retention in WarpStream is based on when the data was **written to WarpStream**, not the original record timestamps from the source cluster. The retention clock resets when Orbit replicates data into the destination cluster. This means that if a topic in the source cluster has a 7-day retention and you start replicating it with `begin_fetch_at_latest_offset: false`, all replicated historical data will be retained for the full 7 days from the time it was replicated into WarpStream — even if the records were originally produced days or weeks ago in the source cluster. ### When syncing consumer group offsets, which topics are included? Orbit only copies consumer group offsets for topics that are **managed by Orbit** in the destination cluster. This means the topic must have been created (or taken over) by Orbit and still be under Orbit's management. If a consumer group in the source cluster has committed offsets for topics that Orbit is not managing in the destination, those offsets will not be copied. Additionally, if a consumer group in the destination cluster has active member clients, Orbit will stop copying offsets for that group entirely. Orbit also never allows a committed offset for a group to revert to a smaller value — offsets will only move forward. See [What is the difference between "managed by Orbit" and matching a `topic_mappings` entry?](#what-is-the-difference-between-managed-by-orbit-and-matching-a-topic_mappings-entry) for more details on this distinction. There is currently no way to filter which specific managed topics have their consumer group offsets synced — it is all-or-nothing via the `copy_offsets_enabled` flag. # Auto Migration Migrate to WarpStream with zero client restarts ## Overview Orbit Auto Migration is a way to seamlessly migrate Kafka producers from a source cluster to a destination WarpStream cluster. Migrating Kafka producers traditionally requires a single coordinated maintenance window— every producer application has to be stopped, replication lag has to be drained to zero, and every producer has to be restarted pointing at the destination cluster. For deployments with more than a handful of producer applications, that means coordinating dozens or hundreds of restarts inside a tight window, often across multiple teams, with limited rollback options if something goes wrong. Orbit Auto migration removes the coordinated restart from the cutover. Producers are reconfigured to target a set of WarpStream agents ahead of time, at whatever pace suits the team that owns each application. Until cutover, the agents transparently forward writes to the source cluster; in parallel, Orbit replicates records from source to WarpStream. When you initiate the cutover, WarpStream briefly returns retriable errors to producers, waits for replication lag to drain, and then serves writes directly. Standard Kafka client retry behavior carries producers through the block window without a restart. For the manual approach, see [Manual Migration](https://docs.warpstream.com/warpstream/kafka/orbit#migration). ### How a Migration Works {% hint style="info" %} You need at least v826 of the WarpStream agent to use Orbit Auto Migration. {% endhint %} In a typical migration, you start with a collection of producers and consumers writing to and reading from your source Kafka cluster. The source can be any Kafka-compatible cluster including open-source Kafka, MSK, or even another WarpStream cluster. The goal is to end up with all of those producers and consumers connected to the destination WarpStream cluster, with no records lost in the process. Orbit Auto migration removes the manual coordination required by a traditional cutover, in which producers are paused in lockstep, replication lag is watched by hand, and every producer is restarted inside a single maintenance window. Once producers have been repointed at WarpStream, each topic is cut over independently via a single click or API call. To pull this off, Auto Migration treats every migration as a per-topic operation. Each topic moves through a number of states, initiated by the config you apply to Orbit, that determines where producer writes go and what producers see at each phase of the cutover: | State | What it means | Producer experience | | ----------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `REJECT` | Default for Orbit-managed topics. Auto migration is not active for this topic. | Produces to WarpStream are rejected. Producers should still directly target the source cluster. | | `PROXY` | Auto migration is active. WarpStream forwards every produce request to source; Orbit replicates source → WarpStream as usual. | Normal success responses (records actually land at source). | | `MIGRATING` | Cutover in progress. WarpStream returns a retriable error to all producers while Orbit drains the remaining lag. | Retriable errors. Well-configured Kafka clients automatically retry with backoff. | | `COMPLETE` | Cutover complete. Orbit no longer manages the topic; produces go directly to WarpStream. | Normal success responses (records land directly in WarpStream). | `COMPLETE` is terminal- once a topic is migrated, it cannot be moved back. To understand the migration process in more detail, it's helpful to walkthrough a migration end-to-end. #### Step 1 — Prerequisites Before starting, confirm all of the following are in place: * **An Orbit pipeline replicating from source to destination.** Auto migration sits on top of standard Orbit replication. Orbit must already be replicating every topic you intend to migrate from your source cluster to the destination WarpStream cluster, and replication lag should be trending down. See [Configuring Orbit](https://docs.warpstream.com/warpstream/kafka/orbit#orbit-configuration) for further details. * **WarpStream credentials for your producers and consumers.** Each producer and consumer that will connect to WarpStream needs credentials on the destination cluster. See [here](https://docs.warpstream.com/warpstream/kafka/orbit#specify-credentials) for further information. * **ACLs on the destination WarpStream cluster.** If you use Kafka ACLs, configure equivalent ACLs on WarpStream so that producers and consumers have the permissions they expect once repointed. #### Step 2 — Migrate consumers Once Orbit is replicating and ACLs are in place, you can migrate your consumer applications to WarpStream. Note that in order for consumers to resume from the right position and avoid duplicate processing, Orbit must be configured to preserve source offsets. You can find details on how to migrate Consumers [here](https://docs.warpstream.com/warpstream/kafka/orbit#consumers). #### Step 3 — Configure auto migration Auto migration is configured under two top-level blocks in your Orbit config: * `auto_migration` — auto migration settings (e.g. which topics are enabled). * `auto_migration_source_cluster_credentials` — credentials for the proxy producer embedded in the agents that forwards client writes to source. A typical config looks like this: ```yaml auto_migration: enable: false # global default auto_migration_source_cluster_credentials: sasl_username_env: ORBIT_PROXY_SASL_USERNAME sasl_password_env: ORBIT_PROXY_SASL_PASSWORD sasl_mechanism: scram-512 use_tls: true topic_mappings: - source_topic: orders destination_topic: orders auto_migration: enable: true # opt this topic in - source_topic: legacy_audit destination_topic: legacy_audit # no auto_migration block — this topic stays in REJECT ``` Once you apply the config, every Orbit-managed topic for which `enable` resolves to `true` transitions from `REJECT` to `PROXY`- producing to those topics via the WarpStream agents will transparently forward writes to the source cluster. Topics with `enable: false` stay in `REJECT` and are unaffected. **Topic-level configs** Fields under the top-level `auto_migration:` block apply as **global defaults** to every Orbit-managed topic. Any field can be overridden per-topic under `topic_mappings[].auto_migration:`— a field set per-topic overrides the global value, and a field left unset inherits it. This is what lets you roll out selectively (keep `enable: false` globally and flip it on a few topics at a time). | Field | Default | Per-topic overridable | Description | | -------- | ------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable` | `false` | Yes | Turn auto migration on for matched topics. When `true`, the topic transitions from `REJECT` to `PROXY`: writes to the topic via WarpStream are transparently forwarded to source. | **Proxy credentials** Auto migration uses its own dedicated credentials to talk to the source cluster, separate from the credentials Orbit uses for replication reads. The Orbit config exposes them as two distinct blocks: * `source_cluster_credentials` — used by Orbit for **replication reads**. * `auto_migration_source_cluster_credentials` — used by the **proxy producer** embedded in the agents to forward client writes to source. The proxy is a regular Kafka producer client. It only ever calls three Kafka APIs: * `Metadata` — to discover partition leaders. * `Produce` — to forward client writes. * `InitProducerID` — to allocate a source-cluster producer ID for each idempotent producer it proxies. Like Orbit's replication credentials, proxy credentials are read from environment variables— you specify the env-var **name** in the config, not the secret value itself. See [Source Cluster Credentials](https://docs.warpstream.com/warpstream/kafka/orbit#specify-credentials) for how env vars are wired in. You have two options for the proxy principal: * **Separate proxy principal (recommended for least privilege).** Provision a fresh SASL principal on the source cluster with only the permissions listed below. Your Orbit replication principal stays read-only. ```yaml source_cluster_credentials: sasl_username_env: ORBIT_SASL_USERNAME sasl_password_env: ORBIT_SASL_PASSWORD sasl_mechanism: scram-512 use_tls: true auto_migration_source_cluster_credentials: sasl_username_env: ORBIT_SASL_PROXY_USERNAME # different env var as source_cluster_credentials sasl_password_env: ORBIT_SASL_PROXY_PASSWORD sasl_mechanism: scram-512 use_tls: true ``` * **Reuse the Orbit replication principal.** If your existing Orbit replication principal already has (or can be granted) the produce permissions below, you can reuse it: point each `*_env` field under `auto_migration_source_cluster_credentials` at the same env-var names you set under `source_cluster_credentials`. While simpler operationally, the trade-off is that one principal now has both read and write on the source. 
source_cluster_credentials:
    sasl_username_env: ORBIT_SASL_USERNAME
    sasl_password_env: ORBIT_SASL_PASSWORD
    sasl_mechanism: scram-512
    use_tls: true

  auto_migration_source_cluster_credentials:
    sasl_username_env: ORBIT_SASL_USERNAME    # same env var as source_cluster_credentials
    sasl_password_env: ORBIT_SASL_PASSWORD
    sasl_mechanism: scram-512
    use_tls: true
The proxy supports the same authentication options as Orbit replication— SASL/PLAIN, SASL/SCRAM-256, SASL/SCRAM-512, mTLS (PEM or JKS), TLS-only, and plaintext. Field semantics for `sasl_username_env`, `mtls_client_cert_env`, `jks_key_store_file_path_env`, etc. are identical to `source_cluster_credentials`; see [Source Cluster Credentials](https://docs.warpstream.com/warpstream/kafka/orbit#specify-credentials) for the full reference. **Required ACLs on the source cluster** If your source cluster has ACLs enabled, the proxy principal needs the following ACLs on every topic that will be auto-migrated: | Operation | Resource | Why | | ---------- | -------- | ----------------------------------------------------------------------------------- | | `Write` | `Topic` | For `Produce`. | | `Describe` | `Topic` | For `Metadata`. (Modern Kafka grants `Describe` implicitly when `Write` is granted) | **Required ACLs on the destination cluster** If ACLs are enabled on the destination WarpStream cluster, the agent enforces them as normal during `PROXY`— including for produces forwarded to source. Configure the full set of ACLs your clients will need after cutover (produce, consume, consumer-group, transactional, cluster-level— whatever applies), and do so *before* enabling enforcement. The proxy authorizes every client request against the destination cluster's ACLs and rejects unauthorized requests with the usual Kafka authorization errors. For example, if a client with principal `foo` produces to topic `bar` while ACLs are enabled, `foo` must have `WRITE` on `bar` on the destination cluster, even though the write is being proxied to source— the proxy enforces destination ACLs in exactly the same way it will after cutover. To validate destination ACLs are correct before turning enforcement on, enable [**ACL shadowing**](https://docs.warpstream.com/warpstream/kafka/manage-security/configure-acls#acl-shadowing) on the destination cluster. The agent evaluates every produce against the configured ACLs but does not block— it emits an `ACL_SHADOW_DENIED` diagnostic and associated logs for any produce that *would* have been rejected. Review the diagnostics, fix gaps in your ACL configuration, then flip ACLs from shadowed to enforced. #### Step 4 — Cut over producers (Proxy phase) With auto migration enabled, your topics are now in `PROXY` state. Reconfigure your producer applications to point their `bootstrap.servers` at WarpStream. **There is no time pressure**— you can do this over hours, days, or weeks, in any order, as you finish redeploying each application. While a topic is in `PROXY` state: * WarpStream forwards every produce request to your source cluster. * Producers see the same success/failure semantics they would see producing directly to source. * Orbit continues replicating from the source to WarpStream. {% hint style="warning" %} **Producer compatibility** While a topic is in `PROXY` (or `MIGRATING`), the following Kafka producer features are **not supported**: * **Transactional produce**— not supported. Producers must have it disabled before being repointed at WarpStream. Consult the relevant documentation for your client to see how to do this. Once a topic reaches `COMPLETE`, normal WarpStream produce semantics apply and these constraints no longer hold. {% endhint %} You can verify a topic is being proxied by checking the Migration tab in the Console or by querying the topic's migration state via the API— its state will show `PROXY` and lag will reflect normal Orbit replication. **Tuning your Producers** Before you point your producers at WarpStream, you have to tune it. Check out the [config tuning section](#tuning-the-producers) for more details. #### Step 5 — Initiate migration When all producers for a topic have been pointed at WarpStream, writes are being proxied, and replication lag is stable, you're ready to cut over. #### Initiating Migrations can be initiated either from the UI or via the API. **UI:** In the WarpStream console, open the **Migration** tab, select one or more topics in PROXY state, and click **Migrate**. **API:** Use the [Initiate Orbit Topic Auto Migration endpoint](/warpstream/reference/api-reference/orbit-auto-migration/initiate). Topics can be selected by literal name, by regex, or both, and a `dry_run` flag is available to preview which topics would be initiated without actually doing so. #### Migration Parameters For each migration request, there are certain parameters that can be set. These apply per-request— if you roll back to `PROXY` and re-initiate later, you can supply different values for the second attempt (e.g. tighter lag thresholds once you have a better sense of the topic's traffic profile). Any field omitted from the request falls back to the default shown below. | Parameter | Type | Required | Default | Description | | --------------------------- | ---- | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `max_time_lag_seconds` | int | No | `180` (3m) | Per-topic time-lag ceiling, in seconds, at the moment of initiation. **Mutually exclusive** with `max_offset_lag`. Must be ≥ 0. Used by default when both lag fields are unset. | | `max_offset_lag` | int | No | `0` (unused) | Per-topic offset-lag ceiling at the moment of initiation; topics above it are reported in `errors` and not initiated. **Mutually exclusive** with `max_time_lag_seconds`. Must be ≥ 0. Prefer `max_time_lag_seconds` unless you have a specific reason. | | `migration_timeout_seconds` | int | No | `300` (5m) | Maximum seconds a topic may remain in `MIGRATING` before the scheduler rolls it back to `PROXY`. Must be ≥ 0. | > Set **at most one** of `max_offset_lag` and `max_time_lag_seconds` — supplying both positive values is rejected with `400 conflicting_lag_thresholds`. #### The Migration Phase Once initiation is accepted, the topic enters the `MIGRATING` state. WarpStream returns a retriable error to all producers for that topic— well-behaved Kafka clients automatically retry with backoff, so your applications do not crash. No new records enter the source cluster via the proxy while Orbit drains the remaining lag. The dominant factor in how long a migration takes is **how quickly Orbit replication lag drops to 0** for each topic— a migration cannot succeed until it does. To reduce migration time, the most useful lever is tuning your Orbit configuration (concurrency, fetch sizes, and related settings) to drain lag faster. See the [Orbit performance tuning guide](https://docs.warpstream.com/warpstream/kafka/orbit#tuning-orbit) for details. #### Cutover Once lag has reached zero and any inflight requests have been drained, the topic is promoted to `COMPLETE`. From this point, producer retries succeed and writes go directly into WarpStream. #### Timeout and rollback If lag does not reach zero within the configured `migration_timeout_seconds` (default 300 seconds/5 minutes), the topic automatically rolls back to `PROXY` state and producers resume forwarding to source. #### Post-migration Once a topic is in `COMPLETE` state, Orbit stops managing it and producer writes go directly to WarpStream; the producer experience returns to normal and no further action is needed. `SERVE` is terminal. Repeat Step 4 for any remaining topics— topics migrate independently of one another. ### Aborting a migration If you've initiated a migration but want to back out, for example, you've spotted some misconfiguration, you can abort while a topic is in `MIGRATING` state. Abort rolls the topic back to a `PROXY` state immediately, without waiting for `migration_timeout_seconds` to elapse, and producers resume forwarding to source. Topics in `COMPLETE` state cannot be aborted— migration is one-way. Migrations can be aborted either by clicking the `Abort` action on the relevant row in the Migration tab in the Console or programmatically via the API, using the [Abort Orbit Topic Auto Migration endpoint](/warpstream/reference/api-reference/orbit-auto-migration/abort). ### Tuning the producers While a topic is in `PROXY` state, records are forwarded to your source cluster. However, we've made it such that the latency profile that producers experience matches what they'd see producing straight to WarpStream, not the source cluster's. Because of this, you should start by following the normal WarpStream guidance when tuning producers— see [Configure Clients](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client) and [Tuning for Performance](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/tuning-for-performance). This also means that after the clients cutover, the latency profile should remain mostly the same, so your producer config carries over and your producers keep operating as normal with minimal disruption. In some cases, there are two additional configs that may that need to be tuned, depending on your existing configuration, to ensure producers can migrate smoothly. These include: * Producer buffer size * Retry deadline for records #### Producer Buffer Size While a topic in in `MIGRATING` state, the agent will fence all produce requests with retryable errors. As a result, the producer’s record buffer can fill up. If it reaches maximum capacity, the client will either block until space frees up or fail with a timeout error. If your application cannot tolerate this, increase the client's buffer size. The exact size required depends on your workload and cutover plan. As a rule of thumb, size the buffer to hold all records submitted during the `MIGRATING` phase, computed as `peak produce rate * migration_timeout_seconds`. We recommend being conservative with the estimate and setting it 1.5X what you expect the buffer to hold. For example, if a producer’s throughput for a topic you plan to migrate is 1 MBps and the estimated migration duration is 20 seconds, then the buffer size needs to be at least 20MB. You also need to reserve enough buffer space for other topics that the producer writes to. To estimate the migration duration, you can look at the Orbit time lag for the topics you want to migrate, or simply use the `migration_timeout_seconds` you expect to set. In general, the Orbit time lag represents how long the migration phase will take. #### Retry Deadline Every producer client enforces a deadline, expressed either as a delivery timeout or as a maximum number of attempts, after which a record that has only ever received retryable errors is failed permanently. Once the retry deadline is passed, the record is lost and is not recoverable. If a producer’s retry budget is shorter than the migration time, then the record is lost. We recommend being conservative when setting the retry deadline and set it to a value that's higher than the `migration_timeout_seconds` in the Orbit config (defaults to 5m). if you scale `migration_timeout_seconds` to a higher value, you should bump the retry deadline to 1 minute beyond that. #### Client configs You should first tune the producers based on the [Tuning for Clients](/warpstream/kafka/configure-kafka-client/tuning-for-performance) page. Then you should set the following configs specifically for auto migration. **Librdkafka / confluent-kafka / confluent-kafka-javascript** | Producer Settings | Recommended Value | | ------------------------------ | ---------------------------------- | | `queue.buffering.max.messages` | depends on the workload throughput | | `queue.buffering.max.kbytes` | depends on the workload throughput | | `message.timeout.ms` | `360000` | | `message.send.max.retries` | `2147483647` (the default) | | `retry.backoff.max.ms` | `5000` | **Java Client** | Producer Settings | Recommended Value | | --------------------- | ---------------------------------- | | `buffer.memory` | depends on the workload throughput | | `delivery.timeout.ms` | `360000` (6m) | | `retries` | `MAX_INT` (default) | **Franz-go** | Producer Settings | Recommended Value | | ----------------------- | ---------------------------------- | | `MaxBufferedRecords` | depends on the workload throughput | | `MaxBufferedBytes` | `0` (unlimited - default) | | `RecordDeliveryTimeout` | unset (no timeout - default) | **Segment kafka-go** Kafka-go backs off *exponentially* between retries, starting at `WriteBackoffMin` and growing up to `WriteBackoffMax`, for up to `MaxAttempts` attempts. Since the ceiling is reached within a few attempts, the total time a record stays retryable is approximately `MaxAttempts * WriteBackoffMax`. To ensure records survive the 5 minute block timeout, we recommend leaving `WriteBackoffMax` at `1 second` and setting `MaxAttempts` to `360`, giving it 6 minutes to retry. | Producer Settings | Recommended Value | | ----------------- | ----------------- | | `MaxAttempts` | 360 | | `WriteBackoffMax` | `1s` (default) | **Sarama** Sarama performs a flat constant sleep on every retry, up to `Producer.Retry.Max` number of retries. The total time a record stays retryable is therefore `Producer.Retry.Max × Producer.Retry.Backoff`. To ensure records survive the 5 minute block phase, we recommend setting `Producer.Retry.Backoff` to 1 second and allowing up to 360 retries, which gives 6 minutes of headroom. | Producer Settings | Recommended Value | | ------------------------ | ----------------- | | `Producer.Retry.Max` | 360 | | `Producer.Retry.Backoff` | `1s` | ### Monitoring migration The Migration tab in the WarpStream console is the main place to watch a migration. It shows every Orbit-managed topic with a state badge (`REJECT` / `PROXY` / `MIGRATING` / `COMPLETE`), per-topic offset lag and time lag, and a state-summary bar with topic counts in each state. Similar data is available programmatically via the [Orbit Topic Auto Migration Status endpoint](/warpstream/reference/api-reference/orbit-auto-migration/status).
#### Metrics Key metrics to watch include: * **Orbit replication lag** (offset lag and time lag, per topic)— directly visible in the Migration tab. This is the single biggest input to how long a migration will take. If lag isn't dropping, the migration won't progress. See the [Orbit observability section](/warpstream/kafka/orbit#observability) for more details on how to monitor Orbit's offset and time lag. * **Source-cluster produce throughput vs. WarpStream destination throughput**— useful to confirm the proxy is forwarding everything and that Orbit is keeping up. * **WarpStream proxy throughput**— the agent emits metrics for the number of compressed bytes and number of records that were proxied to the source cluster. See the [Metrics page](/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs#auto-migration) for more details on the metrics. #### Diagnostics & Events The Migration tab also surfaces migration diagnostics— checks that flag potential issues during and after a migration. These include, among others: * **Unproxied source writes**— records are being written directly to the source cluster, bypassing the WarpStream proxy. This usually means a producer wasn't reconfigured to point at WarpStream, and those records risk being lost when the source is decommissioned. * **Post-migration source writes**— records landed on the source cluster *after* a topic finished migrating. Unlike the case above, these records are *not* replicated to WarpStream and will be lost when the source is decommissioned— a strong signal a producer is still pointing at the source. The tab also shows a feed of migration events— a per-topic lifecycle log covering initiation, completion, timeouts, and aborts. These events are stored as `orbit_logs` and are searchable in the [Events Explorer](/warpstream/reference/events). Migration events require [Events](/warpstream/reference/events#enabling-events) to be enabled on the cluster and at least one Agent with the [`jobs` role](/warpstream/kafka/advanced-agent-deployment-options/splitting-agent-roles) running **v829 or higher**. ### FAQ / Troubleshooting **A migration keeps timing out and rolling back to PROXY.** The cause is replication lag failing to reach zero within `migration_timeout_seconds`. There are three things to check: 1. **Is lag decreasing during migration but just not fast enough?** If so, increase `migration_timeout_seconds` to give Orbit a longer window to drain. 2. **Is Orbit replication itself the bottleneck?** Tune Orbit's concurrency and fetch settings to drain lag faster— see [Orbit performance tuning](https://docs.warpstream.com/warpstream/kafka/orbit/auto-migration?cache=1781287057). 3. **Is something keeping lag elevated?** The most common cause is a producer still writing directly to the source cluster, bypassing WarpStream— those writes during a migration keep lag from reaching zero. Audit producer configurations to confirm every producer for the topic has its `bootstrap.servers` pointed at WarpStream. Check that the "Orbit Auto Migration Unproxied Source Writes" diagnostic is not firing. **How can I make a migration finish faster?** Migration time is dominated by how quickly Orbit can drain replication lag to zero. Until lag drops below the topic's configured threshold, initiation is rejected; once a topic starts `MIGRATING`, it stays there until lag reaches zero (or `migration_timeout_seconds` elapses and it rolls back). The most useful lever is therefore Orbit itself— increasing Orbit's concurrency, tuning fetch sizes, and ensuring the agents replicating the topic aren't resource-constrained. See [Orbit performance tuning](https://docs.warpstream.com/warpstream/kafka/orbit#tuning-orbit) for the relevant settings. **Can I migrate one partition at a time?** No. Migration state is per-topic. All partitions of a topic must drain to zero lag before the topic can flip from `MIGRATING` to `COMPLETE`. **Can I roll a topic back from `COMPLETE`?** No. `COMPLETE` is terminal. # Tableflow Setup This page describes how to setup WarpStream Tableflow. ## Introduction Tableflow automates the tedious process of transforming a topic in an Apache Kafka-compatible data streaming system into an Apache Iceberg table. Instead of writing custom code and manually configuring a data pipeline for each table you want to build, Tableflow allows you to declaratively specify which topics to build tables from and what schema and data format to expect. When schemas inevitably need to change, you can update the schema in Tableflow's editor and WarpStream will handle the schema migration automatically. Compaction and table maintenance is included out-of-the-box with no tuning required. Tableflow continuously compacts the table in the background with intelligent heuristics to ensure readers get the best performance. Tableflow is available as Bring-Your-Own-Cloud (BYOC) where the compute and storage live inside your cloud account inside your VPC. The raw data for your table is only ever stored inside your object storage bucket and never leaves your VPC during the table ingestion and maintenance process. Tableflow maintains a metadata store inside WarpStream Cloud as the Iceberg metadata layer that is periodically synced into your object storage bucket. ## Getting Started To get started with Tableflow, you first need to create a Tableflow cluster from the WarpStream Console, or using one of [infrastructure-as-code](/warpstream/agent-setup/infrastructure-as-code) deployment options. The WarpStream Agents that join this cluster will only perform Tableflow operations and do not expose the Apache Kafka protocol. Please refer to our [other documentation](/warpstream/agent-setup/deploy) for how to install and configure the WarpStream Agents in your environment as the process does not differ for Tableflow. As part of deploying the Agents, you'll also need to setup and configure an object storage bucket and/or provide the Agents with access to one of your existing buckets. See our [object storage configuration documentation](/warpstream/agent-setup/different-object-stores) for more details on that. Once the Agents are running, you can open the Configuration table and start defining your source clusters, topics, tables, and schemas. ## Managed Tables Tableflow tables are fully managed by WarpStream, or what we call "managed tables". You cannot use another system for performing writes, compactions, or other table maintenance operations. This is in contrast to a connector-based approach where you would be forced to combine multiple distinct systems or operations together to implement all of these functions. ## Configuration Tableflow is configured with and is fully controllable from a single YAML file which can be edited through the WarpStream console or the [Pipelines API](/warpstream/reference/api-reference/pipelines). ### Overview Currently, there are two methods for defining schemas: * an `inline` mode which doesn't require using an external Schema Registry. This supports all schema types (JSON, Avro, Protobuf). The full schema must be fully contained in the YAML config: you cannot import external schemas with this mode. * a `schema_registry` mode which requires using an external Schema Registry. Protobuf and JSON schemas are supported from agent version `v813`+, Avro schemas are supported from agent version `v820`+. When using the Schema Registry, a schema may reference and import other schemas (as long as they are also registered in the Schema Registry), and Tableflow will resolve the final schema. An example YAML for the `inline` mode is the following: ```yaml source_clusters: - name: tableflow_cluster_1 bootstrap_brokers: - hostname: localhost port: 9092 - name: tableflow_cluster_2 bootstrap_brokers: - hostname: broker1.kafkaserver.com port: 9092 - hostname: broker2.kafkaserver.com port: 9092 credentials: sasl_username_env: SASL_USERNAME_ENV_VAR sasl_password_env: SASL_PASSWORD_ENV_VAR use_tls: true sasl_mechanism: plain tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_json_logs_topic source_format: json schema_mode: inline input_schema: | { "type": "object", "properties": { "environment": { "type": "string" }, "service": { "type": "string" }, "status": { "type": "string" }, "message": { "type": "string" } }, "required": ["environment", "service", "status", "message"] } - source_cluster_name: tableflow_cluster_2 source_topic: example_avro_events_topic source_format: avro schema_mode: inline input_schema: | { "type": "record", "name": "ExampleAvroEvent", "fields": [ { "name": "event_id", "type": "string" }, { "name": "user_id", "type": "long" }, { "name": "session_id", "type": "string" } ] } destination_bucket_url: s3://my-bucket-name ``` An example YAML for the `schema_registry` mode is the following: ```yaml source_clusters: - name: tableflow_cluster_1 bootstrap_brokers: - hostname: localhost port: 9092 - name: tableflow_cluster_2 bootstrap_brokers: - hostname: broker1.kafkaserver.com port: 9092 - hostname: broker2.kafkaserver.com port: 9092 credentials: sasl_username_env: SASL_USERNAME_ENV_VAR sasl_password_env: SASL_PASSWORD_ENV_VAR use_tls: true sasl_mechanism: plain schema_registries: - name: schema_registry_1 url: https://schema_registry_1.com:9094 credentials: username_env: SR_USERNAME password_env: SR_PASSWORD tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_protobuf_1_topic source_format: protobuf schema_mode: schema_registry schema_registry: name: schema_registry_1 subject: example-protobuf-1-value - source_cluster_name: tableflow_cluster_2 source_topic: example_protobuf_2_topic source_format: protobuf schema_mode: schema_registry schema_registry: name: schema_registry_1 subject: example-protobuf-2-value destination_bucket_url: s3://my-bucket-name ``` The YAML specifies * The source clusters Tableflow should connect to. * The schema registries Tableflow should connect to (for the `schema_registry` mode). * For each cluster, the topic that Tableflow should create Iceberg tables from. * For each topic, the schema to deserialize the Kafka records with, either as an inline schema, or as referenced by a subject of a schema registry. * The destination bucket to store the table. ### Configure Source Clusters Source clusters are the Apache Kafka-compatible systems like WarpStream that store the topics you'd like to convert to tables. You define clusters by giving them a name, a list of brokers, and credentials if they are needed. You define source clusters at the root of the configuration YAML. ```yaml source_clusters: - name: tableflow_cluster bootstrap_brokers: - hostname: broker.kafkaserver.com port: 9092 ``` You can define multiple source clusters so a single Tableflow cluster can centralize data from multiple clusters into one unified place. ### Configure Schema Registries Schema registries are the Confluent-compatible schema registries that store the schemas you'd like to use to deserialize the data from your topics. In `schema_mode: schema_registry`, Protobuf and JSON tables require agent `v813`+ and Avro tables require agent `v820`+. You define schema registries by giving them a name, a URL and credentials if they are needed. You define schema registries at the root of the configuration YAML. ```yaml schema_registries: - name: my_schema_registry url: https://schema-registry.example.com credentials: username_env: SR_USERNAME password_env: SR_PASSWORD use_tls: true ``` All credential fields reference environment variable names. The fields you may define for the credentials are the following: | Field | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `username_env` | Env var holding the basic-auth username. Must be set together with `password_env`. | | `password_env` | Env var holding the basic-auth password. Must be set together with `username_env`. | | `use_tls` | Enables TLS / mTLS for the registry connection. | | `mtls_client_cert_env` | Env var holding the path to the client certificate (PEM). Must be set together with `mtls_client_private_key_env`. | | `mtls_client_private_key_env` | Env var holding the path to the client private key (PEM). Must be set together with `mtls_client_cert_env`. | | `mtls_client_private_key_password_env` | Env var holding the private key password. Only allowed if `mtls_client_private_key_env` is also set. | | `server_ca_cert_env` | Env var holding the path to the server CA certificate (PEM). | Please note that for all those credentials, contrary to the [Configure Connection and Credentials section](https://docs.warpstream.com/warpstream/tableflow/tableflow#configure-connection-and-credentials) below, the agents do not automatically add a `TABLEFLOW_` prefix to the values of the fields before using the environment variables. So the environment variables in the Agent should be exactly the same as those stated in the configuration. You can define multiple schema registries. ### Configure Connection and Credentials If credentials are needed to connect to the Kafka cluster, the connection information can be provided under the `credentials` block for each cluster. {% hint style="info" %} Note that if the source Kafka cluster is a WarpStream cluster, credentials still need to be provided if authentication is required. This is different from the Managed Data Pipelines setup where credentials are injected automatically. {% endhint %} #### **TLS** ```yaml source_clusters: - name: tableflow_cluster ... credentials: use_tls: true tls_insecure_skip_verify: false ``` * `use_tls` specifies whether the Agents should use TLS when connecting to your source clusters. * `tls_insecure_skip_verify` specifies whether a client verifies the server's certificate chain and host name. #### **SASL** ```yaml source_clusters: - name: tableflow_cluster ... credentials: sasl_username_env: SASL_USERNAME_ENV_VAR sasl_password_env: SASL_PASSWORD_ENV_VAR sasl_mechanism: plain use_tls: true ``` * Both the `sasl_username_env` and the `sasl_password_env` fields refer to environment variable names. The Agents will append a `TABLEFLOW_` prefix to the values of the fields before using the environment variables, so the environment variables in the Agent should be configured as `TABLEFLOW_SASL_USERNAME_ENV_VAR` and `TABLEFLOW_SASL_PASSWORD_ENV_VAR` respectively. * The default value of `sasl_mechanism` is `plain`. Supported mechanisms include: `plain`, `scram-256`, and `scram-512`. #### **mTLS PEM encoded certs** ```yaml source_clusters: - name: tableflow_cluster ... credentials: mtls_client_cert_env: MTLS_CERT_PATH_ENV_VAR mtls_client_key_env: MTLS_KEY_PATH_ENV_VAR mtls_server_ca_cert_env: MTLS_SERVER_CA_CERT_PATH_ENV_VAR use_tls: true ``` * The `mtls_client_cert_env`, `mtls_client_key_env`, and `mtls_server_ca_cert_env` fields refer to environment variable names. The Agents will append a `TABLEFLOW_` prefix to the values of the fields before using the environment variables, so the environment variables in the Agent should be configured as `MTLS_CERT_PATH_ENV_VAR`, `MTLS_KEY_PATH_ENV_VAR`, and `MTLS_SERVER_CA_CERT_PATH_ENV_VAR` respectively. * `mtls_client_cert_env` specifies the environment variable that contains the path to the X.509 certificate file in PEM format. * `mtls_client_key_env` specifies the environment variable that contains the path to the X.509 private key file in PEM format. * `mtls_server_ca_cert_env` is optional and specifies the environment variable that contains the path to the X.509 certificate file in PEM format for the client certificate authority. "If not specified, the host's root certificate pool will be used for client certificate verification. ### Configure the Destination Bucket URL To specify where your table data should be stored, use the `destination_bucket_url` field at the root of the configuration YAML. This configures the default destination bucket URL for all tables. ```yaml destination_bucket_url: s3://bucket-name?region=us-east-1 ``` Alternatively, you can specify per-table bucket URL overrides within each tables configuration: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_json_logs_topic destination_bucket_url: s3://bucket-name-for-this-table?region=us-east-1 ``` {% hint style="warning" %} The destination bucket URL for a table can only be changed if a table is completely empty. This means that once your table has started ingesting data successfully, the destination bucket **cannot** be changed. {% endhint %} Check our [object storage configuration documentation](/warpstream/agent-setup/different-object-stores) for more details on how to configure this URL for various different cloud providers, as well as for a complete list of permissions that the Agents will require. Note that Tables will be created under the `/warpstream/_tableflow` path. Optionally, [a prefix can be specified in the bucket URL](/warpstream/agent-setup/different-object-stores#using-a-bucket-prefix), which will result in Tables being created under the `/prefix/warpstream/_tableflow` path. ### Configure Tables The next step is defining the tables to be created. An optional table name can be defined for each table and by default the table will be named `__`. ``` tables: - name: my_table ``` The table name is surfaced in the console, metrics/logs (requires Agent v823+), the REST catalog, and external catalog integrations. It can be renamed, but doing so can potentially break monitoring and any query paths that reference the old name. ### Source Kafka Topics Each table you define has exactly one source Kafka topic from a cluster listed under the [clusters configuration section](#configure-source-clusters). {% hint style="info" %} Tableflow currently supports append-only tables. If you ingest data from a compacted topic in the source cluster, rows will not be deduplicated and any tombstones may not comply with the schema. Support for compacted topics is coming soon. {% endhint %} To configure the source Kafka topic and how messages are decoded, the following fields are required: * `source_cluster_name`: the source cluster TableFlow connects to. This must match one of the clusters declared in the [Source Clusters](#configure-source-clusters) section. * `source_topic`: the topic TableFlow ingests records from. * `source_format`: the record format (`avro`, `json` or `protobuf`). * `schema_mode`: either `inline` or `schema_registry`. The `schema_registry` mode also requires a `schema_registry` section containing: * the `name` of the source schema registry. It must match the name of one source schema registries declared (see [Source Schema Registry](https://docs.warpstream.com/warpstream/tableflow/tableflow#configure-schema-registries-agent-v813) section) * the `subject` of that schema registry in which all versions of your schema will be registered. An example declaration for the `schema_registry` mode is the following: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_protobuf_events_topic name: my_events_table source_format: protobuf schema_mode: schema_registry schema_registry: name: my_schema_registry subject: example_protobuf_events_topic-value ``` As for the `inline` mode, the schema must be provided (see the [Schema Definitions](https://docs.warpstream.com/warpstream/tableflow/tableflow#schema-definitions) section). An example declaration for the `inline` mode is the following: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_json_logs_topic name: my_logs_table source_format: json schema_mode: inline input_schema: | { "type": "object", "required": ["entry_id", "date", "lines"], "properties": { "entry_id": { "type": "string" }, "date": { "type": "string", "format": "date" }, "description": { "type": "string" }, "reference": { "type": "string" }, "entry_type": { "type": "string", "enum": ["standard", "adjusting", "closing", "reversing"] }, "lines": { "type": "array", "items": { "type": "object", "required": ["account_code", "account_name", "debit", "credit"], "properties": { "account_code": { "type": "string" }, "account_name": { "type": "string" }, "debit": { "type": "number" }, "credit": { "type": "number" }, "department": { "type": "string" }, "cost_center": { "type": "string" } } } } } } - source_cluster_name: tableflow_cluster_2 source_topic: example_avro_events_topic source_format: avro schema_mode: inline input_schema: | { "type": "record", "name": "ExampleAvroEventsTopic", "fields": [ { "name": "event_id", "type": "string" }, { "name": "user_id", "type": "long" }, { "name": "session_id", "type": "string" }, { "name": "profile", "type": { "type": "record", "name": "Profile", "fields": [ { "name": "country", "type": "string" }, { "name": "language", "type": "string" } ] } }, { "name": "device", "type": { "type": "record", "name": "Device", "fields": [ { "name": "type", "type": "string" }, { "name": "os", "type": { "type": "record", "name": "DeviceOS", "fields": [ { "name": "name", "type": "string" }, { "name": "version", "type": "string" }, { "name": "vendor", "type": ["null", "string"] } ] } }, { "name": "browser", "type": { "type": "record", "name": "DeviceBrowser", "fields": [ { "name": "name", "type": "string" }, { "name": "version", "type": "string" } ] } }, { "name": "screen", "type": { "type": "record", "name": "DeviceScreen", "fields": [ { "name": "width", "type": "int" }, { "name": "height", "type": "int" }, { "name": "pixel_ratio", "type": "double" } ] } }, { "name": "model", "type": ["null", "string"] } ] } }, { "name": "cookies", "type": { "type": "array", "items": "string" } }, { "name": "event_attributes", "type": { "type": "map", "values": "string" } } ] } ``` Note that the Kafka Topic is read with an `IsolationLevel` of `read_committed`. It's currently not possible to change it. Please ask warpstream support if you are interested in creating Tableflow Tables from `read_uncommitted` data. ### Schema Definitions Tableflow uses schemas for two related but different purposes: * `input_schema` tells Tableflow how to decode records from the source Kafka topic. * `schema` defines the final Iceberg table schema that will be written and queried (the output schema). `schema_mode` controls where schema definitions come from: * `inline`: schema is declared directly in the Tableflow config. * `schema_registry`: schema is declared in an external Schema Registry. The guidance in this section is about how to declare schemas when using `schema_mode: inline`. For `input_schema`, provide a raw schema string in the same format as your source records: * If your records are JSON, provide a JSON Schema. * If your records are Avro, provide an Avro schema. * If your records are Protobuf, provide a `.proto` schema. #### Schema Definition for the `schema_registry` mode For the `schema_registry` mode, there is no need to declare a schema inline as it will be fetched from the external Schema Registry. However, each table requires a `schema_registry` section containing the `name` of the Schema Registry to connect to, and the `subject` that registers every schema version to fetch. ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_protobuf_events_topic source_format: protobuf schema_mode: schema_registry schema_registry: name: my_schema_registry subject: example_protobuf_events_topic-value ``` #### Recommended Default: Define Only `input_schema` In most cases, defining only `input_schema` is enough. Tableflow uses the input schema to decode source records and infer the table schema automatically. Use this pattern when: * The table should have the same logical shape as the source records. * You are not using transforms that rename, flatten, remove, or add fields. * You are happy with Tableflow choosing the corresponding table types for your input types. ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_events_topic source_format: json schema_mode: inline input_schema: | { "type": "object", "properties": { "event_id": { "type": "string" }, "user_id": { "type": "integer" }, "created_at": { "type": "string", "format": "date-time" } }, "required": ["event_id"] } ``` Here, `input_schema` is a raw JSON Schema string because `source_format` is `json`. Tableflow infers the Iceberg table schema from the input schema, so there is no need to declare a separate output schema. #### Example: Json Input Schema Supported types are: `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `string`, `uuid`, `fixed`, and `binary`. ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_events_topic source_format: json schema_mode: inline input_schema: | { "type": "object", "properties": { "event_id": { "type": "string" }, "user_id": { "type": "integer" }, "created_at": { "type": "string", "format": "date-time" } }, "required": ["event_id"] } ``` **Required and Nullable Fields (Agent v801+)** Fields listed in the `required` array are validated at decode time: if a required field is missing from the JSON record, the record is rejected. By default, all fields are optional. To declare a field as nullable, use a type array with `"null"`: ```json { "type": "object", "properties": { "event_id": { "type": "string" }, "user_id": { "type": "integer" }, "nickname": { "type": ["string", "null"] } }, "required": ["event_id", "nickname"] } ``` In this example, `event_id` is required and non-nullable — the record is rejected if it is missing or `null`. `nickname` is required and nullable — `{"nickname": null}` is accepted but `{}` (missing) is rejected. `user_id` is optional — it can be missing, `null`, or present. The combination of `required` and nullable controls validation as follows: | Configuration | Value present | `null` | Missing | | ------------------------------------------------------- | ------------- | -------- | -------- | | Optional (default) | accepted | accepted | accepted | | Required (`"required"` array) | accepted | rejected | rejected | | Required + Nullable (`"required"` + `["type", "null"]`) | accepted | accepted | rejected | {% hint style="info" %} In Iceberg, there is no distinction between nullable and optional — a column is either required or optional. Tableflow maps this as: a field is optional in Iceberg if it is not in the `required` array **or** if it is nullable. Requiredness and nullability are only distinguished at JSON decode time. {% endhint %} **Map keys in JSON** Note that map keys can only be `string` and are declared using `additionalProperties` in json inline schemas: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_events_topic source_format: json schema_mode: inline input_schema: | { "type": "object", "properties": { "event_id": { "type": "string" }, "user_id": { "type": "integer" }, "event_attributes": { "type": "object", "additionalProperties": {"type": "string"} }, }, "required": ["event_id"] } ``` The `additionalProperties` must contain a valid type and cannot be mixed with explicit fields. The following is **invalid:** ```json { "type": "object", "properties": { "data": { "type": "object", "properties": { "name": {"type": "string"} }, "additionalProperties": {"type": "string"} } } } ``` #### Example: Avro Input Schema Supported types are: `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `string`, `uuid`, `fixed`, and `binary`. Note that map keys can only be `string.` ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_avro_events_topic source_format: avro schema_mode: inline input_schema: | { "type": "record", "name": "Event", "fields": [ { "name": "event_id", "type": "string" }, { "name": "user_id", "type": "long" }, { "name": "amount", "type": ["null", "double"] } ] } ``` #### Example: Protobuf Input Schema (Agent v796+) When `source_format` is `protobuf`, set `wire_format` based on the payload encoding: * `raw`: protobuf binary payload with no prefix * `confluent`: Confluent wire format (magic byte + schema ID prefix) The list of supported types are: `boolean`, `int32`, `sint32`, `uint32`, `fixed32`, `sfixed32`, `int64`, `sint64`, `uint64`, `fixed64`, `sfixed64`, `float`, `double`, `string`, and `bytes` . ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_protobuf_events_topic source_format: protobuf wire_format: raw schema_mode: inline input_schema: | syntax = "proto3"; message Event { string event_id = 1; int64 user_id = 2; Status status = 3; } enum Status { UNKNOWN = 0; ACTIVE = 1; INACTIVE = 2; } ``` {% hint style="warning" %} Enum values are stored by name in Iceberg but identified by number on the wire. This has implications for schema evolution: * adding new enum values is safe * renaming enum values is **forbidden** in WarpStream's TableFlow because renaming would cause inconsistent data (old records would have old names, new records would have new names) * removing old enum values is safe. But note that if we decode a record whose number is not in the current schema, it will be stored in Iceberg as the number in string form (e.g. `"99"`) **Note:** WarpStream's TableFlow also validates that sibling enum fields have identical sets if they share any value name. This prevents accidental inconsistencies between different fields using the same enum type. {% endhint %} The Protobuf types are converted as follows to the Iceberg types: | Protobuf Type | Iceberg Type | Notes | | --------------------------- | --------------- | ------------------------------------------------------------------------ | | `boolean` | `boolean` | | | `int32` | `integer` | | | `sint32` | `integer` | | | `sfixed32` | `integer` | | | `uint32` | `decimal(10,0)` | Stored as decimal(10,0) to prevent overflow (max `uint32` > max `int32`) | | `fixed32` | `decimal(10,0)` | Stored as decimal(10,0) to prevent overflow (max `uint32` > max `int32`) | | `int64` | `long` | | | `sint64` | `long` | | | `sfixed64` | `long` | | | `uint64` | `decimal(20,0)` | Stored as decimal(20,0) to prevent overflow (max `uint64` > max `int64`) | | `fixed64` | `decimal(20,0)` | Stored as decimal(20,0) to prevent overflow (max `uint64` > max `int64`) | | `float` | `float` | | | `double` | `double` | | | `enum` | `string` | Stored as the enum value name | | `string` | `string` | | | `bytes` | `binary` | | | `message` | `struct` | | | `map` | `map` | | | `repeated` | `list` | | | `oneof` | `struct` | Converted to a struct where each option is an optional field | | `google.protobuf.Timestamp` | `timestamptz` | Nanosecond precision is truncated to microseconds (Iceberg limitation) | **Note:** Iceberg does not have unsigned integer types. To prevent overflow when storing large unsigned values: * `uint32` and `fixed32` are stored as `decimal(10,0)` (the max `uint32` number `4,294,967,295` has 10 digits) * `uint64` and `fixed64` are stored as `decimal(20,0)` (the max `uint64` number `18,446,744,073,709,551,615` has 20 digits) #### JSON Schema Features Currently Not Supported When using JSON `input_schema`, Tableflow currently rejects JSON Schema documents that use the following keywords: * `prefixItems` * `contains` * `patternProperties` * `dependentRequired` * `dependentSchemas` * `if` * `then` * `else` * `$defs` * `$ref` * `allOf` * `oneOf` * `anyOf` If any of these are present, schema conversion fails with an unsupported-feature validation error. ### When to Declare `schema` (Output Schema) Define `schema` only when you need control over the final table shape. Common reasons include: * You use transforms that change the record shape (rename, flatten, remove, or add fields). * You want full control over the table type chosen for a field (for example, storing a JSON number as a specific decimal precision and scale). * You want to control nullability explicitly. The input schema describes records before transforms. The output schema describes records after transforms. Conceptually: 1. Source Kafka record 2. → decoded using `input_schema` 3. → transformed using `transforms`, if configured 4. → written using `schema`, if configured 5. → otherwise written using the schema inferred from `input_schema` When you define `schema`, use a YAML object with a `fields` list. The types here are expressed using \[Iceberg types]\(nested structure explicitly) so do not define `schema` as a raw JSON Schema, Avro schema, or `.proto` string. Raw schema strings should only be used for `input_schema`. ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_cdc_topic source_format: avro schema_mode: inline input_schema: | { "type": "record", "name": "Envelope", "fields": [ { "name": "payload", "type": { "type": "record", "name": "Payload", "fields": [ { "name": "after", "type": { "type": "record", "name": "After", "fields": [ { "name": "field1", "type": "string" }, { "name": "field2", "type": "string" } ] } } ] } } ] } transforms: - transform_type: bento transform: | root.field1 = this.payload.after.field1 root.field2 = this.payload.after.field2 schema: fields: - name: field1 type: string - name: field2 type: string ``` In this example: * `input_schema` matches the nested source record and is a raw Avro schema string. * The transform flattens the record. * `schema` describes the flattened table as a YAML object. #### Output Schema Format When you define `schema`, use YAML: ```yaml schema: fields: - name: event_id type: string - name: amount type: decimal decimal_precision: 12 decimal_scale: 2 optional: true - name: metadata type: struct fields: - name: source type: string - name: version type: long ``` #### Supported Field Types **Primitive Types** Each field has a `name`, a `type`, and an optional `optional` flag (defaults to `false`): ```yaml - name: type: optional: true | false ``` Supported primitive types are [Iceberg primitive types](https://iceberg.apache.org/spec/#parquet): `int`, `long`, `float`, `double`, `string`, `boolean`, `date`, `timestamp`, `timestamptz`, `uuid`, and `binary`. Parameterized types are also supported: * `decimal` — requires `decimal_precision` (1–38) and `decimal_scale` (0 to precision): ```yaml - name: amount type: decimal decimal_precision: 12 decimal_scale: 2 ``` * `fixed` — requires `fixed_length`: ```yaml - name: checksum type: fixed fixed_length: 16 ``` **Struct** A `struct` is a tuple of named, typed fields. Fields inside a struct follow the same rules as top-level fields and can be of any type, including other structs: ```yaml - name: address type: struct fields: - name: street type: string - name: city type: string - name: zip type: string optional: true ``` **Map** A `map` is a key/value pair. The key field and value field are defined as child fields. Map keys are required; map values can be optional. Keys must be of type `string`: ```yaml - name: headers type: map fields: - name: key type: string - name: value type: string optional: true ``` **List** A `list` contains a single child field named `element`. Elements can be of any type: ```yaml - name: tags type: list fields: - name: element type: string ``` Lists of structs are also supported: ```yaml - name: line_items type: list fields: - name: element type: struct fields: - name: product_id type: string - name: quantity type: int - name: price type: decimal decimal_precision: 10 decimal_scale: 2 ``` #### Field IDs Are Deprecated Do not define field IDs in new Tableflow schemas. Tableflow assigns and manages field IDs internally. User-defined field IDs are deprecated and will be ignored. Existing legacy examples that include `id` should not be copied into new configurations. This applies to table schema field IDs only. Protobuf field numbers inside a `.proto` `input_schema` are still part of the Protobuf schema and should remain there.
Deprecated schema definitions This section contains documentation for the old way of declaring schemas definitions in tableflow. This was cumbersome because it required you to manually specify field-ids and used a schema definition that didn't necessarily map 1 to 1 with the type used to store your data. We still support it for backward compatibility but you shouldn't use it. As shown in the above example, schemas specified with the `inline` mode contain a list of fields. Each field is named and has a unique integer id that will be mapped to the field ID for your Iceberg table as well as a type that will be used as the Iceberg date type for the corresponding column. **Primitive Types** The syntax for defining a primitive field looks like the following: ```yaml - { name: , id: , type: , optional: { true | false } } ``` where `field-type` is one of the supported fields for your input record type (refer to the Protobuf / Avro / Json sections above). **Nested Types** **For Avro and JSON only** A `struct` is specified as a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. Fields can be of any type. ```yaml - name: id: type: struct fields: - { name: , id: , type: , optional: { true | false} } - { name: , id: , type: , optional: { true | false} } - { name: , id: , type: , optional: { true | false} } ``` **For Avro, JSON and Protobuf** A `map` is specified as a key/value pair. Both the key field and value field have an integer id that is unique in the table schema. While map values can be either optional or required, map keys are required. For both Avro and JSON schemas, map keys can only be of type `string`, but values can be of any type. For Protobuf, map keys can be of any map key type allowed by the Protobuf specs, namely: `string`, `int32`, `int64`, `uint32`, `uint64`, `sint32`, `sint64`, `sfixed32`, `sfixed64`, `fixed32`, `fixed64`, `boolean`. ```yaml - name: id: type: map fields: - { name: , id: , type: } - { name: , id: , optional: { true | false} } ``` Note that for Protobuf, the `proto_field_number` must be added to the map field but should not be set for the key nor value. So a full example for a Protobuf map is: ```yaml - name: id: proto_field_number: type: map fields: - { name: , id: , type: } - { name: , id: , optional: { true | false} } ``` A `list` (in JSON and Avro) or `repeated` (in Protobuf) is specified with a single element field. The element field is named `element` and has an integer id that is unique in the table schema. Elements can be either optional or required and can be of any type. ```yaml - name: id: type: list fields: - { name: element, id: , type: } ``` Note that for Protobuf, the `proto_field_number` must be added to the repeated field but should not be set for the element. So a full example for a Protobuf repeated is: ```yaml - name: id: proto_field_number: type: repeated fields: - { name: element, id: , type: } ``` **For Avro only** AVRO's binary encoding does not include field names nor type information and instead values are concatenated strictly based on the schema. Consequently, the decoder interprets the stream of bytes strictly according to the sequence defined in the schema. **You must list fields in your table schema in the same order as in the producer’s AVRO schema.** Reordering fields (e.g. by id or by logical group) can cause decode failures such as `avro: ReadBool: invalid bool` or `unexpected EOF`. If you see these errors, compare your schema field order to the producer’s (e.g. the `.avsc` file or the schema in your schema registry) and align the order. **For Protobuf only** A `message` is defined using `type: message`. A `proto_field_number` must be provided for every nested field. ```yaml - name: id: proto_field_number: type: message fields: - { name: , id: , proto_field_number: , type: , optional: { true | false} } - { name: , id: , proto_field_number: , type: , optional: { true | false} } - { name: , id: , proto_field_number: , type: , optional: { true | false} } ``` An `enum` is stored as a string in the Iceberg table using the enum value name. You must define all enum values with their corresponding numbers. ```yaml - name: id: proto_field_number: type: enum enum_values: - { name: , number: <0> } - { name: , number: <1> } - { name: , number: <2> } ``` {% hint style="warning" %} Enum values are stored by name in Iceberg but identified by number on the wire. This has implications for schema evolution: * adding new enum values is safe * renaming enum values is **forbidden** in WarpStream's TableFlow because renaming would cause inconsistent data (old records would have old names, new records would have new names) * removing old enum values is safe. But note that if we decode a record whose number is not in the current schema, it will be stored in Iceberg as the number in string form (e.g. `"99"`) **Note:** WarpStream's TableFlow also validates that sibling enum fields have identical sets if they share any value name. This prevents accidental inconsistencies between different fields using the same enum type. {% endhint %} A `oneof` field is defined with `type: oneof`. Each option must be explicitly set to optional and with a `proto_field_number`, but the oneof field itself must be defined as required and without any `proto_field_number`. ```yaml - name: id: type: oneof fields: - { name: , id: , proto_field_number: , type: , optional: true } - { name: , id: , proto_field_number: , type: , optional: true } - { name: , id: , proto_field_number: , type: , optional: true } ```
### Field Type Remapping (Agent v796+) {% hint style="info" %} **Requires Agent v796 or higher.** {% endhint %} Field type remapping lets you override specific Iceberg column types in the inferred table schema without declaring a full output `schema`. Use it when the input schema is correct for decoding, but the default Iceberg type is not the type you want to expose in the table. This is also useful when Tableflow infers a wider type than you need (for example, `long` when `int` is sufficient) and you want tighter typing for storage efficiency or downstream compatibility. For example, if your JSON producer serializes numeric IDs as strings, you can keep the input schema as `string` (so decoding works) but store the column as `long` in Iceberg: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_events_topic source_format: json schema_mode: inline input_schema: | { "type": "object", "properties": { "user_id": { "type": "string" }, "event_name": { "type": "string" } } } field_type_remapping: - path: user_id type: long ``` Here, `user_id` is decoded as a JSON string but the Iceberg table column is created as `long`. The input schema is not modified. #### Requirements * `schema_mode` must be `inline`. * No explicit output `schema` can be defined. Remapping only applies when Tableflow infers the table schema from `input_schema`. #### Syntax Each entry specifies a `path` (dot-separated field path) and a `type` (Iceberg primitive type): ```yaml field_type_remapping: - path: type: ``` Nested fields use dot notation: ```yaml field_type_remapping: - path: device.screen.width type: int ``` For lists and maps, use the internal child names from the schema conversion. For arrays use `element` and for maps use `key` and `value`: ```yaml field_type_remapping: - path: page_views.element.time_on_page_ms type: int - path: event_attributes.value type: binary ``` Only scalar leaf fields can be remapped. Struct, list, map, and other container types cannot be remapped. #### Supported Target Types The target `type` must be a valid [Iceberg primitive type](https://iceberg.apache.org/spec/#parquet). Simple types include: `int`, `long`, `float`, `double`, `string`, `boolean`, `date`, `timestamp`, `timestamptz`, `uuid`, and `binary`. Parameterized types are also supported: ```yaml field_type_remapping: - path: amount type: decimal(12, 2) - path: checksum type: fixed[16] ``` Decimal precision must be between 1 and 38, and scale must be non-negative and no greater than precision. #### Common Use Cases **String to date:** ```yaml input_schema: | { "type": "object", "properties": { "event_date": { "type": "string" } } } field_type_remapping: - path: event_date type: date ``` Valid input: `{ "event_date": "2026-05-18" }`. Date strings must be in `YYYY-MM-DD` format. **String to decimal:** ```yaml input_schema: | { "type": "object", "properties": { "price": { "type": "string" } } } field_type_remapping: - path: price type: decimal(10, 2) ``` Valid input: `{ "price": "123.45" }`. #### Important Considerations {% hint style="warning" %} * Field type remapping does not change how source records are decoded. Decoding always uses `input_schema`. * Paths must match resolved schema field names exactly. Missing or duplicate paths are rejected. * Changing an existing table's columns type in a non-widening manner is an incompatible Iceberg schema change. Use `recreation_key` to trigger a table rebuild when intentionally changing column types. * Decimal scale handling truncates extra fractional digits rather than rounding. If exact decimal precision matters, prefer using a string field type. * Config validation confirms the target is a valid table schema type, but runtime success depends on whether actual values can be converted. For example, JSON `"123"` can become `int`, but `"not-a-number"` will fail. {% endhint %} ### Partitioning Tableflow supports unpartitioned, timestamp partitioned tables using the record timestamp, and custom partitioning. Tables are unpartitioned by default. To change the partition spec of a table, simplify deploy an updated configuration with the new spec and Tableflow will handle the partition evolution. The partitioning scheme is specified on a per-table basis. For convenience the `partitioning_scheme` option can be used to define unpartitioned tables and timestamp partitioned tables using the record timestamp. Supported values for `partitioning_scheme` include `unpartitioned`, `hour`, `day`, `month`, and `year`. ```yaml tables: - source_topic: example_json_logs_topic ... partitioning_scheme: hour ... ``` If a custom partitioning scheme is needed, then the `custom_partitioning` option can be used as follows: ```yaml tables: - source_topic: example_json_logs_topic ... custom_partitioning: - source_field_path: name: transform: { name: } - source_field_path: name: transform: { name: } ... ``` | Field | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `source_field_path` | **The Input Column**. The input field to partition on. Use the field path from the source schema, for example `user_id`, `created_at`, or `customer.address.country`. For WarpStream metadata fields, use paths like `warpstream.timestamp` or `warpstream.partition` | | `name` | **The Partition Alias**. This is the name given to the partition itself. It does not have to match the source field name. It must start with a letter and contain only alphanumerics or underscores. | | `transform` | An object defining the transform to be applied to the source column to produce a partition value. | The `transform` object requires a `name` and, depending on the type, additional parameters: * Time-based: `year`, `month`, `day`, `hour`. * `bucket`: Requires an `n` field (e.g., `{ name: "bucket", n: 16 }`) to specify the number of buckets. * `truncate`: Requires a `w` field (e.g., `{ name: "truncate", w: 10 }`) to specify the width of the truncation. * `identity`: Uses the source value as-is. {% hint style="info" %} Note that to use custom partitioning the Agent version needs to be at least v748. {% endhint %} For example, to create hourly partitions on the Kafka timestamp and bucket the Kafka partitions into fours bins the configuration would look like this: ```yaml tables: - source_topic: example_json_logs_topic ... custom_partitioning: - source_field_path: warpstream.timestamp name: timestamp_hour transform: { name: hour } - source_field_path: warpstream.bucket name: partition_bucket transform: { name: bucket, n: 4 } ... ``` ### Sorting {% hint style="warning" %} **Requires Agent v810 or higher.** Every Agent in the cluster must be on a supported version before you configure sorting. An Agent that is too old cannot create a sorted table, so enabling sorting while any of the cluster's Agents are still on an older version will break ingestion for that table until all of them are upgraded. {% endhint %} Tableflow can physically sort the records within each table's data files by one or more fields. Sorting clusters similar values from a sort field together within a file, which improves compression and allows query engines to prune data files and Parquet row groups. This can significantly speed up queries that filter or range-scan on the sort fields. Sorting is opt-in: a table is sorted only when you set the `sorting:` option and is left unsorted otherwise. `sorting:` is an ordered list of fields. The order is significant: records are sorted by the first field, ties are broken by the second field, and so on. ```yaml tables: - source_topic: example_json_logs_topic ... sorting: - field_path: warpstream.timestamp direction: asc null_order: nulls_last transform: { name: day } - field_path: user_id direction: desc ... ``` | Field | Description | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `field_path` | **(Required)** The dot-separated path of the field to sort by, taken from the table schema (for example `user_id` or `customer.address.country`). WarpStream metadata fields such as `warpstream.timestamp` are also valid. | | `direction` | The sort direction. One of `asc` (default) or `desc`. | | `null_order` | Where null values are placed. One of `nulls_last` (default) or `nulls_first`. | | `transform` | An optional transform applied to the field before sorting. Supports the same transforms as [custom partitioning](#partitioning): `identity` (default), `year`, `month`, `day`, `hour`, `bucket` (requires `n`), and `truncate` (requires `w`). | Only primitive fields can be used as sort fields. Supported types are `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `string`, `binary`, `uuid`, and `fixed`. `struct`, `list`, and `map` container fields cannot be sorted directly; reference a primitive leaf field with dot notation instead. {% hint style="warning" %} **Decimal sort fields require Agent v822 or higher.** {% endhint %} A table can have at most **5** sort fields. Every additional field also adds ordering work during ingestion and compaction, so sort only by the fields you actually query on. Contact support if you need a higher limit. {% hint style="info" %} Sorting only affects newly written data; existing data is not rewritten to match a new sort order. This applies both when you first enable sorting and when you later change a table's sort fields. {% endhint %} {% hint style="warning" %} Enabling sorting sends the minimum and maximum value of each sort field to the WarpStream control plane, to be used for Iceberg metadata generation. See [Security and Privacy Considerations](/warpstream/reference/security-and-privacy-considerations) for the full list of metadata WarpStream stores. {% endhint %} ### Column Statistics {% hint style="warning" %} **Requires Agent v810 or higher.** As with sorting, every Agent in the cluster must be on a supported version before you configure `statistics:` on additional columns, or ingestion for that table will break until all of them are upgraded. {% endhint %} Tableflow can track per-file minimum and maximum value bounds for specific columns. These bounds let query engines prune data files that cannot match a query's filter, which accelerates selective queries on the tracked columns. Use the `statistics:` option to list the columns to track, by dot-separated field path: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_events_topic source_format: json schema_mode: inline input_schema: | { "type": "object", "properties": { "user_id": { "type": "integer" }, "country": { "type": "string" }, "created_at": { "type": "string", "format": "date-time" } } } statistics: - user_id - country ``` Tableflow always tracks statistics for the built-in `warpstream.timestamp` and `warpstream.partition` fields, as well as for every field used in the table's [`sorting:`](#sorting) configuration. Use `statistics:` to track additional columns. The supported column types are the same as for [sorting](#sorting): primitive fields only (`boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `string`, `binary`, `uuid`, and `fixed`). Container fields (`struct`, `list`, and `map`) cannot be tracked directly. {% hint style="warning" %} **Decimal column statistics require Agent v822 or higher.** {% endhint %} A table can track statistics for at most **10** columns. Each tracked column also adds to the per-file metadata Tableflow maintains, so track only the columns you filter on. Contact support if you need a higher limit. Bounds for `string` and `binary` columns are stored as short truncated prefixes rather than full values, so pruning on those columns is conservative: a query may scan some files that do not actually match, but it never skips a file that could. {% hint style="warning" %} Enabling column statistics sends the minimum and maximum value of each tracked field to the WarpStream control plane, to be used for Iceberg metadata generation. See [Security and Privacy Considerations](/warpstream/reference/security-and-privacy-considerations) for the full list of metadata WarpStream stores. {% endhint %} {% hint style="info" %} The per-file minimum and maximum bounds are written into the table's Iceberg metadata, so external query engines (not just WarpStream) can use them to skip files that cannot match a query's filter. {% endhint %} ### Transforms (Agent v730+) Tableflow supports applying stateless transformations to ingested records. This can be helpful to massage the data into the desired shape before inserting it into the table without having to reprocess the data into a completely different topic first. For example, imagine the topic contains a [CDC change stream from debezium](https://debezium.io/documentation/reference/stable/integrations/serdes.html) that looks like the following: ```json { "schema": {...}, "payload": { "op": "u", "source": { ... }, "ts_ms" : "...", "ts_us" : "...", "ts_ns" : "...", "before" : { "field1" : "oldvalue1", "field2" : "oldvalue2" }, "after" : { "field1" : "newvalue1", "field2" : "newvalue2" } } } ``` Without a transform, the table schema would have to be defined as follows: ```yml source_format: json input_schema: | { "type": "object", "properties": { "payload": { "type": "object", "properties": { "after": { "type": "object", "properties": { "field1": { "type": "string" }, "field2": { "type": "string" } } } } } } } ``` This is unfortunate because users querying the data would always have to write their queries in the form: `SELECT payload.after.field1` instead of simply `SELECT field1`. Transforms solve this problem by rewriting the structure of the record before applying the table schema. Define `input_schema` to match the source records, apply transforms to reshape the data, and define `schema` as a YAML object to describe the final table shape. For a JSON workload: ```yaml source_format: json input_schema: | { "type": "object", "properties": { "payload": { "type": "object", "properties": { "after": { "type": "object", "properties": { "field1": { "type": "string" }, "field2": { "type": "string" } } } } } } } transforms: - transform_type: bento transform: | root.field1 = this.payload.after.field1 root.field2 = this.payload.after.field2 schema: fields: - name: field1 type: string - name: field2 type: string ``` For an Avro workload, `input_schema` is especially important because Avro records cannot be deserialized without a schema. The same pattern applies: `input_schema` is a raw Avro schema matching the source records, and `schema` is a YAML object describing the post-transform table: ```yaml source_format: avro input_schema: | { "type": "record", "name": "InputRecord", "fields": [ { "name": "payload", "type": { "type": "record", "name": "Payload", "fields": [ { "name": "after", "type": { "type": "record", "name": "After", "fields": [ { "name": "field1", "type": "string" }, { "name": "field2", "type": "string" } ] } } ] } } ] } transforms: - transform_type: bento transform: | root.field1 = this.payload.after.field1 root.field2 = this.payload.after.field2 schema: fields: - name: field1 type: string - name: field2 type: string ``` In summary, records before transforms must match `input_schema` and records after transforms must match `schema`. If `schema` is not defined, Tableflow infers the table schema from `input_schema`. If transforms only normalize values without changing field names, nesting, or types, you may still be able to omit `schema` and rely on inference. Separately, keep in mind that transforms can be chained: ```yaml transforms: - transform_type: bento transform: | root.field1 = this.payload.after.field1 - transform_type: bento transform: | root.field2 = this.payload.after.field2 ``` Tableflow transformations are executed by running arbitrary Bento Bloblang programs, but are limited to "pure" Bloblang functions that have no external side-effects. Bloblang is a rich turing-complete programming language with many [features](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/about), [functions](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/functions), [methods](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/methods), [conditionals](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/walkthrough#conditionals), and even [error-handling](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/walkthrough#error-handling). You can read more about Bloblang and its capabilities in the [Bento Bloblang documentation](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/about), but but the basics are quite straightforward and can be grasped with a few examples. The key thing to understand about Bloblang transformations is that they're mapping functions that mutate the input record into the desired shape. Within the context of a Tableflow Bloblang mapping, the `this` keyword refers to the input record and the `root` keyword refers to the output record. See the examples below to learn how to perform the most common transformations. {% hint style="success" %} The Bento website has a [powerful and interactive Bloblang playground](https://warpstreamlabs.github.io/bento/docs/guides/bloblang/playground) that can be used to experiment with Bloblang mapping programs. {% endhint %} In `schema_mode: schema_registry`, bento transforms are supported. To branch on writer schema metadata during transform execution, use `warpstream.sr.*` fields (requires agent `v817`+; see [Schema Registry mode](#schema-registry-mode)). #### Rename a field ```python root.new_field = this.old_field ``` #### Delete a field ```python root.unwanted_field = deleted() ``` #### Add a field ```python root.uppercase_name = this.name.uppercase() ``` #### Drop / filter out an entire record ```python if this.name == "foo" { root = deleted() } else { root.name = this.name.uppercase() } ``` #### Type Conversions ```python this.user_age = this.user_age.string() ``` ### Data Retention and TTL By default, data is retained in the table indefinitely. Optionally, a retention period can specified using the `retention_ttl` field. Retention must be expressed in units of hour (`h`). 
tables:
    - source_topic: example_json_logs_topic
      retention_ttl: 720h ## 30d
      ...
### Starting ingestion at the latest offsets By default, data is ingested from the start of the topic. You can optionally override this setting to instead start at the latest offsets (skipping any data previously stored in your input topic) via the `start_ingestion_at` field: 
tables:
    - source_topic: example_json_logs_topic
      start_ingestion_at: latest
      ...
The supported values are: * `latest` * `earliest` ### Dead Letter Queue (DLQ) Mode {% hint style="info" %} **Requires Agent v737 or higher.** {% endhint %} By default, Tableflow stops ingestion when it sees records that are incompatible with the provided schema to avoid head of line blocking. This behavior can be overridden using the `dlq_mode` field. Supported values include: * `stop`, which blocks ingestion upon encountering an invalid record (*this is the default*) * `skip`, which skips invalid records during ingestion * `keep`, (**Requires Agent v792 or higher**) which parks invalid records to an internal topic that can then be re-ingested. You can fix those records using transforms if they were not complying with the table schema for instance. 
tables:
    - source_topic: example_json_logs_topic
      dlq_mode: stop
      ...
#### DLQ Mode Keep settings {% hint style="info" %} **Requires Agent v792 or higher.** {% endhint %} Setting the DLQ mode to `keep` ensures continuous ingestion from your main topic even when some records cannot be processed. Those records will be sent to an internal topic so you can then manually trigger ingestion for from the UI.
You can control this behavior via the `dlq_keep_settings` settings in your table configuration (required when setting the mode to `keep`). `dlq_keep_settings` supports: * `retention`: how long records are stored in the DLQ topic (for instance `10h` or `5d`) * `dlq_replay_mode`: the mode used when replaying traffic from the DLQ (must be `skip` or `stop`) * `skip`: ignores records that fail ingestion again during the replay * `stop`: halts the entire replay process if any DLQ record fails to ingest * `circuit_breaker`: controls what happens when too many records are being DLQ'ed * `on_open`: defines the fallback behavior if the circuit breaker trips. Supported values are `skip` or `stop`. When triggered, this value overrides the `keep` behavior with this new setting. * `last_n_records`: the only circuit breaker policy allowed for now * `count`: the number of consecutive invalid records the system must encounter before tripping the circuit breaker and triggering the `on_open` logic. Here's an example of a complete DLQ `keep` configuration 
tables:
    - source_topic: example_json_logs_topic
      dlq_mode: "keep"
      dlq_keep_settings:
        retention: "24h"
        # Will stop the replay if we fail to ingest any DLQ record while replaying the data
        dlq_replay_mode: "stop"
        circuit_breaker:
          # Will stop if we write more than 50 records in the DLQ topic
          on_open: "stop"
          last_n_records:
            count: 50
      ...
#### Re-processing DLQ records As mentioned in the previous section, if you have some records that were written to the DLQ, you can go into the UI and manually replay them. However, if your records were put into the DLQ it often means that there was an issue ingesting them, which could be because of an invalid input or table schema. If this happens you can write a transform and scope it to your DLQ records, for instance if your `session_id` field was wrong you could do: 
tables:
    - source_topic: example_json_logs_topic
      transforms:
      - transform_type: bento
        transform: |
          let is_dlq_replay = this.warpstream.is_dlq

          if $is_dlq_replay {
            session_id = "fixed_by_dlq_replay"
          }
      ...
Records coming from the DLQ are automatically injected with metadata fields that you can access within your transforms. | Field | Description | | ----------------------------------- | ----------------------------------------------------------------- | | `warpstream.dlq_timestamp` | The exact time the record was written to the DLQ. | | `warpstream.dlq.failure_reason` | A brief error message explaining why the record failed ingestion. | | `warpstream.dlq.original_offset` | The offset of the record in the source topic. | | `warpstream.dlq.original_partition` | The partition of the record in the source topic. | | `warpstream.dlq.source_topic` | The name of the source topic. | ### Compression codecs {% hint style="info" %} **Requires Agent v748 or higher.** {% endhint %} Tableflow supports a few compression codecs for the stored data files. The default one is `snappy` . This codec can be overridden using the `compression` field. Supported values include: * `snappy` * `gzip` * `lz4` * `zstd` * `brotli` * `none` To disable compression 
tables:
    - source_topic: example_json_logs_topic
      compression: zstd
      ...
### Skipping raw record values {% hint style="info" %} **Requires Agent v749 or higher.** {% endhint %} If you don't need to access the raw record values (the ones coming from the kafka topics) you can set the `skip_raw_record_values` to true in your config. This will result in smaller data files. 
tables:
    - source_topic: example_json_logs_topic
      skip_raw_record_values: true
      ...
### Handling topic re-creation To define the table ingestion behavior when the source topic is recreated, use the `topic_recreation_policy` setting.\ \ Currently, the only supported policy is `recreate_table`. This ensures data integrity by creating a new table (with a different identifier) whenever the system detects that the source topic has been re-created. 
tables:
    - source_topic: example_json_logs_topic
      topic_recreation_policy: recreate_table
      ...
### Pausing a Table To temporarily stop ingestion for a specific table without removing it from your configuration, set `paused` to `true`: ```yaml tables: - source_topic: example_json_logs_topic paused: true ... ``` To resume, set `paused: false` (or remove the field) and deploy the updated configuration. Ingestion will pick up from where it left off. ### Schema Registry mode Tableflow supports Protobuf and JSON schemas from an external Confluent-compatible Schema Registry from agent version `v813`+ and Avro schemas from agent version `v820`+. In `schema_mode: schema_registry`, records are expected to be serialized with Confluent's wire format. You may omit `wire_format` (defaults to Confluent) or set `wire_format: confluent` explicitly. A config using the `schema_registry` mode requires some specific sections: * a `schema_registries` section (see [Configure Source Schema Registry](https://docs.warpstream.com/warpstream/tableflow/tableflow#configure-schema-registries-agent-v813)). * inside each `tables` section, a `schema_registry` subsection that contains the `name` of the source schema registry and the `subject` in which schema versions are registered. An example config for that mode is the following: ```yaml source_clusters: - name: tableflow_cluster_1 bootstrap_brokers: - hostname: localhost port: 9092 schema_registries: - name: schema_registry_1 url: https://schema_registry_1.com:9094 credentials: username_env: SR_USERNAME password_env: SR_PASSWORD tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_protobuf_1_topic source_format: protobuf schema_mode: schema_registry schema_registry: name: schema_registry_1 subject: example-protobuf-1-value - source_cluster_name: tableflow_cluster_1 source_topic: example_json_events_topic source_format: json schema_mode: schema_registry schema_registry: name: schema_registry_1 subject: example-json-events-value destination_bucket_url: s3://my-bucket-name ``` #### Important requirements and recommendations for use in Schema Registry mode There are some requirements for the schema registry integration to work properly: * The Schema Registry must be [Confluent-compatible](https://docs.confluent.io/platform/current/schema-registry/index.html), so that we can query it. * The records must have been serialized using a [Confluent-compatible serializer](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html), so that we can decode their headers. In the Schema Registry mode, the ingested records are always deserialized using the schema they were serialized with and written to Parquet files using the latest schema that was synced (which is the last registered schema in the subject). This means that we need to convert the record in the writer schema version to the latest schema version. This step requires for every new version to be backwards-compatible with any existing version used by a producer. Therefore, we recommend setting your schema registry to have a [compatibility level](https://docs.confluent.io/platform/current/schema-registry/develop/api.html#compatibility-concepts) set to `BACKWARD_TRANSITIVE` to guarantee that a record produced with any schema version will be convertible to the latest version. That being said, you may relax this configuration to only `BACKWARD` compatibility level if you always produce with the two most recent schema versions for example. {% hint style="warning" %} If the latest schema version synced by TableFlow is not backwards-compatible with the schema used to produce a record, we may DLQ that record or decode it incorrectly. {% endhint %} Any schema evolution must be [Iceberg backwards-compatible](https://docs.warpstream.com/warpstream/tableflow/tableflow#schema-migrations). In the Schema Registry mode, contrary to the inline mode, that check is made asynchronously after fetching the latest version from the schema registry. The same is true for checking that some sections of the config (like partitioning, sorting, ...) are compatible with that schema. It's preferable that you wait until your new schema has been synced (and thus that we have made sure it is valid) before you start producing records with it. {% hint style="info" %} Our recommended flow when deploying a schema change is the following: * Register your new version in the Schema Registry * This new schema version will be processed by TableFlow: checking that it is Iceberg backward-compatible and that it's compatible with the rest of the config (partitioning, sorting, ...) * Once that version has been validated and synced, you can start upgrading your producers to use that new version. To that end, we also highly recommend that you disable auto schema registration on your serializer (see [docs](https://docs.confluent.io/platform/current/schema-registry/security/index.html#disabling-auto-schema-registration)). {% endhint %} {% hint style="warning" %} If you do not follow our recommended flow, the following may happen: * If you start producing records with a schema version newer than the version that was synced, ingestion of those records will be transiently paused until we have synced a newer version. * If you start producing records with a schema version newer than the version that was synced AND it turns out the version cannot be synced because it is invalid (for example not backwards-compatible), ingestion of these records will be transiently paused until you have submitted a new version that is valid. {% endhint %} #### Using transforms with the schema registry mode In `schema_mode: schema_registry`, bento transforms are supported. During transform execution, writer schema metadata is available under `warpstream.sr` (these fields are not stored in the Iceberg table): {% hint style="warning" %} **Requires Agent v817 or higher.** To use `warpstream.sr.writer_schema_id` and `warpstream.sr.writer_schema_version` in your Bento transform logic every Agent in the Tableflow cluster must be on at least v817. {% endhint %} | Field | Description | | ------------------------------------- | ---------------------------------------------------- | | `warpstream.sr.writer_schema_id` | Confluent schema ID used to produce the record | | `warpstream.sr.writer_schema_version` | Schema Registry subject version of the writer schema | Example table config with a transform that tags records based on writer schema version: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_json_events_topic source_format: json schema_mode: schema_registry schema_registry: name: schema_registry_1 subject: example-json-events-value transforms: - transform_type: bento transform: | root = this root.era = if this.warpstream.sr.writer_schema_version < 3 { "legacy" } else { "current" } ``` From agent version `v830`+, you can also define a `schema` inline. For example: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_json_events_topic source_format: json schema_mode: schema_registry schema_registry: name: schema_registry_1 subject: example-json-events-value transforms: - transform_type: bento transform: | root = this root.field1 = this.field1 + 10 root.field2 = "my_new_field" schema: fields: - name: field1 type: int - name: field2 type: string ``` {% hint style="warning" %} Note that the `schema` defined inline will **always** be the one used to write parquet files. The schema used for decoding records remains the schema whose ID is stored in each record's header. This means that if the schemas you encode your records with evolve, you are responsible for keeping your `transforms` and `schema` defined inline in sync with the decoding schemas. {% endhint %} ### Table Schema Tableflow creates an Iceberg table with a struct schema, containing all the fields from the configured schema as well as the following default fields: | Name | Field ID | Type | | -------------------- | -------- | --------- | | warpstream | 10000000 | struct | | warpstream.partition | 10000001 | int | | warpstream.offset | 10000002 | long | | warpstream.key | 10000003 | binary | | warpstream.value | 10000004 | binary | | warpstream.timestamp | 10000005 | timestamp | The `warpstream.value` field can be ommited with the `skip_raw_record_values` [option](#skipping-raw-record-values-agent-v749). ## Schema Migrations Tableflow follows the [Apache Iceberg schema evolution rules](https://iceberg.apache.org/spec/#schema-evolution). Schema migrations are supported for adding columns, changing fields from required to optional, and widening integer and floating point types. To perform any of these operations, update the schema in the Configuration editor and deploy it. In the next few syncs of the table metadata into your bucket, the schema change will be reflected. {% hint style="warning" %} Ensure that you execute a schema change before attempting to send data with the new schema. If you fail to do this, you will potentially lose the data written with the newer schema. {% endhint %} ### Breaking Changes Any schema change that is not one of the compatible operations listed above is considered a breaking change and requires [Table Recreation](#table-recreation). Specifically, the following changes are breaking: * **Make an optional field required** — Existing data may contain nulls for that field, so the constraint cannot be applied retroactively. * **Change a type in a non-widening way** — For example, `long` → `int`, `double` → `float`, or `string` → `int`. This includes any data type change that is not a supported numeric widening. * **Drop a column** — Removing a column from the schema is not supported as an in-place migration. * **Reorder columns** — Changing the order of columns is not supported as an in-place migration. * **Rename a column** — Renaming a column is not currently supported as an in-place migration. To apply breaking changes, use the `recreation_key` mechanism described in [Table Recreation](#table-recreation) below. ## Table Recreation Certain operations require an existing table to be completely deleted and recreated. While this is most commonly necessary to apply breaking schema changes (such as converting an optional field to required, performing non-widening type conversions, or dropping columns), a full rebuild may also be required for other operational reasons. The `recreation_key` parameter is designed to automate this workflow. To utilize this feature, assign an initial string value to the `recreation_key` in your table configuration. Any change to this value acts as a direct trigger for table recreation. When you need to rebuild a table, simply update the `recreation_key` to a new string and deploy the new configuration. Tableflow will detect the change, automatically drop the existing table, and provision a new one with the updated config. {% hint style="danger" %} Note that adding a recreation key to a table that does not already specify this option will trigger a recreation. {% endhint %} The following example demonstrates how to use the `recreation_key` to apply backward-incompatible schema changes: ```yaml tables: - source_cluster_name: tableflow_cluster_1 source_topic: example_json_logs_topic source_format: json recreation_key: "v1" schema_mode: inline input_schema: | { "type": "object", "properties": { "environment": { "type": "string" }, "service": { "type": "string" } }, "required": ["environment"] } ``` To make the `service` field required and add a required `severity` field, bump the value (e.g. `"v1"` to `"v2"`) alongside your schema change: ```yaml recreation_key: "v2" input_schema: | { "type": "object", "properties": { "environment": { "type": "string" }, "service": { "type": "string" }, "severity": { "type": "integer" } }, "required": ["environment", "service", "severity"] } ``` After recreation, Tableflow will re-ingest data from the earliest available offset in the source topic (subject to the topic's retention policy) into the new table. {% hint style="warning" %} Changing `recreation_key` will hard-delete the existing table's metadata. Data already written to the object store is not removed. The new table will only contain data that is still available in the source topic based on its retention settings. {% endhint %} ## Configuring Kafka ACLs If your source Kafka cluster has access control enabled, then the principal used to connect to the source Kafka topic needs to have the following access: | Operation | Resource | Purpose | | --------------- | -------- | ----------------------------------------------------------- | | Describe | Topic | List topics in a cluster and get offsets for a given topic. | | DescribeConfigs | Topic | Get topic retention. | | Read | Topic | Poll records from a topic. | ## Table Deletion Tableflow does not delete any tables from the object storage bucket when they are removed from the Configuration in order to prevent accidental data deletion. To delete a table, first delete it from the Configuration and then use your cloud provider's UI or CLI to delete the directory containing your table from within the `warpstream/_tableflow` directory. To programmatically delete and recreate a table (e.g. for incompatible schema changes), see [Table Recreation](#table-recreation). ## Object Storage Path Layout Tableflow writes all table data and metadata into your object storage bucket under a predictable directory structure. Understanding this layout is useful for debugging, removing an entire table directory, or integrating with query engines that need direct file paths. Given a bucket URL of `s3://my-bucket` (or `s3://my-bucket?prefix=my-prefix`), the structure is: ``` s3://my-bucket/[my-prefix/]warpstream/_tableflow/ └── -/ ├── data/ │ ├── 00000000000000000001.parquet │ ├── 00000000000000000002.parquet │ └── ... └── metadata/ ├── v1.metadata.json ├── v2.metadata.json ├── ... ├── version-hint.text ├── snap-.avro └── mani-.avro ``` | Path | Contents | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `warpstream/_tableflow/` | Root directory for all Tableflow tables. | | `-/` | One directory per table. | | `data/` | Parquet data files. | | `metadata/` | Iceberg metadata: JSON metadata files (`v1.metadata.json`, `v2.metadata.json`, ...), manifest files (`mani-*.avro`), manifest lists (`snap-*.avro`), and a `version-hint.text` that points to the latest metadata version. | {% hint style="info" %} When a table is deleted from the configuration or via the API, Tableflow only removes the control plane metadata. The data and metadata files in object storage are **not** deleted. You must clean them up manually using your cloud provider's UI or CLI. {% endhint %} ## Terraform / Infrastructure as Code / APIs Tableflow Agents are deployed using the standard [WarpStream Agent chart](https://github.com/warpstreamlabs/charts/tree/main/charts/warpstream-agent), and there is full support for Tableflow clusters in the [WarpStream Terraform provider](https://registry.terraform.io/providers/warpstreamlabs/warpstream/latest/docs/resources/tableflow_cluster). [Click here](https://github.com/warpstreamlabs/terraform-provider-warpstream/blob/main/examples/tableflow/main.tf) for a complete Terraform example of creating a Tableflow cluster and configuring it to ingest a single topic into an Iceberg table. Tableflow configuration can also be modified with the [pipelines API](/warpstream/reference/api-reference/pipelines/create-pipeline-configuration). ## Observability Tableflow clusters emit metrics for ingestion lag, offset lag, and end-to-end query lag, as well as diagnostics and events for tracking pipeline health. Ingestion lag is also available visually in the WarpStream Console.
For a complete guide on which metrics to monitor, how to interpret diagnostics, how to use events for troubleshooting, and recommended alerts, see [Monitoring Tableflow](/warpstream/tableflow/monitoring). ## Agent Version Requirements Tableflow requires agents running on version v797 or higher. | Feature | Minimum agent version | | ----------------------------------------------------------------------- | --------------------- | | `schema_mode: schema_registry` with `source_format: protobuf` or `json` | v813 | | `schema_mode: schema_registry` with `source_format: avro` | v820 | | `warpstream.sr.*` writer schema metadata in transforms | v817 | {% hint style="info" %} Always check the [Change Log](/warpstream/overview/change-log) for the latest feature additions and bug fixes. When in doubt, use the latest stable Agent version. {% endhint %} ## Tableflow UI The Tableflow UI available in the WarpStream console allows editing the Configuration.
# Monitoring Tableflow Key metrics, diagnostics, events, and recommended alerts for Tableflow clusters. Tableflow Agents expose a Prometheus endpoint on the internal port (default `8080`). Lag metrics and resource counts are also available on the [hosted Prometheus endpoint](/warpstream/agent-setup/monitor-the-warpstream-agents/hosted-prometheus-endpoint) without needing to scrape agents directly. All metrics use the `warpstream_` prefix. If you are using Datadog with Agent `v679` or later, the prefix is `warpstream.` instead (dot, not underscore). See [Set up Monitoring](/warpstream/agent-setup/monitor-the-warpstream-agents) for general setup. ## Ingestion Lag ### Metrics | Metric | Unit | Description | Min version | | -------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------- | | `warpstream_tableflow_query_lag_seconds` | seconds | End-to-end delay until data is queryable: ingestion lag + Iceberg catalog sync staleness. **This is the primary metric to monitor.** | v778+ | | `warpstream_tableflow_ingestion_lag_seconds` | seconds | Time since the last ingested record, per table. **0** when caught up. | v778+ | | `warpstream_tableflow_partition_offset_lag` | records | Number of records not yet ingested per table (high watermark minus last ingested offset). | v776+ | All lag metrics are tagged by `virtual_cluster_id`, `topic`, `table.name`, and `table.uuid`. On the hosted Prometheus endpoint, tags use underscores (`table_name`, `table_uuid`) and an additional `is_dlq` tag distinguishes between the main ingestion pipeline (`is_dlq="false"`) and the [DLQ replay](#dead-letter-queue-dlq) pipeline (`is_dlq="true"`). Ingestion lag is also visible in the WarpStream Console. {% hint style="info" %} `warpstream_tableflow_ingestion_lag_seconds` was previously named `warpstream_tableflow_partition_time_lag_seconds` (available since v710). The old name is deprecated starting in v778. {% endhint %} ### How to interpret these metrics **Query lag** (`warpstream_tableflow_query_lag_seconds`) is the metric you should use to understand how fresh the data is when you query your Iceberg tables. It captures the full delay from when data is produced to Kafka until it becomes visible to query engines like Spark, Trino, or DuckDB. This delay has two components: the ingestion itself and the Iceberg catalog sync. **Ingestion lag** (`warpstream_tableflow_ingestion_lag_seconds`) helps you break down where the delay is coming from: * If **query lag is high but ingestion lag is low**, the bottleneck is the catalog sync — data has been ingested into Iceberg files but the catalog metadata has not been refreshed yet. * If **ingestion lag is high**, it means the ingestion pipeline itself is falling behind. This is typically caused by one of two things: * **An error** — check your cluster's [diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) for any failing health checks, and look for warnings or errors in [events](/warpstream/reference/events) (`tableflow_logs`) to identify the root cause. * **Insufficient capacity** — the agents cannot keep up with the produce rate on the source topics. Add more Tableflow Agents to increase ingestion throughput. {% hint style="warning" %} When ingestion lag is high, make sure your **source Kafka topic retention** is large enough so that records are not deleted before they can be ingested. If retention expires while the pipeline is behind, data will be permanently lost. {% endhint %} ### Recommended alerts * Alert on `warpstream_tableflow_query_lag_seconds` sustained above your freshness SLA (e.g., 600s). * Alert on `warpstream_tableflow_ingestion_lag_seconds` sustained above a threshold (e.g., 300s) to catch pipeline issues early, before they affect queryability. ## Dead Letter Queue (DLQ) When a table is configured with a [DLQ mode](/warpstream/tableflow/tableflow#dead-letter-queue-dlq-mode) (`stop`, `skip`, or `keep`), invalid records are handled according to that mode. ### Metrics | Metric | Unit | Description | | ------------------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------- | | `warpstream_tableflow_dlq_records_counter` | records | Number of records handled by the DLQ during ingestion, tagged by `topic` and `strategy` (`skip` or `keep`). | Use diagnostics and events for additional DLQ monitoring: * Check your cluster's [diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) for any Tableflow-related failures. Failing diagnostics indicate issues such as ingestion being stopped, records being skipped or routed to DLQ, or DLQ replay backlog. * Check [events](/warpstream/reference/events) (`tableflow_logs`) for warnings and errors related to DLQ activity — these provide per-record detail including failure reasons and affected topics. {% hint style="warning" %} If DLQ mode is `stop` and ingestion encounters invalid records, the pipeline will halt. This will also surface as high ingestion lag. {% endhint %} ## Resource Counts The hosted Prometheus endpoint exposes gauges for resource usage on your Tableflow cluster, tagged by `virtual_cluster_id`: | Metric | Description | | --------------------------------------- | ----------------------------------------------- | | `warpstream_tableflow_tables_count` | Number of tables in the cluster. | | `warpstream_tableflow_files_count` | Number of Iceberg data files across all tables. | | `warpstream_tableflow_snapshots_count` | Number of Iceberg snapshots across all tables. | | `warpstream_tableflow_partitions_count` | Number of partitions across all tables. | These are useful for capacity planning and tracking cluster growth over time. ## Diagnostics and Events [Diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics) are proactive health checks that run continuously on your cluster. They surface problems in the Console UI and as `warpstream_diagnostic_failure` gauge metrics (1 = failing, 0 = healthy) that you can alert on. Diagnostics cover infrastructure issues (bucket access, source cluster authentication), resource limits (table count), and ingestion health (DLQ activity, record errors). [Events](/warpstream/reference/events) provide detailed, per-occurrence context for troubleshooting. Tableflow clusters emit `tableflow_logs` (ingestion failures, table lifecycle, compaction, catalog sync, DLQ replay) and `agent_logs` (general agent operations). Events must be [enabled](/warpstream/reference/events#enabling-events) on your cluster. To investigate issues: * Look for diagnostics in a **failing** state in the Console Health tab or by alerting on `warpstream_diagnostic_failure == 1`. * Look for events with `data.log_level == "error"` or `data.log_level == "warn"` in the Events Explorer, scoped to `tableflow_logs`. For general resource alerts (CPU, memory), see [Recommended List of Alerts](/warpstream/agent-setup/monitor-the-warpstream-agents/recommended-list-of-alerts). Tableflow Agents are stateless and can be auto-scaled based on CPU. # Iceberg REST Catalog This page describes how to connect with Tableflow's read-only REST catalog. WarpStream implements a **read-only** Iceberg REST catalog API that provides access to Tableflow tables stored in your clusters. ### Important Details **Read-Only Access:** This catalog is read-only and designed for integrating with query engines like Snowflake, Databricks, and AWS Glue. All write operations return `403 Forbidden`. **Authentication:** Bearer token authentication using WarpStream Agent Keys. Other authentication methods are not currently supported. **Namespaces:** Only a single namespace `default` is supported. **Storage Credentials:** The catalog does not vend storage credentials. Clients must configure their own storage credentials to access table data. #### Construct the Base URL The base URL for connecting to the catalog is formed by `$METADATA_URL/catalogs/iceberg/$VIRTUAL_CLUSTER_ID`. Where `$METADATA_URL` can be found at [Deploy the Agents](/warpstream/agent-setup/deploy#region), and `$VIRTUAL_CLUSTER_ID` is found in the overview tab, in the top right section.
So for example if your metadata URL was: `https://metadata.default.us-east-1.warpstream.com` and your virtual cluster ID was `vci_dl_04ace9b5_5e41_1915_85ff_a20ea5601e48` then your base URL would be: `https://metadata.default.us-east-1.warpstream.com/catalogs/iceberg/vci_dl_04ace9b5_5e41_1915_85ff_a20ea5601e48` ### Authenticate Currently, only Bearer token authentication is supported. You must include your Agent Key in the Authorization header: ``` Authorization: Bearer $AGENT_KEY ``` The Agent key can be found (or created) at the Agent Keys tab in the UI:
You can also manage your Agent Keys using WarpStream's Terraform provider at . ### Supported Endpoints #### Configuration * `GET /v1/config` - Catalog configuration #### Namespaces * `GET /v1/namespaces` - List namespaces * `GET /v1/namespaces/{namespace}` - Get namespace details * `HEAD /v1/namespaces/{namespace}` - Check namespace exists #### Tables * `GET /v1/namespaces/{namespace}/tables` - List tables * `GET /v1/namespaces/{namespace}/tables/{table}` - Load table metadata * `HEAD /v1/namespaces/{namespace}/tables/{table}` - Check table exists * `GET /v1/namespaces/{namespace}/tables/{table}/credentials` - Get credentials (returns empty) #### Views * `GET /v1/views` - List views (returns empty) * `GET /v1/views/{view}` - Returns `404` * `HEAD /v1/views/{view}` - Returns `404` ### Unsupported Operations All namespace/table/view write operations (`POST`, `DELETE`, updates) return `403 Forbidden`. ### Example {% code overflow="wrap" %} ```bash curl -H "Authorization: Bearer aks_..." https://metadata.default.us-east-1.warpstream.com/catalogs/iceberg/vci_dl_04ace9b5_5e41_1915_85ff_a20ea5601e48/v1/namespaces/default/tables/my_table ``` {% endcode %} # Integrate With Query Engines and External Catalogs Tableflow has direct integration with some external catalogs. Additionally, most catalogs have some concept of an "external table" where the catalog will periodically scrape the object storage bucket where the table lives as the mode of integration, or fetch the latest metadata on-demand when running a query. For example, Snowflake calls this the "Iceberg files" integration method. {% hint style="info" %} Snowflake Documentation: {% endhint %} Similar to catalogs, most query engines have a way to specify the location of a table in object storage and query that directly without going through a catalog. This method can be applied to ClickHouse and DuckDB, for example. # Snowflake This pages describes how to integrate Tableflow with Snowflake so that you can query Iceberg tables created by WarpStream in Snowflake. ## WarpStream Iceberg Catalog This guide explains how to query WarpStream Tableflow tables by connecting Snowflake to the WarpStream Iceberg REST catalog. #### 1. Prepare the Base URL and Authentication Get the base URL and authentication for the next step by following the instructions in [Iceberg REST Catalog](/warpstream/tableflow/iceberg-catalog#construct-the-base-url) and [Iceberg REST Catalog](/warpstream/tableflow/iceberg-catalog#authenticate). #### 2. Create Catalog Integration ```sql CREATE OR REPLACE CATALOG INTEGRATION warpstream_catalog CATALOG_SOURCE = ICEBERG_REST TABLE_FORMAT = ICEBERG CATALOG_NAMESPACE = 'default' REFRESH_INTERVAL_SECONDS = 60 REST_CONFIG = ( CATALOG_URI = '$BASE_URL' ) REST_AUTHENTICATION = ( TYPE = BEARER BEARER_TOKEN = '$AGENT_KEY' ) ENABLED = TRUE; ``` Replace: * `$BASE_URL` - Base URL from the previous step. * `$AGENT_KEY` - Your WarpStream Agent Key (starts with `aks_`) from the previous step.
#### 3. Create External Volume Configure an external volume `` pointing to your Tableflow storage bucket by following these instructions: Basically, you will need to configure the correct authentication or privileges (IAM role, GCS principal etc.) depending on the storage provider, and then run this command to create an external volume: ```sql CREATE OR REPLACE EXTERNAL VOLUME STORAGE_LOCATIONS = ( ( NAME = '', STORAGE_PROVIDER = '', STORAGE_BASE_URL = '' ) ); ```
#### 4. Create Iceberg Table ```sql CREATE OR REPLACE ICEBERG TABLE my_table EXTERNAL_VOLUME = '' CATALOG = 'warpstream_catalog' CATALOG_TABLE_NAME = ''; ```
#### 5. Query the Table ```sql -- Count records SELECT COUNT(*) FROM my_table; -- View sample data SELECT * FROM my_table LIMIT 10; ```
# Databricks Unity This pages describes how to integrate Tableflow with Databricks so that you can query Iceberg tables created by WarpStream in Databricks. ## Integration Context & Limitations To query Iceberg tables managed by Tableflow with Databricks, [catalog federation](https://docs.databricks.com/aws/en/query-federation/catalog-federation) is used. With this approach, Unity Catalog will populate a foreign catalog by crawling the external catalog. This allows Unity Catalog to act as a governance layer while the actual metadata remains managed by an external provider. Currently, federation in Unity is limited to a small set of catalogs, such as AWS Glue, Hive Metastore, or Snowflake Horizon, and does not support generic Iceberg REST endpoints. As such, this guide focuses on an example of setting up WarpStream Tableflow for Databricks Unity with AWS Glue as the integration path. To enable access to your data, we use AWS Glue as an intermediate catalog. Your table metadata is synced to AWS Glue, which is then used to populated a Foreign Catalog in Databricks. {% hint style="info" %} If your architecture requires connecting via a different supported catalog (e.g., syncing Tableflow to Snowflake and connecting that to Databricks), please reach out to us for assistance. {% endhint %} ## Schema Limitations & Workarounds Unity Catalog has a known limitation regarding Iceberg schemas: it does not support `NOT NULL` constraints nested within arrays or maps. If your schema contains these fields, queries may fail with the error: `[DELTA_NESTED_NOT_NULL_CONSTRAINT] Delta does not support NOT NULL constraints nested within arrays or maps.` Workarounds: * **Option A: Modify Schema** Update your Iceberg schema to make nested fields optional (nullable). This allows the table to be queried using standard Databricks "SQL Warehouse" compute. * **Option B: Use Cluster Compute** If you cannot modify the schema, you must use "Cluster Compute" (SQL Warehouses are not supported for this config) and enable the following Spark configuration to suppress the error: ```toml spark.databricks.delta.constraints.allowUnenforcedNotNull.enabled = true ``` ## Prerequisites Before starting, make sure you have: * Completed the [AWS Glue integration setup](/warpstream/tableflow/catalogs-and-query-engines/aws-glue) so your WarpStream Tableflow tables are available in AWS Glue. * A Databricks workspace with **Unity Catalog enabled**. * **Databricks Runtime 16.2 or above** for Iceberg table support (currently in Public Preview). * SQL Warehouses must be **Pro** or **Serverless**. * The following privileges on the Unity Catalog metastore (metastore admins have these by default): * `CREATE SERVICE CREDENTIAL` * `CREATE CONNECTION` * `CREATE EXTERNAL LOCATION` * `CREATE CATALOG` ## Integrate via AWS Glue ### 0. Set Up AWS Glue Integration Before setting up the Databricks integration, you must follow the steps at [AWS Glue](/warpstream/tableflow/catalogs-and-query-engines/aws-glue) to have your WarpStream Tableflow tables available in AWS Glue. ### 1. Create a Service Credential for AWS Glue Access Databricks needs access to the **AWS Glue API** to crawl catalog metadata. This requires creating an IAM role with Glue-specific permissions and registering it as a **Service Credential** in Databricks. **Create the IAM Role** Create an IAM role that Databricks can assume, with the following policy. Scope the permissions to your specific Tableflow Glue database to avoid federating your entire Glue catalog: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables", "glue:GetPartitions" ], "Resource": [ "arn:aws:glue:::catalog", "arn:aws:glue:::database/default", "arn:aws:glue:::database/", "arn:aws:glue:::table//*" ] }, { "Effect": "Allow", "Action": [ "sts:AssumeRole" ], "Resource": [ "arn:aws:iam:::role/" ] } ] } ``` Replace the placeholders: * `` — the AWS region where your Glue catalog lives (e.g. `us-east-1`). * `` — your AWS account ID. * `` — the `database_name` configured in your WarpStream Tableflow AWS Glue configuration. * `` — the name of this IAM role (required for the `sts:AssumeRole` self-reference). Key details about resource scoping: * The `default` database ARN **must** be included or Databricks will return an error. * Use `/*` wildcard for tables if you have multiple tables in the database. {% hint style="warning" %} If you have other databases and tables in your AWS Glue catalog, make sure to scope the IAM policy to only the Tableflow database. Otherwise, Databricks will attempt to federate **all** databases and tables in your Glue catalog, which can cause errors or timeouts. {% endhint %} **Register the Service Credential in Databricks** Once the IAM role is created, register it as a service credential in Databricks. * **Instructions:** Follow the [Databricks Guide: Create service credentials](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-services/service-credentials). ### 2. Create the Glue Connection Create a connection object within Databricks to link to your AWS Glue environment. When creating the connection, use the following values: * **Connection type:** Hive Metastore * **Metastore type:** AWS Glue * **Credential:** Select the service credential created in Step 1 * **AWS Account ID:** The AWS account ID where your Glue catalog is (same account used in the Glue setup) * **AWS Region:** The region where your Glue catalog is * **Instructions:** Follow the [Databricks Guide: Create the connection](https://docs.databricks.com/aws/en/query-federation/hms-federation-glue#create-the-connection). {% hint style="info" %} If you are using the Databricks Catalog Explorer UI, the connection wizard can also create the foreign catalog in the same flow. Users who follow the UI wizard may be able to combine the connection and catalog creation (Step 4) into a single step. {% endhint %} ### 3. Create a Storage Credential and External Location for S3 Access Databricks also needs access to **S3** to read the actual Iceberg data files. This requires a **separate** IAM role (distinct from the service credential in Step 1) with S3 read permissions, registered as a **Storage Credential** in Databricks. You then create an **External Location** that points to the S3 bucket where Tableflow stores its data. The IAM role for the storage credential needs the following S3 permissions on your Tableflow bucket: * `s3:GetObject` * `s3:ListBucket` * `s3:GetBucketLocation` * **Instructions:** Follow the [Databricks Guide: Create storage credential and external location](https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manual). ### 4. Create the Foreign Catalog {% hint style="danger" %} **Critical:** When creating the catalog, you **must** set the **Storage location** field to your S3 bucket path (e.g., `s3://your-tableflow-bucket`). If this is omitted, Databricks will fail to read the Iceberg data. {% endhint %} Create the foreign catalog to mount the Glue database. When creating the catalog: * Set the **Storage location** to the S3 path where Tableflow stores data (e.g. `s3://your-tableflow-bucket`). * Set the **Authorized paths** to match your S3 bucket path. Tables outside these paths won't be queryable. This should be the same bucket path used in the external location from Step 3. * **Instructions:** Follow the [Databricks Guide: Create a foreign catalog](https://docs.databricks.com/aws/en/query-federation/hms-federation-glue#step-3-create-a-foreign-catalog). ### 5. Query the Data Once the catalog is created, your WarpStream tables will automatically appear in the Databricks UI (Catalog Explorer). You can now query them using standard SQL, Notebooks, or BI tools just like any other native table. To reference a table in your queries, use the full three-level namespace: ```sql SELECT * FROM [catalog_name].[glue_database_name].[table_name]; ``` # BigQuery This page describes how to integrate Tableflow with Google BigQuery so that you can query Iceberg tables created by WarpStream directly in BigQuery. {% hint style="warning" %} **For new GCP deployments, we recommend using** [**BigLake**](/warpstream/tableflow/catalogs-and-query-engines/biglake) **instead.** BigLake registers tables in the BigLake Metastore catalog, making them automatically available in BigQuery and in any other query engine that supports the Iceberg REST Catalog protocol. {% endhint %} Tableflow can automatically register tables in BigQuery and update a table's metadata location to point to the latest snapshot. ### Prerequisites In order for this to work, the WarpStream Agents need to be upgraded to at least **v737**. ## 1. Create the BigQuery Dataset Create a BigQuery dataset to hold your Tableflow tables. The dataset must exist before enabling the integration. ```bash bq mk --dataset --location= : ``` {% hint style="warning" %} **Critical Requirement:** The BigQuery Dataset location must match your GCS bucket region. * If your bucket is in `us-east1`, your dataset must be in `us-east1`. * If they do not match, BigQuery will be unable to read the data files. {% endhint %} ## 2. Grant IAM Permissions The Tableflow agent service account requires the following roles: | Role | Purpose | | ---------------------------- | --------------------------------- | | `roles/bigquery.dataEditor` | Create and update external tables | | `roles/storage.objectViewer` | Read Iceberg metadata from GCS | Grant them via: ```bash # 1. Grant BigQuery access gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \ --role="roles/bigquery.dataEditor" # 2. Grant GCS access (required for schema detection) gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \ --role="roles/storage.objectViewer" ``` ## 3. Add Table Configuration Add the following BigQuery configuration to your table config: ```yaml # Global defaults for all BigQuery tables bigquery_defaults: project_id: dataset_id: tables: # Example Table: 'events' - source_topic: events # ... other table settings ... bigquery_table_config: enabled: true table_id: "events" ``` #### Top-Level Defaults (`bigquery_defaults`) These defaults apply to all tables unless overridden per-table. | Field | Description | | ------------ | ---------------------------------------------------- | | `project_id` | The GCP project ID containing the BigQuery dataset | | `dataset_id` | The BigQuery dataset ID where tables will be created | #### Per-Table Configuration (`bigquery_table_config`) | Field | Description | | ------------ | ---------------------------------------------------- | | `enabled` | Set to `true` to enable BigQuery sync for this table | | `table_id` | The BigQuery table name to create/update | | `project_id` | Override the default `project_id` for this table | | `dataset_id` | Override the default `dataset_id` for this table | ## 4. Query the Data Once enabled, your tables will appear in the BigQuery console. Query them using standard SQL: ```sql SELECT * FROM .. LIMIT 100; ``` {% hint style="warning" %} To write efficient queries on partitioned tables please read . {% endhint %} # BigLake This page describes how to integrate with Google BigLake Metastore so that you can query Iceberg tables created by Tableflow directly in BigQuery via BigLake. Integrating with BigLake Metastore is the recommended way to make Tableflow Iceberg tables queryable in BigQuery. This approach uses the [BigLake Metastore Iceberg REST Catalog](https://docs.cloud.google.com/biglake/docs/blms-rest-catalog) API to handle table registration, giving you native Iceberg support and keeping your catalog automatically in sync as new snapshots are taken. {% hint style="success" %} **BigLake vs BigQuery integration:** BigLake is preferred over the [BigQuery integration](/warpstream/tableflow/catalogs-and-query-engines/bigquery) because tables are registered in BigLake Metastore, which exposes the standard Iceberg REST Catalog protocol. This means the tables are automatically queryable from BigQuery *and* from any engine that can connect to an Iceberg REST Catalog, such as Spark, Trino, and Presto. The legacy BigQuery integration only creates BigQuery external tables, so the tables are only visible to BigQuery. {% endhint %} ## Prerequisites 1. Please upgrade your WarpStream Agents to at least **v769**. 2. You need a [BigLake Metastore Iceberg REST catalog](https://docs.cloud.google.com/biglake/docs/blms-rest-catalog) created in the same region as your GCS bucket. ### Create a BigLake Metastore Catalog If you don't already have one, create a BigLake Metastore Iceberg REST catalog. You can create either a single-bucket catalog, or a multi-bucket catalog. Note that [multi-bucket catalogs are now recommended by Google](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog#bucket_type). To create a single-bucket catalog: ```bash gcloud beta biglake iceberg catalogs create \ \ --project \ --catalog-type gcs-bucket ``` {% hint style="info" %} The catalog name must match the GCS bucket name (e.g., if your bucket is `gs://my-bucket`, use `my-bucket` as the catalog name). This is a BigLake requirement for `gcs-bucket` type catalogs. See the [Google Cloud documentation](https://docs.cloud.google.com/biglake/docs/blms-rest-catalog#create_a_catalog) for details. {% endhint %} To create a mult-bucket catalog: ``` gcloud biglake iceberg catalogs create \ \ --project \ --catalog-type biglake \ --default-location gs://[/PATH] \ --restricted-locations gs://[/PATH],gs://[/PATH] ``` {% hint style="info" %} Contrary to single-bucket catalogs, the multi-bucket catalog name does not need to match any GCS bucket name. {% endhint %} {% hint style="info" %} You do not need to manually create a BigQuery dataset. Tableflow automatically creates the namespace via the BigLake REST Catalog API. {% endhint %} ## 1. Authentication and IAM Permissions The BigLake integration uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials). The agent authenticates to both the BigLake REST Catalog API and GCS using the service account it runs as — no additional credential configuration is needed. The agent service account requires the following role: | Role | Purpose | | --------------------- | ------------------------------------------------- | | `roles/biglake.admin` | Create and update tables in the BigLake Metastore | Or if you prefer more granular permissions, you can create a custom role with the following permissions: * `biglake.catalogs.get` * `biglake.namespaces.create` * `biglake.namespaces.get` * `biglake.tables.delete` * `biglake.tables.get` * `biglake.tables.register` And add the role to your service account via: ```bash gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \ --role="roles/biglake.admin" ``` {% hint style="info" %} The agent already has GCS access for reading and writing Iceberg data. The only additional permission needed for BigLake is `roles/biglake.admin`. {% endhint %} ## 2. Add Table Configuration Add the following `biglake_table_config` block to each table you would like to sync: ```yaml tables: - source_topic: events # ... other table settings ... biglake_table_config: enabled: true project_id: "" namespace: "" table_name: "" ``` ### Configuration Fields To use a single-bucket catalog, the following fields are required: | Field | Required | Description | | ------------ | -------- | ---------------------------------------------------------------------------------------------------- | | `enabled` | Yes | Set to `true` to enable BigLake sync for this table | | `project_id` | Yes | The GCP project ID (used for billing/quota attribution via `x-goog-user-project`) | | `namespace` | Yes | The BigLake namespace where the table will be registered. Created automatically if it doesn't exist. | | `table_name` | Yes | The table name to create/update in the BigLake catalog | Multi-bucket catalogs are supported from agent version `v828+` onward. For a multi-bucket catalog, there are additional fields required: | Field | Required | Description | | -------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `is_multi_bucket_catalog` | No | Set to `true` if the catalog is a multi-bucket catalog. Requires `catalog_id` and `catalog_storage_location` to be set. | | `catalog_id` | Yes if `is_multi_bucket_catalog: true` | The BigLake catalog ID from the GCP console. Optional for single-bucket catalogs (and defaults to the destination bucket name is omitted). | | `catalog_storage_location` | Yes if `is_multi_bucket_catalog: true` | One of the allowed locations of your catalog. For multi-bucket catalogs, that must correspond to either the "GCS Warehouse" field or one of the "Allowed Paths" from the Google console (eg: `gs://my-bucket/some/subpath`). Optional for single-bucket and defaults to `gs://bucket-name` if omitted. | {% hint style="info" %} For single-bucket catalogs only, the agent automatically discovers the BigLake catalog using the destination GCS bucket. No catalog resource path is needed in the configuration. For multi-bucket catalogs however, you **must** set the `catalog_id`and `catalog_storage_location` explicitly. {% endhint %} {% hint style="info" %} Tableflow automatically creates the namespace if it doesn't exist. Note that a namespace is pinned to one location the first time it's created. If you use a multi-bucket catalog and have multiple tables using that catalog, make sure that you choose a different namespace if they don't share the same `catalog_storage_location`. {% endhint %} ## 3. Query the Data Once enabled, Tableflow will automatically register the table in BigLake and keep the metadata location up to date. ### From BigQuery BigLake tables are queryable from BigQuery using the 4-part `Project.Catalog.Namespace.Table` syntax: ```sql SELECT * FROM `...` LIMIT 100; ``` For example, if your project is `my-project`, your catalog is `my-catalog`, and you configured namespace `my_ns` with table `my_table`: ```sql SELECT * FROM `my-project.my-catalog.my_ns.my_table` LIMIT 100; ``` ### From Other Query Engines Any query engine that supports the Iceberg REST Catalog protocol (e.g. Spark, Trino, Presto) can connect directly to the BigLake Metastore REST Catalog endpoint and query the tables. # DuckDB To query Tableflow tables with DuckDB, you'll first need to install and load the [Iceberg extension](https://duckdb.org/docs/stable/core_extensions/iceberg/overview.html). ```sql INSTALL iceberg; LOAD iceberg; ``` Once you've installed the Iceberg extension, you'll need to connect to the object store system where you've chosen to store your Tableflow tables. * [AWS S3 and S3-compatible systems](https://duckdb.org/docs/stable/core_extensions/httpfs/s3api) * [Microsoft Azure](https://duckdb.org/docs/stable/core_extensions/azure) * [Google Cloud Storage with HMAC Keys](https://duckdb.org/docs/stable/guides/network_cloud_storage/gcs_import) Once you've successfully connected to the object store, you can query your tables using the standard syntax for the Iceberg extension: ```sql SELECT * from iceberg_scan('s3://'); ``` and to query a specific snapshot version: ```sql SELECT * from iceberg_scan('s3:///metadata/v.metadata.json'); ``` # ClickHouse To query Tableflow tables with ClickHouse, you'll first need to configure credentials for your object store where you've chosen to store your tables and use the [Iceberg table engine](https://clickhouse.com/docs/engines/table-engines/integrations/iceberg). #### AWS S3 For testing purposes, you can define credentials inline in the `CREATE TABLE` statement. For production, we recommend using the configuration file to define a "named collection" for your credentials. ```xml http://test.s3.amazonaws.com/clickhouse-bucket/ test test ``` Then you can run your `CREATE TABLE` statement against the named collection instead of inline credentials: ```sql CREATE TABLE iceberg_table ENGINE=IcebergS3(iceberg_conf, filename = '') ``` #### GCS For connecting to GCS, you'll need to create GCS HMAC credentials and use them in place of the `access_key_id` and `secret_access_key` for GCS. #### Partition Pruning To take advantage of partition pruning in ClickHouse, you must set a session variable. ```sql SET use_iceberg_partition_pruning = 1; ``` # AWS Glue Tableflow can automatically register tables in Glue and update a table's metadata location to point to the latest snapshot. ### Prerequisites Tableflow uses the AWS SDK for Go v2 to access Glue, so credentials can be specified using one of the supported ways in the SDK’s [default credentials chain](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-gosdk.html#specifying-credentials). To use STS AssumeRole instead, set the `WARPSTREAM_TABLEFLOW_ASSUME_ROLE_ARN` environment variable on the Agent (and optionally `WARPSTREAM_TABLEFLOW_ASSUME_ROLE_DURATION_MINUTES`). In either case, the same identity is used to access Glue and S3. The Glue integration requires the Agent version to be at least [version 710](/warpstream/overview/change-log#release-v710). Additionally, the Agents must have the appropriate IAM policy for Glue attached. Specifically, the following permissions are needed on the catalogs, databases and tables that you want Tableflow to manage: * `glue:GetTable` * `glue:CreateTable` * `glue:UpdateTable` * `glue:GetTableVersions` * `glue:BatchDeleteTableVersion` The IAM policy should look like the following: ```json { "Version":"2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:CreateTable", "glue:GetTable", "glue:UpdateTable", "glue:GetTableVersions", "glue:BatchDeleteTableVersion" ], "Resource": [ "arn:aws:glue:::catalog", "arn:aws:glue:::database/", "arn:aws:glue:::table//" ] } ] } ``` ### Configuration To enable this feature, update the configuration YAML with the following: ```yaml tables: - source_topic: "example_json_logs_topic" ... aws_glue_table_config: enabled: { true | false } catalog_id: '' database_name: '' table_name: '' schema: ... ``` **Required parameters** `enabled` Specifies whether the Glue integration should run. `database_name` Specifies the database in which to create the table. This database needs to exist already as Tableflow will not try to create one automatically. `table_name` Specifies the name the Glue table should be created with. This can be different from the name of the table in Tableflow. **Optional parameters** `catalog_id` Specifies the ID of the Data Catalog in which to create the Table. If none is supplied, the AWS account ID will be used. Note that the `database_name` and `table_name` parameter should match the resources from the IAM policy. # Amazon Athena To query Tableflow tables using Athena, we recommend using Tableflow's [AWS Glue integration](https://docs.warpstream.com/warpstream/tableflow/catalogs-and-query-engines/aws-glue) to manage the table. This ensures that the query runs against latest metadata version file. You can also use the "CREATE TABLE" command directly in Athena. This will create a new Iceberg table and register it with the AWS Glue Data Catalog, but it requires manual updates to the latest metadata version file. ```sql CREATE TABLE [db_name.]table_name (col_name data_type [COMMENT col_comment] [, ...] ) // schema of table here LOCATION 's3://demo-bucket/_tableflow/path/to/table' TBLPROPERTIES ( 'table_type' = 'ICEBERG', 'format' = 'parquet' ); ``` {% hint style="info" %} Athena create table always creates an empty table. Before querying the table, we need to update the table property of "metadata\_location" to point to latest metadata version file.\ \ Go to AWS Glue Data Catalog -> Tables -> Select your Tableflow table -> Actions -> Edit Table -> update "metadata\_location" in Table properties {% endhint %} # Hive Metadata Store Tableflow can automatically register tables in a Hive Metastore and update a table's metadata location to point to the latest snapshot. Tables are registered as external Iceberg tables using the Hive Metastore Thrift protocol. #### Prerequisites In order for this to work, the WarpStream Agents need to be upgraded to at least version **v769**. Additionally, the WarpStream Agents must have network access to the Hive Metastore Thrift endpoint. {% hint style="info" %} Authentication: Only NOSASL authentication is currently supported. Kerberos and other authentication mechanisms are not yet available. If you need Kerberos support, please reach out to us. {% endhint %} ### 1. Ensure Network Connectivity The WarpStream Agents must be able to reach the Hive Metastore Thrift endpoint over the network. The default Hive Metastore Thrift port is 9083. ### 2. Add Table Configuration Add the following Hive Metastore configuration to your table config: ```yaml tables: - source_topic: "example_topic" ... hive_table_config: enabled: true thrift_uri: "thrift://:9083" namespace: "" table_name: "" schema: ... ``` #### Required Parameters | Field | Description | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enabled | Set to true to enable Hive Metastore sync for this table. | | thrift\_uri | The Thrift URI of the Hive Metastore. Must use the thrift:// scheme (e.g., thrift://hive-metastore.example.com:9083). If the port is omitted, 9083 is used by default. | | namespace | The Hive database (namespace) where the table will be created. If the database does not exist, Tableflow will create it automatically. | | table\_name | The name of the table to create or update in the Hive Metastore. This can be different from the name of the table in Tableflow. | ### 3. Query the Data Once enabled, your tables will appear in the Hive Metastore as external Iceberg tables. You can query them from any query engine that supports the Hive Metastore catalog, such as Trino, Spark, or Presto. For example, using Trino: ```sql SELECT * FROM hive.my_database.my_table LIMIT 100; ``` Or using Spark SQL: ```sql SELECT * FROM my_database.my_table LIMIT 100; ``` ### How It Works When the Hive Metastore integration is enabled for a table, Tableflow will: 1. **Create the namespace** if it does not exist in the Hive Metastore. 2. **Create the table** as an external Iceberg table if it does not exist, setting the metadata\_location to the latest Iceberg metadata file in your object storage bucket. 3. **Update the table** on subsequent syncs by updating metadata\_location to point to the latest snapshot. The previous metadata location is preserved in the previous\_metadata\_location table property. # API Reference For detailed information on managing your tables programmatically, see the [Tableflow HTTP API Reference](/warpstream/reference/api-reference/tableflow). # Setup This page explains how to setup WarpStream's BYOC Schema Registry. ## Overview WarpStream’s BYOC Schema Registry serves as a central repository to store and retrieve schemas used to serialize / deserialize and validate your Kafka records. The Schema Registry exposes a REST server that is API-compatible with Confluent's Schema Registry (more details about API compatibility [below](#protocol-support)). We highly recommend watching the overview video below, which provides a comprehensive overview of WarpStream's schema features. {% embed url="" %} WarpStream’s BYOC Schema Registry employs the same zero-disk (diskless Kafka), stateless architecture as WarpStream BYOC. It stores data directly to object storage with no intermediary disks, separates storage from compute, and separates the data plane from the control plane. All schemas are stored in your object store, while metadata that requires consensus (e.g. schemaID, versioning, etc) are offloaded to WarpStream’s control plane. WarpStream’s Schema Registry is embedded natively into the WarpStream Agents. This makes deploying schema registries as easy as deploying the stateless WarpStream Agents. Unlike traditional Kafka schema registry, in which only the leader node is capable of performing writes to the underlying Kafka log, any WarpStream agent is capable of both writing to and reading from the registry. Furthermore, scaling the number of Schema Registry Agents during traffic spikes is trivial due to the stateless nature of WarpStream Agents. To learn more about security and privacy concerns for WarpStream BYOC Schema Registry clusters, check out the schema registry section of [the security and privacy documentation](/warpstream/reference/security-and-privacy-considerations#data-isolation-for-byoc-schema-registry-clusters). Check out this overview video to learn more: {% embed url="" %} ### Protocol Support WarpStream's Schema Registry supports `Avro` and `protobuf` schemas, with `JSON Schema` support coming soon. WarpStream's BYOC Schema Registry supports most APIs specified in Confluent's Schema Registry [API documentation](https://docs.confluent.io/platform/current/schema-registry/develop/api.html). However, it doesn't support advanced features like [data contracts](https://docs.confluent.io/platform/current/schema-registry/fundamentals/data-contracts.html) and [client-side field level encryption](https://docs.confluent.io/cloud/current/security/encrypt/csfle/overview.html). For a full list of features not supported, check out the [protocol documentation](/warpstream/kafka/reference/protocol-and-feature-support#schema-registry). ## Run the Schema Registry Locally Once you [install the Agent binary](https://docs.warpstream.com/warpstream/getting-started/install-the-warpstream-agent), you can have a Schema Registry running locally on your laptop within seconds for you to test against. For instructions on how to run a Schema Registry locally, check out [this doc](https://docs.warpstream.com/warpstream/byoc/run-the-agent-locally). ## Create a Schema Registry To create a Schema Registry Virtual Cluster, you can either create it from the WarpStream console or the API. #### Creating from the Console To create a Schema Registry from [the console](https://console.warpstream.com/), navigate to the Schema Registries tab and click the Create Schema Registry button. #### Creating via API To create a Schema Registry Virtual Cluster via API, invoke the `/create_virtual_cluster` endpoint and specify the `virtual_cluster_type` as `byoc_schema_registry` as follows: {% code overflow="wrap" %} ```bash curl https://api.warpstream.com/api/v1/create_virtual_cluster \ -H 'warpstream-api-key: XXXXXXXXXX' \ -H 'Content-Type: application/json' \ -d '{"virtual_cluster_name": "XXXXXXXXXX", "virtual_cluster_type": "byoc_schema_registry", "virtual_cluster_region": "us-east-1", "virtual_cluster_cloud_provider": "aws"}' ``` {% endcode %} The response object will contain the Schema Registry's Virtual Cluster ID as well as the agent key necessary to deploy the Agent. Note that Virtual Cluster IDs of Schema Registries always begin with `vci_sr_`. See our [Create Cluster API documentation](https://docs.warpstream.com/warpstream/reference/api-reference/virtual-clusters/create) for more details. ## Deploy the BYOC Schema Registry Agents In the [playground/demo mode](https://docs.warpstream.com/warpstream/byoc/run-the-agent-locally), we automatically deploy both a Kafka agent and a Schema Registry agent to make them easier to experiment with. For real clusters, you would have to deploy Schema Registry agents separately from your Kafka agents. After you obtain a Schema Registry Virtual Cluster ID and an agent key, you can deploy the Schema Registry Agent the exact same way you would deploy your Kafka Agents, using the same Agent binary. The only difference is that the Agent will host a Schema Registry HTTP server instead of a Kafka TCP server when deployed. See our [deployment docs](https://docs.warpstream.com/warpstream/byoc/deploy) on how to deploy the Agent. ## Monitor the Schema Registry Agents You can monitor your Schema Registry Agents using logs and metrics emitted by the Agents. See the[ ](https://docs.warpstream.com/warpstream/byoc/monitor-the-warpstream-agents#observability)schema registry section in our monitoring[ documentation](/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs#schema-registry) for more details. ## Client Configuration WarpStream’s BYOC schema registry is API-compatible with Confluent's schema registry. To obtain a Schema Registry URL that points to your Schema Registry Agents, navigate to [WarpStream console](https://console.warpstream.com/)'s and click the`Connect` tab. Once you have the Schema Registry URL, you can slot it into your schema registry client. For example, here is how you can initialize Franz-go's Schema Registry client: {% code overflow="wrap" %} ```go url := "api-80ba097c-d4ef-4e0b-8e86-d05b80fee6ed.discoveryv2.prod-z.us-east-1.warpstream.com:9094" srClient, err := sr.NewClient( sr.URLs(url), ) if err != nil { return fmt.Errorf("error initialiazing schema registry client: %w", err) } ``` {% endcode %} Alternatively if you use Kubernetes deployments, you can also use the service name from the deployed WarpStream chart as the schema registry URL, for example: `warpstream-agent:9094` #### Configuring Client to Eliminate InterZone Networking Costs To ensure your schema registry clients connect to agents within the same availability zone, you need to ensure there is at least one agent in the same availability zone as your clients. You also need to specify the client’s availability zone by embedding the availability zone into the Schema Registry URL. For example, to specify that a client is in `asia-southeast-1a`, embed the AZ into the URL like this: `api-80ba097c-d4ef-4e0b-8e86-d05b80fee6ed.azasia-southeast-1a.discoveryv2.prod-z.us-east-1.warpstream.com:9094`. Note the `az` prefix before the name of the availability zone. This is not required for production usage, but it can help reduce costs for high volume schema registry workloads. ## Using Agent Groups If you ever need to split your Schema Registry cluster into different "groups" that are isolated at the network / service discovery layer, you can use [Agent Groups](https://docs.warpstream.com/warpstream/byoc/advanced-agent-deployment-options/agent-groups). All you have to do is to pass the `-agentGroup` flag to the Agent binary. Check out [the agent groups documentation](https://docs.warpstream.com/warpstream/byoc/advanced-agent-deployment-options/agent-groups#configuration) for more information. Note that the Schema Registry URL displayed in the console is currently not "agent group aware" and will randomly return Agent IP addresses from different groups. To use the Agent Group functionality, you should instead use URLs that target the Agents in the specific agent group. For example, if you're using the official WarpStream Kubernetes chart. You can specify the `agentGroup` via the `extraArgs` property. You can then use the Kubernetes service name generated by the chart as the schema registry URL.[
](https://docs.warpstream.com/warpstream/byoc/advanced-agent-deployment-options/splitting-agent-roles) ## Authentication WarpStream BYOC Schema Registry currently supports: * Basic Authentication * TLS Encryption * mTLS Authentication **TLS Encryption** To configure your WarpStream agent to enable TLS encryption, set the `WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLED`environment variable to true. After creating your TLS certicicates, you also have to set the `WARPSTREAM_TLS_SERVER_CERT_FILE`environment variable to the public key of the certificate and set the `WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE`to the private key of the certificate. For more information about how to configure TLS encryption for your WarpStream cluster, check out the [TLS Encryption documentation](/warpstream/kafka/manage-security/protect-data-in-motion-with-tls-encryption#configure-tls-encryption-for-a-warpstream-cluster). **Mutual TLS (mTLS) Authentication** To enable mTLS authentication, set the `WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION`environment variable to true. It's also recommended to set the environment variable `WARPSTREAM_TLS_CLIENT_CA_CERT_FILE`to the public keys of the certificate authorities that sign your client certificates. Note that mTLS authentication requires TLS encryption. For more information on how to configure mTLS authentication for your WarpStream cluster, check out the [mTLS documentation](/warpstream/kafka/manage-security/mutual-tls-mtls#configure-warpstream-agents). **Basic Authentication** When basic authentication is enabled, the WarpStream Agent uses the username/password encoded in the HTTP request's Authorization header to authenticate your Schema Registry clients. To configure your WarpStream agent to enable basic authentication, set the `WARPSTREAM_SCHEMA_REGISTRY_BASIC_AUTH_ENABLED`environment variable to true. For more information about how to set up basic authentication for your WarpStream cluster, check out the [basic authentication documentation](/warpstream/schema-registry/manage-security/basic-authentication). ## Integrating with WarpStream Schema Validation You can configure your Kafka agents to perform server-side schema validation that checks whether the data actually conforms to the expected schema. To enable your Kafka agents to fetch schemas from your WarpStream schema registry, you need to do two things: * Set the `-schemaValidationVirtualClusterID` flag when deploying the Kafka agent. * Make sure the agent has permissions to read existing files from the object storage bucket that holds the schemas for your BYOC Schema Registry. Check out the [schema validation docs](/warpstream/schema-registry/schema-validation#using-warpstreams-byoc-schema-registry-for-schema-validation) for more details on how to configure the permissions. The best part is that when performing schema validation alongside WarpStream's BYOC Schema Registry, you don't need any Schema Registry agents running! This is because the Kafka agent can just fetch your schemas directly from object storage. Check out the [schema validation docs](/warpstream/schema-registry/schema-validation#using-warpstreams-byoc-schema-registry-for-schema-validation) for more details. ## Limits Here are some enforced limits for each Schema Registry: * The number of schema versions is limited to 100,000 versions. You can track how many schema versions you have with the `warpstream_schema_versions_count`. For more details, check out the [metrics documentation](https://docs.warpstream.com/warpstream/byoc/monitor-the-warpstream-agents#observability). * The size limit of each schema is limited to 1MB. If you need an increase for one or more of these limits, contact us at . ## Debugging Schemas with Confluent's VS Code Plugin WarpStream console allows you to inspect your subjects, compatibility rules, and versions. However, WarpStream console cannot show the schemas themselves because WarpStream's control plane doesn't have access to your schemas. If you want to debug your schemas, there are a couple of tools you can use. One of the tools is [Confluent's VS Code plugin](https://marketplace.visualstudio.com/items?itemName=confluentinc.vscode-confluent). Confluent's VS Code extension allows you to connect to your WarpStream BYOC Schema Registry and inspect your schemas. You just have to provide the URL and the necessary credentials to the plugin.
Once connected, you can inspect the schemas like below:
# Enforce Schemas This page describes how to configure the topic and agent to perform schema validation. ## How Schema Validation Works Historically, schemas stored in schema registries are used only by clients to serialize/deserialize and validate messages. With WarpStream, you can configure the agents to not only validate that the record contains a valid schema ID, but that the record actually conforms to the corresponding schema. The agent can then reject or emit metrics when it receives invalid records. {% hint style="info" %} Note that enabling schema validation will increase the CPU usage of the agent. {% endhint %} Currently, WarpStream supports two types of schema registries: * Kafka-compatible Schema Registry * AWS Glue Schema Registry {% hint style="info" %} For how to encode the records into the right serialization format, check out the [Serialization Format](#serialization-format) section. {% endhint %} Here is a brief overview of how schema validation works in WarpStream: * The producer serializes data with the schema retrieved from the schema registry and encodes it into the right serialization format. * The producer send the data to a WarpStream agent. * On receiving the message, the WarpStream agent decodes the message to obtain the schema ID (or Schema Version Id in the case of AWS Glue Schema Registry). * The agent uses the schema ID to fetch the remote schema. * Finally, the agent verifies if the data actually conforms to the schema and rejects (or emit metrics) any invalid records. This process is illustrated in the diagram below:
Currently, WarpStream supports the following schema formats: `Avro` and `JSON Schema` (with `Protobuf` coming soon). Check out this overview video to learn more: {% embed url="" %} ## Serialization Format Records must be serialized according to which schema registry you are using. For Kafka-compatible schema registries, you must encode records using [Confluent's Wire Format](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#wire-format). The first byte of the encoded record is always `0`. It is then followed by a 4-byte schema ID. Finally, the serialized data is appended to the end of the record.
For AWS Glue, the encoding starts off with a magic byte of 3, followed by one byte which denotes whether the data is compressed and 16 bytes of UUID which represents the schema. Finally, the data is appended at the end.
## Topic Level Configurations for Schema Validation Schema validation is configurable per topic. The following configurations can be provided when a topic is created or altered. Note that there are additional configurations depending on the schema registry type, which will be discussed in the next section.
ConfigurationDescription
warpstream.key.schema.validationBoolean config that indicates whether to validate the record key.
warpstream.value.schema.validationBoolean config that indicates whether to validate the record value.
warpstream.schema.validation.warning.only

When an invalid record is detected, the Agent allows the record to be written, but emits a metric indicating that the record is invalid instead of rejecting the record.

The metric (counter) emitted is: schema_registry_validation_invalid_record

Defaults to true.

warpstream.schema.registry.type

The type of schema registry that the schemas live in. Supported values include:

  • "STANDARD": Any schema registries that are compatible with Confluent's schema registry
  • "AWS_GLUE": AWS Glue's Schema Registry.

Defaults to "STANDARD"

#### Topic Level Configurations for Kafka-compatible Schema Registries Below are topic-level configurations for Kafka-compatible schema registries.
ConfigurationDescription
warpstream.key.subject.name.strategy

Config that determines which schemas are allowed for the record key.

Allowed values: TopicNameStrategy, RecordNameStrategy, TopicRecordNameStrategy. See more details below.

warpstream.value.subject.name.strategy

Config that determines which schemas are allowed for the record key.

Allowed values: TopicNameStrategy, RecordNameStrategy, TopicRecordNameStrategy. See more details below.

#### **Subject Name Strategy:** Each schema in the Schema Registry is registered under a subject. During schema validation, the agent looks up the subject for the schema ID and verifies that the subject conforms to the subject name strategy. There are three subject name strategies: | Strategy | Definition | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TopicNameStrategy |

The subject is derived from the topic name with the following format:

  • \-key for the record key
  • \-value for the record value.
| | RecordNameStrategy | The subject is the schema’s fully-qualified record name. | | TopicRecordNameStrategy | The subject is a combination of the topic name and the record name with the following format: \-\ | The fully-qualified record name for Avro is the record’s namespace + record name. For JSON Schema, the record name is the `title`. ## Using WarpStream's BYOC Schema Registry for Schema Validation Our goal is to make it as easy as possible to perform schema validation with WarpStream's BYOC Schema Registry. To configure the agent to perform schema validation using schemas from your BYOC Schema Registry cluster, you need to do two things: * Set the `-schemaValidationVirtualClusterID` flag to your Schema Registry's Virtual Cluster ID when deploying your agent. Alternatively, you can set the environment variable `WARPSTREAM_SCHEMA_VALIDATION_VIRTUAL_CLUSTER_ID` to your Virtual Cluster ID. * Configure your agent to have the permission to read existing files from the object storage bucket that holds the schemas for your BYOC Schema Registry. In the case of AWS, this is the `GetObject` permission. In the case of GCP, this is the `storage.objects.get` permission. Once the flag is set and the object storage permissions are provided, the agent will automatically fetch schemas from the object storage that holds the schemas for your BYOC Schema Registry when performing schema validation. ## Connecting to External Kafka-Compatible Schema Registry To allow the agent to connect to a Kafka-specific schema registry, set the `-schemaRegistryURL` flag to the URL of the schema registry. Alternatively, you can also set the `WARPSTREAM_SCHEMA_REGISTRY_URL` environment variable. ### Authentication Most schema registry implementations support some form of authentication. WarpStream supports connecting to external schema registries with MTLS, TLS, or basic authentication. #### Basic Authentication For basic authentication, supply the `username` and `password` as follows: * set the `-externalSchemaRegistryBasicAuthUsername` flag to the username of the schema registry. Alternatively, set the `WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_USERNAME` environment variable * set the `-externalSchemaRegistryBasicAuthPassword` flag to the password of the schema registry. Alternatively, set the `WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_BASIC_AUTH_PASSWORD` environment variable #### TLS/MTLS For `mTLS`, the agent needs both a certificate and a private key to enable the schema registry server to authenticate the agent. You can use the `-externalSchemaRegistryTlsClientCertFile` and `-externalSchemaRegistryTlsClientPrivateKeyFile` to pass in the **file paths** to the agent certificate and private key, respectively. Alternatively, you can use `WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_CERT_FILE` and `WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_CLIENT_PRIVATE_KEY_FILE` environment variables. For TLS and mTLS, you can optionally add a **file path** to the root certificate authority certificate file which the Agent will use to verify the schema registry server's certificate. Use the `-externalSchemaRegistryTlsServerCACertFile` flag, or the `WARPSTREAM_EXTERNAL_SCHEMA_REGISTRY_TLS_SERVER_CA_CERT_FILE` environment variable. ## Connecting to AWS Glue Schema Registry The agent must be deployed in AWS to connect to an AWS Glue schema registry. In addition, you'll also need to make sure the Agent containers have the appropriate permissions to read from the schema registry. Below is an example Terraform configuration for an AWS IAM policy document that provides WarpStream with the appropriate permissions to access an AWS Glue schema registry. 
data "aws_iam_policy_document" "warpstream_aws_glue_policy_document" {
  statement {
    sid = "AWSGlueSchemaRegistryReadonlyAccess"
    actions = [
      "glue:GetSchemaVersion"
    ]

    effect    = "Allow"
    resources = [ "*" ]
  }
}
Note that there is currently a [terraform bug](https://github.com/awslabs/aws-glue-schema-registry/issues/68) that prevents providing the specific registry arn for the iam policy for AWS Glue. You would have to use `"*"` instead as showed in the example. ## Limitations Here are the list of schema features that are currently not supported: #### JSON Schema * Regular Expressions * Remote references (WarpStream supports referencing schemas that are part of the schema stored in the schema registry but not to remote schemas) * [Conditional schema validation](/warpstream/kafka/reference/protocol-and-feature-support) (`dependentRequired`) * `$dynamicRef` and `$dynamicAnchor` * `contentMediaType` # Schema Linking Replicate and migrate schemas to WarpStream BYOC Schema Registry. ## Overview WarpStream Schema Linking can migrate schemas from Confluent-compatible schema registries to WarpStream’s BYOC Schema Registry. During migration, your schemas will never leave your cloud environment and object storage buckets. In addition to migrating schemas, WarpStream Schema Linking also preserves schema IDs, subjects, subject versions, compatibility rules, etc. It even preserves whether subject versions are soft deleted. This means that after the migration, the destination schema registry should behave identically to the source schema registry from an API-level. WarpStream Schema Linking is embedded natively into the WarpStream Agent, so you don’t need to run any additional infrastructure beyond the WarpStream Agents to migrate schemas. Check out this overview video to learn more: {% embed url="" %} ## Getting Started Getting started with WarpStream Schema Linking is easy. First, deploy an agent for the schema registry you want to migrate schemas to. The agent must be able to run jobs, which is enabled by default or can be set explicitly with the `-roles` flag (check out the [Agent Roles documentation](https://docs.warpstream.com/warpstream/byoc/advanced-agent-deployment-options/splitting-agent-roles#configuring-agent-roles) for more details). For example: `warpstream agent -virtualClusterID vci_sr_XXXXXXXX -apiKey XXXXXXXX -bucketURL s3://my-warpstream-bucket -roles jobs,proxy` However, before deploying your Schema Linking configuration to production, let’s try migrating schemas into an ephemeral WarpStream Playground schema registry. You can start one with the command: `warpstream playground` This command will automatically create a schema registry and deploy a schema registry agent locally. The command will print a URL to a temporary WarpStream playground account. Open the URL in your browser and navigate to the schema registry. Next, click the Schema Linking tab. You’ll see a text editor that allows you to modify the config. From here, you can edit the configuration of your pipeline, pause it, resume it, and roll your configuration forwards and backwards. To learn more about the Schema Linking config, check out the [Config section](#configuration). Once you are done editing your config, you can click Save.
The configuration is now saved and deployed, but it’s not running yet. Click the toggle button above the Deploy button to change the pipeline’s state from PAUSED to RUNNING.
If you click on Full Details, you can see sync statistics such as how many subject versions were found in the source registry, how many newly migrated subject versions there are, etc.
When a Schema Linking pipeline is set up, the first thing it will do is to set the destination schema contexts' mode to `IMPORT`. This prevents anyone except for the pipeline to write to the schema registry. The destination schema context must be empty before the migration, so that the schema IDs can be preserved. If the pipeline detects that the destination schema context is not empty, it will automatically stop and fail. To confirm that the destination schema registry is in `IMPORT` mode, you can click on the Contexts tab.
Right now, your BYOC Schema Registry is just a read-replica of the source schema registry. To be able to write to the destination schema registry, you need to switch the schema contexts' modes from `IMPORT`to `READ_WRITE`. To do that, edit the config and set the `irreversible_switch_to_read_write_mode` field to true. After that, click the deploy button so that the pipeline will use the latest config. This operation is not reversible and it will terminate any in progress syncs.
After a while, you should see a new entry under Sync Stats stating that the schema contexts have been set to READ\_WRITE mode. The pipeline will also automatically stop running.
To confirm that, you can go to the Contexts tab and you can see that the default schema context is in `READ_WRITE` mode.
Now, you can read and write to your newly migrated WarpStream BYOC Schema Registry! ## How Syncing Works WarpStream Schema Linking continuously migrates the schema registry. This means that you can keep making changes to your source schema registry and the pipeline will eventually detect and apply those changes to the destination schema registry. Note that there are limitations for what you can do to the source schema registry during migration, specifically hard deleting subjects. Check out the [limitations section](#limitations) for more details. Once Schema Linking is deployed, the pipeline will periodically sync the source registry with the destination registry. You can configure how frequently the syncs occur (e.g. once every 5 minutes, once every hour, etc). During each sync, the Agents will fetch subjects, subject versions, and compatibility rules from the source schema registry using HTTP requests with Confluent’s Schema Registry API. The pipeline will then perform a diff between the source and destination to figure out what needs to be migrated. This means that after the initial sync, only newly registered schemas will be fetched and migrated. ## Configuration WarpStream Schema Linking is fully controllable from a single YAML config file which can be edited through the WarpStream console or the [Pipelines API](/warpstream/reference/api-reference/pipelines). ### Overview ```yaml sync_every_seconds: 300 context_type: "DEFAULT" source_schema_registry: hostname: "localhost" # schema registry hostname. port: 9094 # You can also optionally specify credentials here ``` Here’s a quick summary of the YAML file above: * The `sync_every_seconds` is set to `300`, which means that after each successful sync, the sync engine will wait for 300 seconds (5 minutes) before initiating a new sync. * The `context_type` is set to `"DEFAULT"`. This means that schemas will be copied from the default schema context of the source registry to the default schema context of the destination registry. Check out the [context types section](#context-types) on other context types. * The `source_schema_registry` field specifies the hostname, port, and credentials for the source schema registry HTTP server. ### Specifying Credentials #### TLS ```yaml source_schema_registry: hostname: "localhost" port: 9094 credentials: use_tls: true # Whether TLS verification should be skipped. tls_insecure_skip_verify: false ``` To make the Agents use TLS when connecting to the source schema registry, set the `use_tls` flag to true. ### Basic Auth ```yaml source_schema_registry: hostname: "localhost" port: 9094 credentials: basic_auth_username_env: BASIC_AUTH_USERNAME_ENV basic_auth_password_env: BASIC_AUTH_PASSWORD_ENV ``` To provide username/password for basic auth, you do not put the raw username/password in the config. Instead, you provide the environment variables that point to the username/password. When deploying your agents, set the environment variables `BASIC_AUTH_USERNAME_ENV` and `BASIC_AUTH_PASSWORD_ENV` to the basic auth’s username and env, respectively. ### mTLS ```yaml source_schema_registry: hostname: "localhost" port: 9094 credentials: use_tls: true mtls_client_cert_env: MTLS_CERT_PATH_ENV mtls_client_key_env: MTLS_KEY_PATH_ENV ``` When deploying your agents, set the environment variables `MTLS_CERT_PATH_ENV` and `MTLS_KEY_PATH_ENV` to the file paths to the PEM-encoded certificate and private key files. ### Context Types WarpStream’s BYOC Schema Registry supports schema contexts. From a higher level, each schema context can be viewed as a separate “sub-registry”, with an isolated group of schema IDs and subject names. Learn more about schema contexts with [Confluent’s schema contexts documentation](https://docs.confluent.io/platform/7.9/schema-registry/schema-linking-cp.html#schema-contexts). When setting up WarpStream schema linking, you can specify the source and destination schema contexts. Let’s check out the different context types. ### **Default Context** Each schema registry has a default context. When you register schemas and subjects without specifying an explicit context, you are writing to the default context. ```yaml sync_every_seconds: 300 context_type: "DEFAULT" source_schema_registry: hostname: "localhost" port: 9094 ``` If you specify the `context_type` to be `DEFAULT`, the schema migrator will migrate schemas and subjects from the source registry’s default schema context to the destination registry’s default schema context. ### **Context Mappings** To pick which schema contexts to migrate to/from, you can specify the `context_type` to be `CONTEXT_MAPPINGS`. Then use the `context_mappings`field to provide a list of source/destination schema contexts. ```yaml source_schema_registry: hostname: "localhost" port: 9094 sync_every_seconds: 300 context_type: "CONTEXT_MAPPINGS" context_mappings: - source_context: "." destination_context: ".dest_foo" - source_context: ".source_bar" destination_context: ".dest_bar" - source_context: ".source_baz" destination_context: ".dest_baz" ``` The config above specifies that the default schema context from the source registry is migrated to the `.dest_foo`, `.source_bar` is migrated into `.dest_bar`, and `.source_baz` is migrated into `.dest_baz`. ## **Limitations** Since WarpStream Schema Linking preserves schema IDs, the destination schema context must be empty before migration. If not, the pipeline will fail and stop. During migration, **hard deleting a subject is allowed only if you don’t register new schemas to that subject before the pipeline also hard deletes the subject** from the destination schema registry. During syncing, the Schema Linking pipeline assumes that a subject version always points to the same schema ID. This is true unless you hard delete a subject and register a schema under the subject. In that case, the subject version assignment resets to 1 and the assumption no longer holds. However, if you hard delete and wait long enough for the pipeline to detect that the subject is hard deleted and deletes it from the destination schema registry, you can then register new subject versions and the pipeline will replicate that correctly, since it treats it as a new subject. WarpStream's BYOC Schema Registry is not fully compatible with Confluent's Schema Registry. This means that things like `metadata`, `ruleSet` and `subject aliases` will not be migrated to the destination schema registry. Check out the [list of features](/warpstream/kafka/reference/protocol-and-feature-support#schema-registry) that WarpStream's BYOC Schema Registry doesn't support. # Manage Security This section explains how to manage security for WarpStream's Schema Registry product. ## Basic Authentication WarpStream's BYOC Schema Registry supports basic authentication. The following topic explains how to configure basic authentication in WarpStream. * [Basic Authentication](#basic-authentication) --- [Next Page](/warpstream/llms-full.txt/1)