> For the complete documentation index, see [llms.txt](https://docs.warpstream.com/warpstream/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.warpstream.com/warpstream/kafka/orbit/auto-migration.md). # Auto Migration ## Overview Orbit Auto Migration is a way to seamlessly migrate Kafka producers from a source cluster to a destination WarpStream cluster. Migrating Kafka producers traditionally requires a single coordinated maintenance window— every producer application has to be stopped, replication lag has to be drained to zero, and every producer has to be restarted pointing at the destination cluster. For deployments with more than a handful of producer applications, that means coordinating dozens or hundreds of restarts inside a tight window, often across multiple teams, with limited rollback options if something goes wrong. Orbit Auto migration removes the coordinated restart from the cutover. Producers are reconfigured to target a set of WarpStream agents ahead of time, at whatever pace suits the team that owns each application. Until cutover, the agents transparently forward writes to the source cluster; in parallel, Orbit replicates records from source to WarpStream. When you initiate the cutover, WarpStream briefly returns retriable errors to producers, waits for replication lag to drain, and then serves writes directly. Standard Kafka client retry behavior carries producers through the block window without a restart. For the manual approach, see [Manual Migration](https://docs.warpstream.com/warpstream/kafka/orbit#migration). ### How a Migration Works {% hint style="info" %} You need at least v807 of the WarpStream agent to use Orbit Auto Migration. {% endhint %} In a typical migration, you start with a collection of producers and consumers writing to and reading from your source Kafka cluster. The source can be any Kafka-compatible cluster including open-source Kafka, MSK, or even another WarpStream cluster. The goal is to end up with all of those producers and consumers connected to the destination WarpStream cluster, with no records lost in the process. Orbit Auto migration removes the manual coordination required by a traditional cutover, in which producers are paused in lockstep, replication lag is watched by hand, and every producer is restarted inside a single maintenance window. Once producers have been repointed at WarpStream, each topic is cut over independently via a single click or API call. To pull this off, Auto Migration treats every migration as a per-topic operation. Each topic moves through a number of states, initiated by the config you apply to Orbit, that determines where producer writes go and what producers see at each phase of the cutover: | State | What it means | Producer experience | | ----------- | ----------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `REJECT` | Default for Orbit-managed topics. Auto migration is not active for this topic. | Produces to WarpStream are rejected. Producers should still directly target the source cluster. | | `PROXY` | Auto migration is active. WarpStream forwards every produce request to source; Orbit replicates source → WarpStream as usual. | Normal success responses (records actually land at source). | | `MIGRATING` | Cutover in progress. WarpStream returns a retriable error to all producers while Orbit drains the remaining lag. | Retriable errors. Well-configured Kafka clients automatically retry with backoff. | | `COMPLETE` | Cutover complete. Orbit no longer manages the topic; produces go directly to WarpStream. | Normal success responses (records land directly in WarpStream). | `COMPLETE` is terminal- once a topic is migrated, it cannot be moved back. To understand the migration process in more detail, it's helpful to walkthrough a migration end-to-end. #### Step 1 — Prerequisites Before starting, confirm all of the following are in place: * **An Orbit pipeline replicating from source to destination.** Auto migration sits on top of standard Orbit replication. Orbit must already be replicating every topic you intend to migrate from your source cluster to the destination WarpStream cluster, and replication lag should be trending down. See [Configuring Orbit](https://docs.warpstream.com/warpstream/kafka/orbit#orbit-configuration) for further details. * **WarpStream credentials for your producers and consumers.** Each producer and consumer that will connect to WarpStream needs credentials on the destination cluster. See [here](https://docs.warpstream.com/warpstream/kafka/orbit#specify-credentials) for further information. * **ACLs on the destination WarpStream cluster.** If you use Kafka ACLs, configure equivalent ACLs on WarpStream so that producers and consumers have the permissions they expect once repointed. #### Step 2 — Migrate consumers Once Orbit is replicating and ACLs are in place, you can migrate your consumer applications to WarpStream. Note that in order for consumers to resume from the right position and avoid duplicate processing, Orbit must be configured to preserve source offsets. You can find details on how to migrate Consumers [here](https://docs.warpstream.com/warpstream/kafka/orbit#consumers). #### Step 3 — Configure auto migration Auto migration is configured under two top-level blocks in your Orbit config: * `auto_migration` — auto migration settings (e.g. which topics are enabled). * `auto_migration_source_cluster_credentials` — credentials for the proxy producer embedded in the agents that forwards client writes to source. A typical config looks like this: ```yaml auto_migration: enable: false # global default auto_migration_source_cluster_credentials: sasl_username_env: ORBIT_PROXY_SASL_USERNAME sasl_password_env: ORBIT_PROXY_SASL_PASSWORD sasl_mechanism: scram-512 use_tls: true topic_mappings: - source_topic: orders destination_topic: orders auto_migration: enable: true # opt this topic in - source_topic: legacy_audit destination_topic: legacy_audit # no auto_migration block — this topic stays in REJECT ``` Once you apply the config, every Orbit-managed topic for which `enable` resolves to `true` transitions from `REJECT` to `PROXY`- producing to those topics via the WarpStream agents will transparently forward writes to the source cluster. Topics with `enable: false` stay in `REJECT` and are unaffected. **Topic-level configs** Fields under the top-level `auto_migration:` block apply as **global defaults** to every Orbit-managed topic. Any field can be overridden per-topic under `topic_mappings[].auto_migration:`— a field set per-topic overrides the global value, and a field left unset inherits it. This is what lets you roll out selectively (keep `enable: false` globally and flip it on a few topics at a time). | Field | Default | Per-topic overridable | Description | | -------- | ------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable` | `false` | Yes | Turn auto migration on for matched topics. When `true`, the topic transitions from `REJECT` to `PROXY`: writes to the topic via WarpStream are transparently forwarded to source. | **Proxy credentials** Auto migration uses its own dedicated credentials to talk to the source cluster, separate from the credentials Orbit uses for replication reads. The Orbit config exposes them as two distinct blocks: * `source_cluster_credentials` — used by Orbit for **replication reads**. * `auto_migration_source_cluster_credentials` — used by the **proxy producer** embedded in the agents to forward client writes to source. The proxy is a regular Kafka producer client. It only ever calls two Kafka APIs: * `Metadata` — to discover partition leaders. * `Produce` — to forward client writes. Like Orbit's replication credentials, proxy credentials are read from environment variables— you specify the env-var **name** in the config, not the secret value itself. See [Source Cluster Credentials](https://docs.warpstream.com/warpstream/kafka/orbit#specify-credentials) for how env vars are wired in. You have two options for the proxy principal: * **Separate proxy principal (recommended for least privilege).** Provision a fresh SASL principal on the source cluster with only the permissions listed below. Your Orbit replication principal stays read-only. ```yaml source_cluster_credentials: sasl_username_env: ORBIT_SASL_USERNAME sasl_password_env: ORBIT_SASL_PASSWORD sasl_mechanism: scram-512 use_tls: true auto_migration_source_cluster_credentials: sasl_username_env: ORBIT_SASL_PROXY_USERNAME # different env var as source_cluster_credentials sasl_password_env: ORBIT_SASL_PROXY_PASSWORD sasl_mechanism: scram-512 use_tls: true ``` * **Reuse the Orbit replication principal.** If your existing Orbit replication principal already has (or can be granted) the produce permissions below, you can reuse it: point each `*_env` field under `auto_migration_source_cluster_credentials` at the same env-var names you set under `source_cluster_credentials`. While simpler operationally, the trade-off is that one principal now has both read and write on the source. 
source_cluster_credentials:
    sasl_username_env: ORBIT_SASL_USERNAME
    sasl_password_env: ORBIT_SASL_PASSWORD
    sasl_mechanism: scram-512
    use_tls: true

  auto_migration_source_cluster_credentials:
    sasl_username_env: ORBIT_SASL_USERNAME    # same env var as source_cluster_credentials
    sasl_password_env: ORBIT_SASL_PASSWORD
    sasl_mechanism: scram-512
    use_tls: true
The proxy supports the same authentication options as Orbit replication— SASL/PLAIN, SASL/SCRAM-256, SASL/SCRAM-512, mTLS (PEM or JKS), TLS-only, and plaintext. Field semantics for `sasl_username_env`, `mtls_client_cert_env`, `jks_key_store_file_path_env`, etc. are identical to `source_cluster_credentials`; see [Source Cluster Credentials](https://docs.warpstream.com/warpstream/kafka/orbit#specify-credentials) for the full reference. **Required ACLs on the source cluster** If your source cluster has ACLs enabled, the proxy principal needs the following ACLs on every topic that will be auto-migrated: | Operation | Resource | Why | | ---------- | -------- | ----------------------------------------------------------------------------------- | | `Write` | `Topic` | For `Produce`. | | `Describe` | `Topic` | For `Metadata`. (Modern Kafka grants `Describe` implicitly when `Write` is granted) | **Required ACLs on the destination cluster** If ACLs are enabled on the destination WarpStream cluster, the agent enforces them as normal during `PROXY`— including for produces forwarded to source. Configure the full set of ACLs your clients will need after cutover (produce, consume, consumer-group, transactional, cluster-level— whatever applies), and do so *before* enabling enforcement. The proxy authorizes every client request against the destination cluster's ACLs and rejects unauthorized requests with the usual Kafka authorization errors. For example, if a client with principal `foo` produces to topic `bar` while ACLs are enabled, `foo` must have `WRITE` on `bar` on the destination cluster, even though the write is being proxied to source— the proxy enforces destination ACLs in exactly the same way it will after cutover. To validate destination ACLs are correct before turning enforcement on, enable [**ACL shadowing**](https://docs.warpstream.com/warpstream/kafka/manage-security/configure-acls#acl-shadowing) on the destination cluster. The agent evaluates every produce against the configured ACLs but does not block— it emits an `ACL_SHADOW_DENIED` diagnostic and associated logs for any produce that *would* have been rejected. Review the diagnostics, fix gaps in your ACL configuration, then flip ACLs from shadowed to enforced. ### Step 4 — Cut over producers (Proxy phase) With auto migration enabled, your topics are now in `PROXY` state. Reconfigure your producer applications to point their `bootstrap.servers` at WarpStream. **There is no time pressure**— you can do this over hours, days, or weeks, in any order, as you finish redeploying each application. While a topic is in `PROXY` state: * WarpStream forwards every produce request to your source cluster. * Producers see the same success/failure semantics they would see producing directly to source. * Orbit continues replicating from the source to WarpStream. {% hint style="warning" %} **Producer compatibility** While a topic is in `PROXY` (or `MIGRATING`), the following Kafka producer features are **not supported**: * **Idempotent produce**— not supported. (Support may be added in a future release.) * **Transactional produce**— not supported. Producers using either feature must have it disabled before being repointed at WarpStream. Consult the relevant documentation for your client to see how to do this. Once a topic reaches `COMPLETE`, normal WarpStream produce semantics apply and these constraints no longer hold- for example, you can re-enable idempotent produce. {% endhint %} You can verify a topic is being proxied by checking the Migration tab in the Console or by querying the topic's migration state via the API— its state will show `PROXY` and lag will reflect normal Orbit replication. **Tuning your Producers** While a topic is in `PROXY` state, records are forwarded to your source cluster. However, we've made it such that the latency profile that producers experience matches what they'd see producing **straight to WarpStream**, not the source cluster's. Because of this, you can just follow the normal WarpStream guidance when tuning producers— see [Configure Clients](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client) and [Tuning for Performance](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/tuning-for-performance). After cutover the latency profile stays the same, so your producer config carries over and your producers keep operating as normal. ### Step 5 — Initiate migration When all producers for a topic have been pointed at WarpStream, writes are being proxied, and replication lag is stable, you're ready to cut over. #### Initiating Migrations can be initiated either from the UI or via the API. **UI:** In the WarpStream console, open the **Migration** tab, select one or more topics in PROXY state, and click **Migrate**. **API:** Use the [Initiate Orbit Topic Auto Migration endpoint](https://app.gitbook.com/o/xuJtdDASiqpJSjc2Fs19/s/jB7FxO8ty4EXO4HsQP4E/~/edit/~/changes/1143/reference/api-reference/orbit-auto-migration/initiate). Topics can be selected by literal name, by regex, or both, and a `dry_run` flag is available to preview which topics would be initiated without actually doing so. #### Migration Parameters For each migration request, there are certain parameters that can be set. These apply per-request— if you roll back to `PROXY` and re-initiate later, you can supply different values for the second attempt (e.g. tighter lag thresholds once you have a better sense of the topic's traffic profile). Any field omitted from the request falls back to the default shown below. | Parameter | Type | Required | Default | Description | | --------------------------- | ---- | -------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `max_time_lag_seconds` | int | No | `180` (3m) | Per-topic time-lag ceiling, in seconds, at the moment of initiation. **Mutually exclusive** with `max_offset_lag`. Must be ≥ 0. Used by default when both lag fields are unset. | | `max_offset_lag` | int | No | `0` (unused) | Per-topic offset-lag ceiling at the moment of initiation; topics above it are reported in `errors` and not initiated. **Mutually exclusive** with `max_time_lag_seconds`. Must be ≥ 0. Prefer `max_time_lag_seconds` unless you have a specific reason. | | `migration_timeout_seconds` | int | No | `300` (5m) | Maximum seconds a topic may remain in `MIGRATING` before the scheduler rolls it back to `PROXY`. Must be ≥ 0. | > Set **at most one** of `max_offset_lag` and `max_time_lag_seconds` — supplying both positive values is rejected with `400 conflicting_lag_thresholds`. #### The Migration Phase Once initiation is accepted, the topic enters the `MIGRATING` state. WarpStream returns a retriable error to all producers for that topic— well-behaved Kafka clients automatically retry with backoff, so your applications do not crash. No new records enter the source cluster via the proxy while Orbit drains the remaining lag. The dominant factor in how long a migration takes is **how quickly Orbit replication lag drops to 0** for each topic— a migration cannot succeed until it does. To reduce migration time, the most useful lever is tuning your Orbit configuration (concurrency, fetch sizes, and related settings) to drain lag faster. See the [Orbit performance tuning guide](https://docs.warpstream.com/warpstream/kafka/orbit#tuning-orbit) for details. #### Cutover Once lag has reached zero and any inflight requests have been drained, the topic is promoted to `COMPLETE`. From this point, producer retries succeed and writes go directly into WarpStream. #### Timeout and rollback If lag does not reach zero within the configured `block_timeout_seconds` (default 300 seconds/5 minutes), the topic automatically rolls back to `PROXY` state and producers resume forwarding to source. #### Post-migration Once a topic is in `COMPLETE` state, Orbit stops managing it and producer writes go directly to WarpStream; the producer experience returns to normal and no further action is needed. `SERVE` is terminal. Repeat Step 4 for any remaining topics— topics migrate independently of one another. ### Aborting a migration If you've initiated a migration but want to back out, for example, you've spotted some misconfiguration, you can abort while a topic is in `MIGRATING` state. Abort rolls the topic back to a `PROXY` state immediately, without waiting for `migration_timeout_seconds` to elapse, and producers resume forwarding to source. Topics in `COMPLETE` state cannot be aborted— migration is one-way. Migrations can be aborted either by clicking the `Abort` action on the relevant row in the Migration tab in the Console or programmatically via the API, using the [Abort Orbit Topic Auto Migration endpoint](https://app.gitbook.com/o/xuJtdDASiqpJSjc2Fs19/s/jB7FxO8ty4EXO4HsQP4E/~/edit/~/changes/1143/reference/api-reference/orbit-auto-migration/abort). ### Monitoring migration The Migration tab in the WarpStream console is the main place to watch a migration. It shows every Orbit-managed topic with a state badge (`REJECT` / `PROXY` / `MIGRATING` / `COMPLETE`), per-topic offset lag and time lag, and a state-summary bar with topic counts in each state. Similar data is available programmatically via the [Orbit Topic Auto Migration Status endpoint](https://app.gitbook.com/o/xuJtdDASiqpJSjc2Fs19/s/jB7FxO8ty4EXO4HsQP4E/~/edit/~/changes/1143/reference/api-reference/orbit-auto-migration/status).
#### Metrics Key metrics to watch include: * **Orbit replication lag** (offset lag and time lag, per topic)— directly visible in the Migration tab. This is the single biggest input to how long a migration will take. If lag isn't dropping, the migration won't progress. See the [Orbit observability section](/warpstream/kafka/orbit.md#observability) for more details on how to monitor Orbit's offset and time lag. * **Source-cluster produce throughput vs. WarpStream destination throughput**— useful to confirm the proxy is forwarding everything and that Orbit is keeping up. * **WarpStream proxy throughput**— the agent emits metrics for the number of compressed bytes and number of records that were proxied to the source cluster. See the [Metrics page](/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs.md#auto-migration) for more details on the metrics. #### Diagnostics & Events The Migration tab also surfaces migration diagnostics— checks that flag potential issues during and after a migration. These include, among others: * **Unproxied source writes**— records are being written directly to the source cluster, bypassing the WarpStream proxy. This usually means a producer wasn't reconfigured to point at WarpStream, and those records risk being lost when the source is decommissioned. * **Post-migration source writes**— records landed on the source cluster *after* a topic finished migrating. Unlike the case above, these records are *not* replicated to WarpStream and will be lost when the source is decommissioned— a strong signal a producer is still pointing at the source. The tab also shows a feed of migration events— a per-topic lifecycle log covering initiation, completion, timeouts, and aborts. ### FAQ / Troubleshooting **A migration keeps timing out and rolling back to PROXY.** The cause is replication lag failing to reach zero within `migration_timeout_seconds`. There are three things to check: 1. **Is lag decreasing during migration but just not fast enough?** If so, increase `migration_timeout_seconds` to give Orbit a longer window to drain. 2. **Is Orbit replication itself the bottleneck?** Tune Orbit's concurrency and fetch settings to drain lag faster— see [Orbit performance tuning](https://docs.warpstream.com/warpstream/kafka/orbit/auto-migration?cache=1781287057#:~:text=resource%2Dconstrained.%20See-,Orbit%20performance%20tuning,-for%20the%20relevant). 3. **Is something keeping lag elevated?** The most common cause is a producer still writing directly to the source cluster, bypassing WarpStream— those writes during a migration keep lag from reaching zero. Audit producer configurations to confirm every producer for the topic has its `bootstrap.servers` pointed at WarpStream. Check that the "Orbit Auto Migration Unproxied Source Writes" diagnostic is not firing. **How can I make a migration finish faster?** Migration time is dominated by how quickly Orbit can drain replication lag to zero. Until lag drops below the topic's configured threshold, initiation is rejected; once a topic starts `MIGRATING`, it stays there until lag reaches zero (or `migration_timeout_seconds` elapses and it rolls back). The most useful lever is therefore Orbit itself— increasing Orbit's concurrency, tuning fetch sizes, and ensuring the agents replicating the topic aren't resource-constrained. See [Orbit performance tuning](https://docs.warpstream.com/warpstream/kafka/orbit#tuning-orbit) for the relevant settings. **Can I migrate one partition at a time?** No. Migration state is per-topic. All partitions of a topic must drain to zero lag before the topic can flip from `MIGRATING` to `COMPLETE`. **Can I roll a topic back from `COMPLETE`?** No. `COMPLETE` is terminal. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.warpstream.com/warpstream/kafka/orbit/auto-migration.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.