Migrating Control Plane Regions

This feature is only available for WarpStream Kafka clusters. If you need this for Schema Registry or TableFlow, contact us.

WarpStream Cluster's Control Plane (the part that WarpStream owns and hosts) lives in a specific (or multiple, if you're using Multi-Region Clusters) cloud provider region. If you have a cluster for which you want to move the Control Plane from one region to another (to reduce latency or switch cloud providers), or to convert it from a single-region control plane to a multi-region control plane, or any combination thereof, WarpStream offers a way to do so with virtually no downtime.

This will not affect the location of your actual data, of your agents, or anything else. It's mostly useful when moving live workloads around or upgrading a production cluster from single-region to multi-region for improved availability guarantees.

Changing a Cluster's Region

To change a cluster's region, there are four simple steps:

• Initiate the process by migrating the cluster metadata from the console

• Rolling restart your consumers with new config

• Rolling restart your agents with new config

• Click the finish migration button in the console

After this, the cluster will behave as if it had lived in the destination region(s) all along.

The only important thing to remember during this whole process, is that each step must be fully completed before proceeding onto the next.

It's also important that you actually go through with all the steps, as the cluster will technically be more sensitive (outages on either region can affect it) during the migration.

Initiate the migration

Go to the cluster list page, click on "Operations" and open up the "Migrate Cluster" modal. Select your destination region(s) and hit "Migrate".

This will put the cluster in a state where agents configured to talk to the old region(s) will still work normally, but the Control Plane will be migrated to the new region almost instantly (within a few seconds). Agents will automatically start rerouting their traffic to the new control plane region without any manual intervention.

Through this process, the only step where you might see some errors/backpressure from the Agents is this one. The cluster might appear unavailable for a few seconds right after you click the button (less than 10s).

After you click this button and until the end of the migration, a new tab will appear under this cluster's detail page, "Regional Migration", which will track your progress through the rest of the steps and tell you exactly what you need to do to finish it.

Update your client configurations

When you migrate the cluster, it gets a new Bootstrap URL for your kafka clients to connect to. Don't worry, until you finish the migration, the old ones will keep working! Head over to the "Connect" tab (or the "Regional Migration" tab) to grab the new URL and roll all your clients talking to this cluster to use it.

It's very important that you do this before continuing to avoid client downtime, as the next steps will make the old Bootstrap URLs stop working.

If your Kafka clients were not using this kind of Bootstrap URL previously, and instead were communicating through a Kubernetes DNS or some other kind of resolution mechanism, you can ignore this step entirely and there is no need to update them.

Update your agent configurations

Update the -region or -multiregion flag on your WarpStream agents to the new region. You can see exactly what value you must set in the "Deploy" tab of the cluster detail page, or on the "Regional Migration" tab. Once you fully roll the agents out with the new flag, and no agent is trying to talk to the old region, you'll see that the Regional Migration wizard marks this step as done. It dynamically watches for agent messages to the old region to keep track of this. You won't be able to click on the finish button until this happens.

Finish the migration

Once you are done with the above steps, simply click the finish migration button on the wizard to finish the process. You can now migrate to any other region again if you wish, or even back to the original one.

A note on Terraform

We don't allow regional migrations through Terraform. Changing the region field in Terraform will try to delete the cluster (which should hopefully be deletion protected). To prevent disasters, if your migrating cluster is managed through terraform, remember to:

• Update the cluster's terraform code to the new region right after the first step

• Run a plan to make sure it's clean