Skip to content
WarpStream Documentation

Point-in-Time Recovery

Roll a WarpStream Kafka cluster back to an earlier snapshot of its metadata to recover from accidental deletions or bad changes.

Point-in-Time Recovery (PITR) restores a cluster back to an earlier snapshot of its state. You can use it to go back to a healthy state of your cluster if something went very wrong, like hard deleting a topic by accident or producing some poisoned records.

A recovery rolls back the cluster’s metadata (topics, their configuration, partitions and offsets, and consumer groups’ committed offsets) to the selected snapshot. Your data stays in your object storage bucket the whole time, and your Agents, credentials, and Bootstrap URLs don’t change.

To be able to restore a cluster, all of the data that was on the cluster at the restore point should still be available in your warpstream data bucket. This also includes all of the intermediate file objects that got compacted away right after, since the rolled back state won’t know about those compactions.

The way WarpStream compaction works, we keep compacted source files around for a while before garbage collecting them. This is controlled by the cluster’s Data Plane Garbage Collector Minimum Age setting, in the “Configuration” tab of the cluster detail page.

  • The default is 2 hours. It can be set to anything from 2 hours to 14 days.
  • Raising it extends the window, at the cost of keeping deleted files in your bucket for longer, increasing the amount of data in your bucket at any given time.
  • A new value only applies from the moment you change it. If you raise it from 2 hours to 7 days, you can’t restore to a point more than 2 hours old until that much time has passed.
  • A snapshot must be at least 10 minutes newer than the start of the window, so that the window doesn’t move past it while the recovery is running.

In short, if you want to be able to recover from mistakes you might not notice right away, raise the minimum age ahead of time. It can’t be extended after the fact.

A cluster can only be restored if:

  • It is a single-region BYOC Kafka cluster.
  • It has never used Ripcord. Ripcord deletes files from object storage in a way that bypasses the garbage collector’s minimum age, so a cluster that has ever used it can’t be restored.
  • No other migration, like a control plane region migration, is in progress.

To restore a cluster, there are four steps:

  • Choose a snapshot from the console
  • Scale your Agents down to 0 and start the recovery
  • Scale your Agents back up
  • Click the finish recovery button in the console

Each step must be fully completed before proceeding onto the next, the UI will guide you. For example, it won’t let you start the recovery if agents are still running).

Open the “Recovery” tab of the cluster detail page. It shows the restorable window, and checks whether the cluster meets the requirements above.

Pick the point in time you want to go back to. The console shows the closest snapshot taken before that time and the closest one taken after it. Pick the one before if you want to be sure the cluster is restored to a state from before the mistake happened. Snapshots on an active cluster typically happen every few seconds, so that’s the resolution available for the recovery.

Scale your Agent deployment down to 0, then click “Re-check agents” in the console if it still sees them. Once no Agents are running, click “Start a Recovery” on the snapshot you picked and confirm.

The cluster is rewound to the snapshot almost instantly. The “Recovery” tab then tracks the rest of the steps.

Scale your Agent deployment back up. You don’t need to change the Agents’ configuration.

Then check your clients are working as expected. Your producers and consumers should have restarted and none of them should hold on to state from the discarded timeline.

Once your Agents are back up and the cluster is healthy, click “Finish recovery” in the “Recovery” tab to return the cluster to normal operation.

The restored cluster continues from the snapshot’s state, so producers and consumers see the cluster as it was at that time:

  • Records produced after the snapshot are gone. New records are assigned offsets starting from each partition’s offset in the snapshot, so they can reuse offsets that records from the discarded timeline had.
  • Consumer groups keep their committed offsets from the snapshot, and resume from them when they rejoin. Group membership is reset, so every consumer rejoins its group. This means you won’t be reprocessing all the events from the start as if these were new consumer groups. You might process a few duplicates at the edge of the restore, on groups that hadn’t committed their offsets at the time of the snapshot.
  • Idempotent and transactional producers are assigned new producer IDs. Producer IDs from the discarded timeline are never reissued.
  • Transactions that were still open in the snapshot are aborted.