Rolling Restarts and Upgrades

This page explains how rolling restarts and upgrades work in WarpStream.

Graceful Shutdown

The WarpStream Agents perform a graceful shutdown routine when they receive a sigterm signal. The graceful shutdown routine strives to minimize disruption to the cluster and your Kafka clients.

By default, this graceful shutdown process takes 300 seconds (5 minutes) to complete. However, many container orchestration frameworks will not wait that long for a container to shutdown gracefully.

For example, in Kubernetes the default graceful termination window is 30 seconds. We recommend increasing this value to 600 seconds (10 minutes). In Kubernetes, the configuration value for this is called terminationGracePeriodSeconds.

In addition, for this graceful shutdown to work smoothly with your application, we recommend setting the metadata refresh interval on your client to 1 minute. See our client tuning documentation for more details.

If you want to change the graceful shutdown duration, you can tune the value of the -gracefulShutdownDuration flag or WARPSTREAM_GRACEFUL_SHUTDOWN_DURATION environment variable on your Agents. The value will be parsed as a duration (I.E 30s and 5m are both valid values). If you do this, you may also need to update the graceful termination window in Kubernetes or whatever software you're using to schedule your containers as described above.

Rolling Restarts

Rolling restarts are handled by WarpStream gracefully as long as the Agent containers are allowed to complete their graceful shutdown process, and clients are appropriately tuned to refresh their cluster metadata frequently as describe in the section above.

Rolling Upgrades

Rolling upgrades behave just like rolling restarts and leverage the same graceful shutdown mechanism. The only difference is that the Agent docker image tag is changed. Upgrades are always backwards compatible and no manual upgrade steps are required except deploying the new Docker image version in an incremental manner.