Lightning topics

This page explains how to configure topics as Lightning Topics and what impact that will have.

Overview

Lightning Topics are a special topic type in WarpStream where the Agents skip committing metadata to the control plane in the critical path of a Produce() request. Instead, they journal Produce() requests to object storage, and then commit them to the control plane asynchronously.

As a result, Lightning Topics have dramatically lower Produce() request latency than regular topics. E2E latency is not impacted.

Lightning Topics provide the exact same durability guarantees as regular topics: any acknowledged Produce() request is guaranteed to be durable and (eventually) consumable. However, acknowledged data is not immediately visible to consumers due to the async commit which has a few broader implications. See the caveats sections for more details.

Lightning Topics are not a substitute for Ripcord. If your goal is to configure your Agents to continue processing Produce() requests even when the control plane is unavailable, you'll need to configure Ripcord.

Lightning Topics are tool for reducing the Produce() request latency for specific topics, but they don't guarantee that the workload will continue functioning in the face of control plane unavailability.

You need at least v743 of the WarpStream agent to produce records to a lightning topic.

Configuration

Converting an existing topic to a Lightning Topic can be accomplished by adding the Kafka topic configuration property: warpstream.topic.type and setting its value to lightning . This can be done via the Kafka API using standard Kafka tooling, or via the WarpStream UI, or via our Terraform provider.

resource "warpstream_topic" "topic" {
  topic_name         = "logs"
  partition_count    = 1
  virtual_cluster_id = warpstream_virtual_cluster.example.id

  config {
    name  = "retention.ms"
    value = "604800000"
  }
  
  config {
    name  = "warpstream.topic.type"
    value = "lightning"
  }
}

It is also possible to configure the default topic type of all newly created topics in a cluster. For this you can modify the broker configuration default.warpstream.topic.type and set the value to lightning using any standard Kafka tooling, or change the default topic type in the Cluster Settings page of the WarpStream console.

You can also set this at the virtual cluster level in terraform.

resource "warpstream_virtual_cluster" "my_cluster" {
  name = "vcn_my_cluster"
  tier = "pro"
  configuration = {
    default_topic_type = "lightning"
  }
}

Caveats

1. Offsets returned from Produce() requests will always be 0. Applications cannot rely on the returned offsets in the ProduceResponse. Almost no applications rely on this, but it's good to be aware of it.

2. The idempotent producer Kafka feature will not work (it must be disabled on all clients producing to Lightning Topics). (e.g. if your producer is configured with enable.idempotence , its produce requests will be rejected)

3. Kafka Transactions will not work (Transactions must be disabled on all clients producing to the Lightning Topics). (e.g. if your producer is configured with transactional.id , its produce requests will be rejected)

4. External consistency is no longer guaranteed. For example, if batch A is produced at time T0 and then a successful acknowledgement is received at T1, a batch B that is produced at T2 is not guaranteed to show up in the log after batch A.

Differences with Ripcord

Ripcord and Lightning Topics are implemented in a very similar way. Agents running in Ripcord mode are effectively treating all Kafka topics as Lightning Topics, although Ripcord mode also makes a few other changes to make the Agents more resilient to control plane unavailability.

If your objective is to reduce the latency of Produce() requests, use Lightning Topics. If your objective is to ensure that your WarpStream cluster can still accept Produce() requests if the WarpStream control plane is unavailable, configure Ripcord.