# Lightning topics

## Overview

Lightning Topics are a special topic type in WarpStream where the Agents skip committing metadata to the control plane in the critical path of a `Produce()` request. Instead, they journal `Produce()` requests to object storage, and then commit them to the control plane asynchronously.

As a result, Lightning Topics have dramatically lower `Produce()` request latency than regular topics. E2E latency is not impacted.

Lightning Topics provide the exact same durability guarantees as regular topics: any acknowledged `Produce()` request is guaranteed to be durable and (eventually) consumable. However, acknowledged data is not immediately visible to consumers due to the async commit which has a few broader implications. See the [caveats sections](#caveats) for more details.

{% hint style="danger" %}
Lightning Topics are not a substitute for [Ripcord](https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/ripcord). If your goal is to configure your Agents to continue processing `Produce()` requests even when the control plane is unavailable, you'll need to configure Ripcord.

Lightning Topics are tool for reducing the `Produce()` request latency for specific topics, but they don't guarantee that the workload will continue functioning in the face of control plane unavailability.
{% endhint %}

{% hint style="info" %}
You need at least v743 of the WarpStream agent to produce records to a lightning topic.
{% endhint %}

## Configuration

Converting an existing topic to a Lightning Topic can be accomplished by adding the Kafka topic configuration property: `warpstream.topic.type` and setting its value to `lightning` . This can be done via the Kafka API using standard Kafka tooling, or via the WarpStream UI, or via our Terraform provider.

```hcl
resource "warpstream_topic" "topic" {
  topic_name         = "logs"
  partition_count    = 1
  virtual_cluster_id = warpstream_virtual_cluster.example.id

  config {
    name  = "retention.ms"
    value = "604800000"
  }
  
  config {
    name  = "warpstream.topic.type"
    value = "lightning"
  }
}
```

It is also possible to configure the default topic type of all newly created topics  in a cluster. For this you can modify the broker configuration `warpstream.default.topic.type` and set the value to `lightning` using most standard Kafka tooling (note that some Kafka tools don't allow setting configs which aren't available in Kafka), or change the default topic type in the Cluster Settings page of the WarpStream console.

You can also set this at the virtual cluster level in terraform.

```hcl
resource "warpstream_virtual_cluster" "my_cluster" {
  name = "vcn_my_cluster"
  tier = "pro"
  configuration = {
    default_topic_type = "lightning"
  }
}
```

## Caveats

1. Offsets returned from `Produce()` requests will always be 0. Applications cannot rely on the returned offsets in the ProduceResponse. Almost no applications rely on this, but it's good to be aware of it.
2. The idempotent producer Kafka feature will not work (it must be disabled on all clients producing to Lightning Topics).\
   (e.g. if your producer is configured with `enable.idempotence` , its produce requests will be rejected)
3. Kafka Transactions will not work (Transactions must be disabled on all clients producing to the Lightning Topics).\
   (e.g. if your producer is configured with `transactional.id` , its produce requests will be rejected)
4. External consistency is no longer guaranteed. For example, if batch A is produced at time T0 and then a successful acknowledgement is received at T1, a batch B that is produced at T2 **is not guaranteed to show up in the log after batch A.**

## Differences with Ripcord

[Ripcord](https://docs.warpstream.com/warpstream/kafka/advanced-agent-deployment-options/ripcord) and Lightning Topics are implemented in a very similar way. Agents running in Ripcord mode are effectively treating all Kafka topics as Lightning Topics, although Ripcord mode also makes a few other changes to make the Agents more resilient to control plane unavailability.

If your objective is to reduce the latency of `Produce()` requests, use Lightning Topics. If your objective is to ensure that your WarpStream cluster can still accept `Produce()` requests if the WarpStream control plane is unavailable, configure Ripcord.