> For the complete documentation index, see [llms.txt](https://docs.warpstream.com/warpstream/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/client-metrics-kip-714.md).

# Client Metrics (KIP-714)

**This feature requires your Virtual Cluster's Agents to run on v805 or higher.**

[KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) lets a Kafka cluster ask connected clients to ship their internal producer/consumer metrics to the broker on a regular interval, with no application code changes. WarpStream implements the broker side of KIP-714: you create one or more named **client metrics subscriptions** on a Virtual Cluster, and any matching client will start pushing OTLP-encoded metrics to the Agents on the schedule you configured. The Agents decode each push and write one event per data point to the cluster's events stream, where you can search and aggregate them from the [Events Explorer](/warpstream/reference/events.md) or the [MCP Server](/warpstream/reference/mcp-server.md).

{% hint style="info" %}
KIP-714 is a Kafka protocol feature, so it works with any KIP-714-aware client (modern Java client, librdkafka, etc.). You do not need to install any agent or sidecar in your application.
{% endhint %}

## Quick start

The fastest way to get telemetry flowing is to create a single subscription that targets every client at a 1-minute interval and asks for all metrics:

{% code overflow="wrap" %}

```bash
kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --alter \
    --entity-type client-metrics \
    --entity-name all-clients \
    --add-config 'interval.ms=60000,metrics=,match='
```

{% endcode %}

An empty `metrics=` value is the KIP-714 wildcard meaning "subscribe to every metric the client exposes". An empty `match=` matches every client. After a minute or two, open the **Events** tab on your Virtual Cluster in the [WarpStream Console](https://console.warpstream.com), pick the `client_metrics` event type, and you should start seeing data points roll in. See [Viewing client metrics](#viewing-client-metrics) below for how to drill in.

Once you are comfortable, replace this catch-all with one or more narrower subscriptions targeting specific applications and metric prefixes.

## Configuring subscriptions

A subscription is a named record with three configurable fields:

* `interval.ms` — how often matched clients push, in milliseconds.
  * Default: 300000 (5 minutes). Min: 100. Max: 3600000 (1 hour).
* `metrics` — a comma-separated list of metric-name prefixes the client should send. An entry is a prefix match (`org.apache.kafka.producer.` matches every producer metric). An empty list means "all metrics".
* `match` — a comma-separated list of `key=regex` selectors that pick which clients this subscription applies to. An empty list matches every client.

Up to 100 subscriptions are allowed per Virtual Cluster.

There are three interchangeable ways to manage subscriptions: the web console, the public HTTP API via WarpStream's Terraform provider, and the standard Kafka AdminClient.

### Using `kafka-configs.sh` (Kafka AdminClient)

WarpStream supports the KIP-714 admin surface: `IncrementalAlterConfigs`, `AlterConfigs`, `DescribeConfigs`, and `ListConfigResources` all accept `client-metrics` as the entity type.

Create or update a subscription:

{% code overflow="wrap" %}

```bash
kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --alter \
    --entity-type client-metrics \
    --entity-name producers \
    --add-config 'interval.ms=60000,metrics=org.apache.kafka.producer.,match=client_id=app-.*'
```

{% endcode %}

If a value contains a comma, wrap it in `[...]` so `kafka-configs.sh` does not treat the inner commas as separators between configs:

{% code overflow="wrap" %}

```bash
kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --alter \
    --entity-type client-metrics \
    --entity-name app-and-staging \
    --add-config 'interval.ms=30000,metrics=[org.apache.kafka.producer.,org.apache.kafka.consumer.],match=[client_id=app-.*,client_software_name=apache-kafka-java]'
```

{% endcode %}

List existing subscriptions:

```bash
kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --describe \
    --entity-type client-metrics
```

Describe one by name:

```bash
kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --describe \
    --entity-type client-metrics \
    --entity-name producers
```

Delete a subscription by clearing all of its fields:

{% code overflow="wrap" %}

```bash
kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --alter \
    --entity-type client-metrics \
    --entity-name producers \
    --delete-config 'interval.ms,metrics,match'
```

{% endcode %}

### Using the WarpStream HTTP API

Four endpoints under `/api/v1/` manage subscriptions. See the [Client Metrics API reference](/warpstream/reference/api-reference/client-metrics.md) for full request and response schemas.

| Operation                                                                               | Endpoint                               |
| --------------------------------------------------------------------------------------- | -------------------------------------- |
| [List Subscriptions](/warpstream/reference/api-reference/client-metrics/list.md)        | `list_client_metrics_subscriptions`    |
| [Describe Subscription](/warpstream/reference/api-reference/client-metrics/describe.md) | `describe_client_metrics_subscription` |
| [Update Subscriptions](/warpstream/reference/api-reference/client-metrics/update.md)    | `update_client_metrics_subscriptions`  |
| [Delete Subscriptions](/warpstream/reference/api-reference/client-metrics/delete.md)    | `delete_client_metrics_subscriptions`  |

The read endpoints are also exposed as MCP tools, so an AI assistant connected to your cluster can describe what subscriptions exist and how they are configured.

Note that the HTTP field is `interval_ms`, while the Kafka admin config name is `interval.ms`. Both refer to the same value.

### Match selectors

Match selectors filter which clients a subscription applies to. The supported keys mirror the KIP-714 specification:

| Key                       | What the broker matches against                                                                   |
| ------------------------- | ------------------------------------------------------------------------------------------------- |
| `client_instance_id`      | The CONNECT-time UUID assigned to a single client instance.                                       |
| `client_id`               | The Kafka `client.id` configured on the producer/consumer.                                        |
| `client_software_name`    | The client library name reported via ApiVersions (for example `apache-kafka-java`, `librdkafka`). |
| `client_software_version` | The client library version reported via ApiVersions.                                              |
| `client_source_address`   | The client's source IP as the Agent observes it.                                                  |
| `client_source_port`      | The client's source TCP port as the Agent observes it.                                            |

A few things to keep in mind:

* **Patterns are full-string regex.** Each pattern is anchored as `^(?:pattern)$`. `client_id=app` will not match `app-1`; write `client_id=app.*` (or the equivalent `^app.*$`).
* **Duplicate keys: the last selector wins.** Inside one `match` value, `client_id=^app$,client_id=^never$` matches nothing, because the second `client_id` selector overrides the first.
* **`client_software_*` selectors require ApiVersions v3+.** Older clients do not advertise their software name and version, so those selectors never match them. Use `client_id` instead if you need to target legacy clients.
* **Multiple matching subscriptions are merged.** If a client matches more than one subscription, the Agent unions the metric prefixes and uses the smallest `push_interval_ms` across them. There is no priority ordering.

### Subscription examples

Push every producer metric, from every Java client, every minute:

```
interval.ms = 60000
metrics     = org.apache.kafka.producer.
match       = client_software_name=apache-kafka-java
```

Capture detailed latency from one specific application at 5-second resolution:

```
interval.ms = 5000
metrics     = org.apache.kafka.producer.node.request.latency,org.apache.kafka.consumer.node.request.latency
match       = client_id=checkout-service-.*
```

Sample everything from staging at 1-minute resolution, while leaving production untouched:

```
interval.ms = 60000
metrics     =
match       = client_id=staging-.*
```

### Event payload

Each data point in a `PushTelemetry` request becomes one event. The `data` payload looks like this:

```json
{
    "metric_schema_version": 1,
    "metric_name": "org.apache.kafka.producer.request.total",
    "metric_type": "gauge",
    "metric_unit": "1",
    "labels": {
        "client_id": "prod-app",
        "node_id": "1"
    },
    "time_unix_nano": 1737059620000000000,
    "point": {
        "value": 42
    },
    "source": {
        "kind": "client",
        "client_instance_id": "T48WSCxIRkZPj6r5...",
        "client_instance_id_uuid": "4f8f1648-...",
        "subscription_id": -123456789,
        "matched_subscription_names": ["prod-apps"],
        "terminating": false,
        "ingested_at_unix_nano": 1737059623000000000
    }
}
```

Useful fields to filter and group by:

* `data.metric_name` — the OTLP metric name (typically the Kafka client metric, e.g. `org.apache.kafka.producer.request.total`).
* `data.metric_type` — `gauge`, `sum`, `histogram`, `exponential_histogram`, or `summary`.
* `data.labels` — OTLP resource, scope, and data-point attributes flattened into a string map. This is where `client_id` lives, plus any per-data-point dimensions the client emits (such as `node_id`, `topic`, or `partition`).
* `data.point.value` — for gauges and sums; histogram/summary types nest their fields here too.
* `data.source.client_instance_id` — the per-client UUID, useful for following one client over time.
* `data.source.matched_subscription_names` — which subscription(s) caused this push, useful for attributing data to its owner.

## How it works

When a client first connects, it sends `GetTelemetrySubscriptions` (Kafka API key 71). The Agent looks at all configured subscriptions on that Virtual Cluster, finds the ones whose `match` selectors apply to this client, and replies with:

* a `subscription_id` (a hash of the resolved configuration; it changes whenever a subscription is added, removed, or modified),
* the smallest `push_interval_ms` across matching subscriptions,
* the union of subscribed metric prefixes,
* the broker's accepted compression codecs.

The client then sends `PushTelemetry` (API key 72) every `push_interval_ms`, carrying an OTLP `MetricsData` payload (compressed if both sides agree on a codec). The Agent decodes the payload and emits one CloudEvent per data point to the cluster's `client_metrics` events stream.

## Defaults and limits

* **Default push interval (no subscription matches):** 5 minutes. The client will poll on this schedule but send no data points.
* **Push interval bounds:** 100ms minimum, 1 hour maximum.
* **Maximum subscriptions per cluster:** 100. [Contact us](https://www.warpstream.com/contact-us) if you need a higher limit.
* **Maximum push payload (after decompression):** 1 MiB per request. Clients that have more data than this in one push will skip metrics until the next interval.
* **Supported compression codecs (broker preference order):** `zstd`, `lz4`, `gzip`, `snappy`. Clients pick whichever they prefer from this list; uncompressed pushes are also accepted.
* **Temporality:** delta only. Cumulative-temporality metrics are rejected.

## Notes and limitations

* **Throttling state is per-Agent.** Each Agent independently enforces the configured push interval against the clients connected to it; that state is not shared. A client that reconnects to a different Agent may briefly bypass the per-interval throttle. This matches the KIP-714 specification and is rarely visible in practice because the events are still emitted with their `time_unix_nano`.
* **OTLP attribute keys with `.` are normalized to `_`.** For example, an OTLP attribute named `kafka.client.id` becomes the event label `kafka_client_id`.
* **The broker-derived `client_id` always wins.** If a client emits a `client_id` data-point attribute that disagrees with the `client.id` it set on the connection, the Agent overwrites it with the connection-level value and logs a one-shot warning per request.
* **Empty `metrics=` means all metrics.** This is the KIP-714 wildcard, useful for exploration but expensive on busy clusters; switch to a list of prefixes once you know what you care about.

## See also

* [Events Explorer](/warpstream/reference/events.md)
* [Important Metrics and Logs](/warpstream/agent-setup/monitor-the-warpstream-agents/important-metrics-and-logs.md)
* [Diagnostics](/warpstream/agent-setup/monitor-the-warpstream-agents/diagnostics.md)
* [Apache Kafka KIP-714](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.warpstream.com/warpstream/kafka/configure-kafka-client/client-metrics-kip-714.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.