Ctrlk
WarpStream.comSlackDiscordContact UsCreate Account
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Client Metrics (KIP-714)

Configure your Kafka clients to push runtime metrics to WarpStream using KIP-714 client metrics subscriptions, then query them from the Events Explorer.

This feature requires your Virtual Cluster's Agents to run on v809 or higher.

KIP-714 lets a Kafka cluster ask connected clients to ship their internal producer/consumer metrics to the broker on a regular interval, with no application code changes. WarpStream implements the broker side of KIP-714: you create one or more named client metrics subscriptions on a Virtual Cluster, and any matching client will start pushing OTLP-encoded metrics to the Agents on the schedule you configured. The Agents decode each push and write one event per data point to the cluster's events stream, where you can search and aggregate them from the Events Explorer or the MCP Server.

KIP-714 is a Kafka protocol feature, so it works with any KIP-714-aware client (modern Java client, librdkafka, etc.). You do not need to install any agent or sidecar in your application.

Quick start

The fastest way to get telemetry flowing is to create a single subscription that targets every client at a 1-minute interval and asks for all metrics:

kafka-configs.sh --bootstrap-server $BOOTSTRAP \
    --alter \
    --entity-type client-metrics \
    --entity-name all-clients \
    --add-config 'interval.ms=60000,metrics=,match='

An empty metrics= value is the KIP-714 wildcard meaning "subscribe to every metric the client exposes". An empty match= matches every client. After a minute or two, open the Events tab on your Virtual Cluster in the WarpStream Console, pick the client_metrics event type, and you should start seeing data points roll in. See Viewing client metrics below for how to drill in.

Once you are comfortable, replace this catch-all with one or more narrower subscriptions targeting specific applications and metric prefixes.

Configuring subscriptions

A subscription is a named record with three configurable fields:

  • interval.ms — how often matched clients push, in milliseconds.

    • Default: 300000 (5 minutes). Min: 100. Max: 3600000 (1 hour).

  • metrics — a comma-separated list of metric-name prefixes the client should send. An entry is a prefix match (org.apache.kafka.producer. matches every producer metric). An empty list means "all metrics".

  • match — a comma-separated list of key=regex selectors that pick which clients this subscription applies to. An empty list matches every client.

Up to 100 subscriptions are allowed per Virtual Cluster.

There are three interchangeable ways to manage subscriptions: the web console, the public HTTP API via WarpStream's Terraform provider, and the standard Kafka AdminClient.

Using kafka-configs.sh (Kafka AdminClient)

WarpStream supports the KIP-714 admin surface: IncrementalAlterConfigs, AlterConfigs, DescribeConfigs, and ListConfigResources all accept client-metrics as the entity type.

Create or update a subscription:

If a value contains a comma, wrap it in [...] so kafka-configs.sh does not treat the inner commas as separators between configs:

List existing subscriptions:

Describe one by name:

Delete a subscription by clearing all of its fields:

Using the WarpStream HTTP API

Four endpoints under /api/v1/ manage subscriptions. See the Client Metrics API reference for full request and response schemas.

Operation
Endpoint

List Subscriptions

list_client_metrics_subscriptions

Describe Subscription

describe_client_metrics_subscription

Update Subscriptions

update_client_metrics_subscriptions

Delete Subscriptions

delete_client_metrics_subscriptions

The read endpoints are also exposed as MCP tools, so an AI assistant connected to your cluster can describe what subscriptions exist and how they are configured.

Note that the HTTP field is interval_ms, while the Kafka admin config name is interval.ms. Both refer to the same value.

Match selectors

Match selectors filter which clients a subscription applies to. The supported keys mirror the KIP-714 specification:

Key
What the broker matches against

client_instance_id

The CONNECT-time UUID assigned to a single client instance.

client_id

The Kafka client.id configured on the producer/consumer.

client_software_name

The client library name reported via ApiVersions (for example apache-kafka-java, librdkafka).

client_software_version

The client library version reported via ApiVersions.

client_source_address

The client's source IP as the Agent observes it.

client_source_port

The client's source TCP port as the Agent observes it.

A few things to keep in mind:

  • Patterns are full-string regex. Each pattern is anchored as ^(?:pattern)$. client_id=app will not match app-1; write client_id=app.* (or the equivalent ^app.*$).

  • Duplicate keys: the last selector wins. Inside one match value, client_id=^app$,client_id=^never$ matches nothing, because the second client_id selector overrides the first.

  • client_software_* selectors require ApiVersions v3+. Older clients do not advertise their software name and version, so those selectors never match them. Use client_id instead if you need to target legacy clients.

  • Multiple matching subscriptions are merged. If a client matches more than one subscription, the Agent unions the metric prefixes and uses the smallest push_interval_ms across them. There is no priority ordering.

Subscription examples

Push every producer metric, from every Java client, every minute:

Capture detailed latency from one specific application at 5-second resolution:

Sample everything from staging at 1-minute resolution, while leaving production untouched:

Event payload

Each data point in a PushTelemetry request becomes one event. The data payload looks like this:

Useful fields to filter and group by:

  • data.metric_name — the OTLP metric name (typically the Kafka client metric, e.g. org.apache.kafka.producer.request.total).

  • data.metric_type — gauge, sum, histogram, exponential_histogram, or summary.

  • data.labels — OTLP resource, scope, and data-point attributes flattened into a string map. This is where client_id lives, plus any per-data-point dimensions the client emits (such as node_id, topic, or partition).

  • data.point.value — for gauges and sums; histogram/summary types nest their fields here too.

  • data.source.client_instance_id — the per-client UUID, useful for following one client over time.

  • data.source.matched_subscription_names — which subscription(s) caused this push, useful for attributing data to its owner.

How it works

When a client first connects, it sends GetTelemetrySubscriptions (Kafka API key 71). The Agent looks at all configured subscriptions on that Virtual Cluster, finds the ones whose match selectors apply to this client, and replies with:

  • a subscription_id (a hash of the resolved configuration; it changes whenever a subscription is added, removed, or modified),

  • the smallest push_interval_ms across matching subscriptions,

  • the union of subscribed metric prefixes,

  • the broker's accepted compression codecs.

The client then sends PushTelemetry (API key 72) every push_interval_ms, carrying an OTLP MetricsData payload (compressed if both sides agree on a codec). The Agent decodes the payload and emits one CloudEvent per data point to the cluster's client_metrics events stream.

Defaults and limits

  • Default push interval (no subscription matches): 5 minutes. The client will poll on this schedule but send no data points.

  • Push interval bounds: 100ms minimum, 1 hour maximum.

  • Maximum subscriptions per cluster: 100. Contact us if you need a higher limit.

  • Maximum push payload (after decompression): 1 MiB per request. Clients that have more data than this in one push will skip metrics until the next interval.

  • Supported compression codecs (broker preference order): zstd, lz4, gzip, snappy. Clients pick whichever they prefer from this list; uncompressed pushes are also accepted.

  • Temporality: delta only. Cumulative-temporality metrics are rejected.

Notes and limitations

  • Throttling state is per-Agent. Each Agent independently enforces the configured push interval against the clients connected to it; that state is not shared. A client that reconnects to a different Agent may briefly bypass the per-interval throttle. This matches the KIP-714 specification and is rarely visible in practice because the events are still emitted with their time_unix_nano.

  • OTLP attribute keys with . are normalized to _. For example, an OTLP attribute named kafka.client.id becomes the event label kafka_client_id.

  • The broker-derived client_id always wins. If a client emits a client_id data-point attribute that disagrees with the client.id it set on the connection, the Agent overwrites it with the connection-level value and logs a one-shot warning per request.

  • Empty metrics= means all metrics. This is the KIP-714 wildcard, useful for exploration but expensive on busy clusters; switch to a list of prefixes once you know what you care about.

See also

PreviousConfiguring Kafka Client ID FeaturesNextHosted Metadata Endpoint

Last updated

Was this helpful?