OpenTelemetry Collector

This page describes how to connect the OpenTelemetry Collector to WarpStream using the Kafka exporter.

The Kafka exporteris no longer bundled as part of the standard otel collector, so you'll need to use one of the contrib releases.

Unauthenticated / BYOC WarpStream Cluster (Example)

The sample configuration file below demonstrates how to configure the OpenTelemetry collector with a Kafka exporter that writes to an unauthenticated WarpStream cluster. This setup makes the most sense when you're self-hosting the WarpStream Agents in your own cloud account / VPC using the BYOC product.

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  kafka:
    brokers: ["<YOUR_WARPSTREAM_BOOTSTRAP_URL>:9092"]
    topic: "traces"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, kafka]

Authenticated WarpStream Cluster (Example)

The sample configuration file below demonstrates how to configure the OpenTelemetry collector with a Kafka exporter that writes to an authenticated WarpStream cluster.

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  kafka:
    brokers: ["<YOUR_WARPSTREAM_BOOTSTRAP_URL>:9092"]
    topic: "traces"
    auth:
      sasl:
        username: $USERNAME
        password: $PASSWORD
        mechanism: PLAIN
        version: 1
      tls:
        insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, kafka]

Configuring Zone Aware Routing

v0.92.0 of the collector supports specifying Kafka client IDs like so:

receivers:
  otlp:
    protocols:
      http:
        cors:
          allowed_origins:
            - "*"

processors:
  batch:

exporters:
  logging:
    loglevel: debug
  kafka:
    brokers: ["<YOUR_WARPSTREAM_BOOTSTRAP_URL>:9092"]
    client_id: "warpstream_az=us-east-1a"
    topic: "traces"
    auth:
      sasl:
        username: $USERNAME
        password: $PASSWORD
        mechanism: PLAIN
        version: 1
      tls:
        insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, kafka]

See our documentation on configuring WarpStream features via the Kafka Client ID for more details.