Datadog Integration

Datadog metrics

Starting from Warpstream Agent v679 all metrics on Datadog will start with warpstream. and no longer warpstream_ . All references to metrics in our doc will keep mentioning metrics starting with warpstream_ so you have to do the conversion when you are using Datadog and a Warpstream Agent recent enough.

You can fall back to the previous behavior by setting the WARPSTREAM_DATADOG_NORMALIZER_PREFIX_WITH_DOT environment variable to false.

This change comes along the official release of our Datadog integration, making all the Warpstream Agent metrics free if you install the integration (and use the new naming convention)

Integration & Statsd Client (Push)

We recommend installing the official Datadog integration. We also have a pre-made Datadog Dashboard that you can easily import to get started.

The WarpStream Agents embed the Datadog statsd metrics client and can push directly to the Datadog agent by setting the -enableDatadogMetrics flag or adding WARPSTREAM_ENABLE_DATADOG_METRICS=true as an environment variable. Additionally, to configure the Datadog client properly, the DD_AGENT_HOST environment variable needs to be set to the host IP. For Agents running in AWS with IMDS enabled this step can be skipped.

The integration may complain in Datadog that it's missing some integration metric. You can disregard this, the python integration steps are optional and do not provide any additional insights on the WarpStream Agent health.

Prometheus Exporter (pull)

An alternative is to follow the Datadog instructions for scraping Prometheus/OpenTelemetry metrics using the Datadog Agent. Configuration will vary from environment to environment, but you should end up with something like the following configuration (Kubernetes example):

spec:
  template:
    metadata:
      annotations:
        ad.datadoghq.com/warpstream-agent.checks: |
          {
            "openmetrics": {
              "init_config": {},
              "instances": [
                {
                  "openmetrics_endpoint": "http://%%host%%:8080/metrics",
                  "metrics": [".*"],
                  "send_distribution_buckets": true,
                  "collect_counters_with_distributions": true,
                  "max_returned_metrics": 2000
                }
              ]
            }
          }

Which specifies that the Datadog Agent should scrape the WarpStream agent at port 8080 for metrics, and that it should scrape all the custom metrics that the WarpStream agent exposes.

The Datadog Agent will scrape only 2,000 metrics by default. This limit may be too low for WarpStream if you have many topics and consumer groups and/or have high cardinality metrics enabled. If you observe dropped or missing metrics, consider increasing this value.