ShadowTraffic

ShadowTraffic is a containerized service for declaratively generating data, packed with knobs to perfectly mimic your production traffic to Kafka-compatible, and other destinations.

A video walkthrough can be found below:

Prerequisites

This guide does not instruct you on using ShadowTraffic; please refer to their docs for details. You'll need:

1. WarpStream account - get access to WarpStream by registering here.
2. WarpStream cluster up and running.

3. A ShadowTraffic account.

Step 1: Retrieve the WarpStream Bootstrap Broker URL and SASL credentials

Obtain the Bootstrap Broker from the WarpStream console by navigating to your cluster and clicking the Connect tab. If you don't have SASL credentials, you can also create a set of credentials from the console.

Save these values as you will need them in the next step:

export BOOTSTRAP_HOST=<YOUR_BOOTSTRAP_BROKER> \
SASL_USERNAME=<YOUR_SASL_USERNAME> \
SASL_PASSWORD=<YOUR_SASL_PASSWORD>;

Step 2: Configure ShadowTraffic JSON

Open the following file with your favorite editor and with the required permissions to edit it:

{
  "generators" : [ {
    "topic" : "<YOUR_TOPIC>",
    "value" : {
      "customerId" : {
        "_gen" : "uuid"
      },

Next, set up your credentials in the connections section of the JSON, as noted below:

  "connections" : {
    "warpstream" : {
      "kind" : "kafka",
      "producerConfigs" : {
        "bootstrap.servers" : "<YOUR_BOOTSTRAP_BROKER>",
        "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='<YOUR_SASL_USERNAME>' password='<YOUR_SASL_PASSWORD>';",
        "value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
        "key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
        "sasl.mechanism": "PLAIN",
        "security.protocol": "SASL_SSL"
      }
    }
  }

Save the JSON file.

Step 3: Run ShadowTraffic

With your ShadowTraffic JSON configured, you can now produce your sample data to WarpStream. Follow the ShadowTraffic instructions, but the command will be something like this:

docker run   --env-file license.env   -v $(pwd)/parade.json:/home/<YOUR_JSON>.json   shadowtraffic/shadowtraffic:latest   --config /home/config.json   --watch

Next steps

ShadowTraffic can now produce data to topics in your WarpStream cluster.