ShadowTraffic
ShadowTraffic is a containerized service for declaratively generating data, packed with knobs to perfectly mimic your production traffic to Kafka-compatible, and other destinations.
A video walkthrough can be found below:
Prerequisites
This guide does not instruct you on using ShadowTraffic; please refer to their docs for details. You'll need:
WarpStream account - get access to WarpStream by registering here.
WarpStream cluster up and running.
A ShadowTraffic account.
Step 1: Retrieve the WarpStream Bootstrap Broker URL and SASL credentials
Obtain the Bootstrap Broker from the WarpStream console by navigating to your cluster and clicking the Connect tab. If you don't have SASL credentials, you can also create a set of credentials from the console.
Save these values as you will need them in the next step:
export BOOTSTRAP_HOST=<YOUR_BOOTSTRAP_BROKER> \
SASL_USERNAME=<YOUR_SASL_USERNAME> \
SASL_PASSWORD=<YOUR_SASL_PASSWORD>;
Step 2: Configure ShadowTraffic JSON
Open the following file with your favorite editor and with the required permissions to edit it:
{
"generators" : [ {
"topic" : "<YOUR_TOPIC>",
"value" : {
"customerId" : {
"_gen" : "uuid"
},
Next, set up your credentials in the connections
section of the JSON, as noted below:
"connections" : {
"warpstream" : {
"kind" : "kafka",
"producerConfigs" : {
"bootstrap.servers" : "<YOUR_BOOTSTRAP_BROKER>",
"sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='<YOUR_SASL_USERNAME>' password='<YOUR_SASL_PASSWORD>';",
"value.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
"key.serializer": "io.shadowtraffic.kafka.serdes.JsonSerializer",
"sasl.mechanism": "PLAIN",
"security.protocol": "SASL_SSL"
}
}
}
Save the JSON file.
Step 3: Run ShadowTraffic
With your ShadowTraffic JSON configured, you can now produce your sample data to WarpStream. Follow the ShadowTraffic instructions, but the command will be something like this:
docker run --env-file license.env -v $(pwd)/parade.json:/home/<YOUR_JSON>.json shadowtraffic/shadowtraffic:latest --config /home/config.json --watch
Next steps
ShadowTraffic can now produce data to topics in your WarpStream cluster.
Last updated
Was this helpful?