Tinybird

This page describes how to integrate WarpStream with Tinybird, ingest data into Tinybird from WarpStream, and then create an API endpoint for your applications to access the result set.

A video walkthrough can be found below:

Prerequisites

1. WarpStream account - get access to WarpStream by registering here.
2. Tinybird account - get access to Tinybird by registering here.
3. Serverless WarpStream cluster up and running.

Step 1: Create a topic in your WarpStream cluster

Obtain the Bootstrap Broker from the WarpStream console by navigating to your cluster and clicking the Connect tab. If you don't have SASL credentials yet, you can also create a set of credentials from the console.

Store these values for easy reference; they will be needed in Tinybird. If you are going to produce records to your topic from the command line, then export them in a terminal window:

export BOOTSTRAP_HOST=<YOUR_BOOTSTRAP_BROKER> \
SASL_USERNAME=<YOUR_SASL_USERNAME> \
SASL_PASSWORD=<YOUR_SASL_PASSWORD>;

Then, create a topic in the WarpStream console if you don't already have one.

Step 2: Produce some records

You can use the WarpStream CLI to produce messages to your topic if you don't already have an active topic to work with:

warpstream kcmd -bootstrap-host $BOOTSTRAP_HOST -tls -username $SASL_USERNAME -password $SASL_PASSWORD -type produce -topic decodable_demo --records '{"action": "click", "user_id": "user_0", "page_id": "home"},,{"action": "hover", "user_id": "user_0", "page_id": "home"},,{"action": "scroll", "user_id": "user_0", "page_id": "home"},,{"action": "click", "user_id": "user_1", "page_id": "home"},,{"action": "click", "user_id": "user_1", "page_id": "home"},,{"action": "click", "user_id": "user_2", "page_id": "home"}'

Note that the WarpStream CLI uses double commas (,,) as a delimiter between JSON records.

Step 3: Connect Tinybird to WarpStream

In the Tinybird Web Console, next to "DATA PROJECT," click "+" and select "Data Source," select "Kafka," and file in the connection information.

Tinybird Connection Panel

Tinybird Kafka Connection Details

Walk through the next steps to select your Topic and set your Configuration. Name the data source, and then click "Create Data Source," which will present a data preview and graph of ingested data, similar to the following:

Step 4: Configure your Tinybird pipeline

Once again, next to "DATA PROJECT," click "+" and select "Pipe." Here you will create your SQL code to power your Tinybird API Endpoint. Tinybird will validate your code and provide a subset of the qualified dataset. In this example, we are looking for any "action" that does not have a status of "ACCEPT," we've named this "rejects." Once satisfied with the results, click "Create API Endpoint."

Step 5: Deploy your Tinybird endpoint

Tinybird will present various code snippets to enable you to deploy your endpoint using the pipeline you just created.

TinyBird API deployment options

Taking the HTTP option and copying it into a browser yields the following result:

Next Steps

Congratulations! You've set up a stream processing pipeline between WarpStream and Tinybird.

Next, check out the WarpStream docs for configuring the WarpStream Agent, or review the Tinybird Docs to learn more about what is possible with WarpStream and Tinybird!