Tinybird

This page describes how to integrate WarpStream with Tinybird, ingest data into Tinybird from WarpStream, and then create an API endpoint for your applications to access the result set.

A video walkthrough can be found here.

Prerequisites

  1. WarpStream account - get access to WarpStream by registering here.

  2. Tinybird account - get access to Tinybird by registering here.

  3. Serverless WarpStream cluster up and running.

Step 1: Create a topic in your WarpStream cluster

Obtain the Bootstrap Broker from the WarpStream console by navigating to your cluster and then clicking the Connect tab. If you don't have SASL credentials yet, you can also create a set of credentials from the console.

Store these values for easy reference; they will be needed in Tinybird. If you are going to produce records to your topic from the command line, then export them in a terminal window:

export BOOTSTRAP_HOST=<YOUR_BOOTSTRAP_BROKER> \
SASL_USERNAME=<YOUR_SASL_USERNAME> \
SASL_PASSWORD=<YOUR_SASL_PASSWORD>;

Then, create a topic in the WarpStream console if you don't already have one.

Step 2: Produce some records

You can use the WarpStream CLI to produce messages to your topic if you don't already have an active topic to work with:

warpstream kcmd -bootstrap-host $BOOTSTRAP_HOST -tls -username $SASL_USERNAME -password $SASL_PASSWORD -type produce -topic decodable_demo --records '{"action": "click", "user_id": "user_0", "page_id": "home"},,{"action": "hover", "user_id": "user_0", "page_id": "home"},,{"action": "scroll", "user_id": "user_0", "page_id": "home"},,{"action": "click", "user_id": "user_1", "page_id": "home"},,{"action": "click", "user_id": "user_1", "page_id": "home"},,{"action": "click", "user_id": "user_2", "page_id": "home"}'

Note that the WarpStream CLI uses double commas (,,) as a delimiter between JSON records.

Step 3: Connect Tinybird to WarpStream

In the Tinybird Web Console, next to "DATA PROJECT," click "+" and select "Data Source," select "Kafka," and file in the connection information.

Walk through the next steps to select your Topic and set your Configuration. Name the data source, and then click "Create Data Source," which will present a data preview and graph of ingested data, similar to the following:

Step 4: Configure your Tinybird pipeline

Once again, next to "DATA PROJECT," click "+" and select "Pipe." Here you will create your SQL code that will power your Tinybird API Endpoint. Tinybird will validate your code and provide a subset of the dataset that qualifies. In this example, we are looking for any "action" that does not have a status of "ACCEPT," we've named this "rejects." Once we are satisfied with the results, click on "Create API Endpoint."

Step 5: Deploy your Tinybird endpoint

Tinybird will present various code snippets to enable you to deploy your endpoint using the pipeline you just created.

Taking the HTTP option and copying it into a browser yields the following result:

Next Steps

Congratulations! You've set up a stream processing pipeline between WarpStream and Tinybird.

Next, check out the WarpStream docs for configuring the WarpStream Agent, or review the Tinybird Docs to learn more about what is possible with WarpStream and Tinybird!

Last updated

Logo

Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. Kinesis is a trademark of Amazon Web Services.