Materialize
This page describes how to set up a connection between WarpStream and Materialize, ingest data into Materialize, and create a materialized view of this data.
A video walkthrough can be found below:
Prerequisites
WarpStream account - get access to WarpStream by registering here.
Materialize account - get access to Materialize by registering here.
WarpStream cluster is up and running.
Step 1: Create a topic in your WarpStream cluster
Obtain the Bootstrap Broker from the WarpStream console by navigating to your cluster and then clicking the Connect tab. If you don't have SASL credentials yet, you can also create a set of credentials from the console.
Store these values as environment variables for easy reference:
Then, create a topic using the WarpStream CLI:
You should see the following output in your Terminal:
Created topic materialize_click_streams.
Step 2: Produce some records
Using the WarpStream CLI, produce several messages to your topic:
Note that the WarpStream CLI uses double commas (,,)
as a delimiter between JSON records.
Step 3: Connect Materialize with WarpStream
First, provide your WarpStream credentials as SECRET
objects in the Materialize console:
Then, establish a CONNECTION
object (note: replace $BOOTSTRAP_HOST with its value from Step 1):
Finally, create a SOURCE
object to being consuming messages from your WarpStream topic:
Step 4: Verify that Materialize is ingesting data from WarpStream
In the Materialize Console, run:
You should see the results displaying the records that you produced in Step 2.
Step 5: Create a materialized view
First, create the materialized view:
Then, query the result:
You should see a result showing user_0
with a count of 3 records.
Step 6: Produce another record to your WarpStream topic
In your Terminal, produce another record to your topic:
In the Materialize Console, query the materialized view again:
You should now see two records, the record for user_0
with a count
of 3, and a new record for user_1
with a count
of 1.
Next Steps
After validating that your WarpStream cluster is connected to Materialize and that Materialize can consume and process data from WarpStream, you are ready to set up an actual data pipeline between WarpStream and Materialize.
For more information on how to set up and configure Materialize, head over to the Materialize Docs page.
Last updated