Administrate your BYOC clusters with Serverless

This page explains how you can leverage Serverless to call Kafka Admin APIs from anywhere.

If you create a virtual cluster in a region that hosts WarpStream Serverless services (us-east-1 and ap-southeast-1) then it is possible to use the Serverless public endpoint to call Kafka Admin APIs.

Let's go through an example to understand how it works.

First let's create a BYOC virtual cluster in us-east-1:

Click on "Create Cluster"

Then, go to the "deploy" tab and you should see something like this:

If there is no "Admin Bootstrap URL" at the bottom left, it means there is no serverless service in this region.

Take note of the "Admin Bootstrap URL". In this case, "serverless.warpstream.com:9092".

Now we need to create some credentials to use this endpoint. Use your virtual cluster's "Credentials" tab:

Click on "Create Credentials"

Hold onto these credentials by copying the export command to your clipboard.

Open a shell and paste the export command. Install the WarpStream CLI if it's not already installed.

curl https://console.warpstream.com/install.sh | sh

Now run the following command using the same Admin Bootstrap URL:

$ warpstream cli broker-metadata -bootstrap-host serverless.warpstream.com

You should see something like this:

2024/05/15 11:55:55 INFO panic on invariant violation is enabled
running broker-metadata sub-command with bootstrap-host: serverless.warpstream.com and bootstrap-port: 9092
SASL credentials were detected from flags or environment, using credentials with username: ccun_d74aabd7f08d3c07a727c30cfc22a9307980723ea2934ce1ddf0cae050cd5d74

Broker Details
---------------
  serverless.prod-z.us-east-1.warpstream.com:9092 (NodeID: 1) [UNKNOWN_RACK]

You can now run Kafka Admin commands such as creating and listing topics:

$ warpstream cli create-topic -topic testtopic -bootstrap-host serverless.warpstream.com
2024/05/15 11:57:29 INFO panic on invariant violation is enabled
running create-topic sub-command with bootstrap-host: serverless.warpstream.com and bootstrap-port: 9092
SASL credentials were detected from flags or environment, using credentials with username: ccun_d74aabd7f08d3c07a727c30cfc22a9307980723ea2934ce1ddf0cae050cd5d74

created topic "testtopic" successfully, topic ID: AAAAAAAAAAAAAAAAAAAAAA==

$ warpstream cli list-topics -bootstrap-host serverless.warpstream.com
2024/05/15 11:58:04 INFO panic on invariant violation is enabled
running list-topics sub-command with bootstrap-host: serverless.warpstream.com and bootstrap-port: 9092
SASL credentials were detected from flags or environment, using credentials with username: ccun_d74aabd7f08d3c07a727c30cfc22a9307980723ea2934ce1ddf0cae050cd5d74

Topic Details
---------------
  testtopic: 1 partition(s)

However, fetch and produce requests will fail. WarpStream's Serverless agents cannot access your bucket:

$ warpstream cli  produce -topic testtopic -records testrecord  -bootstrap-host serverless.warpstream.com
2024/05/15 11:59:01 INFO panic on invariant violation is enabled
running produce sub-command with bootstrap-host: serverless.warpstream.com and bootstrap-port: 9092
SASL credentials were detected from flags or environment, using credentials with username: ccun_d74aabd7f08d3c07a727c30cfc22a9307980723ea2934ce1ddf0cae050cd5d74

failed to produce, err: record failed after being retried too many times

Note that this works with any Kafka client, the WarpStream CLI being one example. This means you can leverage an existing Kafka Terraform provider using our Serverless public endpoint, even if your cluster is BYOC.