Hosted Metadata Endpoint

This page explains how you can leverage WarpStream's hosted metadata endpoint to administer your WarpStream BYOC clusters from anywhere.

You can use WarpStream's hosted serverless.warpstream.com:9092 endpoint to easily run any Kafka administrative tasks for your BYOC clusters, like creating topics. The hosted endpoint can handle almost all Kafka protocol requests except for Produce and Fetch because WarpStream does not have any access to your data with our BYOC product.

Instructions

First, identify your cluster's region and ID from the "Overview" tab (e.g., ap-southeast-1 and vci_xxx).

Next, create a new dedicated agent key in the "Agent Keys" tab (e.g., aks_xxx).

Finally, arrange the credentials in this format and configure your Kafka client to connect with TLS enabled using SASL PLAIN:

SASL_USERNAME=<region>::<virtualCluster>
SASL_PASSWORD=<agentKey>

Concretely that would look like:

SASL_USERNAME=ap-southeast-1::vci_02e1aa27_5024_695d_819b_dc2d1719959d
SASL_PASSWORD=aks_29c445a02ae375e31c316eab7c69e2f1709b16bdcc044ab3a2489da19ae9239b

Example: franz-go

var (
    virtualCluster = "vci_xxx"
    agentKey       = "aks_xxx"
    region         = "xxx" // e.g. "us-east-1"
)

opts := []kgo.Opt{
	kgo.SeedBrokers("serverless.warpstream.com:9092"),
	kgo.DialTLS(),
	kgo.SASL(plain.Auth{
		User: fmt.Sprintf("%s::%s", region, virtualCluster),
		Pass: agentKey,
	}.AsMechanism()),
}

adm, err := kadm.NewOptClient(opts...)

Example: terraform