LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Orbit
    • Port Forwarding (K8s)
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
Powered by GitBook
On this page

Was this helpful?

  1. BYOC

Hosted Metadata Endpoint

This page explains how you can leverage WarpStream's hosted metadata endpoint to administer your WarpStream BYOC clusters from anywhere.

PreviousClient Configuration Auto-tuningNextManaged Data Pipelines

Last updated 5 months ago

Was this helpful?

You can use WarpStream's hosted serverless.warpstream.com:9092 endpoint to easily run any Kafka administrative tasks for your BYOC clusters, like creating topics. The hosted endpoint can handle almost all Kafka protocol requests except for Produce and Fetch because WarpStream does not have any access to your data with our BYOC product.

Instructions

First, identify your cluster's region and ID from the "Overview" tab (e.g., ap-southeast-1 and vci_xxx).

Next, create a new dedicated agent key in the "Agent Keys" tab (e.g., aks_xxx).

Finally, arrange the credentials in this format and configure your Kafka client to connect with TLS enabled using SASL PLAIN:

SASL_USERNAME=<region>::<virtualCluster>
SASL_PASSWORD=<agentKey>

Concretely that would look like:

SASL_USERNAME=ap-southeast-1::vci_02e1aa27_5024_695d_819b_dc2d1719959d
SASL_PASSWORD=aks_29c445a02ae375e31c316eab7c69e2f1709b16bdcc044ab3a2489da19ae9239b

Example: franz-go

var (
    virtualCluster = "vci_xxx"
    agentKey       = "aks_xxx"
    region         = "xxx" // e.g. "us-east-1"
)

opts := []kgo.Opt{
	kgo.SeedBrokers("serverless.warpstream.com:9092"),
	kgo.DialTLS(),
	kgo.SASL(plain.Auth{
		User: fmt.Sprintf("%s::%s", region, virtualCluster),
		Pass: agentKey,
	}.AsMechanism()),
}

adm, err := kadm.NewOptClient(opts...)

Example: terraform

See our terraform example on Github.