LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Port Forwarding (K8s)
    • Orbit
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
Powered by GitBook
On this page

Was this helpful?

  1. BYOC

Port Forwarding (K8s)

This page provides documentation on how to solve a common problem with WarpStream: connecting to Agents deployed in K8s by port-forwarding.

PreviousWarpStream Schema LinkingNextOrbit

Last updated 7 months ago

Was this helpful?

If you're running WarpStream in Kubernetes, you may have noticed that port-forwarding to the Agents does not work immediately. For example, if you run the follow comand in one terminal:

kubectl port-forward $POD_NAME 9092

and then run the following command in another terminal, you'll get an error:

warpstream cli diagnose-connection

You'll get an error like this:

running diagnose-connection sub-command with bootstrap-host: localhost and bootstrap-port: 9092


Broker Details
---------------
  10.212.3.4:9092 (NodeID: 206702995) [asia-southeast1-a]
failed to communicate with Agent returned as part of Kafka Metadata response, err: <nil>, this usually means that the provided bootstrap host: localhost:9092 is accessible on the current network, but the URL that the Agent is advertising as its broker host/ip: 10.212.3.4:9092 is not accessible on this network. If this is occurring during local development whilst running the Agent in a docker container, consider adding the following flag to the docker run command: --env "WARPSTREAM_PRIVATE_IP_OVERRIDE=127.0.0.1" which will force the Agent to advertise its hostname/IP address as localhost for development purposes.

The reason for this is that your initial client connection will reach the Agent successfully, but then the client will perform a Metadata request for service discovery and the WarpStream control plane will instruct the client to connect to other Agents using their internal IP addresses (the ones reachable in K8s, but not from your laptop). .

To work around this issue, WarpStream has a feature to instruct the service discovery system to report all the other Agents hostnames as localhost so that when your client performs service discovery via the Metadata request and gets redirected, it will just re-use the existing port-forward. This works because all the WarpStream Agents are stateless, none are special, and any Agent can successfully handle any Kafka protocol request.

To enable this feature from your laptop, simply add the following string to your Kafka client ID: ws_host_override=localhost . Using the WarpStream CLI, that would look like this:

warpstream cli diagnose-connection -client-id ws_host_override=localhost

This time it should just work, enjoy your local development experience!

running diagnose-connection sub-command with bootstrap-host: localhost and bootstrap-port: 9092


Broker Details
---------------
  LoCALHOst:9092 (NodeID: 206702995) [asia-southeast1-a]
    ACCESSIBLE ✅

  LOcALhOST:9092 (NodeID: 576636702) [asia-southeast1-b]
    ACCESSIBLE ✅

  LoCAlhost:9092 (NodeID: 1053567971) [asia-southeast1-c]
    ACCESSIBLE ✅
You can read more about WarpStream service discovery in this documentation for more details