Port Forwarding (K8s)

This page provides documentation on how to solve a common problem with WarpStream: connecting to Agents deployed in K8s by port-forwarding.

If you're running WarpStream in Kubernetes, you may have noticed that port-forwarding to the Agents does not work immediately. For example, if you run the follow comand in one terminal:

kubectl port-forward $POD_NAME 9092

and then run the following command in another terminal, you'll get an error:

warpstream cli diagnose-connection

You'll get an error like this:

running diagnose-connection sub-command with bootstrap-host: localhost and bootstrap-port: 9092


Broker Details
---------------
  10.212.3.4:9092 (NodeID: 206702995) [asia-southeast1-a]
failed to communicate with Agent returned as part of Kafka Metadata response, err: <nil>, this usually means that the provided bootstrap host: localhost:9092 is accessible on the current network, but the URL that the Agent is advertising as its broker host/ip: 10.212.3.4:9092 is not accessible on this network. If this is occurring during local development whilst running the Agent in a docker container, consider adding the following flag to the docker run command: --env "WARPSTREAM_PRIVATE_IP_OVERRIDE=127.0.0.1" which will force the Agent to advertise its hostname/IP address as localhost for development purposes.

The reason for this is that your initial client connection will reach the Agent successfully, but then the client will perform a Metadata request for service discovery and the WarpStream control plane will instruct the client to connect to other Agents using their internal IP addresses (the ones reachable in K8s, but not from your laptop). You can read more about WarpStream service discovery in this documentation for more details.

To work around this issue, WarpStream has a feature to instruct the service discovery system to report all the other Agents hostnames as localhost so that when your client performs service discovery via the Metadata request and gets redirected, it will just re-use the existing port-forward. This works because all the WarpStream Agents are stateless, none are special, and any Agent can successfully handle any Kafka protocol request.

To enable this feature from your laptop, simply add the following string to your Kafka client ID: ws_host_override=localhost . Using the WarpStream CLI, that would look like this:

warpstream cli diagnose-connection -client-id ws_host_override=localhost

This time it should just work, enjoy your local development experience!

running diagnose-connection sub-command with bootstrap-host: localhost and bootstrap-port: 9092


Broker Details
---------------
  LoCALHOst:9092 (NodeID: 206702995) [asia-southeast1-a]
    ACCESSIBLE ✅

  LOcALhOST:9092 (NodeID: 576636702) [asia-southeast1-b]
    ACCESSIBLE ✅

  LoCAlhost:9092 (NodeID: 1053567971) [asia-southeast1-c]
    ACCESSIBLE ✅