Configure the WarpStream Agent Within a Container or Behind a Proxy

When deploying a WarpStream agent within a Docker container or behind a proxy, ensuring proper connectivity involves more than just reaching the bootstrap server. This guide clarifies key concepts and steps to guarantee a seamless connection setup.

To explore in detail the functionality of WarpStream's service discovery mechanism, check: Service Discovery.

Connecting to Bootstrap Server

At first, a client may successfully connect to the bootstrap server using an address like localhost:9092, or a bootstrap URL, such as api-example.discovery.prod-z.us-east-1.warpstream.com:9092. However, this initial connection is just the beginning of the communication process.

Local Agent Address

For proper functioning, subsequent interactions between the WarpStream agents, as well as between the agents and your Kafka clients, rely on the local hostname and port of the agent, often distinct from the bootstrap address. A local connection might be structured as 192.0.1.1:9092.

Configuration Details

1. Selection for Address:

   • Utilize the -advertiseHostnameStrategy flag or the WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable to select the appropriate hostname advertisement strategy:

     • auto-ip4 (default in agent mode): Discovers and uses the internal IPv4 of the agent's host for internal connectivity.

     • auto-ip6: Discovers and uses the internal IPv6 of the agent's host for internal connectivity.

     • local (default in playground/demo mode): For local development, where the agent advertises its hostname as localhost for client connections.

2. Override the External Address and Port for Client Accessibility:

   • If the IP and port advertised by Warpstream are not reachable by your Kafka clients, use the following options to ensure the agent’s advertised address and port are accessible for client connections:

     1. Global Overrides (All Clients):

        • Set environment variables on the Warpstream agent to change the advertised address and port for all clients:

          • WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE

          • WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE

     2. Client-Specific Overrides:

        • Ideal for scenarios with multiple clients or load balancers.

        • Set the ws_host_override parameter within the client's ID when creating the Kafka client (check Configuring Kafka Client ID Features for more details):

        kgo.NewClient(..., 
            kgo.ClientID("ws_host_override=agent-lb.yourcompany.com"),
        )

3. Override for Inter-Agent Communication:

   • Besides making sure agents and clients can connect, it’s also important that agents can talk to each other. They need to communicate to efficiently share files and data.

   • If an agent’s local IP isn’t reachable by other agents in the same group, you can fix this by using the -advertiseHostnameStrategy flag or the WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable. Set it to custom so you can enter a specific address that all agents can reach. Then, provide the custom hostname by setting either the -advertiseHostnameCustom flag or the WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM environment variable.

   • Note: if WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE is not set, this custom address will also be used as the external address.

Advertise the WarpStream Agents behind a traditional load balancer

Similarly, to run the Agent behind a load balancer, use WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE to indicate how the clients should discover the Agents. In most cases, the value of WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE should just be the DNS name of the load balancer.

Importance of Overrides

WarpStream agents utilize their private IP and ports for ongoing connections after the initial bootstrap. Without the correct configurations, clients might connect to bootstrap successfully yet experience issues when progressing beyond the initial phase.