Configure the WarpStream Agent Within a Container or Behind a Proxy

When deploying a WarpStream agent within a Docker container or behind a proxy, ensuring proper connectivity involves more than just reaching the bootstrap server. This guide clarifies key concepts and steps to guarantee a seamless connection setup.

For a comprehensive understanding, a valuable resource to explore is the functionality of WarpStream's service discovery mechanism: Service Discovery.

Connecting to Bootstrap Server

At first, a client may successfully connect to the bootstrap server using an address like localhost:9092, or a magic URL, such as api-example.discovery.prod-z.us-east-1.warpstream.com:9092. However, this initial connection is just the beginning of the communication process.

Local Agent Address

For proper functioning, subsequent interactions within the WarpStream ecosystem rely on the local hostname and port of the agent, often distinct from the bootstrap address. A local connection might be structured as 192.0.1.1:9092.

Configuration Details

1. Selection for Address:

   • Utilize the -advertiseHostnameStrategy flag or the WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable to select the appropriate hostname advertisement strategy:

     • auto-ip4 (default in agent mode): Discovers and uses the internal IPv4 of the agent's host for internal connectivity.

     • auto-ip6: Discovers and uses the internal IPv6 of the agent's host for internal connectivity.

     • local (default in playground/demo mode): For local development, where the agent advertises its hostname as localhost for client connections.

2. Override for External Address and Port:

   • If the default IP and port are not reachable by your Kafka clients, use WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE and WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE to ensure the agent’s advertised address and port are accessible for client connections.

3. Override for Inter-Agent Communication:

   • Besides making sure agents and clients can connect, it’s also important that agents can talk to each other. They need to communicate to efficiently share files and data.

   • If an agent’s local IP isn’t reachable by other agents in the same group, you can fix this by using the -advertiseHostnameStrategy flag or the WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable. Set it to custom so you can enter a specific address that all agents can reach. Then, provide the custom hostname by setting either the -advertiseHostnameStrategy flag or the WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM environment variable.

   • Note: if WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE is not set, this custom address will also be used as the external address.

Advertise the WarpStream Agents behind a traditional load balancer

Similarly, if you want to run the Agent behind a load balancer, you still need to leverage the advertiseHostnameStrategy flag or WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable described above to indicate how the Agents should discover each other for distributed file caching. You should also use WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE to indicate how the clients should discover the Agents. In most cases, the value of WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE should just be the DNS name of the load balancer.

Importance of Overrides

WarpStream agents utilize their private IP and ports for ongoing connections after the initial bootstrap. When a client automatically connects to a new agent for load balancing or other purposes, this internal connectivity is critical for efficient operation. Without the correct configurations, clients might connect to bootstrap successfully yet experience issues when progressing beyond the initial phase.

Implementing these guidelines and understanding the significance of internal connectivity can ensure the smooth functioning of WarpStream agents within Docker containers.