Links

Configure the WarpStream Agent Within a Container or Behind a Proxy

When deploying a WarpStream agent within a Docker container or behind a proxy, ensuring proper connectivity involves more than just reaching the bootstrap server. This guide clarifies key concepts and steps to guarantee a seamless connection setup.
For a comprehensive understanding, a valuable resource to explore is the functionality of WarpStream's service discovery mechanism: Service Discovery.

Connecting to Bootstrap Server

At first, a client may successfully connect to the bootstrap server using an address like localhost:9092, or a magic URL such as api-example.discovery.prod-z.us-east-1.warpstream.com:9092. However, this initial connection is just the beginning of the communication process.

Internal Agent Connection

For proper functioning, subsequent interactions within the WarpStream ecosystem rely on the internal hostname and port of the agent, often distinct from the bootstrap address. An internal connection might be structured as 192.0.1.1:9092.

Address Overrides for Complete Connectivity

To ensure comprehensive connectivity, consider the following points:
  1. 1.
    Dual Connectivity: Merely making the bootstrap address reachable is insufficient. Clients must be able to establish connections to both the bootstrap server and the agent's advertised hostname.
  2. 2.
    Override Mechanism: In situations where the agent's internal IP and port aren't directly reachable, you have several configuration knobs available:
    • You can use the -advertiseHostnameStrategy flag (or the equivalent WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable) to decide which strategy to use to select the hostname that the Agent will advertise in WarpStream's service discovery system:
      • auto-ip4 (default in agent mode): automatically try to discover the internal IPv4 of the agent's host
      • auto-ip6: automatically try to discover the internal IPv6 of the agent's host
      • local (default in playground/demo mode): for local development only, will instruct the Agent to advertise its hostname as localhost
      • custom: will use whatever value is passed with advertiseHostnameCustom or WARPSTREAM_ADVERTISE_HOSTNAME_CUSTOM
    • Utilize WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE to specify the port used to listen to the Kafka protocol

Advertise the WarpStream Agents behind a traditional load balancer

If you want to run the Agent behind a load balancer, you still need to leverage the advertiseHostnameStrategy flag or WARPSTREAM_ADVERTISE_HOSTNAME_STRATEGY environment variable described above to indicate how the Agents should discover each other for distributed file caching. And you should also use WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE to indicate how the clients should discover the Agents. In most cases the value of WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE should just be the DNS name of the load balancer.

Importance of Overrides

WarpStream agents utilize their private IP and ports for ongoing connections after the initial bootstrap. When a client automatically connects to a new agent for load balancing or other purposes, this internal connectivity is critical for efficient operation. Without the correct configurations, clients might connect to bootstrap successfully yet experience issues when progressing beyond the initial phase.
By implementing these guidelines and understanding the significance of internal connectivity, you can ensure the smooth functioning of WarpStream agents within Docker containers.
Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. Kinesis is a trademark of Amazon Web Services.