Comment on page
Configure Authentication for the WarpStream Agent
This page describes how to configure authentication for the WarpStream Agent so it can be exposed safely over the internet.
The WarpStream Agent has built-in support for SASL/PLAIN authentication, which makes it possible to securely expose them over the internet when combined with TLS. This is useful if you need to access your WarpStream topics from outside your VPC, or integrate with cloud hosted compute engines like Materialize, ClickHouse, MongoDB Stream Processing, or RisingWave.
Before proceeding, please keep in mind two very important points:
- 1.The WarpStream Agent does not perform TLS termination currently. To expose it safely over the internet, you must run some kind of TLS termination in front of them using something like an AWS Network Load Balancer, or a platform with TLS termination built-in like fly.io.
- 2.SASL authentication only protects the Apache Kafka protocol port (default
9092). The HTTP port (default8080)in the Agent that exposes the AWS Kinesis API, distributed file cache, and prometheus metrics is not protected by SASL and should not be exposed to the public internet.
The Agent should not be exposed directly to the internet via plain TCP. In addition, the load balancer that performs TLS termination for the Agent should only expose the Apache Kafka protocol port, as that is the only one that will be authenticated via SASL.
We will provide more detailed examples on how to configure TLS termination in various environments in the future.
However, one important thing to keep in mind is that by default the WarpStream Agent is designed to automatically integrate with the WarpStream discovery system under the assumption that they're running inside a VPC. If the Agent is deployed behind a load balancer that is performing TLS termination, then the configuration needs to be modified so that service discovery works properly.
Specifically, the
WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE environment value should be set to whatever the host / DNS name of the load balancer is. For example, an AWS Network Load Balancer DNS name will look something like:WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=$NLB_NAME-XXXXXXXXXXXX.elb.us-east-1.amazonaws.com
or if the Agent was deployed on fly.io it would look something like:
WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=$APP_NAME.fly.dev
If the load balancer is not exposing the Kafka protocol externally over the default port of
9092, then you will also need to add another environment variable called WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE that indicates which port the load balancer is exposing the Kafka protocol over. For example, if it was over port 16553, then the environment variable would look like:WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE=16553
For more details on how WarpStream and Kafka's service discovery mechanisms interact and why overriding these environment variables is necessary, see our "Service Discovery" reference documentation.
The WarpStream Agents support SASL/PLAIN, but not SASL/SCRAM.
First, deploy the Agent like you normally would following our "Configure the WarpStream Agent for Production" guide. However, you'll need to make one small addition which is to add the
-requireAuthentication flag, or add an environment variable: WARPSTREAM_REQUIRE_AUTHENTICATION=true .Once authentication is enabled on the Agent, it will enforce that all Apache Kafka clients that connect to them authenticate themselves via SASL, otherwise it will refuse the connection.
In order to connect an Apache Kafka client to the authenticated WarpStream Agent, you'll need to create a set of credentials. You can do that by navigating to the "Virtual Clusters" section of the WarpStream Console, and then clicking "View Credentials" on the Virtual Cluster that you want to create a set of credentials for.
Click "View Credentials"
Once you're on the credentials view, you can create a new set SASL credentials by clicking the "New Credentials" button.
Click "New Credentials"
Every set of WarpStream SASL credentials are scoped to a specific Virtual Cluster and Agent Pool, so you'll have to select which Agent Pool you'd like to create the credentials for as well.
Once you're done creating the credentials, the admin console will show you the username and password one time. Store these values somewhere safe, as you'll never be able to view them again. WarpStream does not store them in plaintext, so we cannot retrieve them for you.
Save your credentials somewhere safe!
In the case that you lose your credentials, you can create a new set of credentials in the admin console following the same steps as above, up to a limit of 100 credentials.
Lets test the credentials we just created. Make sure that your Agent is enforcing authentication by trying to connect to it with invalid credentials first:
warpstream kcmd -type diagnose-connection -bootstrap-host $TLS_TERMINATOR_HOST_NAME -tls -username ccun_invalid_username -password ccp_invalid_password
If the Agent is configured to require authentication, this should result in an error message like:
failed to fetch broker metadata: SASL_AUTHENTICATION_FAILED: SASL Authentication failed.
Now let's try with the valid credentials we created previously:
warpstream kcmd -type diagnose-connection -bootstrap-host $TLS_TERMINATOR_HOST_NAME -tls -username ccun_XXXXXXXXXX -password ccp_XXXXXXXXXX
This time you should see a successful message like the following:
Broker Details
---------------
$TLS_TERMINATOR_HOST_NAME:9092 (NodeID: 0) [UNKNOWN_RACK]
ACCESSIBLE ✅
GroupCoordinator: $TLS_TERMINATOR_HOST_NAME:9092 (NodeID: 0)
ACCESSIBLE ✅