Comment on page
Configure Authentication for the WarpStream Agent
This page describes how to configure authentication for the WarpStream Agent so it can be exposed safely over the internet.
The WarpStream Agent has built-in support for SASL/PLAIN authentication, which makes it possible to securely expose them over the internet when combined with TLS. This is useful if you need to access your WarpStream topics from outside your VPC, or integrate with cloud hosted compute engines like Materialize, ClickHouse, MongoDB Stream Processing, or RisingWave.
Before proceeding, please keep in mind two very important points:
- 2.SASL authentication only protects the Apache Kafka protocol port (default
9092). The HTTP port (default
8080)in the Agent that exposes the AWS Kinesis API, distributed file cache, and prometheus metrics is not protected by SASL and should not be exposed to the public internet.
The Agent should not be exposed directly to the internet via plain TCP. In addition, the load balancer that performs TLS termination for the Agent should only expose the Apache Kafka protocol port, as that is the only one that will be authenticated via SASL.
We will provide more detailed examples on how to configure TLS termination in various environments in the future.
However, one important thing to keep in mind is that by default the WarpStream Agent is designed to automatically integrate with the WarpStream discovery system under the assumption that they're running inside a VPC. If the Agent is deployed behind a load balancer that is performing TLS termination, then the configuration needs to be modified so that service discovery works properly.
WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDEenvironment value should be set to whatever the host / DNS name of the load balancer is. For example, an AWS Network Load Balancer DNS name will look something like:
or if the Agent was deployed on fly.io it would look something like:
If the load balancer is not exposing the Kafka protocol externally over the default port of
9092, then you will also need to add another environment variable called
WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDEthat indicates which port the load balancer is exposing the Kafka protocol over. For example, if it was over port
16553, then the environment variable would look like:
The WarpStream Agents support SASL/PLAIN, but not SASL/SCRAM.
First, deploy the Agent like you normally would following our "Configure the WarpStream Agent for Production" guide. However, you'll need to make one small addition which is to add the
-requireAuthenticationflag, or add an environment variable:
Once authentication is enabled on the Agent, it will enforce that all Apache Kafka clients that connect to them authenticate themselves via SASL, otherwise it will refuse the connection.
In order to connect an Apache Kafka client to the authenticated WarpStream Agent, you'll need to create a set of credentials. You can do that by navigating to the "Virtual Clusters" section of the WarpStream Console, and then clicking "View Credentials" on the Virtual Cluster that you want to create a set of credentials for.
Click "View Credentials"
Once you're on the credentials view, you can create a new set SASL credentials by clicking the "New Credentials" button.
Click "New Credentials"
Every set of WarpStream SASL credentials are scoped to a specific Virtual Cluster and Agent Pool, so you'll have to select which Agent Pool you'd like to create the credentials for as well.
Once you're done creating the credentials, the admin console will show you the username and password one time. Store these values somewhere safe, as you'll never be able to view them again. WarpStream does not store them in plaintext, so we cannot retrieve them for you.
Save your credentials somewhere safe!
In the case that you lose your credentials, you can create a new set of credentials in the admin console following the same steps as above, up to a limit of 100 credentials.
Lets test the credentials we just created. Make sure that your Agent is enforcing authentication by trying to connect to it with invalid credentials first:
warpstream kcmd -type diagnose-connection -bootstrap-host $TLS_TERMINATOR_HOST_NAME -tls -username ccun_invalid_username -password ccp_invalid_password
If the Agent is configured to require authentication, this should result in an error message like:
failed to fetch broker metadata: SASL_AUTHENTICATION_FAILED: SASL Authentication failed.
Now let's try with the valid credentials we created previously:
warpstream kcmd -type diagnose-connection -bootstrap-host $TLS_TERMINATOR_HOST_NAME -tls -username ccun_XXXXXXXXXX -password ccp_XXXXXXXXXX
This time you should see a successful message like the following:
$TLS_TERMINATOR_HOST_NAME:9092 (NodeID: 0) [UNKNOWN_RACK]
GroupCoordinator: $TLS_TERMINATOR_HOST_NAME:9092 (NodeID: 0)