Mutual TLS (mTLS)

Mututal TLS (mTLS) authentication requires TLS encryption, this page shows you how to configure both at the same time and is a superset of configurations required just for Configuring the WarpStream Agents to terminate TLS.

Creating TLS keys and Certificates for the WarpStream Agents

Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization.

If running WarpStream in Kubernetes using cert-manager can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities.

We recommend that either the certificate is made with IP SANs if using IP addresses or SANS if using hostnames to connect to your WarpStream Agents.

Creating TLS keys and Certificate for the Kafka/Schema Registry Clients

When using mutual TLS, it is highly recommended to use a private certificate authority for certificates. Most large organizations typically already have their own internal private certificate authority that you can use to generate client certificates.

For alternatives if your organization does not have an existing private certificate authority, every cloud provider offers a managed solution, for example AWS Private CA. You can also create a certificate authority manually using the openssl command line however that can be complex to manage in production environments.

When using mutual TLS is it recommended to give each application their own unique certificate with their own unique Distinguished Name(DN). If using ACLs having unique certificates with unique DN's will allow each application to have it's own set of permissions.

Configure WarpStream Agents

Once your certificates are generated to must set the WARPSTREAM_TLS_SERVER_CERT_FILE environment variable to the public key of the certificate and set WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE to the private key of the certificate.

To enabled mutual TLS you must set the WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION environment variable to true.

Once authentication is enabled on the Agent, it will enforce that all Apache Kafka clients that connect to them authenticate themselves via mTLS. Otherwise, it will refuse the connection.

It is also highly recommended to set the environment variable WARPSTREAM_TLS_CLIENT_CA_CERT_FILE to the public keys of the certificate authorities that sign your client certificates. If this environment variable is not set WarpStream defaults to trusting all client certificates from your Operating System's root certificate store. This store typically contains all the public keys for all the publicly accepted certificate authorities. If this environment variable is not set any certificate from any public certificate authority will be valid and authenticated against your WarpStream agent which could cause external data leaks.

By default, the Agent will use the Distinguished Name(DN) from the client TLS certificate as the principal for ACLs. A custom TLS mapping rule regex can be provided using the -tlsPrincipalMappingRule flag to extract a name from the DN. For example, the rule CN=([^,]+) will extract the Common Name(CN) from the DN, and use that as the ACL principal.

For example, given a certificate with the DN of CN=test_principal,O=ACME Corp, if -tlsPrincipalMappingRule is set to CN=([^,]+) then the name test_principal will be used as the mTLS ACL principal.

Configure Kafka and Schema Registry clients

For configuring mTLS in your Kafka/Schema Registry clients it is recommended to review the documentation for your client. Every client configures mTLS differently and those configurations may change version to version.

We recommend using the Confluent Platform documentation to learn how to configure Java-based clients for mTLS.