Protect Data in Motion with TLS Encryption

TLS Encryption Overview

By default Kafka clients communicating with WarpStream Agents use PLAINTEXT, meaning that all data is sent in plain text (unencrypted). To encrypt data in motion (or data in transit) between Kafka clients and your WarpStream Agents you should configure them to use TLS Encryption.

WarpStream supports Transport Layer Security (TLS) encryption based on OpenSSL, an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS).

Enabling TLS encryption might have a performance impact due to overhead of encrypting and decrypting data. This performance impact can vary depending on the operating system, linux kernel version, and CPU used. We recommend using the newest and best possible versions of your Operating system, Kernel, and CPU to minimize any possible impacts.

TLS uses private-key/certificate pairs, which are used during the TLS handshake process.

• Each WarpStream Agent needs a private-key/certificate pair, and the Kafka client uses the certificate to authenticate to the WarpStream Agent.

• Each logical client needs a private-key/certificate pair if client authentication is enabled, and the WarpStream Agent uses the certificate to authenticate the Kafka client.

Mutual (mTLS) Authentication

If you configure TLS encryption, you can optionally configure mutual (mTLS) authentication. You can configure just TLS encryption (by default, TLS encryption includes certificate authentication of the server) and use a separate mechanism for client authentication (for example, mTLS or SASL). By default, TLS encryption enables one-way authentication in which the client authenticates the server certificate. For bidirectional authentication, where the broker also authenticates the client certificate, you can use mTLS.

When you use mTLS Authentication, the WarpStream Agent authenticates the Kafka client and the Kafka client also authenticates the WarpStream Agent. This bidirectional, or mutual, authentication provides an additional layer of security for your WarpStream cluster.

Configure TLS Encryption for a WarpStream Cluster

Configuring TLS for a WarpStream Cluster can be done in one of two ways. Which option is the best depends on your requirements and deployment environment.

Option 1: Configuring the WarpStream Agents to terminate TLS

This option is the recommended configuration when using TLS. This ensures that there is full end-to-end Kafka client to WarpStream Agent encryption. This comes at the dis-advantage of needing to configure each WarpStream Agent with a TLS certificate.

1. Create x509 encoded TLS certificates

   • Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization.

   • If running WarpStream in Kubernetes using cert-manager can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities.

   • We recommend that either the certificate is made with IP SANs if using IP addresses or SANS if using hostnames to connect to your WarpStream Agents.

2. Configure the WarpStream Agents to load the certificates

   • Once your certificates are generated to must set the WARPSTREAM_TLS_SERVER_CERT_FILE environment variable to the public key of the certificate and set WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE to the private key of the certificate.

Option 2: Configuring a Load Balancer to terminate TLS

This option can be simpler to implement for WarpStream clusters behind a load balancer. This option requires that you Advertise the WarpStream Agents behind a traditional load balancer for it to work as expected. The dis-advantage of this option is that the communication between the Load Balancer and WarpStream Agents are not encrypted. If you have a requirement for full end-to-end encryption we do not recommend using this option. This option also cannot use mTLS as an authentication mechanism due to TLS termination on Load Balancers not being able to pass-through the client certificate.

Every Load Balancer is configured differently for TLS termination, bellow is information from the 3 major Cloud Providers for how to configure TLS termination on their Load Balancers.

• AWS

  • EKS NLB

  • Network Load Balancer

• GCP

  • GKE - As of 2024 GKE does not support TLS termination on Kubernetes Services with Load Balancer types

  • Network Load Balancer SSL Proxy

• Azure

  • As of 2024 Azure Layer 4 load balancers do not support TLS termination

Configure TLS Encryption for Kafka clients

For configuring TLS encryption in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures TLS differently and those configurations may change version to version.

We recommend using the Confluent Platform documentation to learn how to configure Java-based clients for TLS encryption.