Protect Data in Motion with TLS Encryption
Last updated
Was this helpful?
Last updated
Was this helpful?
By default Kafka/Schema Registry clients communicating with WarpStream Agents use PLAINTEXT
, meaning that all data is sent in plain text (unencrypted). To encrypt data in motion (or data in transit) between your clients and your WarpStream Agents, you should configure them to use TLS Encryption.
WarpStream supports encryption based on , an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS).
Enabling TLS encryption might have a performance impact due to overhead of encrypting and decrypting data. This performance impact can vary depending on the operating system, linux kernel version, and CPU used. We recommend using the newest and best possible versions of your Operating system, Kernel, and CPU to minimize any possible impacts.
TLS uses private-key/certificate pairs, which are used during the TLS handshake process.
Each WarpStream Agent needs a private-key/certificate pair, and the Kafka client uses the certificate to authenticate to the WarpStream Agent.
Each logical client needs a private-key/certificate pair if client authentication is enabled, and the WarpStream Agent uses the certificate to authenticate the Kafka client.
If you configure TLS encryption, you can optionally configure . You can configure just TLS encryption (by default, TLS encryption includes certificate authentication of the server) and use a separate mechanism for client authentication (for example, mTLS or SASL). By default, TLS encryption enables one-way authentication in which the client authenticates the server certificate. For bidirectional authentication, where the broker also authenticates the client certificate, you can use mTLS.
When you use , the WarpStream Agent authenticates the Kafka client and the Kafka client also authenticates the WarpStream Agent. This bidirectional, or mutual, authentication provides an additional layer of security for your WarpStream cluster.
Configuring TLS for a WarpStream Cluster can be done in one of two ways. Which option is the best depends on your requirements and deployment environment.
This option is the recommended configuration when using TLS. This ensures that there is full end-to-end Kafka client to WarpStream Agent encryption. This comes at the disadvantage of needing to configure each WarpStream Agent with a TLS certificate.
Create x509 encoded TLS certificates
Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization.
We recommend that either the certificate is made with IP SANs
if using IP addresses or SANS
if using hostnames to connect to your WarpStream Agents.
Configure the WarpStream Agents to load the certificates
Once your certificates are generated to must set the WARPSTREAM_TLS_SERVER_CERT_FILE
environment variable to the public key of the certificate and set WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE
to the private key of the certificate.
Configure the WarpStream Agents to enable TLS encryption
To enable TLS encryption for your Kafka agent, set the -kafkaTLS
agent flag. Alternatively, you can set the WARPSTREAM_TLS_ENABLED
environment variable to true.
To enable TLS encryption for your Schema Registry agent, set the -schemaRegistryTLS
agent flag. Alternatively, you can set the WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLED
environment variable to true.
By default WarpStream Agents will load the TLS certificate and key from the agent's local filesystem. The agents can be configured to instead load the TLS certificate and key from blob storage by using the -tlsBlobURL
flag or WARPSTREAM_TLS_BLOB_URL
environment variable. For example -tlsBlobURL=s3://my-tls-bucket
.
Every Load Balancer is configured differently for TLS termination, bellow is information from the 3 major Cloud Providers for how to configure TLS termination on their Load Balancers.
AWS
GCP
GKE - As of 2024 GKE does not support TLS termination on Kubernetes Services with Load Balancer types
Azure
As of 2024 Azure Layer 4 load balancers do not support TLS termination
For configuring TLS encryption in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures TLS differently and those configurations may change version to version.
By default the WarpStream Agent uses the default Golang TLS settings when serving the Kafka Protocol over TLS. These defaults change over time but and may not be suitable for all environments.
These profiles can be set via the -tlsProfile
flag or WARPSTREAM_TLS_PROFILE
environment variable.
All profiles use the following TLS Curves
All TLS 1.3 connections use the following Cipher Suites:
golang-default
old
This profile supports older TLS clients with the minimum TLS version set to 1.0 with support for the following Cipher Suites for TLS 1.0, 1.1 and 1.2:
intermediate
This profile supports intermediate TLS clients with the minimum TLS version set to 1.2 with support for the following Cipher Suites:
modern
This profile supports modern TLS clients and TLS 1.3 only.
Some organizations use transparent HTTPS Proxies to decrypt and inspect HTTPS traffic. This may cause the WarpStream Agent to be unable to communicate with the WarpStream Control Plane.
You may see the following error if that is the case
An example values.yaml
of this configuration via our Kubernetes Helm Chart
is as follows:
If running WarpStream in Kubernetes using can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities.
This option can be simpler to implement for WarpStream clusters behind a load balancer. This option requires that you for it to work as expected. The disadvantage of this option is that the communication between the Load Balancer and WarpStream Agents are not encrypted. If you have a requirement for full end-to-end encryption we do not recommend using this option. This option also cannot use mTLS as an authentication mechanism due to TLS termination on Load Balancers not being able to pass-through the client certificate.
We recommend using the to learn how to configure Java-based clients for TLS encryption.
The WarpStream agent supports setting different TLS Profiles based on recommendations from .
The ability to configure TLS Profiles was added in Agent Version .
Use the Golang defaults for TLS Version, Curves and Cipher Suites. These will change over time. See documentation for details.
Starting in version the WarpStream agent as the ability to provide an HTTPS Proxy Certificate Authority for the Agent to trust. This can be done with the -httpsProxyCACertFile
flag or WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE
environment variable.