Protect Data in Motion with TLS Encryption
Last updated
Was this helpful?
Last updated
Was this helpful?
By default Kafka/Schema Registry clients communicating with WarpStream Agents use PLAINTEXT, meaning that all data is sent in plain text (unencrypted). To encrypt data in motion (or data in transit) between your clients and your WarpStream Agents, you should configure them to use TLS Encryption.
WarpStream supports encryption based on , an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS).
Enabling TLS encryption might have a performance impact due to overhead of encrypting and decrypting data. This performance impact can vary depending on the operating system, linux kernel version, and CPU used. We recommend using the newest and best possible versions of your Operating system, Kernel, and CPU to minimize any possible impacts.
TLS uses private-key/certificate pairs, which are used during the TLS handshake process.
Each WarpStream Agent needs a private-key/certificate pair, and the Kafka client uses the certificate to authenticate to the WarpStream Agent.
Each logical client needs a private-key/certificate pair if client authentication is enabled, and the WarpStream Agent uses the certificate to authenticate the Kafka client.
If you configure TLS encryption, you can optionally configure . You can configure just TLS encryption (by default, TLS encryption includes certificate authentication of the server) and use a separate mechanism for client authentication (for example, mTLS or SASL). By default, TLS encryption enables one-way authentication in which the client authenticates the server certificate. For bidirectional authentication, where the broker also authenticates the client certificate, you can use mTLS.
When you use , the WarpStream Agent authenticates the Kafka client and the Kafka client also authenticates the WarpStream Agent. This bidirectional, or mutual, authentication provides an additional layer of security for your WarpStream cluster.
Configuring TLS for a WarpStream Cluster can be done in one of two ways. Which option is the best depends on your requirements and deployment environment.
This option is the recommended configuration when using TLS. This ensures that there is full end-to-end Kafka client to WarpStream Agent encryption. This comes at the disadvantage of needing to configure each WarpStream Agent with a TLS certificate.
Create x509 encoded TLS certificates
Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization.
We recommend that either the certificate is made with IP SANs if using IP addresses or SANS if using hostnames to connect to your WarpStream Agents.
Configure the WarpStream Agents to load the certificates
Once your certificates are generated to must set the WARPSTREAM_TLS_SERVER_CERT_FILE environment variable to the public key of the certificate and set WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE to the private key of the certificate.
Configure the WarpStream Agents to enable TLS encryption
To enable TLS encryption for your Kafka agent, set the -kafkaTLSagent flag. Alternatively, you can set the WARPSTREAM_TLS_ENABLEDenvironment variable to true.
To enable TLS encryption for your Schema Registry agent, set the -schemaRegistryTLSagent flag. Alternatively, you can set the WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLEDenvironment variable to true.
By default WarpStream Agents will load the TLS certificate and key from the agent's local filesystem. The agents can be configured to instead load the TLS certificate and key from blob storage by using the -tlsBlobURL flag or WARPSTREAM_TLS_BLOB_URL environment variable. For example -tlsBlobURL=s3://my-tls-bucket.
Every Load Balancer is configured differently for TLS termination, bellow is information from the 3 major Cloud Providers for how to configure TLS termination on their Load Balancers.
AWS
GCP
GKE - As of 2024 GKE does not support TLS termination on Kubernetes Services with Load Balancer types
Azure
As of 2024 Azure Layer 4 load balancers do not support TLS termination
For configuring TLS encryption in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures TLS differently and those configurations may change version to version.
If running WarpStream in Kubernetes using can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities.
This option can be simpler to implement for WarpStream clusters behind a load balancer. This option requires that you for it to work as expected. The disadvantage of this option is that the communication between the Load Balancer and WarpStream Agents are not encrypted. If you have a requirement for full end-to-end encryption we do not recommend using this option. This option also cannot use mTLS as an authentication mechanism due to TLS termination on Load Balancers not being able to pass-through the client certificate.
We recommend using the to learn how to configure Java-based clients for TLS encryption.