LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Port Forwarding (K8s)
    • Orbit
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
Powered by GitBook
On this page
  • TLS Encryption Overview
  • Mutual (mTLS) Authentication
  • Configure TLS Encryption for a WarpStream Cluster
  • Option 1: Configuring the WarpStream Agents to terminate TLS
  • Option 2: Configuring a Load Balancer to terminate TLS
  • Configure TLS Encryption for Kafka clients
  • TLS Profiles
  • Transparent HTTPS Proxies

Was this helpful?

  1. BYOC
  2. Advanced Agent Deployment Options

Protect Data in Motion with TLS Encryption

PreviousAgent GroupsNextLow Latency Clusters

Last updated 3 days ago

Was this helpful?

TLS Encryption Overview

By default Kafka/Schema Registry clients communicating with WarpStream Agents use PLAINTEXT, meaning that all data is sent in plain text (unencrypted). To encrypt data in motion (or data in transit) between your clients and your WarpStream Agents, you should configure them to use TLS Encryption.

WarpStream supports encryption based on , an open source cryptography toolkit that provides an implementation of the Transport Layer Security (TLS).

Enabling TLS encryption might have a performance impact due to overhead of encrypting and decrypting data. This performance impact can vary depending on the operating system, linux kernel version, and CPU used. We recommend using the newest and best possible versions of your Operating system, Kernel, and CPU to minimize any possible impacts.

TLS uses private-key/certificate pairs, which are used during the TLS handshake process.

  • Each WarpStream Agent needs a private-key/certificate pair, and the Kafka client uses the certificate to authenticate to the WarpStream Agent.

  • Each logical client needs a private-key/certificate pair if client authentication is enabled, and the WarpStream Agent uses the certificate to authenticate the Kafka client.

Mutual (mTLS) Authentication

If you configure TLS encryption, you can optionally configure . You can configure just TLS encryption (by default, TLS encryption includes certificate authentication of the server) and use a separate mechanism for client authentication (for example, mTLS or SASL). By default, TLS encryption enables one-way authentication in which the client authenticates the server certificate. For bidirectional authentication, where the broker also authenticates the client certificate, you can use mTLS.

When you use , the WarpStream Agent authenticates the Kafka client and the Kafka client also authenticates the WarpStream Agent. This bidirectional, or mutual, authentication provides an additional layer of security for your WarpStream cluster.

Configure TLS Encryption for a WarpStream Cluster

Configuring TLS for a WarpStream Cluster can be done in one of two ways. Which option is the best depends on your requirements and deployment environment.

Option 1: Configuring the WarpStream Agents to terminate TLS

This option is the recommended configuration when using TLS. This ensures that there is full end-to-end Kafka client to WarpStream Agent encryption. This comes at the disadvantage of needing to configure each WarpStream Agent with a TLS certificate.

  1. Create x509 encoded TLS certificates

    • Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization.

    • We recommend that either the certificate is made with IP SANs if using IP addresses or SANS if using hostnames to connect to your WarpStream Agents.

  2. Configure the WarpStream Agents to load the certificates

    • Once your certificates are generated to must set the WARPSTREAM_TLS_SERVER_CERT_FILE environment variable to the public key of the certificate and set WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE to the private key of the certificate.

  3. Configure the WarpStream Agents to enable TLS encryption

    • To enable TLS encryption for your Kafka agent, set the -kafkaTLSagent flag. Alternatively, you can set the WARPSTREAM_TLS_ENABLEDenvironment variable to true.

    • To enable TLS encryption for your Schema Registry agent, set the -schemaRegistryTLSagent flag. Alternatively, you can set the WARPSTREAM_SCHEMA_REGISTRY_TLS_ENABLEDenvironment variable to true.

By default WarpStream Agents will load the TLS certificate and key from the agent's local filesystem. The agents can be configured to instead load the TLS certificate and key from blob storage by using the -tlsBlobURL flag or WARPSTREAM_TLS_BLOB_URL environment variable. For example -tlsBlobURL=s3://my-tls-bucket.

Option 2: Configuring a Load Balancer to terminate TLS

Every Load Balancer is configured differently for TLS termination, bellow is information from the 3 major Cloud Providers for how to configure TLS termination on their Load Balancers.

  • AWS

  • GCP

    • GKE - As of 2024 GKE does not support TLS termination on Kubernetes Services with Load Balancer types

  • Azure

    • As of 2024 Azure Layer 4 load balancers do not support TLS termination

Configure TLS Encryption for Kafka clients

For configuring TLS encryption in your Kafka clients it is recommended to review the documentation for your Kafka client. Every Kafka client configures TLS differently and those configurations may change version to version.

TLS Profiles

By default the WarpStream Agent uses the default Golang TLS settings when serving the Kafka Protocol over TLS. These defaults change over time but and may not be suitable for all environments.

These profiles can be set via the -tlsProfile flag or WARPSTREAM_TLS_PROFILE environment variable.

All profiles use the following TLS Curves

tls.X25519
tls.CurveP256
tls.CurveP384
tls.CurveP521
tls.X25519MLKEM768

All TLS 1.3 connections use the following Cipher Suites:

tls.TLS_AES_128_GCM_SHA256
tls.TLS_AES_256_GCM_SHA384
tls.TLS_CHACHA20_POLY1305_SHA256

golang-default

old

This profile supports older TLS clients with the minimum TLS version set to 1.0 with support for the following Cipher Suites for TLS 1.0, 1.1 and 1.2:

tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
tls.TLS_RSA_WITH_AES_128_GCM_SHA256
tls.TLS_RSA_WITH_AES_256_GCM_SHA384
tls.TLS_RSA_WITH_AES_128_CBC_SHA256
tls.TLS_RSA_WITH_AES_128_CBC_SHA
tls.TLS_RSA_WITH_AES_256_CBC_SHA
tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA

intermediate

This profile supports intermediate TLS clients with the minimum TLS version set to 1.2 with support for the following Cipher Suites:

tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305

modern

This profile supports modern TLS clients and TLS 1.3 only.

Transparent HTTPS Proxies

Some organizations use transparent HTTPS Proxies to decrypt and inspect HTTPS traffic. This may cause the WarpStream Agent to be unable to communicate with the WarpStream Control Plane.

You may see the following error if that is the case

Post "https://metadata.default.${region}$.${cloud}$.warpstream.com/api/v1/agent/agentpool": tls: failed to verify certificate: x509: certificate signed by unknown authority

An example values.yaml of this configuration via our Kubernetes Helm Chart is as follows:

extraEnv:
  - name: WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE
    # The path to where the configmap is mounted.
    # In this example the item inside the configmap is
    # called `ca.crt` so it get's created as that file.
    value: /etc/https-proxy/ca.crt

volumeMounts:
  - name: https-proxy-ca
    # The path to where the confimap should be mounted.
    mountPath: /etc/https-proxy

volumes:
  - name: https-proxy-ca
    configMap:
      # Name of the configmap in Kubenretes that contains 
      # the x509 PEM file for the HTTPS Proxy.
      name: https-proxy-ca

If running WarpStream in Kubernetes using can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities.

This option can be simpler to implement for WarpStream clusters behind a load balancer. This option requires that you for it to work as expected. The disadvantage of this option is that the communication between the Load Balancer and WarpStream Agents are not encrypted. If you have a requirement for full end-to-end encryption we do not recommend using this option. This option also cannot use mTLS as an authentication mechanism due to TLS termination on Load Balancers not being able to pass-through the client certificate.

We recommend using the to learn how to configure Java-based clients for TLS encryption.

The WarpStream agent supports setting different TLS Profiles based on recommendations from .

The ability to configure TLS Profiles was added in Agent Version .

Use the Golang defaults for TLS Version, Curves and Cipher Suites. These will change over time. See documentation for details.

Starting in version the WarpStream agent as the ability to provide an HTTPS Proxy Certificate Authority for the Agent to trust. This can be done with the -httpsProxyCACertFile flag or WARPSTREAM_HTTPS_PROXY_CA_CERT_FILE environment variable.

Transport Layer Security (TLS)
OpenSSL
mutual (mTLS) authentication
mTLS Authentication
cert-manager
EKS NLB
Network Load Balancer
Network Load Balancer SSL Proxy
Confluent Platform documentation
Mozilla SSL
Golang TLS
Advertise the WarpStream Agents behind a traditional load balancer
v657
v657