LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Port Forwarding (K8s)
    • Orbit
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
Powered by GitBook
On this page
  • Creating TLS keys and Certificates for the WarpStream Agents
  • Creating TLS keys and Certificate for the Kafka/Schema Registry Clients
  • Configure WarpStream Agents
  • Configure Kafka and Schema Registry clients

Was this helpful?

  1. BYOC
  2. Authentication

Mutual TLS (mTLS)

PreviousSASL AuthenticationNextBasic Authentication

Last updated 1 month ago

Was this helpful?

Mututal TLS (mTLS) authentication requires TLS encryption, this page shows you how to configure both at the same time and is a superset of configurations required just for .

Creating TLS keys and Certificates for the WarpStream Agents

Every organization has different policies on how to create and manage certificate. We recommend talking with your IT team for how to best create certificates in your organization.

If running WarpStream in Kubernetes using can be the easiest way to create certificates. It supports a wide range of certificate providers including private certificate authorities.

We recommend that either the certificate is made with IP SANs if using IP addresses or SANS if using hostnames to connect to your WarpStream Agents.

Creating TLS keys and Certificate for the Kafka/Schema Registry Clients

When using mutual TLS, it is highly recommended to use a private certificate authority for certificates. Most large organizations typically already have their own internal private certificate authority that you can use to generate client certificates.

For alternatives if your organization does not have an existing private certificate authority, every cloud provider offers a managed solution, for example . You can also create a certificate authority manually using the openssl command line however that can be complex to manage in production environments.

When using mutual TLS is it recommended to give each application their own unique certificate with their own unique Distinguished Name(DN). If using having unique certificates with unique DN's will allow each application to have it's own set of permissions.

Configure WarpStream Agents

Once your certificates are generated to must set the WARPSTREAM_TLS_SERVER_CERT_FILE environment variable to the public key of the certificate and set WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE to the private key of the certificate.

First, to enable TLS you must set the WARPSTREAM_TLS_ENABLED environment variable to true, or set the kafkaTLS flag. Then, to enable mutual TLS you must set the WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION environment variable to true, or set the requireMTLSAuthentication flag.

Once authentication is enabled on the Agent, it will enforce that all Apache Kafka clients that connect to them authenticate themselves via mTLS. Otherwise, it will refuse the connection.

It is also highly recommended to set the environment variable WARPSTREAM_TLS_CLIENT_CA_CERT_FILE to the public keys of the certificate authorities that sign your client certificates. If this environment variable is not set WarpStream defaults to trusting all client certificates from your Operating System's root certificate store. This store typically contains all the public keys for all the publicly accepted certificate authorities. If this environment variable is not set any certificate from any public certificate authority will be valid and authenticated against your WarpStream agent which could cause external data leaks.

By default, the Agent will use the Distinguished Name(DN) from the client TLS certificate as the principal for ACLs. A custom TLS mapping rule regex can be provided using the -tlsPrincipalMappingRule flag to extract a name from the DN. For example, the rule CN=([^,]+) will extract the Common Name(CN) from the DN, and use that as the ACL principal.

For example, given a certificate with the DN of CN=test_principal,O=ACME Corp, if -tlsPrincipalMappingRule is set to CN=([^,]+) then the name test_principal will be used as the mTLS ACL principal.

Configure Kafka and Schema Registry clients

For configuring mTLS in your Kafka/Schema Registry clients it is recommended to review the documentation for your client. Every client configures mTLS differently and those configurations may change version to version.

We recommend using the to learn how to configure Java-based clients for mTLS.

Confluent Platform documentation
cert-manager
AWS Private CA
ACLs
Configuring the WarpStream Agents to terminate TLS