LogoLogo
WarpStream.comSlackDiscordContact UsCreate Account
  • Overview
    • Introduction
    • Architecture
      • Service Discovery
      • Write Path
      • Read Path
      • Life of a Request (Simplified)
    • Change Log
  • Getting Started
    • Install the WarpStream Agent / CLI
    • Run the Demo
    • "Hello World" for Apache Kafka
  • BYOC
    • Run the Agents Locally
    • Deploy the Agents
      • Object Storage Configuration
      • Kubernetes Known Issues
      • Rolling Restarts and Upgrades
    • Infrastructure as Code
      • Terraform Provider
      • Helm charts
      • Terraform Modules
    • Monitoring
      • Pre-made Datadog Dashboard
      • Pre-made Grafana Dashboard
      • Important Metrics and Logs
      • Recommended List of Alerts
      • Monitoring Consumer Groups
      • Hosted Prometheus Endpoint
    • Client Configuration
      • Tuning for Performance
      • Configure Clients to Eliminate AZ Networking Costs
        • Force Interzone Load Balancing
      • Configuring Kafka Client ID Features
      • Known Issues
    • Authentication
      • SASL Authentication
      • Mutual TLS (mTLS)
      • Basic Authentication
    • Advanced Agent Deployment Options
      • Agent Roles
      • Agent Groups
      • Protect Data in Motion with TLS Encryption
      • Low Latency Clusters
      • Network Architecture Considerations
      • Agent Configuration Reference
      • Reducing Infrastructure Costs
      • Client Configuration Auto-tuning
    • Hosted Metadata Endpoint
    • Managed Data Pipelines
      • Cookbooks
    • Schema Registry
      • WarpStream BYOC Schema Registry
      • Schema Validation
      • WarpStream Schema Linking
    • Port Forwarding (K8s)
    • Orbit
    • Enable SAML Single Sign-on (SSO)
    • Trusted Domains
    • Diagnostics
      • GoMaxProcs
      • Small Files
  • Reference
    • ACLs
    • Billing
      • Direct billing
      • AWS Marketplace
    • Benchmarking
    • Compression
    • Protocol and Feature Support
      • Kafka vs WarpStream Configuration Reference
      • Compacted topics
    • Secrets Overview
    • Security and Privacy Considerations
    • API Reference
      • API Keys
        • Create
        • Delete
        • List
      • Virtual Clusters
        • Create
        • Delete
        • Describe
        • List
        • DescribeConfiguration
        • UpdateConfiguration
      • Virtual Clusters Credentials
        • Create
        • Delete
        • List
      • Monitoring
        • Describe All Consumer Groups
      • Pipelines
        • List Pipelines
        • Create Pipeline
        • Delete Pipeline
        • Describe Pipeline
        • Create Pipeline Configuration
        • Change Pipeline State
      • Invoices
        • Get Pending Invoice
        • Get Past Invoice
    • CLI Reference
      • warpstream agent
      • warpstream demo
      • warpstream cli
      • warpstream cli-beta
        • benchmark-consumer
        • benchmark-producer
        • console-consumer
        • console-producer
        • consumer-group-lag
        • diagnose-record
        • file-reader
        • file-scrubber
      • warpstream playground
    • Integrations
      • Arroyo
      • AWS Lambda Triggers
      • ClickHouse
      • Debezium
      • Decodable
      • DeltaStream
      • docker-compose
      • DuckDB
      • ElastiFlow
      • Estuary
      • Fly.io
      • Imply
      • InfluxDB
      • Kestra
      • Materialize
      • MinIO
      • MirrorMaker
      • MotherDuck
      • Ockam
      • OpenTelemetry Collector
      • ParadeDB
      • Parquet
      • Quix Streams
      • Railway
      • Redpanda Console
      • RisingWave
      • Rockset
      • ShadowTraffic
      • SQLite
      • Streambased
      • Streamlit
      • Timeplus
      • Tinybird
      • Upsolver
    • Partitions Auto-Scaler (beta)
    • Serverless Clusters
Powered by GitBook
On this page

Was this helpful?

  1. Reference
  2. Protocol and Feature Support

Kafka vs WarpStream Configuration Reference

The purpose of this page is to compare configuration options in Kafka and WarpStream 1:1 as applicable.

The primary goal of this page is to reassure experienced Kafka operators migrating to WarpStream that most of the configuration and settings they're accustomed to tuning and managing in Apache Kafka have no equivalent in WarpStream and don't need to be managed at all due to WarpStream's architecture.

group.max.session.timeout.ms

Not supported.

auto.create.topics.enable

Same. Managed via Kafka API or WarpStream console.

auto.leader.rebalance.enable

N/A. WarpStream Agents do not have leaders.

controlled.shutdown.enable

WarpStream Agents accept a "gracefulShutdownDuration" flag that defaults to 80 seconds.

controlled.shutdown.max.retries

N/A. Nothing can go wrong during an Agent clean shutdown because they're stateless.

default.replication.factor

N/A. Doesn't apply to WarpStream because the Agents are stateless and data is stored durably in object storage at all times. Response is hard-coded to 3 to signify high durability.

delete.topic.enable

Same, set as a cluster level configuration via the Kafka API.

kafka.http.metrics.host

Not supported.

kafka.http.metrics.port

Agent flag "httpPort" to expose Prometheus metrics.

kafka.log4j.dir

Not supported.

kerberos.auth.enable

N/A

leader.imbalance.check.interval.seconds

N/A. WarpStream Agents do not have leaders.

leader.imbalance.per.broker.percentage

N/A. WarpStream Agents do not have leaders.

log.cleaner.dedupe.buffer.size

Not supported. WarpStream Agents have an equivalent to this setting internally, but its not configurable by default.

log.cleaner.delete.retention.ms

Topic-level config not supported. Broker-level config, "delete.retention.ms", supported.

log.cleaner.enable

N/A. Compacted topics are always compacted.

log.cleaner.min.cleanable.ratio

N/A. WarpStream uses different heuristics to determine when to compact the topic. This is handled automatically.

log.cleaner.threads

N/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.

log.retention.bytes

Not supported. Response hard-coded to -1 to signify "no limit".

log.retention.check.interval.ms

N/A. Retention is enforced using a different mechanism called the "dead scanner" in WarpStream, and that process is managed automatically.

log.retention.hours

Same as Kafka.

Logcleaner ratio

N/A. WarpStream uses different heuristics to determine when to compact the topic. This is handled automatically.

log.roll.hours

N/A. WarpStream uses object storage and doesn't have physical log files that need to be rolled.

log.segment.bytes

N/A. Object storage file size is an implementation detail of WarpStream compaction, and WarpStream files contain data from many different topic-partitions.

message.max.bytes

Agent environment variable WARPSTREAM_MAX_PRODUCE_RECORD_SIZE_BYTES

min.insync.replicas

N/A. Response hard-coded to 1 as object storage handles replication / durability.

num.io.threads

N/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.

num.partitions

Same as Kafka.

num.replica.fetchers

N/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.

offsets.topic.num.partitions

N/A. WarpStream virtual clusters do not publish offset topics. Offsets are stored in WarpStream control plane / metadata store.

offsets.topic.replication.factor

N/A. WarpStream virtual clusters do not publish offset topics. Offsets are stored in WarpStream control plane / metadata store.

port

Agent environment variable "WARPSTREAM_KAFKA_PORT".

replica.fetch.max.bytes

N/A. WarpStream Agents are stateless and therefore have no replicas.

replica.lag.max.messages

N/A. WarpStream Agents are stateless and therefore have no replicas.

replica.lag.time.max.ms

N/A. WarpStream Agents are stateless and therefore have no replicas.

unclean.leader.election.enable

N/A. WarpStream has no leaders, and therefore unclean leader elections are impossible.

zookeeper.session.timeout.ms

N/A. WarpStream does not depend on Zookeeper.

zookeeper.connection.timeout.ms

N/A. WarpStream does not depend on Zookeeper.

security.inter.broker.protocol

Not supported.

ssl.client.auth

Agent environment variable "WARPSTREAM_REQUIRE_MTLS_AUTHENTICATION"

ssl.truststore.location

Agent environment variable "WARPSTREAM_TLS_CLIENT_CA_CERT_FILE"

ssl.truststore.password

Not supported.

ssl.keystore.location

Agent environment variables "WARPSTREAM_TLS_SERVER_CERT_FILE", "WARPSTREAM_TLS_SERVER_PRIVATE_KEY_FILE"

ssl.keystore.password

N/A

ssl.key.password

N/A

ssl.protocol

Agent environment variable "WARPSTREAM_TLS_ENABLED".

ssl.enabled.protocols

N/A

ssl.keystore.type

N/A

ssl.truststore.type

N/A

broker.id.generation.enable

N/A. Broker IDs in WarpStream are virtual and generated automatically.

sasl.kerberos.service.name

N/A

num.network.threads

N/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically.

num.recovery.threads.per.data.dir

N/A. WarpStream is written in Go, not Java, and doesn't have user-controlled threadpools. Concurrency is managed automatically. Also WarpStream Agents are stateless and have no concept of a recovery.

socket.send.buffer.bytes

N/A

socket.receive.buffer.bytes

N/A

socket.request.max.bytes

N/A

replica.fetch.wait.max.ms

N/A. WarpStream Agents are stateless and therefore have no replicas.

replica.socket.timeout.ms

N/A. WarpStream Agents are stateless and therefore have no replicas.

replica.socket.receive.buffer.bytes

N/A. WarpStream Agents are stateless and therefore have no replicas.

replica.high.watermark.checkpoint.interval.ms

N/A. WarpStream Agents are stateless and therefore have no replicas.

controller.socket.timeout.ms

N/A

controller.message.queue.size

N/A

zookeeper.sync.time.ms

N/A. WarpStream does not depend on Zookeeper.

queued.max.requests

N/A

fetch.purgatory.purge.interval.requests

N/A

producer.purgatory.purge.interval.requests

N/A

authorizer.class.name

N/A. There is only one ACL Authorized implementation in WarpStream.

allow.everyone.if.no.acl.found

Same as Kafka.

broker.id

N/A. WarpStream Broker IDs are generated automatically.

log.dirs

N/A. WarpStream Agents are stateless and store data exclusively in object storage.

zookeeper.connect

N/A. WarpStream does not depend on Zookeeper.

listeners

Agent environment variables: "WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE" and "WARPSTREAM_KAFKA_PORT"

PreviousProtocol and Feature SupportNextCompacted topics

Last updated 4 months ago

Was this helpful?