Protocol and Feature Support
The current implementation of the Apache Kafka protocol in WarpStream supports the basic ability to create topics, delete topics, produce data, consume data, and use consumer groups to load balance consumers and track offsets. Specifically, the following Apache Kafka messages are currently supported:
Produce
InitProducerID
Fetch
ListOffsets
Metadata
OffsetCommit
OffsetFetch
FindCoordinator
JoinGroup
Heartbeat
SyncGroup
OffsetDelete
ApiVersions
CreateTopics
DeleteTopics
ListGroups
AlterConfigs
DescribeConfigs
DescribeCluster
DescribeGroups
DeleteGroup
LeaveGroup
CreateACLs
DescribeACLs
DeleteACLs
CreatePartitions
(for self-hosted deployments; requires Agent v530 or higher)
We're continuously adding support for more Apache Kafka features and message types. Please contact us for specific feature requests or if you notice any discrepancies.
Note that because of WarpStream's stateless architecture, many of the Apache Kafka protocol messages are irrelevant. For example, messages like:
AlterReplicaLogDirs
ElectLeaders
ListPartitionReassignments
AlterPartitionReassignments
DescribeQuorum
UnregisterBroker
ControlledShutdown
StopReplica
LeaderAndIsr
have no meaning or value when using WarpStream because data durability and replication is managed by the underlying object store. Partitions do not have assigned "leaders", and clean shutdown is automated by virtue of the Agents being stateless.
Transactions / Atomicity
WarpStream does not currently support Apache Kafka's transactional APIs, but we intend to in the near future.
However, unlike Apache Kafka, all data that is sent in a single Produce
message is committed completely atomically regardless of how many topics or partitions are in the request. Most Apache Kafka clients do not directly expose the ability to construct a Produce
batch manually and control which messages it contains. If you need support for this feature in the near future, please contact us.
Note that the PutRecords
API in the WarpStream Kinesis protocol implementation is fully atomic.
Exactly Once Semantics
WarpStream does not currently support Apache Kafka's exactly once semantics, but we intend to in the near future. Please contact us if this is of interest to you.
Schema Registry
WarpStream does not have a Schema Registry built into the Agent binaries, however, since it implements the Kafka protocol, you can use your existing Schema Registry implementation with WarpStream. If you're interested in having a Schema Registry implementation built into WarpStream, please contact us.
Supported Clients
We have tested WarpStream compatibility the most with the official librdkafka library which has wrappers in almost every programming language. In addition, we've tested compatibility with the Go-specific libraries: franz-go and segmentio, although we recommend Go users use the franz-go library. Please check our documentation on tuning your clients for maximum performance with WarpStream.
Known Incompatibilities
The current implementation any of the __tagged__ fields in the protocol and ignores them entirely.
The current implementation does not enforce throttling and ignores all throttling-related fields/settings.
All Kafka protocol requests have a maximum timeout of 15s (except for JoinGroup and SyncGroup).
Last updated