Protocol and Feature Support

The current implementation of the Apache Kafka protocol in WarpStream supports the basic ability to create topics, delete topics, produce data, consume data, and use consumer groups to load balance consumers and track offsets. Specifically, the following Apache Kafka messages are currently supported:

  1. Produce

  2. InitProducerID

  3. Fetch

  4. ListOffsets

  5. Metadata

  6. OffsetCommit

  7. OffsetFetch

  8. FindCoordinator

  9. JoinGroup

  10. Heartbeat

  11. SyncGroup

  12. OffsetDelete

  13. ApiVersions

  14. CreateTopics

  15. DeleteTopics

  16. ListGroups

  17. AlterConfigs

  18. DescribeConfigs

  19. DescribeCluster

  20. DescribeGroups

  21. DeleteGroup

  22. LeaveGroup

  23. CreateACLs

  24. DescribeACLs

  25. DeleteACLs

  26. CreatePartitions

We're continuously adding support for more Apache Kafka features and message types. Please contact us for specific feature requests or if you notice any discrepancies.

Note that because of WarpStream's stateless architecture, many of the Apache Kafka protocol messages are irrelevant. For example, messages like:

  1. AlterReplicaLogDirs

  2. ElectLeaders

  3. ListPartitionReassignments

  4. AlterPartitionReassignments

  5. DescribeQuorum

  6. UnregisterBroker

  7. ControlledShutdown

  8. StopReplica

  9. LeaderAndIsr

have no meaning or value when using WarpStream because data durability and replication is managed by the underlying object store. Partitions do not have assigned "leaders", and clean shutdown is automated by virtue of the Agents being stateless.

Transactions / Atomicity

WarpStream does not currently support Apache Kafka's transactional APIs, but we intend to in the near future.

However, unlike Apache Kafka, all data that is sent in a single Produce message is committed completely atomically regardless of how many topics or partitions are in the request. Most Apache Kafka clients do not directly expose the ability to construct a Produce batch manually and control which messages it contains. If you need support for this feature in the near future, please contact us.

Note that the PutRecords API in the WarpStream Kinesis protocol implementation is fully atomic.

Exactly Once Semantics

WarpStream does not currently support Apache Kafka's exactly once semantics, but we intend to in the near future. Please contact us if this is of interest to you.

Schema Registry

WarpStream has a BYOC Schema Registry built into the Agent binary. Check out the documentation for how it works.

WarpStream BYOC Schema Registry supports most APIs listed in Confluent Schema Registry's API documentation. Here are the list of features it doesn't support:

  • it doesn't support data contracts, so the metadata and ruleSet fields in the schemas are ignored.

  • it doesn't support /mode endpoints.

  • it doesn't support /exporters endpoints.

  • it doesn't support subject aliases.

Record Retention Based on Custom Timestamps

Kafka implements record retention using the timestamps within the records themselves. If the client sets the timestamp using the CREATE_TIME timestamp type, it can send a record with a timestamp far in the future or the past. This will result in the record being deleted based on the timestamp rather than real-time passing.

However, WarpStream differs in this aspect. In Warpstream, retention is based solely on the real-time when the record was created. Although you can set a custom timestamp for the record, it will not be used to calculate retention. The retention mechanism in Warpstream strictly adheres to the actual creation time of the record.

Supported Clients

WarpStream should work with any correctly implemented Kafka client. Officially we support librdkafka, franz-go, as well as the standard Java client. Please check our documentation on tuning your clients for maximum performance with WarpStream.

Known Incompatibilities

  1. The current implementation any of the __tagged__ fields in the protocol and ignores them entirely.

  2. The current implementation does not enforce throttling and ignores all throttling-related fields/settings.

  3. All Kafka protocol requests have a maximum timeout of 15s (except for JoinGroup and SyncGroup).

Last updated