Topic Configuration Reference
This page lists the supported configuration for WarpStream topics. Configuration not listed here will be silently ignored by the WarpStream Kafka API.
Configuration items that start with warpstream. are WarpStream specific configurations, native Kafka tooling like the kafka-configs CLI may not be able to set these configurations. It is recommended to set these configurations using our REST API for topics, WarpStream terraform provider, or a non-Java library like franz-go to manage these configurations.
message.timestamp.type
Define whether the timestamp in the message is message create time or log append time. The value should be either CreateTime or LogAppendTime
Default:
CreateTime
Valid Values:
[CreateTime, LogAppendTime]
cleanup.policy
This config designates the retention policy to use on topics. The “delete” policy (which is the default) will discard old message when their retention time or size limit has been reached. The “compact” policy will enable topic compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. “delete,compact”). In this case, old messages will be discarded per the retention time and size configuration, while retained messages will be compacted.
In WarpStream cleanup.policy cannot be changed after a topic is created.
Default:
delete
Valid Values:
[compact, delete]
retention.ms
This configuration controls the maximum time we will retain messages in a topic partition. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied.
Default:
86400000 (1 day)
Valid Values:
[-1,3600000…,3153600000000]
min.compaction.lag.ms
The minimum time a message will remain uncompacted in the topic partition. Only applicable for topics that are being compact.
Default:
0
Valid Values:
[0,…,3153600000000]
delete.retention.ms
The amount of time to retain delete tombstone markers for compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan).
Default:
86400000 (1 day)
Valid Values:
[-1,3600000,…,3153600000000]
warpstream.compression.type.fetch
The compression algorithm that is used to return compressed batches to consumers for this topic.
Default:
agent (lz4)
Valid Values:
[agent, lz4, zstd]
warpstream.schema.registry.type
The schema registry type to use when schema validation is enabled.
Default:
STANDARD
Valid Values:
[STANDARD, AWS_GLUE]
warpstream.schema.validation.warning.only
Should schema validations only return warnings and not block producing of messages to the topic.
Default:
true
Valid Values:
[true, false]
warpstream.key.schema.validation
Should message key schemas be validated when messages are being produced to the topic.
Default:
false
Valid Values:
[true, false]
warpstream.key.subject.name.strategy
warpstream.value.schema.validation
Should message value schemas be validated when messages are being produced to the topic.
Default:
false
Valid Values:
[true, false]
warpstream.value.subject.name.strategy
warpstream.partitions_auto_scaler.enabled
Default:
false
Valid Values:
[true, false]
warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second
The maximum thoughput per partition a topic should have before more partitions are added through autoscaling.
Default:
0
Valid Values:
[0,…]
warpstream.partitions_auto_scaler.max_partition_count
The maximum number of partitions that the topic can autoscale to.
Default:
0
Valid Values:
[0,…,3153600000000]
warpstream.deletion.protection.enabled
Should the topic have delete protection enabled. If enabled the topic cannot be deleted until disabled.
Default:
false
Valid Values:
[true, false]
Last updated
Was this helpful?