Topic Configuration Reference

This page lists the supported configuration for WarpStream topics. Configuration not listed here will be silently ignored by the WarpStream Kafka API.

Configuration items that start with warpstream. are WarpStream specific configurations, native Kafka tooling like the kafka-configs CLI may not be able to set these configurations. It is recommended to set these configurations using our REST API for topics, WarpStream terraform provider, or a non-Java library like franz-go to manage these configurations.

message.timestamp.type

Define whether the timestamp in the message is message create time or log append time. The value should be either CreateTime or LogAppendTime

Type:
string

Default:

CreateTime

Valid Values:

[CreateTime, LogAppendTime]

cleanup.policy

This config designates the retention policy to use on topics. The “delete” policy (which is the default) will discard old message when their retention time or size limit has been reached. The “compact” policy will enable topic compaction, which retains the latest value for each key. It is also possible to specify both policies in a comma-separated list (e.g. “delete,compact”). In this case, old messages will be discarded per the retention time and size configuration, while retained messages will be compacted.

In WarpStream cleanup.policy cannot be changed after a topic is created.

Type:
list

Default:

delete

Valid Values:

[compact, delete]

retention.ms

This configuration controls the maximum time we will retain messages in a topic partition. This represents an SLA on how soon consumers must read their data. If set to -1, no time limit is applied.

Type:
long

Default:

86400000 (1 day)

Valid Values:

[-1,3600000…,3153600000000]

min.compaction.lag.ms

The minimum time a message will remain uncompacted in the topic partition. Only applicable for topics that are being compact.

Type:
long

Default:

0

Valid Values:

[0,…,3153600000000]

delete.retention.ms

The amount of time to retain delete tombstone markers for compacted topics. This setting also gives a bound on the time in which a consumer must complete a read if they begin from offset 0 to ensure that they get a valid snapshot of the final stage (otherwise delete tombstones may be collected before they complete their scan).

Type:
long

Default:

86400000 (1 day)

Valid Values:

[-1,3600000,…,3153600000000]

warpstream.compression.type.fetch

The compression algorithm that is used to return compressed batches to consumers for this topic.

Type:
string

Default:

agent (lz4)

Valid Values:

[agent, lz4, zstd]

warpstream.schema.registry.type

The schema registry type to use when schema validation is enabled.

Type:
string

Default:

STANDARD

Valid Values:

[STANDARD, AWS_GLUE]

warpstream.schema.validation.warning.only

Should schema validations only return warnings and not block producing of messages to the topic.

Type:
string

Default:

true

Valid Values:

[true, false]

warpstream.key.schema.validation

Should message key schemas be validated when messages are being produced to the topic.

Type:
string

Default:

false

Valid Values:

[true, false]

warpstream.key.subject.name.strategy

warpstream.value.schema.validation

Should message value schemas be validated when messages are being produced to the topic.

Type:
string

Default:

false

Valid Values:

[true, false]

warpstream.value.subject.name.strategy

warpstream.partitions_auto_scaler.enabled

Should the partition autoscaler be enabled for the topic.

Type:
string

Default:

false

Valid Values:

[true, false]

warpstream.partitions_auto_scaler.per_partition_throughput_uncompressed_bytes_per_second

The maximum thoughput per partition a topic should have before more partitions are added through autoscaling.

Type:
long

Default:

0

Valid Values:

[0,…]

warpstream.partitions_auto_scaler.max_partition_count

The maximum number of partitions that the topic can autoscale to.

Type:
long

Default:

0

Valid Values:

[0,…,3153600000000]

warpstream.deletion.protection.enabled

Should the topic have delete protection enabled. If enabled the topic cannot be deleted until disabled.

Type:
string

Default:

false

Valid Values:

[true, false]

Last updated

Was this helpful?