> For the complete documentation index, see [llms.txt](https://docs.warpstream.com/warpstream/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.warpstream.com/warpstream/tableflow/query-based-deletion.md). # Query-Based Deletion Create, preview, monitor, and cancel query-based deletion requests for Tableflow tables. Tableflow supports removing rows from a table that match a filter starting from agent version `v833`+. Common use cases include GDPR deletion requests and removing ingested credentials or other sensitive data. Deletions apply to the Tableflow table only. Records from the source Kafka topic will not be deleted. To remove an entire table, see [Table Deletion](/warpstream/tableflow/tableflow.md#table-deletion). To remove data based on its age, see [Data Retention and TTL](/warpstream/tableflow/tableflow.md#data-retention-and-ttl). {% hint style="danger" %} **Deleting data is irreversible.** Before creating a deletion request, review the query filter and optional Kafka timestamp range for selecting rows to delete and use the preview feature to see the sample rows to be deleted. {% endhint %} ## Write a Filter Deletion requests use WarpStream's query filter syntax. For example, the following filter matches rows for one user: ``` user_id == "abc" ``` You can combine conditions with `and`, `or`, and parentheses: ``` tenant_id == 42 and (status == "inactive" or deleted_at == NULL) ``` Nested fields use dot-separated paths: ``` payload.user_id == 123 ``` Deletion filters support: * `==`, `!=`, `<`, `<=`, `>`, and `>=` * `and`, `or`, and parentheses * String, number, boolean, and `NULL` values * Nested field paths Deletion filters do not support `LIKE`, `MATCH`, `IN`, `BETWEEN`, arithmetic, array/map access, or type casts. A filter can contain at most 5,000 characters. ### Limit by Kafka Timestamp You can set optional **Min Kafka Timestamp** and **Max Kafka Timestamp** values to bound the timeframe for which the deletion will be applied to. These bounds are inclusive and apply to the Kafka record timestamp, not to a timestamp field in the record itself. The minimum timestamp cannot be later than the maximum timestamp. ### Resolve an Ambiguous Field A field name is ambiguous if a column was removed and another column with the same name was later added to the table. Tableflow rejects the deletion because the columns can have different types or meanings. The validation error lists the iceberg field IDs and types that match the name. Use those IDs to specify which columns the predicates reference. For example say you wanted to filter on `user_id`: ``` user_id == 123 ``` and it errors out saying that two field IDs were found for this column name, one with ID 10 and type string and the other ID 6 and type long. If your intent is to apply the deletion on both versions of the column you should rewrite your query as so: ``` field_id#10 == "123" or field_id#6 == 123 ``` If you only want to delete the data matching it as a long (the one with field ID 6) you should rewrite your query as so: ``` field_id#6 == 123 ``` The right-hand operand must match the data type of the target column. ## Preview a Deletion A preview applies the filter and optional Kafka timestamp range from the deletion query without deleting data. It returns a sample of up to 10 matching rows. To preview a deletion: 1. In the [WarpStream Console](https://console.warpstream.com), open your Tableflow cluster. 2. Select the table, then open the **Deletions** tab. 3. Click **New Deletion Request**. 4. Enter the filter and optional Kafka timestamp bounds. 5. Optionally, enter a comma-separated list of column names under **Preview columns** to control which fields will be displayed in the preview. 6. Click **Preview** and review the sample rows. Note that deletion previews are stateless, so if you leave your page it will loose any ongoing progress. ## Create a Deletion Request After reviewing the preview, click **Create Deletion Request**. The request will appear in the **Deletion Requests** list with a `queued` status. Creating the same filter more than once creates independent deletion requests. Review the existing requests before submitting another one. ## How Deletions Execute Deletion requests are processed asynchronously. They move through the following statuses: * `queued`: The request was accepted and is waiting to be scheduled. * `scheduled`: The request was assigned for processing. Its progress appears in the Console. * `done`: Tableflow finished processing all data covered by the request. * `canceled`: The request was canceled and will not make further progress. Requests normally remain queued for at least four hours. This delay allows Tableflow to process multiple requests together and reduce the amount of work required to scan a table as well as the write amplification. After a request is scheduled, its completion time depends on the table size, the number of matching records, and the available Agent capacity. Tableflow continues to ingest data while deletions runs. {% hint style="info" %} A deletion request is not permanent and will not be applied to future data. Row added to the Tableflow table after the request is scheduled are not covered, even if they match the filter. Create another deletion request to remove those records. {% endhint %} ## Monitor a Deletion Open the table's **Deletions** tab and click **Refresh** to see the latest request status and progress: * **Processed** shows how many files and records Tableflow has scanned. * **Matching rows** shows how many rows matched this request's filter. This value increases while the request is processed. * **Last updated** shows when the request last made progress. * **Duration** shows the total time for a completed request, or the elapsed processing time for a scheduled request. * **Lag** shows how long an active request has gone without making progress. Deletion requests can be processed together. If several requests in the same batch match one row, that row is counted once for each matching request. For this reason, adding the **Matching rows** values from several requests can produce a number larger than the number of rows removed from the table. Additionally, the Agents emit the following metrics for deletions: | Metric | Type | Description | | ------------------------------------------------------- | ------- | --------------------------------------------------------------------------------------------------------------------------- | | `warpstream.tableflow_deletion_pending_requests` | gauge | Number of query-deletion requests currently pending, grouped by status. | | `warpstream.tableflow_deletion_max_request_lag_seconds` | gauge | Maximum query-deletion request lag in seconds by status, measured since each pending request last changed or made progress. | | `warpstream.tableflow_total_deleted_records` | counter | Monotonically increasing count of records deleted from a Tableflow table. | ## Cancel a Deletion You can cancel a request while its status is `queued` or `scheduled`: 1. Open the table's **Deletions** tab. 2. Find the request and click **Cancel**. 3. Confirm the cancellation. Canceling a scheduled request stops remaining work, but it does not restore data that has already been removed. Requests with a `done` or `canceled` status cannot be canceled. ## When Deleted Data Stops Appearing The `done` status means that Tableflow has finished applying the deletion. Tableflow must then publish a new Iceberg snapshot before query engines see the updated table state. This normally takes about five minutes. External catalog synchronization and query-engine metadata caches can add more time. When confirming a deletion, refresh the query engine's table metadata and query the latest snapshot. Queries that explicitly use an older snapshot can continue to return deleted records until that snapshot expires. ## When Data Is Physically Removed Tableflow does not immediately delete replaced files from object storage. Old files are removed asynchronously after they are no longer referenced by a retained Iceberg snapshot. This cleanup can take several hours. A `done` status confirms that the deletion was applied, but it does not confirm that old files have already been physically removed from object storage. ## Limitations * Query-based deletion applies only to Tableflow. It does not delete source Kafka records. * A request applies to data added to the table before the request is scheduled. It does not continuously delete future matching data. * A table can have at most 500 active deletion requests. * The Console retains the 100 most recent `done` or `canceled` requests for each table. * A filter can contain at most 5,000 characters. * Preview returns at most 10 sample records. Creating and canceling deletion requests are recorded in [Audit Logs](/warpstream/reference/audit-logs.md) when Audit Logs are enabled. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.warpstream.com/warpstream/tableflow/query-based-deletion.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.