Lakehouse Runtime Catalog (BigLake)

This page describes how to integrate with Lakehouse runtime catalog (formerly BigLake Metastore) so that you can query Iceberg tables created by Tableflow directly in BigQuery via Lakehouse.

Integrating with Lakehouse runtime catalog is the recommended way to make Tableflow Iceberg tables queryable in BigQuery. This approach uses the REST Catalog API to handle table registration, giving you native Iceberg support and keeping your catalog automatically in sync as new snapshots are taken.

BigLake vs Lakehouse runtime catalog integration: Lakehouse runtime catalog is preferred over the BigQuery integration because the former exposes the standard Iceberg REST Catalog protocol. This means the tables are automatically queryable from BigQuery and from any engine that can connect to an Iceberg REST Catalog, such as Spark, Trino, and Presto. The legacy BigQuery integration only creates BigQuery external tables, so the tables are only visible to BigQuery.

Prerequisites

In order for this to work, please upgrade your WarpStream Agents to at least v769 (v828+ for multi-bucket catalogs). Additionally, the integration uses Google Application Default Credentials (ADC). The agent authenticates to both the BigLake REST Catalog API and GCS using the service account it runs as, and as such the service account requires the following role:

Role	Purpose
roles/biglake.admin	Create and update tables in the BigLake Metastore

Or if you prefer more granular permissions, you can create a custom role with the following permissions:

• biglake.catalogs.get
• biglake.namespaces.create
• biglake.namespaces.get
• biglake.tables.delete
• biglake.tables.get
• biglake.tables.register

To add the role to your service account, run

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
    --role="roles/biglake.admin"

The agent already has GCS access for reading and writing Iceberg data. The only additional permission needed for the Lakehouse integration is roles/biglake.admin.

Create a Catalog

Lakehouse runtime catalog supports two types of catalogs, the multi-bucket catalog and the single-bucket catalog. Google recommends using the multi-bucket catalog as that allows for a custom catalog name and for multiple buckets to be configured for a single catalog.

To create a multi-bucket catalog:

gcloud biglake iceberg catalogs create \
  <CATALOG_NAME> \
  --project <PROJECT_ID> \
  --catalog-type biglake \
  --default-location gs://<PRIMARY_BUCKET>[/PATH] \
  --restricted-locations gs://<OTHER_BUCKET_1>[/PATH],gs://<OTHER_BUCKET_2>[/PATH]

To create a single-bucket catalog:

gcloud beta biglake iceberg catalogs create \
  <GCS_BUCKET_NAME> \
  --project <PROJECT_ID> \
  --catalog-type gcs-bucket

The catalog name must match the GCS bucket name for single-bucket catalogs (e.g., if your bucket is gs://my-bucket, use my-bucket as the catalog name). This is a BigLake requirement for gcs-bucket type catalogs. See the Google Cloud documentation for details.

Configuration

To use the integration, make sure that a table name is provided under the name option in the table configuration section. This will be the name the table is registered under in the catalog but also the name used by Tableflow to reference the table. Then add the following biglake_table_config block to each table you would like to sync:

tables:
    - source_topic: events
      name: "<TABLE_NAME>"
      # ... other table settings ...
      biglake_table_config:
        enabled: true
        project_id: "<PROJECT_ID>"
        namespace: "<NAMESPACE>"

The following explains the configuration options:

Field	Required	Description
enabled	Yes	Set to true to enable sync for this table
project_id	Yes	The GCP project ID (used for billing/quota attribution via x-goog-user-project)
namespace	Yes	The catalog namespace where the table will be registered. Created automatically if it doesn't exist.
is_multi_bucket_catalog	No	Set to true if the catalog is a multi-bucket catalog.
catalog_id	Yes if is_multi_bucket_catalog: true	The catalog ID from the GCP console. Optional for single-bucket catalogs as those have catalog id set to the bucket name.
namespace_location	No	The path to associate with the namespace. Only tables in locations allowed by this path can be registered with the given namespace. For multi-bucket catalogs, this must be a location allowed by the catalog (determined by default_location or restricted_locations). If absent, the table's bucket URL is used.

Tableflow automatically creates the namespace if it doesn't exist. Namespace locations are immutable and can be used to introduce storage boundaries for tables. For example, if a namespace is set up with the location gs://bucket/prod/... , then tables under the path gs://bucket/staging/... cannot be regsitered with that namespace.

Query the Data

Once enabled, Tableflow will automatically register the table in Lakehouse and keep the metadata location up to date.

From BigQuery

Tables are queryable from BigQuery using the 4-part Project.Catalog.Namespace.Table syntax:

SELECT * FROM `<PROJECT_ID>.<CATALOG_ID>.<NAMESPACE>.<TABLE_NAME>` LIMIT 100;

For example, if your project is my-project, your catalog is my-catalog, and you configured namespace my_ns with table my_table:

SELECT * FROM `my-project.my-catalog.my_ns.my_table` LIMIT 100;

From Other Query Engines

Any query engine that supports the Iceberg REST Catalog protocol (e.g. Spark, Trino, Presto) can connect directly to the BigLake Metastore REST Catalog endpoint and query the tables.