Lakehouse Runtime Catalog (BigLake)
This page describes how to integrate with Lakehouse runtime catalog (formerly BigLake Metastore) so that you can query Iceberg tables created by Tableflow directly in BigQuery via Lakehouse.
Integrating with Lakehouse runtime catalog is the recommended way to make Tableflow Iceberg tables queryable in BigQuery. This approach uses the REST Catalog API to handle table registration, giving you native Iceberg support and keeping your catalog automatically in sync as new snapshots are taken.
BigLake vs Lakehouse runtime catalog integration: Lakehouse runtime catalog is preferred over the BigQuery integration because the former exposes the standard Iceberg REST Catalog protocol. This means the tables are automatically queryable from BigQuery and from any engine that can connect to an Iceberg REST Catalog, such as Spark, Trino, and Presto. The legacy BigQuery integration only creates BigQuery external tables, so the tables are only visible to BigQuery.
Prerequisites
In order for this to work, please upgrade your WarpStream Agents to at least v769 (v828+ for multi-bucket catalogs). Additionally, the integration uses Google Application Default Credentials (ADC). The agent authenticates to both the BigLake REST Catalog API and GCS using the service account it runs as, and as such the service account requires the following role:
roles/biglake.admin
Create and update tables in the BigLake Metastore
Or if you prefer more granular permissions, you can create a custom role with the following permissions:
biglake.catalogs.getbiglake.namespaces.createbiglake.namespaces.getbiglake.tables.deletebiglake.tables.getbiglake.tables.register
To add the role to your service account, run
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
--role="roles/biglake.admin"Create a Catalog
Lakehouse runtime catalog supports two types of catalogs, the multi-bucket catalog and the single-bucket catalog. Google recommends using the multi-bucket catalog as that allows for a custom catalog name and for multiple buckets to be configured for a single catalog.
To create a multi-bucket catalog:
To create a single-bucket catalog:
Configuration
To use the integration, make sure that a table name is provided under the name option in the table configuration section. This will be the name the table is registered under in the catalog but also the name used by Tableflow to reference the table. Then add the following biglake_table_config block to each table you would like to sync:
The following explains the configuration options:
enabled
Yes
Set to true to enable sync for this table
project_id
Yes
The GCP project ID (used for billing/quota attribution via x-goog-user-project)
namespace
Yes
The catalog namespace where the table will be registered. Created automatically if it doesn't exist.
is_multi_bucket_catalog
No
Set to true if the catalog is a multi-bucket catalog.
catalog_id
Yes if is_multi_bucket_catalog: true
The catalog ID from the GCP console. Optional for single-bucket catalogs as those have catalog id set to the bucket name.
namespace_location
No
The path to associate with the namespace. Only tables in locations allowed by this path can be registered with the given namespace. For multi-bucket catalogs, this must be a location allowed by the catalog (determined by default_location or restricted_locations). If absent, the table's bucket URL is used.
Query the Data
Once enabled, Tableflow will automatically register the table in Lakehouse and keep the metadata location up to date.
From BigQuery
Tables are queryable from BigQuery using the 4-part Project.Catalog.Namespace.Table syntax:
For example, if your project is my-project, your catalog is my-catalog, and you configured namespace my_ns with table my_table:
From Other Query Engines
Any query engine that supports the Iceberg REST Catalog protocol (e.g. Spark, Trino, Presto) can connect directly to the BigLake Metastore REST Catalog endpoint and query the tables.
Last updated
Was this helpful?