> For the complete documentation index, see [llms.txt](https://docs.warpstream.com/warpstream/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.warpstream.com/warpstream/tableflow/catalogs-and-query-engines/biglake.md). # Lakehouse Runtime Catalog (BigLake) This page describes how to integrate with Lakehouse runtime catalog (formerly BigLake Metastore) so that you can query Iceberg tables created by Tableflow directly in BigQuery via Lakehouse. Integrating with Lakehouse runtime catalog is the recommended way to make Tableflow Iceberg tables queryable in BigQuery. This approach uses the [REST Catalog](https://docs.cloud.google.com/lakehouse/docs/set-up-lakehouse-iceberg-rest-catalog) API to handle table registration, giving you native Iceberg support and keeping your catalog automatically in sync as new snapshots are taken. {% hint style="success" %} **BigLake vs Lakehouse runtime catalog integration:** Lakehouse runtime catalog is preferred over the [BigQuery integration](/warpstream/tableflow/catalogs-and-query-engines/bigquery.md) because the former exposes the standard Iceberg REST Catalog protocol. This means the tables are automatically queryable from BigQuery *and* from any engine that can connect to an Iceberg REST Catalog, such as Spark, Trino, and Presto. The legacy BigQuery integration only creates BigQuery external tables, so the tables are only visible to BigQuery. {% endhint %} ## Prerequisites In order for this to work, please upgrade your WarpStream Agents to at least **v769** (**v828+** for multi-bucket catalogs). Additionally, the integration uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials). The agent authenticates to both the BigLake REST Catalog API and GCS using the service account it runs as, and as such the service account requires the following role: | Role | Purpose | | --------------------- | ------------------------------------------------- | | `roles/biglake.admin` | Create and update tables in the BigLake Metastore | Or if you prefer more granular permissions, you can create a custom role with the following permissions: * `biglake.catalogs.get` * `biglake.namespaces.create` * `biglake.namespaces.get` * `biglake.tables.delete` * `biglake.tables.get` * `biglake.tables.register` To add the role to your service account, run ```bash gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \ --role="roles/biglake.admin" ``` {% hint style="info" %} The agent already has GCS access for reading and writing Iceberg data. The only additional permission needed for the Lakehouse integration is `roles/biglake.admin`. {% endhint %} ## Create a Catalog Lakehouse runtime catalog supports two types of catalogs, the multi-bucket catalog and the single-bucket catalog. Google [recommends](https://docs.cloud.google.com/lakehouse/docs/about-iceberg-rest-catalog-endpoint#storage-multiple-buckets) using the multi-bucket catalog as that allows for a custom catalog name and for multiple buckets to be configured for a single catalog. To create a multi-bucket catalog: ``` gcloud biglake iceberg catalogs create \ \ --project \ --catalog-type biglake \ --default-location gs://[/PATH] \ --restricted-locations gs://[/PATH],gs://[/PATH] ``` To create a single-bucket catalog: ```bash gcloud beta biglake iceberg catalogs create \ \ --project \ --catalog-type gcs-bucket ``` {% hint style="info" %} The catalog name must match the GCS bucket name for single-bucket catalogs (e.g., if your bucket is `gs://my-bucket`, use `my-bucket` as the catalog name). This is a BigLake requirement for `gcs-bucket` type catalogs. See the [Google Cloud documentation](https://docs.cloud.google.com/biglake/docs/blms-rest-catalog#create_a_catalog) for details. {% endhint %} ## Configuration To use the integration, make sure that a table name is provided under the `name` option in the table configuration section. This will be the name the table is registered under in the catalog but also the name used by Tableflow to reference the table. Then add the following `biglake_table_config` block to each table you would like to sync: ```yaml tables: - source_topic: events name: "" # ... other table settings ... biglake_table_config: enabled: true project_id: "" namespace: "" ``` The following explains the configuration options: | Field | Required | Description | | -------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | Yes | Set to `true` to enable sync for this table | | `project_id` | Yes | The GCP project ID (used for billing/quota attribution via `x-goog-user-project`) | | `namespace` | Yes | The catalog namespace where the table will be registered. Created automatically if it doesn't exist. | | is\_multi\_bucket\_catalog | No | Set to `true` if the catalog is a multi-bucket catalog. | | catalog\_id | Yes if `is_multi_bucket_catalog: true` | The catalog ID from the GCP console. Optional for single-bucket catalogs as those have catalog id set to the bucket name. | | namespace\_location | No | The path to associate with the namespace. Only tables in locations allowed by this path can be registered with the given namespace. For multi-bucket catalogs, this must be a location allowed by the catalog (determined by `default_location` or `restricted_locations`). If absent, the table's bucket URL is used. | {% hint style="info" %} Tableflow automatically creates the namespace if it doesn't exist. Namespace locations are immutable and can be used to introduce storage boundaries for tables. For example, if a namespace is set up with the location `gs://bucket/prod/...` , then tables under the path `gs://bucket/staging/...` cannot be regsitered with that namespace. {% endhint %} ## Query the Data Once enabled, Tableflow will automatically register the table in Lakehouse and keep the metadata location up to date. ### From BigQuery Tables are queryable from BigQuery using the 4-part `Project.Catalog.Namespace.Table` syntax: ```sql SELECT * FROM `...` LIMIT 100; ``` For example, if your project is `my-project`, your catalog is `my-catalog`, and you configured namespace `my_ns` with table `my_table`: ```sql SELECT * FROM `my-project.my-catalog.my_ns.my_table` LIMIT 100; ``` ### From Other Query Engines Any query engine that supports the Iceberg REST Catalog protocol (e.g. Spark, Trino, Presto) can connect directly to the BigLake Metastore REST Catalog endpoint and query the tables. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.warpstream.com/warpstream/tableflow/catalogs-and-query-engines/biglake.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.