> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thanx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Apache Iceberg

> Configuring your Apache Iceberg destination.

<Warning>
  **Beta destination**

  Apache Iceberg is currently a beta destination with an upper limit of 1 billion rows per month. Contact your Thanx representative if you have questions about larger volumes.
</Warning>

## Choosing a catalog

Apache Iceberg requires a central catalog to manage table metadata and provide atomic transactions. The data-syncing service supports several catalog options, each with its own setup steps below:

* AWS Glue Catalog
* AWS S3 Tables Catalog
* Iceberg REST Catalog (including R2 Data Catalog and Tabular)
* Google Lakehouse Catalog

## Setting up with AWS Glue Catalog

<Note>
  **How this works**

  * The Glue catalog stores Iceberg table metadata and the pointer to each table's location.
  * The destination S3 bucket stores your Iceberg data and metadata files, and is used during staging.
</Note>

### Prerequisites

* [ ] By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with the data-syncing service's identifier to grant access. It should look similar to the following JSON object with a proper service account identifier:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRoleWithWebIdentity"
      ],
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Condition": {
        "StringEquals": {
          "accounts.google.com:oaud": "<some_organization_identifier>",
          "accounts.google.com:sub": "<some_service_account_identifier>"
        }
      }
    }
  ]
}
```

### Step 1: Set up destination S3 bucket

1. Navigate to the **S3** service page.
2. Click **Create bucket**.
3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**.
4. Click **Create bucket**.

### Step 2: Create policy and IAM role

1. Navigate to the **IAM** service page.
2. Navigate to the **Policies** tab, and click **Create policy**.
3. Click the **JSON** tab, and paste the following policy, being sure to replace `BUCKET_NAME`, `ACCOUNT_ID`, and `DATABASE` with your specific values.

<Note>
  **Why these permissions are necessary**

  * The Glue permissions manage catalog metadata and handle table operations, including cleaning up temporary tables during syncs.
  * The S3 permissions upload data files, list bucket contents, read Iceberg metadata, and manage files during compaction.
  * `glue:CreateDatabase` is only required if the database does not yet exist. If you use an existing Glue database, you can remove this action and provide the name of your pre-existing database instead.
</Note>

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowGlueAccessToDestinationDatabaseAndTables",
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabases",
                "glue:GetDatabase",
                "glue:GetTables",
                "glue:GetTable",
                "glue:GetPartitions",
                "glue:CreateTable",
                "glue:CreateDatabase",
                "glue:UpdateTable",
                "glue:DeleteTable"
            ],
            "Resource": [
                "arn:aws:glue:*:ACCOUNT_ID:catalog",
                "arn:aws:glue:*:ACCOUNT_ID:database/DATABASE",
                "arn:aws:glue:*:ACCOUNT_ID:database/default",
                "arn:aws:glue:*:ACCOUNT_ID:table/DATABASE/*"
            ]
        },
        {
            "Sid": "AllowS3AccessToBucket",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME",
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}
```

<Note>
  **KMS encryption (optional)**

  If your S3 bucket uses KMS encryption (CMK), add the following statement to the policy's `Statement` array to allow data encryption and decryption with your KMS key. SSE-C encryption is not currently supported.

  ```json theme={null}
  {
    "Effect": "Allow",
    "Action": [
      "kms:GenerateDataKey",
      "kms:Decrypt"
    ],
    "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
  }
  ```

  Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values.
</Note>

4. Click **Next: Tags**, click **Next: Review**.
5. Name the policy, add a description, and click **Create policy**.
6. Navigate to the **IAM** service page, then the **Roles** tab, and click **Create role**.
7. Select **Custom trust policy** and paste the provided trust policy to allow AssumeRole access to the new role. Click **Next**.
8. Add the permissions policy created above, and click **Next**.
9. Enter a **Role name**, for example `transfer-role`, and click **Create role**.
10. Once created, search for the role in the Roles list, click the role name, and make a note of the **ARN** value.

### Step 3: Add your destination

Securely share your **bucket name**, **bucket region**, **role ARN**, and **Glue database name** with us to complete the connection.

## Setting up with AWS S3 Tables Catalog

<Note>
  **How this works**

  * The S3 Tables bucket stores your Iceberg data and metadata.
  * A separate staging S3 bucket is required for staging data.
</Note>

### Prerequisites

* [ ] S3 Tables authentication uses role-based access. You will need the trust policy prepopulated with the data-syncing service's identifier to grant access.
* [ ] The IAM role must also have a trust relationship with itself to function correctly with the S3 Tables API. Your final trust policy should include two principals: the data-syncing service and the role itself. Be sure to replace `YOUR_ACCOUNT_ID` and `YOUR_ROLE_NAME` with the appropriate identifiers.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME" },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRoleWithWebIdentity"],
      "Principal": { "Federated": "accounts.google.com" },
      "Condition": {
        "StringEquals": {
          "accounts.google.com:oaud": "<some_oaud_identifier>",
          "accounts.google.com:sub": "<some_service_account_identifier>"
        }
      }
    }
  ]
}
```

### Step 1: Set up S3 Tables bucket

1. Navigate to the **S3** service page.
2. In the left navigation, click **Table buckets**.
3. Click **Create bucket**.
4. Enter a **Bucket name** and choose the same **AWS Region** you plan to use for your destination S3 bucket. This bucket is used as your **S3 Tables bucket**.
5. Click **Create bucket**.

### Step 2: Set up staging S3 bucket

1. Navigate to the **S3** service page.
2. Click **Create bucket**.
3. Enter a **Bucket name** and modify any of the default settings as desired. Note: **Object Ownership** can be set to "ACLs disabled" and **Block Public Access settings for this bucket** can be set to "Block all public access" as recommended by AWS. Make note of the **Bucket name** and **AWS Region**.
4. Click **Create bucket**.

### Step 3: Create policy and IAM role

1. Navigate to the **IAM** service page.
2. Navigate to the **Policies** tab, and click **Create policy**.
3. Click the **JSON** tab, and paste the following policy, replacing `ACCOUNT_ID`, `REGION`, `S3_TABLES_BUCKET_NAME`, and `S3_STAGING_BUCKET_NAME` with the appropriate values.

<Note>
  **Why these permissions are necessary**

  * The S3 Table data permissions read and write Iceberg data files and manage metadata locations in your S3 Tables bucket.
  * The S3 Table management permissions create and manage tables and namespaces (including cleaning up temporary tables during syncs) in your S3 Tables bucket.
  * The S3 permissions write data files to your staging S3 bucket, list bucket contents, and clean up staged or test files.
  * The namespace permissions (`s3tables:CreateNamespace`, etc.) are only required if the namespace does not already exist. If you use an existing namespace, you can remove these actions.
</Note>

```json theme={null}
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowS3TableDataActions",
            "Effect": "Allow",
            "Action": [
                "s3tables:GetTable",
                "s3tables:DeleteTable",
                "s3tables:GetTableData",
                "s3tables:PutTableData",
                "s3tables:GetTableMetadataLocation",
                "s3tables:UpdateTableMetadataLocation"
            ],
            "Resource": "arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/S3_TABLES_BUCKET_NAME/table/*"
        },
        {
            "Sid": "AllowS3TableManagementAndNamespaceActions",
            "Effect": "Allow",
            "Action": [
                "s3tables:GetTableBucket",
                "s3tables:CreateTable",
                "s3tables:ListTables",
                "s3tables:CreateNamespace",
                "s3tables:GetNamespace",
                "s3tables:ListNamespaces",
                "s3tables:DeleteNamespace"
            ],
            "Resource": "arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/S3_TABLES_BUCKET_NAME"
        },
        {
            "Sid": "AllowS3AccessToDestinationBucket",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::S3_STAGING_BUCKET_NAME",
                "arn:aws:s3:::S3_STAGING_BUCKET_NAME/*"
            ]
        }
    ]
}
```

<Note>
  **KMS encryption (optional)**

  If your S3 staging bucket uses KMS encryption (CMK), add the following statement to the policy's `Statement` array to allow data encryption and decryption with your KMS key. SSE-C encryption is not currently supported.

  ```json theme={null}
  {
    "Effect": "Allow",
    "Action": [
      "kms:GenerateDataKey",
      "kms:Decrypt"
    ],
    "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
  }
  ```

  Replace `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID` with your values.
</Note>

4. Click **Next: Tags**, click **Next: Review**.
5. Name the policy, and click **Create policy**.
6. Navigate to the **IAM** service page, then the **Roles** tab, and click **Create role**.
7. Select **Custom trust policy**. Leave the default placeholder trust policy as-is for now (do not paste the final trust policy yet), and click **Next**. The policy must be self-assuming, which is not allowed until the role is created, so it must be updated after the role is created.
8. Add the permissions policy created above, and click **Next**.
9. Enter a **Role name** and click **Create role**.
10. Once created, search for the role in the Roles list and click the role name.
11. In the role detail view, navigate to the **Trust relationships** tab, click **Edit trust policy**, and replace the default trust policy with the trust policy JSON from the **Prerequisites** section above. Click **Update policy** to save.

<Note>
  **AWS IAM propagation delay**

  After updating the trust policy, AWS IAM changes can take 5-10 minutes or longer to propagate. Wait for propagation to complete before testing the connection.
</Note>

### Step 4: Add your destination

Securely share your **S3 Tables bucket ARN**, **destination S3 bucket name**, **destination S3 bucket region**, **role ARN**, and chosen **namespace** with us to complete the connection.

## Setting up with Iceberg REST Catalog

The Iceberg REST catalog is an open standard for interacting with an Iceberg catalog over HTTP. Below are instructions for three implementations.

<Note>
  **Note on credentials**

  The data-syncing service connects to REST catalogs as a standard client using the credentials you provide. Credential vendoring (issuing temporary credentials) is not supported for downstream access.
</Note>

### R2 Data Catalog (Cloudflare)

<Note>
  **Tip: zero egress fees**

  Cloudflare R2 charges no egress fees, making it a cost-effective option if you plan to query your Iceberg data from external locations or other cloud providers.
</Note>

1. Log in to your Cloudflare dashboard.
2. Follow the [Cloudflare documentation to create an R2 bucket](https://developers.cloudflare.com/r2/buckets/create-buckets/). Make a note of the **Bucket Name** and your **R2 Account ID**.
3. Follow the [Cloudflare documentation to create an R2 API token](https://developers.cloudflare.com/r2/api/tokens/) with **Admin Read & Write** permissions. Make a note of the generated **Access Key ID** and **Secret Access Key**.
4. Securely share the following with us to complete the connection:
   * **Catalog URI**: `https://api.cloudflare.com/client/v4/accounts/YOUR_R2_ACCOUNT_ID/r2/catalog`
   * **API Token** (as the credential)
   * **Bucket Name** and **Region**
   * **R2 Access Key ID** and **R2 Secret Access Key**

<Note>
  **R2 catalog path requirement**

  If you customize the folder or path used for the R2 Data Catalog, it must start with `__r2_data_catalog`. The R2 API does not validate this upfront, so an incorrect prefix results in runtime failures when creating or querying tables.
</Note>

### Google BigLake (Lakehouse Catalog)

<Note>
  **How this works**

  * The Google Lakehouse catalog stores Iceberg table metadata.
  * The destination GCS bucket stores your Iceberg data and metadata files, and is used during staging.
  * With Google BigLake, the Iceberg tables become queryable directly from BigQuery. No external table definition or separate mount step is required.
</Note>

**Prerequisites**

* [ ] You will need a Google Cloud service account with permissions to read/write to your GCS bucket, and to manage your Lakehouse catalog.
* [ ] By default, authentication uses role-based access via service account impersonation. You will need the data-syncing service's service account name available to grant access. It should look like `some-name@some-project.iam.gserviceaccount.com`.

1. Navigate to the **Cloud Storage** service page and click **Create bucket**. Provide a name and choose the appropriate region. Make note of the **Bucket name**.
2. Create a Lakehouse catalog in your Google Cloud project by following the [Lakehouse catalog documentation](https://docs.cloud.google.com/lakehouse/docs/lakehouse-iceberg-rest-catalog).

<Warning>
  **Use end-user credentials for the catalog**

  When creating the catalog, configure it to use end-user credentials. Do not use credential vending, since the data-syncing service needs to access the underlying storage directly with the provided service account credentials.
</Warning>

3. Grant your service account the necessary permissions, then allow the data-syncing service to impersonate it:

   **Option A: Preconfigured GCP roles**

   * **Storage Object Admin** (`roles/storage.objectAdmin`) on the destination bucket.
   * **BigLake Editor** (`roles/biglake.editor`) on the target project/catalog, to allow creation and management of Iceberg tables via BigLake.

   **Option B: Fine-tuned custom permissions**

   Cloud Storage permissions (applied to the destination bucket): `storage.buckets.get`, `storage.objects.create`, `storage.objects.delete`, `storage.objects.get`, `storage.objects.list`, `storage.objects.update`.

   BigLake permissions (applied to the target project or catalog): `biglake.catalogs.get`, `biglake.catalogs.list`, `biglake.databases.create`, `biglake.databases.delete`, `biglake.databases.get`, `biglake.databases.list`, `biglake.databases.update`, `biglake.tables.create`, `biglake.tables.delete`, `biglake.tables.get`, `biglake.tables.list`, `biglake.tables.update`.

4. Navigate to **IAM & Admin** > **Service Accounts**, select the service account you created, select the **Permissions** tab, click **Grant Access**, enter the data-syncing service's service account name (from the prerequisites), and select the **Service Account Token Creator** role.

5. Securely share your **Google Cloud Project ID**, **Catalog Name**, **Schema**, **Bucket Name**, and service account credentials with us to complete the connection.

### Tabular

1. Log in to your Tabular organization's dashboard.
2. Navigate to the credentials section and create a new credential with permissions to create tables and write data.
3. Make a note of the generated **Client ID** and **Client Secret**.
4. Securely share the following with us to complete the connection:
   * **Catalog URI**: `https://api.tabular.io/ws`
   * **Client ID**
   * **Client Secret**

## Understanding Iceberg configuration options

<Warning>
  Changing these attributes on an existing destination table does not take effect until you perform a full refresh of the table.
</Warning>

### Managing staged data

During each transfer, batches are first written to a staging prefix in your object storage bucket before they are committed into the final Iceberg table. This prefix is always named `_write_ahead_staging` (for example: `your_folder/_write_ahead_staging/<table_name>/<transfer_id>`, or `_write_ahead_staging/<table_name>/<transfer_id>` if no folder/schema is configured).

We recommend configuring an object storage lifecycle policy to automatically delete objects under the `_write_ahead_staging` prefix after 30 days. This provides a safety net for any orphaned staged files that are not cleaned up due to failed or interrupted runs.

### `retention_window_days`

Sets the number of days for which historical data (previous table snapshots used for time travel or auditing) is retained. Set this value according to your organization's internal data retention policies.

## FAQ

<AccordionGroup>
  <Accordion title="How is data transferred into my Iceberg tables?">
    Batch files are first staged into an object storage bucket, then your chosen catalog atomically commits them into the final Iceberg table. For Glue and REST catalogs, the same S3 bucket is used for both staging and permanent table data, with different prefixes. For S3 Tables, batches are staged into your staging S3 bucket, then the finalized Iceberg data and metadata is written to the managed S3 Tables bucket.
  </Accordion>

  <Accordion title="Should I use AWS Glue or AWS S3 Tables?">
    There are tradeoffs to consider:

    * **Glue Catalog**: Glue stores the table metadata, and your S3 bucket stores both staged files and the final Iceberg data under different prefixes. Snapshot expiry and compaction are handled for you, and you control the S3 layout. This is a good fit if you already use Glue as your central catalog or want to keep data in a single S3 bucket you manage directly.
    * **S3 Tables Catalog**: The S3 Tables bucket is a fully managed table bucket where AWS stores the finalized Iceberg data and metadata. Batches are written to a separate staging S3 bucket, and the catalog writes the final data into the S3 Tables bucket and handles maintenance on your behalf. This is a good fit if you prefer automatic maintenance and plan to query through engines that natively support S3 Tables.
  </Accordion>

  <Accordion title="What is Apache Iceberg and why should I use it?">
    Apache Iceberg is an open table format designed for analytic datasets on object stores. It delivers warehouse-native capabilities such as ACID transactions, time travel, and schema evolution with the simplicity, scalability, and secure permissions model of an object storage bucket. A central catalog provides reliable transactions and enables multiple engines to work concurrently on the same data, so your warehouse stays isolated from data sharing and you can receive data without exposing your internal resources.
  </Accordion>

  <Accordion title="Why do you need permissions to delete data?">
    Iceberg performs background maintenance operations to manage the table's health and performance, including expiring old snapshots and compacting small data files. The writer must have delete permissions to safely remove obsolete files without compromising data integrity.
  </Accordion>

  <Accordion title="Can I send the data to a specific prefix in a bucket?">
    Yes, you can direct data to a specific prefix (warehouse path). We recommend using a completely isolated bucket to receive data, to minimize security risks and reduce the chance of accidental interference with other datasets.
  </Accordion>

  <Accordion title="Do I need to perform maintenance operations on the Iceberg table?">
    No. Snapshot expiry and data compaction are handled for you. Data consumers should not run any non-read queries on the table, except for managed catalogs like R2 and S3 Tables, which automatically run compaction and snapshot expiration.

    <Warning>
      Treat the destination tables as read-only. Executing write or delete operations manually may corrupt the table state and break data synchronization.
    </Warning>
  </Accordion>

  <Accordion title="How do I know when a table has been updated?">
    You can query the table's metadata to see the history of snapshots. Each snapshot represents a version of the table. For example, in Spark SQL:

    ```sql theme={null}
    SELECT snapshot_id, committed_at FROM my_glue_catalog.my_db.my_table.snapshots ORDER BY committed_at DESC LIMIT 1;
    ```

    Most bucket providers also offer the ability to trigger a webhook or lambda when objects are created, which can be configured to monitor the table's metadata directory for new manifest lists.

    A `version-hint.txt` file is written to the metadata directory. This allows tools like PyIceberg and DuckDB to read the table directly from the bucket without needing to connect to the catalog service, by pointing them to the table's root location.
  </Accordion>

  <Accordion title="Are there any limitations on data sizes?">
    There are no enforced size limits, including for JSON fields. Downstream query engines may have their own limits (for example, Amazon Redshift's `SUPER` and `VARCHAR` sizes). Ensure your data fits within your query engine's constraints to avoid query failures.
  </Accordion>

  <Accordion title="Why are two service accounts involved with Google BigLake? Why is service account impersonation required?">
    You create one service account in your project with BigLake/Storage permissions, and the data-syncing service uses its own service account to impersonate yours. This means your private keys are never handled by us, all operations appear in your audit logs, access is via short-lived tokens, and you can revoke access at any time through your own IAM permissions. Direct service account access is not supported.
  </Accordion>

  <Accordion title="Can I mount my Iceberg data to BigQuery?">
    Yes. Set up the destination with the Google Lakehouse Catalog described above: tables written through Google BigLake are queryable directly from BigQuery, with no external table definition or separate mount step required. Mounting is not currently supported for the other catalog options.
  </Accordion>
</AccordionGroup>

## Mounting/reading an Iceberg table

You can mount or read an Iceberg table into your data warehouse of choice. Below are the supported catalog types and links to the corresponding vendor documentation:

* **ClickHouse**: [Glue catalog](https://clickhouse.com/docs/use-cases/data-lake/glue-catalog), [S3 Tables catalog](https://clickhouse.com/docs/engines/table-engines/integrations/iceberg), [REST catalog](https://clickhouse.com/docs/use-cases/data-lake/rest-catalog)
* **DuckDB / MotherDuck**: [Glue catalog](https://duckdb.org/docs/stable/core_extensions/iceberg/amazon_sagemaker_lakehouse), [S3 Tables catalog](https://duckdb.org/2025/03/14/preview-amazon-s3-tables), [REST catalog](https://duckdb.org/docs/stable/core_extensions/iceberg/iceberg_rest_catalogs)
* **Spark**: [Glue catalog](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html#aws-glue-programming-etl-format-iceberg-read-spark), [S3 Tables catalog](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html), [REST catalog](https://iceberg.apache.org/docs/latest/spark-configuration/#rest-catalog)
* **Athena / Redshift**: [Glue catalog](https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg.html), [S3 Tables catalog](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-open-source.html)
* **Snowflake**: [REST catalog](https://docs.snowflake.com/en/user-guide/tables-iceberg-externally-managed-writes)