Delta Lake

​

Choosing a cloud provider

• AWS S3
• Google Cloud Storage
• Azure Blob Storage (with hierarchical namespace / Data Lake Storage Gen2)

​

Setting up with AWS S3

​

Prerequisites

• By default, S3 authentication uses role-based access. You will need the trust policy prepopulated with the data-syncing service’s identifier to grant access. It should look similar to the following JSON object with a proper service account identifier:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRoleWithWebIdentity"
      ],
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": "<some_service_account_identifier>"
        }
      }
    }
  ]
}

​

Step 1: Set up destination S3 bucket

1. Navigate to the S3 service page.
2. Click Create bucket.
3. Enter a Bucket name and modify any of the default settings as desired. Note: Object Ownership can be set to “ACLs disabled” and Block Public Access settings for this bucket can be set to “Block all public access” as recommended by AWS. Make note of the Bucket name and AWS Region.
4. Click Create bucket.

​

Step 2: Create policy and IAM role

1. Navigate to the IAM service page.
2. Navigate to the Policies tab, and click Create policy.
3. Click the JSON tab, and paste the following policy, being sure to replace BUCKET_NAME with the name of the bucket chosen in Step 1. Unlike an append-only object storage destination, Delta Lake requires read and list access in addition to write and delete, since the writer must read existing table metadata to perform transactional commits.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

4. Click Next: Tags, click Next: Review.
5. Name the policy, add a description, and click Create policy.
6. Navigate to the IAM service page, then the Roles tab, and click Create role.
7. Select Custom trust policy and paste the provided trust policy to allow AssumeRole access to the new role. Click Next.
8. Add the permissions policy created above, and click Next.
9. Enter a Role name, for example transfer-role, and click Create role.
10. Once successfully created, search for the created role in the Roles list, click the role name, and make a note of the ARN value.

​

Step 3: Add your destination

​

Setting up with Google Cloud Storage

​

Prerequisites

• By default, GCS authentication uses role-based access via service account impersonation. You will need the data-syncing service’s service account name available to grant access. It should look like some-name@some-project.iam.gserviceaccount.com.

​

Step 1: Create a service account

1. In the GCP console, navigate to the IAM & Admin menu, click into the Service Accounts tab, and click Create service account at the top of the menu.
2. In the first step, name the service account that will be used to transfer data into Cloud Storage and click Create and Continue. Click Continue in the following optional step without assigning any roles.
3. In the Grant users access to this service account step, within the Service account users role field, enter the provided Service account (see prerequisite) and click Done.
4. Once successfully created, search for the created service account in the service accounts list, click the Service account name to view the details, and make a note of the email (note: this is a different email than the service’s service account).
5. Select the permissions tab, find the provided principal name (Service account from the prerequisite), click the Edit principal button (pencil icon), click Add another role, select the Service Account Token Creator role, and click Save.

​

Step 2: Create destination GCS bucket

1. Navigate to the Cloud Storage page.
2. Click Create.
3. Enter a bucket name, choose a region. Note: at the Choose how to control access to objects step, we recommend selecting Enforce public access prevention on this bucket.

4. After choosing your preferences for the remaining steps, click Create.

5. On the Bucket details page for the bucket you created, select the Permissions tab, and click Grant access.
6. Grant access to the principal (Service Account) you created in Step 1 (this is the service account you created, not the service account from the prerequisite), and assign the roles Storage Object User and Storage Legacy Bucket Reader. Click Save. Alternatively, assign a custom role containing at minimum storage.buckets.get, storage.objects.create, storage.objects.get, storage.objects.list, and storage.objects.delete.

​

Step 3: Add your destination

​

Setting up with Azure Blob Storage

​

Step 1: Create Azure storage account

1. In the Azure portal, navigate to the Storage accounts service and click + Create.
2. In the “Basics” tab of the “Create a storage account” form, fill in the required details.
3. In the “Advanced” settings, under “Security” make sure Enable storage account key access is turned on. You may turn off (deselect) “Allow enabling public access on containers”. Under “Data Lake Storage Gen2”, select Enable hierarchical namespace.

4. In the “Networking” settings, you may limit “Network access” to either Enable public access from all networks or Enable public access from selected virtual networks and IP addresses. If the latter is selected, be sure to add the service’s static IP to the address range of the chosen virtual network.

5. In the “Data protection” settings, you must turn off Enable soft delete for blobs, Enable soft delete for containers, and Enable soft delete for file shares.

6. Once the remaining options have been configured to your preference, click Create.

​

Step 2: Create container and access key

1. In the Azure portal, navigate to the Storage accounts service and click on the account that was created in the previous step.
2. In the navigation pane, under “Data storage”, click Containers. Click + Container, choose a name for the container, and click Create.
3. In the navigation pane, under “Security + networking”, click Access keys, and make a note of the generated key under either key1 or key2.

​

Step 3: Add your destination

​

Understanding Delta Lake configuration options

​

retention_window_days

​

column_mapping_mode

​

deletion_vectors_disabled

​

change_data_feed_disabled

​

Reader compatibility

​

FAQ