Step 1: Create an HMAC Access ID and Secret
Consult your object storage platform’s documentation to learn how to generate an HMAC Access ID and Secret.Step 2: Add your destination
Securely share your bucket host, bucket name, chosen folder name, HMAC access ID, and HMAC secret with us to complete the connection.Permissions checklist
- The HMAC access ID and secret have write permissions to the target bucket.
- The bucket is accessible from the data-syncing service’s egress IP.
FAQ
How is the S3 Compatible connection secured?
How is the S3 Compatible connection secured?
We authenticate using the HMAC access ID and secret you provide. These credentials are used to sign requests to the S3-compatible API. Scope the credentials to only the required bucket to limit access.
What file formats are supported?
What file formats are supported?
Parquet (default and recommended), CSV, JSON, and JSONL.
How are large datasets written?
How are large datasets written?
Files are automatically split; multiple files may be written per model per transfer.
How is the data organized in the bucket?
How is the data organized in the bucket?
Data lands in Hive-style partitions per model:
<folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<ext>. Manifest files are written to a _manifests/ directory at the root of the configured folder path.How do I update my HMAC credentials?
How do I update my HMAC credentials?
Generate new HMAC credentials from your object storage platform and update them in the destination configuration.