Skip to main content

​
Prerequisites

  • Have access to your SFTP server to create a dedicated user and configure SSH key authentication.
  • We will provide a public key for you to authorize; it will look similar to <key-type> <ssh_public_key_beginning_with_AAAA> some-comment, where <key-type> is ssh-rsa, ssh-ed25519, or similar.

​
Step 1: Create an SFTP user

Create a dedicated user on your SFTP server for the data-syncing service, and authorize its public key.
  1. Create a group and user for the data transfer.
  1. Switch to the new user and set up the .ssh directory.
  1. Create the authorized_keys file and add the public key we provide.
Write permissions at the SFTP root are requiredThe data-syncing service writes a manifest file per model under a _manifests/ directory at the SFTP home root, in addition to your chosen destination folder. Grant the user write access to both locations.

​
Step 2: Add your destination

Securely share your hostname, port (usually 22), your chosen folder name, username, and delimiter character with us to complete the connection.
Optional: PGP encryptionFor an additional layer of encryption, you may provide a PGP public key (ASCII-armored, RSA 2,048-bit or larger, or ECC). Files written to your SFTP server are encrypted with this key before delivery.

​
Permissions checklist

  • The SFTP user authenticates with the SSH key we provide, added to ~/.ssh/authorized_keys.
  • The user has write access to your chosen destination folder.
  • The user has write access to a _manifests/ directory at the SFTP root.
  • Firewall rules allow egress on port 22 (or your configured port) from the data-syncing service.
  • If PGP encryption is enabled, a valid PGP public key is ready to share.

​
FAQ

Data lands in Hive-style partitions per model: <folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<ext>.
We authenticate using SSH key-based access with a dedicated, least-privileged user scoped to the destination folder.
Parquet (default and recommended), CSV, and JSON/JSONL.
Each transfer writes a manifest file per model under _manifests/, in the format _manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json.
Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys, prioritizing the most recent transfer window; manifest files can help bound the set of files to read.