> ## Documentation Index
> Fetch the complete documentation index at: https://docs.thanx.com/llms.txt
> Use this file to discover all available pages before exploring further.

# SFTP

> Configuring your SFTP destination.

## Prerequisites

* [ ] Have access to your SFTP server to create a dedicated user and configure SSH key authentication.
* [ ] We will provide a public key for you to authorize; it will look similar to `<key-type> <ssh_public_key_beginning_with_AAAA> some-comment`, where `<key-type>` is `ssh-rsa`, `ssh-ed25519`, or similar.

## Step 1: Create an SFTP user

Create a dedicated user on your SFTP server for the data-syncing service, and authorize its public key.

1. Create a group and user for the data transfer.

```bash theme={null}
sudo groupadd sftpwriter
sudo useradd -m -g sftpwriter sftpwriter
```

2. Switch to the new user and set up the `.ssh` directory.

```bash theme={null}
sudo su - sftpwriter
mkdir ~/.ssh
chmod 700 ~/.ssh
cd ~/.ssh
```

3. Create the `authorized_keys` file and add the public key we provide.

```bash theme={null}
touch authorized_keys
chmod 600 authorized_keys
echo "<key-type> <ssh_public_key_beginning_with_AAAA> sftpwriter-public-key" > authorized_keys
```

<Warning>
  **Write permissions at the SFTP root are required**

  The data-syncing service writes a manifest file per model under a `_manifests/` directory at the SFTP home root, in addition to your chosen destination folder. Grant the user write access to both locations.
</Warning>

## Step 2: Add your destination

Securely share your **hostname**, **port** (usually `22`), your chosen **folder name**, **username**, and **delimiter character** with us to complete the connection.

<Note>
  **Optional: PGP encryption**

  For an additional layer of encryption, you may provide a PGP public key (ASCII-armored, RSA 2,048-bit or larger, or ECC). Files written to your SFTP server are encrypted with this key before delivery.
</Note>

## Permissions checklist

* The SFTP user authenticates with the SSH key we provide, added to `~/.ssh/authorized_keys`.
* The user has write access to your chosen destination folder.
* The user has write access to a `_manifests/` directory at the SFTP root.
* Firewall rules allow egress on port `22` (or your configured port) from the data-syncing service.
* If PGP encryption is enabled, a valid PGP public key is ready to share.

## FAQ

<AccordionGroup>
  <Accordion title="How is data organized on the server?">
    Data lands in Hive-style partitions per model: `<folder>/<model_name>/dt=<transfer_date>/<file_part>_<transfer_timestamp>.<ext>`.
  </Accordion>

  <Accordion title="How is the connection secured?">
    We authenticate using SSH key-based access with a dedicated, least-privileged user scoped to the destination folder.
  </Accordion>

  <Accordion title="What file formats are supported?">
    Parquet (default and recommended), CSV, and JSON/JSONL.
  </Accordion>

  <Accordion title="How do I know when a transfer completed?">
    Each transfer writes a manifest file per model under `_manifests/`, in the format `_manifests/<model_name>/dt=<transfer_date>/manifest_{transfer_id}.json`.
  </Accordion>

  <Accordion title="Why do I sometimes see duplicates?">
    Object storage is append-only. The change detection process uses a lookback window to ensure no data is missed, which can create duplicates. Downstream pipelines should deduplicate on primary keys, prioritizing the most recent transfer window; manifest files can help bound the set of files to read.
  </Accordion>
</AccordionGroup>