SFTP files

Read files from a folder

An sftp source lists its directory every poll_interval and reads the files that match pattern. A file is read only once its size and time are unchanged between two listings, so one that is still being uploaded is left alone. Files go oldest first.

• What a file may hold: one message, messages back to back, or an HL7 batch (FHS, BHS, the messages, BTS, FTS), with CR, CRLF or LF line endings. The batch segments are dropped.
• Order: a file's messages are handled in file order, like messages arriving one by one over MLLP: stored (store-and-forward) or delivered (pass-through).
• Lines that are not HL7 are rejected on their own, not glued onto the message before them. With error_directory, they are copied there, with the reasons in <file>.txt.
• When it's done: once every message is accepted, the file is deleted, moved or renamed, as after says. A message that gets AE (the store, or in pass-through the destination, is unavailable) stops the file there; the next poll resumes from that message.
• Crash safety: how far the gateway got through each file is kept in the store after every message. A gateway that stops partway through a file resumes from the next message, without handing any on twice.

# A reference lab's result files on its SFTP server → MLLP into MEDITECH
[[route]]
name = "lab-results-from-sftp"
mode = "store_and_forward"

[route.source]
type = "sftp"
host = "sftp.lab.example.org"            # port = 22
username = "hospital"
private_key = "keys/lab-sftp"             # or private_key_env, or password = "${LAB_SFTP_PASSWORD}"
host_key = "SHA256:…"                     # or known_hosts = "keys/known_hosts"
directory = "/outbox"                     # relative to the home folder without a leading /
pattern = "*.hl7"                         # * and ? wildcards; default *
poll_interval = "30s"
after = "move:/outbox/processed"          # or "delete", or "rename:.done"
error_directory = "/outbox/errors"        # optional: messages the gateway rejected, with the reasons

[route.destination]
type = "mllp"
address = "meditech.hospital.local:6661"

Write files to a folder

An sftp destination writes each message as a file, or adds it to a batch file. Every file is written as <name>.part and renamed when complete, so the receiving system never picks up half a file.

A batch file is closed at whichever comes first of batch.max_messages (default 1000), batch.max_bytes (10 MiB) and batch.max_wait (5 minutes after its first message). Each message is appended to the file on the server before it counts as delivered, so nothing the gateway has acknowledged exists only in memory. In a cluster every node appends to the same batch file, and exactly one closes it.

# MEDITECH → MLLP → HL7 batch files on a partner's SFTP server
[[route]]
name = "orders-to-lab-sftp"
mode = "store_and_forward"
source = { type = "mllp", listen = "10.10.10.10:7895" }

[route.destination]
type = "sftp"
host = "sftp.lab.example.org"
username = "hospital"
private_key = "keys/lab-sftp"
host_key = "SHA256:…"
directory = "inbox"
batch_format = "hl7_batch"                # "single" (default), "concatenated" or "hl7_batch"
file_name = "orders_{timestamp}.hl7"      # {route} {control_id} {timestamp} {uuid}
line_ending = "crlf"                      # "cr" (default), "crlf" or "lf"
batch = { max_messages = 500, max_bytes = 10485760, max_wait = "10m" }

Signing in and the host key

Sign in with a password or a private_key (OpenSSH or PuTTY format, with passphrase if it is encrypted; or private_key_env with the key's text). Ed25519 keys are recommended. The server's host key must match host_key, its SHA256 fingerprint, or an entry in a known_hosts file: the gateway refuses to connect to any other server. Read the fingerprint with host-key, and check it with the server's administrator before you trust it:

hl7-edge-gateway host-key --to sftp.lab.example.org
host_key = "SHA256:EbKoEJqwa3Ffa/gudKmtDQSqn3kSVtKgBbYJxjPRnFU"
# or, in a known_hosts file:
sftp.lab.example.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…

Sign-in failures, a changed host key, missing folders and an unreachable server are retried with backoff, and the console shows the reason. With the JetStream store, one node at a time polls a folder, so a file is never read twice. SFTP sources need no listening port, and in the Helm chart such a route has no port. See the HL7 files over SFTP example.