I am trying to transfer all the part\* files from a directory directly from HDFS dir to sftp server. All the files in hdfs folder is pretty huge, so I do not want to copy them to local file system.

The current setup is

```
hdfs dfs -text "<HDFS_DIR>/part*" > localfile

curl "<sftp_username>:" --key "<private_key_file_path>" --pubkey "<public_key_file_path>" \
    --upload-file local_file "sftp://<SFTP_HOST>/<Upload_dir>"

```

How can I upload the files directly from HDFS to sftp server path without writing the file to local filesystem.

I considered the following options

1. scoop with sftp (Did not find enough resources) - <https://sqoop.apache.org/docs/1.99.7/user/connectors/Connector-SFTP.html>
2. Copy each `part` file to local fs and move it to sftp server (inefficient)
3. hadoop distcp with sftp doesn't work in cdh5. I am using `CDH-5.16.2`

Please let me know which is the best way to accomplish this. Thanks!