Rclone

Rclone is an open source, multi threaded, command line computer program to manage or migrate content on cloud and other high latency storage. Its capabilities include sync, transfer, crypt, cache, union, compress and mount. The rclone website lists fifty supported backends including S3, and Google Drive.
Descriptions of rclone often carry the strapline "Rclone syncs your files to cloud storage". Those prior to 2020 include the alternative "Rsync for Cloud Storage". Users have called rclone "The Swiss Army Knife of cloud storage".
Rclone is well known for its codice_1 and codice_2 commands. It provides further management functions analogous to those ordinarily used for files on local disks, but which tolerate some intermittent and unreliable service. Transfers are optimised for high latency networks. Rclone is commonly a front-end for media servers such as Plex, Emby or Jellyfin to stream content direct from consumer file storage services.
Rclone is widely used at HPC sites to transfer datasets. It has received EU / CERN funding that promotes vendor-neutral synchronisation and sharing of research data held in cloud storage.
Official Ubuntu, Debian, Fedora, Gentoo, Arch, Brew, Chocolatey, and other package managers include rclone.
History.
Nick Craig-Wood was inspired by rsync. Concerns about the noise and power costs arising from home computer servers prompted him to embrace cloud storage. He began developing rclone as open source software in 2012 under the name "swiftsync".
Rclone was promoted to stable version 1.00 in July 2014.
In May 2017 Amazon barred new rclone users from its consumer Amazon Drive file storage product. Amazon Drive had been advertised as offering unlimited storage for £55 per year. Amazon blamed security concerns and also banned other upload utilities. Amazon's AWS S3 service continues to support new rclone users.
The original rclone logo was retired to be replaced with the present one in September 2018.
In March 2020 Nick Craig-Wood resigned from Memset Ltd, a cloud hosting company he founded, to focus on open source software.
Amazon's AWS April 2020 public sector blog explained how the Fred Hutch Cancer Research Center were using rclone in their Motuz tool to migrate very large biomedical research datasets in and out of AWS S3 object stores.
In November 2020 rclone was updated to correct a weakness in the way it generated passwords. Passwords for encrypted remotes can be generated randomly by rclone or supplied by the user. In all versions of rclone from 1.49.0 to 1.53.2 the seed value for generated passwords was based on the number of seconds elapsed in the day, and therefore not truly random. CVE-2020-28924 recommended users upgrade to the latest version of rclone and check the passwords protecting their encrypted remotes.
Release 1.55 of rclone in March 2021 included features sponsored by CERN and their CS3MESH4EOSC project. The work was EU funded to promote vendor-neutral application programming interfaces and protocols for synchronisation and sharing of academic data on cloud storage.
Backends and Commands.
Rclone supports the following services as backends. There are others, built on standard protocols such as WebDAV or S3, that work. WebDAV backends do not directly support server side checksum or modtime.
Remotes are usually defined interactively from these backends, local disk, or memory (as S3), with codice_3. Rclone can further wrap those remotes with one or more of codice_4, codice_5, codice_6, codice_7 or codice_8, remotes.
Once defined, the remotes are referenced by other rclone commands interchangeably with the local drive. Remote names are followed by a colon to distinguish them from local drives. For example, a remote "example_remote" containing a folder, or pseudofolder, "myfolder" is referred to within a command as a path codice_9.
Rclone commands directly apply to remotes, or mount them for file access or streaming. With appropriate cache options the mount can be addressed as if a conventional, block level disk. Commands are provided to serve remotes over SFTP, HTTP, WebDAV, FTP and DLNA. Commands can have sub-commands and flags. Filters determine which files on a remote that rclone commands are applied to.
codice_10 passes commands or new parameters to existing rclone sessions and has an experimental web browser interface.
Crypt remotes.
Rclone's crypt implements encryption of files at rest in cloud storage. It layers an encrypted remote over a pre-existing, cloud or other remote. Crypt is commonly used to encrypt / decrypt media, for streaming, on consumer storage services such as Google Drive.
Rclone's configuration file contains the crypt password. The password can be lightly obfuscated, or the whole rclone.conf file can be encrypted.
Crypt can either encrypt file content and name, or additionally full paths. In the latter case there is a potential clash with encryption for cloud backends, such as Microsoft OneDrive, having limited path lengths. Crypt remotes do not encrypt object modification time or size. The encryption mechanism for content, name and path is available, for scrutiny, on the rclone website. Key derivation is with scrypt.
Top level commands.
 about Get quota information from the remote.
 authorize Remote authorization.
 backend Run a backend specific command.
 cat Concatenates any files and sends them to stdout.
 check Checks the files in the source and destination match.
 cleanup Clean up the remote if possible.
 config Enter an interactive configuration session.
 copy Copy files from source to dest, skipping already copied.
 copyto Copy files from source to dest, skipping already copied.
 copyurl Copy url content to dest.
 cryptcheck Cryptcheck checks the integrity of a crypted remote.
 cryptdecode Cryptdecode returns unencrypted file names.
 dedupe Interactively find duplicate filenames and delete/rename them.
 delete Remove the files in path.
 deletefile Remove a single file from remote.
 genautocomplete Output completion script for a given shell.
 gendocs Output markdown docs for rclone to the directory supplied.
 hashsum Produces a hashsum file for all the objects in the path.
 help Show help for rclone commands, flags and backends.
 link Generate public link to file/folder.
 listremotes List all the remotes in the config file.
 ls List the objects in the path with size and path.
 lsd List all directories/containers/buckets in the path.
 lsf List directories and objects in remote:path formatted for parsing.
 lsjson List directories and objects in the path in JSON format.
 lsl List the objects in path with modification time, size and path.
 md5sum Produces an md5sum file for all the objects in the path.
 mkdir Make the path if it doesn't already exist.
 mount Mount the remote as file system on a mountpoint.
 move Move files from source to dest.
 moveto Move file or directory from source to dest.
 ncdu Explore a remote with a text based user interface.
 obscure Obscure password for use in the rclone config file.
 purge Remove the path and all of its contents.
 rc Run a command against a running rclone.
 rcat Copies standard input to file on remote.
 rcd Run rclone listening to remote control commands only.
 rmdir Remove the empty directory at path.
 rmdirs Remove empty directories under the path.
 selfupdate Update the rclone binary.
 serve Serve a remote over a protocol.
 settier Changes storage class/tier of objects in remote.
 sha1sum Produces an sha1sum file for all the objects in the path.
 size Prints the total size and number of objects in remote:path.
 sync Make source and dest identical, modifying destination only.
 test Run a test command.
 touch Create new file or change file modification time.
 tree List the contents of the remote in a tree like fashion.
 version Show the version number.
Example syntax (Linux).
These examples describe paths and file names but object keys behave similarly.
To install the latest stable rclone version:-
$ curl https://rclone.org/install.sh | sudo bash
To interactively configure "xmpl:" remote from a backend such as S3, Google Drive, or an sftp server:-
$ rclone config
To test the remote and obtain information about it:-
$ rclone about xmpl:
To recursively copy files from directory "remote_stuff", at the remote, to directory "stuff" in the home folder:-
$ rclone copy -v -P xmpl:/remote_stuff ~/stuff
codice_11 enables logging and codice_12, progress information. By default rclone checks the file integrity (hash) after copy; can retry each file up to three times if the operation is interrupted; uses up to four parallel transfer threads, and does not apply bandwidth throttling.
Running the above command again copies any new or changed files at the remote to the local folder but, like default rsync behaviour, will not delete from the local directory, files which have been removed from the remote.
To additionally delete files from the local folder which have been removed from the remote - more like the behaviour of rsync with a codice_13 flag:-
$ rclone sync xmpl:/remote_stuff ~/stuff
And to delete files from the source after they have been transferred to the local directory - more like the behaviour of rsync with a codice_14 flag:-
$ rclone move xmpl:/remote_stuff ~/stuff
To mount the remote directory at a mountpoint in the pre-existing, empty "stuff" directory in the home directory (the ampersand at the end makes the mount command run as a background process):-
$ rclone mount xmpl:/remote_stuff ~/stuff &
Default rclone syntax can be modified. Alternative transfer, filter, conflict and backend specific flags are available. Performance choices include number of concurrent transfer threads; chunk size; bandwidth limit profiling, and cache aggression.
Example syntax (Windows).
These examples describe paths and file names but object keys behave similarly.
To install the latest stable release download the rclone installer archive, from the rclone website, and extract it.
To interactively configure "xmpl:" remote from a backend such as S3, Google Drive, or an sftp server:-
>rclone config
To test the remote and obtain information about it:-
>rclone about xmpl:
To recursively copy files from directory "remote_stuff", at the remote, to directory "stuff" on E drive:-
>rclone copy -v -P xmpl:remote_stuff E:\stuff
codice_11 enables logging and codice_12, progress information. By default rclone checks the file integrity (hash) after copy; can retry each file up to three times if the operation is interrupted; uses up to four parallel transfer threads, and does not apply bandwidth throttling.
Running the above command again copies any new or changed file at the remote to the local directory but will not delete from the local directory.
To additionally delete files removed from the remote also from the local directory:-
>rclone sync xmpl:remote_stuff E:\stufff
And to delete files from the source after they have been transferred to the local directory:-
>rclone move xmpl:remote_stuff E:\stuff
To mount the remote directory from an unused drive letter, or at a mountpoint in a non existent directory:-
>rclone mount xmpl:remote_stuff X:
>rclone mount xmpl:remote_stuff E:\stuff
Default rclone syntax can be modified. Alternative transfer, filter, conflict and backend specific options are available. Performance choices include number of concurrent transfer threads; chunk size; bandwidth limit profiling, and cache aggression.
Academic evaluation.
In 2018, University of Kentucky researchers published a conference paper comparing use of rclone and other command line, cloud data transfer agents for big data. The paper was published as a result of funding by the National Science Foundation.
Later that year, University of Utah's Center for High Performance Computing examined the impact of rclone options on data transfer rates.
Rclone use at HPC research sites.
Examples are University of Maryland, Iowa State University, Trinity College Dublin, NYU, BYU, Indiana University, CSC Finland, Utrecht University, University of Nebraska, University of Utah, North Carolina State University, Stony Brook, Tulane University, Washington State University, Georgia Tech, National Institutes of Health, Wharton, Yale, Harvard, Minnesota, Michigan State, Case Western Reserve University, University of South Dakota, Northern Arizona University, University of Pennsylvania, Stanford, University of Southern California, UC Santa Barbara, UC Irvine, UC Berkeley, and SURFnet.
Rclone use in cybercrime.
May 2020 reports stated rclone had been used by hackers to exploit Diebold Nixdorf ATMs with ProLock ransomware. The FBI issued a Flash Alert MI-000125-MW on 4 May 2020 in relation to the compromise. They issued a further, related alert 20200901–001 in September 2020. Attackers had exfiltrated / encrypted data from organisations involved in healthcare, construction, finance, and legal services. Multiple US government agencies, and industrial entities were affected. Researchers established the hackers spent about a month exploring the breached networks, using rclone to archive stolen data to cloud storage, before encrypting the target system. Reported targets included LaSalle County, and the city of Novi Sad.
The FBI warned January 2021, in Private Industry Notification 20210106–001, of extortion activity using Egregor ransomware and rclone. Organisations worldwide had been threatened with public release of exfiltrated data. In some cases rclone had been disguised under the name svchost. Bookseller Barnes & Noble, US retailer Kmart, games developer Ubisoft and the Vancouver metro system have been reported as victims.
An April 2021 cybersecurity investigation into SonicWall VPN zero-day vulnerability SNWLID-2021-0001 by FireEye's Mandiant team established attackers UNC2447 used rclone for reconnaissance and exfiltration of victims' files. Cybersecurity and Infrastructure Security Agency Analysis Report AR21-126A confirmed this use of rclone in FiveHands ransomware attacks.
Rclone Wars.
In a 2021 Star Wars Day blog article, Managed Security Service Provider Red Canary announced Rclone Wars, an allusion to Clone Wars. The post notes illicit use of other legitimate file transfer utilities in exfiltrate and extort schemes but focuses on MEGAsync, MEGAcmd and rclone. To identify use of renamed rclone executables on compromised devices the authors suggest monitoring for distinctive rclone top level commands and command line flag strings such as codice_17 and codice_18.
Rclone and rsync.
Rsync transfers files with other computers that have rsync installed. It operates at the block, rather than file, level and has a delta algorithm so that it only needs to transfer changes in files. Rsync preserves file attributes and permissions. Rclone has a wider range of content management capabilities, and types of backend it can address, but only works at a whole file / object level. It does not currently preserve permissions and attributes. Rclone is designed to have some tolerance of intermittent and unreliable connections or remote services. Its transfers are optimised for high latency networks. Rclone decides which of those whole files / objects to transfer after obtaining checksums, to compare, from the remote server. Where checksums are not available, rclone can use object size and timestamp.
Rsync is single threaded. Rclone is multi threaded with a user definable number of simultaneous transfers.
Rclone can pipe data between two completely remote locations, sometimes without local download. During an rsync transfer, one side must be a local drive.
Rclone ignores trailing slashes. Rsync requires their correct use. Rclone filters require the use of codice_19 to refer to the contents of a directory. Rsync does not.
Rsync enthusiasts can be rclone enthusiasts too. Rsync continued to influence rclone at September 2020.
Eponymous cloud storage service rsync.net provides remote unix filesystems so that customers can run rsync and other standard Unix tools. They also offer "rclone only" accounts.
In 2016, a poster on Hacker News summarised rclone's relationship to rsync as:- "(rclone) exists to give you rsync to things that aren't rsync. If you want to rsync to things that are rsync, use rsync".
Rclone and AWS CLI.
AWS CLI is part of a suite of tools for managing Amazon's object storage service and compatible S3 products. It is built upon an AWS, python API. Rclone supports storage from a broader range of suppliers. Both can be used to manipulate S3 objects, in a way comparable to local files, at the command line.
A contributor to an rclone forum summarised the difference between using AWS CLI and rclone as:- "rclone works with many cloud providers."
"so learn rclone once and use it again and again."