Get a full local backup of your remote web server with some basic command-line interaction and rsync. Bonus for OS X users: clickable icon backup goodness.
This is one of those handy tricks I discovered way too late, which some of you may not already know.
Problem: You have a web server located somewhere not physically close to you. You use FTP to send and receive files. You’re generally okay with this setup, except for one little chink in the armor: backup. Even if you don’t run remote scripts which generate files on the server (I’m looking at Movable Type here) which you never remember to backup, sooner or later your local copy and the server will lose synch.
Solution: How about a way to backup a perfect copy of the remote server, incrementally, so that each new update only downloads the files that have changed (and not the whole multi-gigabyte site)? It’s as great as it sounds.
Caveat: Though your local computer can run any OS, this only works if the server itself is Unix-based, and you have shell access. If your site runs on IIS, it won’t work. If your host doesn’t provide you with a shell account, it won’t work. In theory, your shell login should be the same as your FTP account, but not necessarily. You may want to get in touch with your host to verify your settings.
Warning: The most important things you should pay attention to are the various path settings. If you get them wrong, and somehow end up moving files to or from the wrong spot, data could become corrupted awfully quickly. The first time you run this, make sure you also have an alternate method of recovering from data loss. Just in case.
If you’re on Windows, you’re going to need some extra software, namely something called an “rsync client”. Though it’s probably overkill, grab Cygwin for now — which is a command line environment that comes with a set of powerful tools, all very much like what you get in a Unix-based OS — and you’ll get rsync with it. Install, then run Cygwin and you should be taken to a Unix-like command line.
So we should all be on the same page at this point, with a command prompt greeting us (shown below). If you already know how to get to your backup directory on the command line, skip ahead to the header “Running rsync”.
Now we want to find the directory that will house our backed-up site. This can be anywhere on your local system, and getting to it is going to depend largely on your computer’s configuration. In my case, I have a partition on my hard drive called ‘Shine’, which is mounted as a separate volume. This is the equivalent to calling a partition the G: drive in Windows. So let’s begin at the root (otherwise known as /) of our system by issuing the “change directory” command: cd /.
We can take a look at what’s in the root by issuing the “list” command: ls.
Let’s break it down piece-by-piece.
rsync - the program name itself, this is just causing it to run.
-aze - these are three options we’re specifying. a sets archive mode, which does things like preserve permissions and use relative paths. z compresses file data to speed up the transfer. e allows us to connect to a remote server. There are more options available, but these are the essential ones for what we’re trying to accomplish.
ssh - ssh, or secure shell, is a method of securely connecting to a remote server. The previous e option told rsync that we wanted to do so, and ssh is the protocol we’re going to use to do it.
username@ - this is your username on the remote server. Again, this may be similar to your FTP program’s login, or it may not. You’ll want to contact your host if you don’t know what your shell login is.
67.19.16.228 - this is the IP address of your web server. You likely won’t be able to just enter yourdomain.com here, so using your IP address is the best bet. However, that’s a pain when you don’t have a static IP, so alternatively this can also be the name of your host’s server. I can use aristotle.multipattern.com in place of an IP address, for example.
:/home/username/public_html/ - this is the full server path to the root of the directory you want to back up. Note the preceding colon, this is important for separating the IP address from the server path. By full server path, I mean you need to know where your site sits within the filesystem of the remote server. You might be able to find this with your FTP program by continuing to navigate up in the hierarchy until you can go no further; then simply chain together the resulting directories you navigated through until you get a full path back down the hierarchy to your web site’s root. Otherwise, you may need to contact your host for the full path.
. - and finally, an important trailing space followed by a single period. This indicates the current local path, which is where we navigated to earlier. Alternatively you could skip the initial step of finding this on the command line and use an absolute path here instead of a period, ie. /Volumes/Shine/Personal/mb-backup.
At this point, if you have the correct data entered, you should be ready to go. Hit return, and if the server is found, it will prompt you for your password. Enter it, then wait. The first sync will take quite a while.
That’s about it if you don’t mind entering the command manually every time you want to backup. But you can also create an alias or a shell script for the entire command that will make life a little easier. In this case, make sure to use the full absolute path on your local server instead of the period, so that the scripts are callable from anywhere.
The user profile file itself is stored in your home directory, which is most likely the directory that loads when you first open up the command line — if not, you can get to it with the command cd ~. It may be difficult to open a file with a preceding period in Windows (if Cygwin even uses this format); unfortunately I can’t really be of much more help here, so the User Manual is once again your friend.
Assuming you’ve managed to create the alias, you can now invoke the backup simply by typing backup on the command line.
The latter three lines are identical to the command-line we generated earlier, and should all be on one line. Save this file wherever you want it, but make sure to give it a “.command” extension. Also very important, make sure that the line break formats are Unix, not Macintosh or DOS.
The very last step will probably be necessary, depending on your system configuration. In the Finder, right-click (Ctrl-click if you have to) on the file and select “Get Info”. In the “Open with” menu, select Terminal from the list. Close the dialogue, and you’re done.
Now whenever you wish to backup your server, all you need to do is double-click the icon and enter your password. If it’s not working as expected, check out this tutorial on executable scripts for more help.
Finally, if this simple set of Unix commands is brand new to you, you may also wish to look into the ability of ssh to lock down your mail, especially if you use a wireless internet connection of any kind.
There’s gold in the Unix command line. It’s worth learning.
I wrote an article on using rsync with Strongspace on Windows, that can be readily adapted to any SSH server.
…but it’s essentially the same for every Unix SSH server.
RsyncX is great for local system backups to external drives, I use it all the time. I think I heard once that you’re only going to want to be running it on OS X systems though, due to resource forks, but I’m not sure about the specifics. I’ve never got it working remotely, anyway.
What a great article! I agree with everything except the Cygwin part–I wouldn’t recommend running it unless you have to, Windows doesn’t play very nice with it.
Oh, and you can get your current path in UNIX by using the ‘pwd’ command. Just type pwd at the prompt and it will tell you your current, full filesystem path.
It’s great to see more people plugging into rsync and how excellent it is. One thing I haven’t been able to figure out is how to do an rsync operation like this, but add in stuff that should be skipped and *not* backed up.
Is there any way to do this outside of creating a copy of the item locally and changing its permissions?
Thanks for sharing this. But what about the more important data from the database? How to solve this?
If you are put off by the lack of feedback as rsync is working you can add the “–progress” option on the command line (note that it can’t appear between the “e” and the “ssh”). Alternatively you could change “-aze ssh” to “-azPe ssh”.
If you don’t want to type your password, you can also generate an SSH key on the command line.
“ssh-keygen -t rsa” will create a keypair in the “.ssh” directory within your home folder. It will ask if you’d like to add a passphrase to your key. If you don’t want a passphrase then you can just press return (but read the warning below).
The keypair is stored in two files: “id_rsa” (the private part) and “id_rsa.pub” (the public part). Now, you need to copy the public part onto your remote machine. It should be added to the file “authorized_keys” in the remote “.ssh” directory (create it if it doesn’t exist).
If you don’t have an authorized_keys file, then you can simply copy “id_rsa.pub” as that file. The following command will do it.
(you’ll need to enter your remote password, hopefully for the last time).
Now, about SSH key passphrases. If you didn’t add a passphrase to your key, then it means anyone with the private part of the key (the id_rsa file) can use the key and potentially gain access to the remote machine. If you can guarantee that file is safe (using FileVault, never letting anyone touch your machine, etc) then that’s ok. If you did add a passphrase, then each time you use the key you’ll be asked for the passphrase. This is just as annoying as having to type your password all the time. Fortunately, there are a couple of OSX tools that handle this for you (SSH Keychain and SSH Agent). It’s up to you. Either way, your private key is valuable and you should protect it.
Excellent primer on rsync. I love that little tool!
Just a note – I believe you have to have a copy of rsync that supports “-E” to use that option. -E helps handle the MacOS metadata so some Linux/Unix hosts might not have rsync patched. Conversely if you are not syncing to/from a Mac then you can omit it. I am not a sysadmin so if anyone can confirm that rsync needs to be configured on a Linux/Unix to support MacOS MD that would be great. I use OS X Tiger and my server are updated to handle OS X MD so I always add -E.
Just SSH to your server, run the command and dl/copy your .sql file for safe keeping.
Just a few more tips, lifted straight from my ./scripts directory. As I don’t run a Mac, YMMV, but this should work on any *nix variant.
The -r is for recursive directory traversal, and the –delete has already been mentioned in comments above.
Just add such scripts to your crontab, to run at the appropriate times, and you should be backed up fairly well, with 3 backups to cover you an any particular time.
Just a boring note: ‘-e ssh’ has been the default in rsync since version 2.6.0, so you don’t need to specify it.
Interesting… I’ve used rsync to keep my dev and live sites in sync and to atomically launch new features.
But I never really put the pieces together on the remote-backup front. Nice article.
This is what I’m talking about! Great post!
As for backing up MySQL databases with rsync - you can just copy the mysql data directory directly. This is much faster than using mysqldump, but will ONLY work for MyISAM tables ( NOT InnoDB! ).
I don’t know enough about rsync, and you may have already considered this, but what about putting your remote directory in Subversion instead?
Thanks all for the suggestions, ssh keys have been on my to-do list for ages. Haven’t had much luck yet, but I’ll keep plugging away at it.
I just about added a line to the original article to this effect. svn or cvs would work perfectly well too. I’ve just found them harder to grasp (and higher maintenance) than rsync. Plus, when working on more than one site the rsync method just seems easier. But that’s just me.
Again, I don’t know if rsync does this, but the added benefit with version control is that you can revert your entire directory and all of its files to a previous state if the need arises. And the need has arisen for me so I’ve found it useful.
I know what you mean about the setup. I guess it depends on your host. I’m on TextDrive and setting up a Subversion repository was pretty simple. After setup, I read through the free Subversion book online to learn a few basic commands. After the initial sync, keeping in sync is just: “svn up”.
But, as I think you were implying, the important thing is do something that works for you, as long as you do something.
My only tip to add to this would be: for finding out the remote path you’re backing up… to save having to traipse up the directory tree on your remote server using FTP, just SSH into it instead and run the ‘pwd’ (print workin’ directory) command.
Why not use the site synchronize method in Dreamweaver?
It’s great :) One click (or perhaps two) and you’re done.
To Neil who was asking about skipping directories, I use unison for automated backups, unison is pretty much rsync with some stuff made easier.
I highly recommend unison for this problem specifically.
I don’t own a Mac so I don’t know for sure, but couldn’t you use Automator to run your shell script file periodically instead of having to remember to run it?
For basic site stuff it’s fine and it’s good to version any actual modifications you make to a webapp’s code… but a lot of webapps do store some dynamic things in there which also need to be backed up.
Wow. First time checking out your site and this is really great stuff even for someone only recently becoming involved with server based web design. Thanks for pointing out some useful knowledge, and keep doing what you do.
Great information! I am considering using this to transfer a complete site from one host to another so preserving permissions is critical as is keeping directory structure (I’ll be using Mac OSX unless I am forced to use Windows).
When it comes time to restore to the new host, what is the correct way to expand everything so that all directory structure and files/permissions are restored to the same state they were in the original site?
Thanks to all of you for some great info!
Just a note on another way to change local directories in the terminal if said directory is open in a window (or you find traversing through the GUI easier than cd/ls). Type cd [space] and simply drag the folder you wish to change to into the Terminal window. Great article - thanks.
i had a problem: very large files (2Gb+) were causing the process to stall. but there’s a –max-size option, so rsync will ignore files over a certain size.
You are reading “Web Server Backup”, an entry posted on 22 September, 2005, to the Script collection. See other posts in this collection.