Why is CiviCRM Upgrade so Complex?
I have been dealing with upgrades a lot lately, and I have the process "down" so that it is very fast.  I am impressed with how fast I can do an upgrade.  The key is shell access - or something that mimics it - because using sftp is very slow.  That might be your problem.  Once I have the data backed up it basically goes like this:


ssh into the server and tar/gz (archive) the existing civicrm directory.  One command, 15 seconds or so.
upload the new version tar/gz.  This of course will vary greatly by internet connection speed, but generally only a couple of minutes or less.
delete the existing directory.  This is what's horrendously slow via most sftp programs I've used.  They usually do a recursive delete on each individual file and directory, and since there's a ton of them in CiviCRM it takes forever.  It takes 1 second in the shell.
Unarchive the new version.  Usually less than a minute.
run the upgrade script.


Very, very fast.

If you aren't a command line person - I'm not, actually - there are some ways around it.  Our systems all have Webmin installed as a helpful control panel, and it's Filemin file manager deletes directories as quickly as the shell commands do.  I also regularly use Cyberduck for SFTP work, and it has a built in tar function, so I can archive/unarchive just as quickly as from the command line.  So, even though I don't really use the command line I am getting the benefits.

I would add that if you are concerned about security the very LAST thing you would want to do is have excessive writable directories on your server, and most built in upgrade systems require these.  Don't give in.  You can reduce writable directories by not using those systems and save you many troubles down the road.