Zfs backup juggling tool -- snapshotting, archiving, retention.
The zfs backing-up tool. ha-ha.
bash shell and zfs utils are needed. zb-pull requires ssh.
Run make install, it installs itself to some sbin/. You can also specify DESTDIR=/usr/local/ or similar.
For local changes (command aliases/wrappers, PATH setting etc.), file $HOME/.zb-rc is sourced before any commands are run.
There is a long-time backup weirdness about that everyone wants some “hourly backups” along with “daily backups”, “monthly backups”, sometimes “weekly”, “yearly”, “full-moon”, “christmas” and “ramadan”.
I don’t like this approach simply for it’s not machine-enough. Instead, I choose to generate the backups regularly, and forget some of the backups from time to time. Obvious way to achieve a good ratio between how many backups to hold vs. their age is “less with the time”, e.g. “for backups that are X hours old, don’t keep backups that are closer than X/10 hours apart”.
This creates a pretty good logarithmic distribution of datapoints in time, can be generally extended to any backup scheme, and looks cool because there is no god damned human timing.
run zb-snap every night (or every hour, if I want it to be denser; it generally doesn’t really matter).
What exactly does zb-cleanup clean up?
if shapshot is older than max_age, delete it right away.
get two historically subsequent snapshots. Determine time in seconds since the newer was created is X seconds, time since the older was created is Y. Obviously X is less than Y.
Calculate density*(Y-X)/Y. If the result is less than 1.0, delete the closer backup.
How to determine your density and other numbers?
Density is “maximum ratio of time between backups to age of backups, in percent”.
Get your density as maximal value from the first column, and max_age as maximum of the second column. Run zb-cleanup periodically with that values. E.g. in our example: zb-cleanup data/set 700 '1 year ago'.
Setup cron to run zb-snap periodically in time interval same as minimum value from the second row - in our case, daily. (probably in morning or somehow off-peak hours).
It doesn’t work from cron!
PATH may be different in cron (which may select wrong date program to run, or not find something other like custom-installed zfs). Edit ~/.zb-rc and fix PATH there.
Some SSH authentication methods may not work from cron environment due to missing ssh-agent, especially the password-protected privkeys. Descriptions of many workarounds are available around the internet.
Make sure that you understand possible security and compatibility implications of this configuration. Specifically, note that some recent SSH installations disable arcfour-family ciphers completely for a good reason. If you have aes CPU extension, aes128-gcm could work quite fast as well.
Can I pull among backups on local machine without SSH?
Yep, use - instead of the SSH connection string.
Be sure to verify that this software really fits your use-case before you use it. Backups are precious.