What are tools that lend themselves for backup and recovery in a CI/CD environment? I have used tar, dump, Amanda, Legato (EMC) Networker and other backup software for making backups of systems.  However, I don't know the best tools for making backups in a CI/CD (Continuous Integration/Continuous Deployment) environment and doing "rolling recoveries" of systems "as you go" which is the case in a DevOps oriented system.

Many of these backup utilities are not necessarily best suited for a CI/CD environment because of the continuous changes taking place on both the development and production environment.

I'm looking at programs like Borg 
and Git based backups like Bup 
These programs would allow for many incremental backups and finer granularity of backups and work well with CD tools like Ansible.  Even Ansible has a couple of way to make backups.  However, there is no clean way to recover the most recent backup in an automated way. 

Relying on some form of caching or mirroring of your content on the cluster and hope it's not corrupted is not a way to go :(

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.
The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles.
Currently I use a combination of scp and tar into an archive directory sorted by date and machine.  I like to find something better if possible.

I would like to know what are the best backup and recovery tools used in the CI/CD environment?  and an example of how you would configure it?

I don't expect a "magic bullet" just some possible solutions.
The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.


Config files, containers and VMs should be backed up and versioned by a tool like git or SVN if you are doing configuration management properly.. Backing up config files and versioning them is the whole point of configuration management tools like, salt, puppet, ansible, and chef.

As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of configuration management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, you should use the many battle-tested techniques and strategies for dealing with this kind of data instead of expecting CI/CD/CM tools to solve problems that they were never intended to address.