How to version the code and the data during the analysis? I am currently looking for a system which will allow me to version both the code and the data in my research.

I think my way of analyzing data is not uncommon, and this will be useful for many people doing bioinformatics and aiming for the reproducibility.

Here are the requrements:


Analysis is performed on multiple machines (local, cluster, server).
All the code is transparently synchronized between the machines.
Source code versioning.
Generated data versioning.
Support for large number of small generated files (>10k). These also could be deleted.
Support for large files (>1Gb). At some point old generated files can  permanently deleted. It would be insane to have transparent synchronization of those, but being able to synchronize them on demand would be nice.


So far I am using git + rsync/scp. But there are several downsides to it.


Synchronization between multiple machines is a bit tedious, i.e. you have to git pull before you start working and git push after each update. I can live with that.
You are not supposed to store large generated data files or large number of files inside your repository.
Therefore I have to synchronize data files manually using rsync, which is error prone.


There is something called git annex. It seems really close to what I need. But:


A bit more work than git, but that's ok.
Unfortunately it seems it does not work well with the large number of files. Often I have more that 10k small files in my analysis. There are some tricks to improve indexing, but it doesn't solve the issue. What I need is one symlink representing the full contents of directory.


One potential solution is to use Dropbox or something similar (like syncthing) in combination with git. But the downside is there will be no connection between the source code version and the data version.

Is there any versioning system for the code and the data meeting the requirements you can recommend?
There is a couple of points to consider here, which I outline below. The goal here should be to find a workflow that is minimally intrusive on top of already using git.

As of yet, there is no ideal workflow that covers all use cases, but what I outline below is the closest I could come to it.

Reproducibility is not just keeping all your data

You have got your raw data that you start your project with.

All other data in your project directory should never just "be there", but have some record of where it comes from. Data processing scripts are great for this, because they already document how you went from your raw to your analytical data, and then the files needed for your analyses.

And those scripts can be versioned, with an appropriate single entry point of processing (e.g. a Makefile that describes how to run your scripts).

This way, the state of all your project files is defined by the raw data, and the version of your processing scripts (and versions of external software, but that's a whole different kind of problem).

What data/code should and should not be versioned

Just as you would not version generated code files, you should not want to version 10k intermediary data files that you produced when performing your analyses. The data that should be versioned is your raw data (at the start of your pipeline), not automatically generated files.

You might want to take snapshots of your project directory, but not keep every version of every file ever produced. This already cuts down your problem by a fair margin.

Approach 1: Actual versioning of data

For your raw or analytical data, Git LFS (and alternatively Git Annex, that you already mention) is designed to solve exactly this problem: add tracking information of files in your Git tree, but do not store the content of those files in the repository (because otherwise it would add the size of a non-diffable file with every change you make).

For your intermediate files, you do the same as you would do with intermediate code files: add them to your .gitignore and do not version them.

This begs a couple of considerations:


Git LFS is a paid service from Github (the free tier is limited to 1 GB of storage/bandwidth per month, which is very little), and it is more expensive than other comparable cloud storage solutions. You could consider paying for the storage at Github or running your own LFS server (there is a reference implementation, but I assume this would still be a substantial effort)
Git Annex is free, but it replaces files by links and hence changes time stamps, which is a problem for e.g. GNU Make based workflows (major drawback for me). Also, fetching of files needs to be done manually or via a commit hook


Approach 2: Versioning code only, syncing data

If your analytical data stays the same for most of your analyses, so the actual need to version it (as opposed to back up and document data provenance, which is essential) may be limited.

The key to get this this working is to put all data files in your .gitignore and ignore all your code files in rsync, with a script in your project root (extensions and directories are an example only):

#!/bin/bash
cd $(dirname $0)
rsync -auvr \
    --exclude "*.r" \
    --include "*.RData" \
    --exclude "dir with huge files that you don't need locally" \
    yourhost:/your/project/path/* .


The advantage here is that you don't need to remember the rsync command you are running. The script itself goes into version control.

This is especially useful if you do your heavy processing on a computing cluster but want to make plots from your result files on your local machine. I argue that you generally don't need bidirectional sync.