Git for Deep Learning - what are the best tools for versioning/tracking machine learning experiments? I'm looking for a tool to track the results of several experiments/iterations in machine learning.

Training a model can take days/weeks, so it's essential to track its performance and be able to easily replicate results and roll back to previous versions. 
I'm looking for a tool that makes the process easy and streamlined like git did for version control.



Useful features would be:


open source with a license that allows commercial application
self-hosted [all the information should reside on computers we own]
python-friendly (ideally, tensorflow-friendly as well)
can store both hyperparameter values as well as references to specific datasets (the latter would be useful to track the effect of hard negative mining, etc.)
cluster-friendly
include basic plotting (it's useful to visualize learning curves)
automatic testing on multiple datasets
user tracking [like "blame" in git]
One such tool is Polyaxon. I'm using it extensively and it's really helping the experimentation pipeline.

It has features like experiment queuing, hyperparameter tuning, clustering, could be self-hosted or on the cloud, plots the metrics, it is independent of the framework and so on ...

I definitely recommend it.