Tool for test/train automation I need to test different datasets as well as different algorithm implementations. The current workflow looks like:


Perform feature extraction from train set 
Train classifier on this features
Feed this classifier to production code
Run production code on a test set, feeding samples one by one
Take results of production code and convert to the same format as test dataset
Show statistics


How to automate this as much as possible? I currently use a bunch of bash/python scripts. I've looked through sklearn pipelines, but not sure they would give any benefit except for running functions in a line.
Maybe it is a bit too much just for this use case, but I have good experience with airflow.

It is an Apache project and quite helpful to automate some stuff. 

Otherwise there are a lot of commercial platform helping you. Just google for data science platform.