What is the right way to keep track of the different things we try? In Machine Learning we usually try many combinations of different features, filters we apply to the data, transformations on the features or the target variables and different versions of hyperparameters.

This fact makes it difficult to keep track of what works and what doesn't if we are not exhaustive with how we keep track of the different combinations we try.

I am wondering if there are any best practices around this problem. My current approach is to keep track of the different combinations naming files making reference to the parts that compose it, for example a hyperparameters pickle file I would name it booster_params_{}_{}_{}_{}.pickle'.format(filter_name, features_name, model_target, params_iteration)

where filter name is the set of filters I'm applying to the data, features name refers to the set of features used, model target to the target I'm modeling and params iteration refers to the version of the hyperparameters.

This seems like an overkill, and that is why I'm looking for ideas on how to tackle this problem.
You can maintain multiple versions of booster_params.pickle either:
1) through a version control system
or
2) manually, e.g. booster_params_v1.pickle, ...v2, etc. and a separate file where you would describe each version