Choose CNN architecture first, then optimize parameters - validation vs test performance to pick architecture?
You shouldn't choose the best model based on the performance on test set. You should run cross-validation and pick the best model from there and then assess performance on a test set. Otherwise you would just getting the best model and overfitting your test set.

Regarding imbalanced data sets, you should be aware of metrics used to assess your model in this case, such as recall, precision, etc., and not only accuracy.
In python, you can do StratifiedKFold, where you will split your data getting the same proportion for train and test set regarding target variable.