How can I be sure that the final model, trained on all data, is correct? The 'by the book' method of delivering final machine learning models is to include all data in the final training (including validation and test sets). To check robustness of my model I use randomly chosen population for training and validation sets with each training (no set random seed). The results on validation and then test sets are pretty satisfactory for my case however they are always different each time, precision spans between 0.7 and 0.9. This is due to fact that each time different data points fall to set with which model is trained.
My question is: how do I know that final training will also generate good model and how to estimate its precision when I do not have anymore unseen data?
The purpose of the test set is to test your model before deploying, otherwise, you would not need the test set in the first place. If you retrain your model by also including the validation and test datasets, of course, you cannot test your model anymore. You need to leave the test dataset separate and not use it for retraining, unless you have more data for testing.