How do I know that model performance improvement is significant?
Your last paragraph is actually the answer but there is no need to train your model several times. You actually need to &quot;validate/evaluate&quot; your model several times which of course can be run on smaller sets.

Have several validation set. Quantity is not the main major but make sure they cover the distribution of data specially all classes.
run both models on those sets and keep results in two lists/arrays
simple/practical solution: If the difference between their means is larger than their individual standard deviations, then you are good to go
a bit more confident: You have two sets of numbers (performance of two models). use a simple anova and monitor the p-value to see if their differences is significant.