Why we need to have the test set remains consistent across multiple runs?
You will judge the performance of the trained model based on certain performance metrics.
If you keep updating your test set, then you will not know whether one run is better than the other.
For e.g. you are trying to predict the value of house. And you have only one data point in your test set and you are using mean absolute error (MAE) as your performance criteria.
Scenario 1: the actual value of house is 100K, your prediction with first model is 101K. MAE = 1000
Scenario 2: the the actual value of house is 25K, your prediction with second model is 26K. MAE = 1000
In reality, prediction is better with model1 as compared to model2. But because you changed your test set, you will not be able to make that conclusion. Had you kept your test set same in scenario2, your prediction would have been much higher making MAE much higher, clearly indicating that model1 is better than model2.
Hope this helps.