XGBoost validation for number of trees
At first glance, your conclusion appears correct, but there are some important caveats to keep in mind.

First, what are the sizes of your training and validation sets?  If your validation set is too small, then the observed difference may not be statistically significant.

Second, you should verify that your validation set is a representative sample.  (i.e. it should come from the same distribution as the training set).  If it's not representative, then it may give poor estimates of performance.

Third, when tuning hyperparameters, it's a good idea to split your dataset into three shards - training, validation, and testing.  You can use the training and validation sets to find optimal hyperparameters (as you have done), then use the testing set to generate a performance estimate for the tuned model.  If you trust the validation accuracy obtained during hyperparameter tuning, you are liable to a subtle form of overfitting where hyperparameters are specialized for the validation set.

Finally, if you have the computational resources, then it's always a good idea to evaluate accuracy with cross-validation rather than a train-test split.  This will give you a more robust estimate of accuracy.

If you've checked all these boxes, then you have good reason to believe that 500 estimators is better than 100 estimators!  


  [S]hould I change the validation data few times and see if similar kinds of increment happens by shifting from 100 to 500 estimators?


Yes, it's always a good idea to try many different configurations of hyperparameters.  You can use scikit-learn's GridSearchCV or RandomizedSearchCV to easily run a search over the hyperparameter space.