How high rmse value can be?
Are you asking about rmse or r-squared?

The size of rmse depends on the general range of your response feature(predicting target). The lower the better.

As to r-squared, both 0.9999 and 0.9769 are very likely to indicate overfitting. If you have to choose between these two models, my suggestion is to evaluate them using generalization error. You can also do cross-validation if no out-of-sample data are available.