Regression performance with Feature Selection
Generally speaking, some ML algorithms include regularization (such as Ridge/Lasso Regression, Random Forest, CatBoost,...) and others don't (k-NN, Gaussian Naive Bayesian,...). Using one that includes regularization may not change the performance when removing less important features.
This applies to your first question. If you use Lasso regularization for example, it will push the weights of &quot;less important&quot; features to zero. So removing them should not change the performance. Using Ridge regularization is a bit different.
So when you say:

MSE increased gradually while I am removing features step by step

There could be multiple reasons for that:

the features you removed contained information and were not &quot;less important&quot; for the regression (see *Note/warning)
even the less important features contain information
you did not retrain your model after removing the &quot;less important&quot; features

Regarding your second question, $R^2$ is related to $MSE$ (it is well explains in this post), so if the $MSE$ increases, $R^2$ should increase as well.
*Note/warning: a feature is often said to be &quot;less important&quot; using one algorithm (such as CatBoost which often uses Gini or entropy). It does not always mean this feature will be &quot;less important&quot; when using it with another algorithm (such as regression).