Dropping features after final evaluation on test data
What you're doing is manual feature selection based on the test set. You're right that it's not correct to proceed this way: in theory, feature selection should be done using only the training set and a validation set, not the final test set. The risk is data leakage: you're modifying the model using information from the test set. Maybe the performance is better without these features on the test set because they happen to be bad for this particular current test set by chance. As a result the model could be overfit, and you wouldn't be able to detect this problem on this test set since it's the source of the overfitting.
So in principle it's always better to separate the data first, keep the test set aside until the final evaluation and use a validation set for intermediate evaluation until the final model (including the set of features) is determined.
In practice, it sometimes happens that we realize that we should have done something differently after applying the model on the final test set. It's a mistake but it's not the end of the world, usually the risk of bias is low. As you said, obviously we don't have to ignore an information which is important for the performance of the model. However if you know that you're going to repeat this procedure with several features, you should defintely do it using a separate validation set (taken from the training data), not the test set: the more you use the test set like this, the higher the risk of data leakage and bias.