What are good practices in reporting RMSE or MAPE estimates for a machine learning model? I have built a regression model to estimate prices given some attributes. Prices are expressed in million USD, with average of 4.4 million USD. I need to provide an estimate of the RMSE and MAPE of the model. The average RMSE and MAPE of my model seem too high, with values of 3.4 million USD and 515 % respectively. However, looking at the RMSE and MAPE distribution in relation to the prices, I can see that there are a few, very low prices, that are difficult to predict and cause the very high mean error estimate. The scatterplot below shows price on the x axis and MAPE on the y axis



If I eliminate those 9 'outliers', the RMSE and MAPE go down to more reasonable values, namely 0.95 million USD and 26.7% respectively. My questions are:


should I eliminate those 'outliers' (i.e. very low prices) before training?
or should I train the model to predict all the data, and then exclude the outliers from the final RMSE/MAPE estimation?
or is there any other best practice in providing the most realistic estimation of the error
or, is there any other error metric that is robust to outliers?


Thank you very much
In general you should not attemt to make your model look better than it actually is. Always stick to the truth. Some ideas:


Outliers: It is not okay to simply drop the outliers. If you want to make a prediction or do inference, you have to ask yourself how relevant such outliers are in practice. Where do they come from (are data reliable or is it an error)? If they are part of the true bigger picture, you cheat if you just drop them. If they are a very special part of the market with specific attributes, you could try to control for these special types (e.g. by adding indicators or changing the parametric form of your model). This question essentially is data and method driven.
Your metrics should of course reflect the final model!
By definition of the RMSE it is the mean (not median). Naturally, you can describe the situation by discussing the influence of the outliers in your own way. But this will not make your model/fit better after all. 


Finally: I think you should NOT ask about a metric that suits your approach and make it look better. You should ask yourself if you can improve your model. You did not specify what type of method you use and what data you work with, but I'm quite sure there are ways to improve the fit. The scatter plot looks like an L-shaped function. So this might be the first thing to start to work on!?