Dealing with outliers and z-scores I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques.

I am building a function to calculate a social welfare score/rank of the countries in the world. While doing so, I come across multiple outliers which essentially skew the results.

I have a few questions:


Which normalization function to use and when? I am currently using the z score.
If i come across an outlier and I know the reason for the outlier and want to avoid it affecting the result, how should I modify the value?eg . replace with mean/median. Which technique should I use and when?
After calculating the z-scores, how to devise a function of the z scores? Is it trial and error based or I can apply some technique to find the best results. Since I do not have a strict result that I am expecting, how do I calculate the coefficients and operations for the z-scores?
Just some quick fixes.
Normalization takes care of scale of data across the columns. e.g.
If one data column is in the ranges of 1000s and other in 10s then normalization will work.

But if you want to take care of the skew-ness one way is to simply take log of the data. It pulls in the outliers. as log increases very slowly for positive values. Do it before the normalization (Z score in your case)


Other ways are taking nth root of the data.