How to normalize data for Neural Network and Decision Forest
I disagree with the other comments.

First of all, I see no need to normalize data for decision trees. Decision trees work by calculating a score (usually entropy) for each different division of the data $(X\leq x_i,X&gt;x_i)$. Applying a transformation to the data that does not change the order of the data makes no difference.

Random forests are just a bunch of decision trees, so it doesn't change this rationale.

Neural networks are a different story. First of all, in terms of prediction, it makes no difference. The neural network can easily counter your normalization since it just scales the weights and changes the bias. The big problem is in the training.

If you use an algorithm like resilient backpropagation to estimate the weights of the neural network, then it makes no difference. The reason is because it uses the sign of the gradient, not its magnitude, when changing the weights in the direction of whatever minimizes your error. This is the default algorithm for the neuralnet package in R, by the way.

When does it make a difference? When you are using traditional backpropagation with sigmoid activation functions, it can saturate the sigmoid derivative.

Consider the sigmoid function (green) and its derivative (blue):



What happens if you do not normalize your data is that your data is multiplied by the random weights and you get things like $s'(9999)=0$. The derivative of the sigmoid is (approximately) zero and the training process does not move along. The neural network that you end up with is just a neural network with random weights (there is no training).

Does this help us to know what the best normalization function is? But of course! First of all, it is crucial to use a normalization that centers your data because most implementation initialize bias at zero. I would normalize between -0.5 and 0.5, $\frac{X-\min{X}}{\max{X}-\min{X}}-0.5$. But standard score is also good.

The actual normalization is not very crucial because it only influences the initial iterations of the optimization process. As long as it is centered and most of your data is below 1, then it might mean you have to use slightly less or more iterations to get the same result. But the result will be the same, as long as you avoid the saturation problem I mentioned.

There is something not here discussed which is regularization. If you use regularization in your objective function, the way you normalize your data will affect the resulting model. I'm assuming your are already familiar with this. If you know that one variable is more prone to cause overfitting, your normalization of the data should take this into account. This is of course completely independent of neural networks being used.