Temporal difference learning with a neural network
Should I scale it before apply learning?


Definitely, you need to either:


Scale your network output so that it lies in the range (-2, 2), or,
Scale your training data so that it lies in the range (-1, 1)


The latter is marginally preferable because it will only need to be computed once, rather than on every pass through your network.


  Can we say anything about it from theoretical point of view?


This is a very standard design, and is simply a consequence of the fact that most commonly chosen activation functions have bounds of [-1, 1], [0, 1], [0, inf) or (-inf, inf). In particular for finite bounded activation functions (i.e. those with outputs bounded by [-1, 1] or [0, 1]), you will need to make sure that the output space of the network is capable of representing the desired target space. For example, for binary classification problems, you will often see people choosing labels of either 0/1 or -1/1 depending on whether the final output is activated by a tanh or sigmoid function.