title: How does ReLu work with zero-centered output domain?
body: <p>In the problem i am trying to solve, my output domain is zero centered, between -1 an 1. When looking up activation functions i noticed that ReLu outputs values between 0 and 1, which basically would mean that your output is all negative or all positive.</p>
  
  <p>This can be mapped back to the appropriate domain through inverse normalization but ReLu is designed to determine the "strength" of a neuron in a single direction, but in my problem, i need to determine the strength of a neuron in one of two direction. If i use tanh, i have to worry about vanishing/exploding gradients, but if i use ReLu, my output will always be "biased" towards positives or negative values because essentially really small values would be have to mapped to a postitive domain and large value a negative domain or visa versa.</p>
  
  <p>Other info: I've used ReLu and it works well but i fear that it is for the wrong reasons. The reason i say this is that it seems for either the pos or neg domain approaching smaller values will mean a stronger connection up to a point, then which it will not be activated at all. Yes the network can technically work (probably harder than it needs to) to keep the entire domain of train outputs in the positive space, but if a value happens to exceed the bounds of the training set it will be non-existent? when in reality it should be even more active</p>
  
  <p>What is the appropriate way to deal with zero centered output domains?</p>