title: Why does my neural network approximate shifted functions so badly?
body: <p>I wrote my own neural network in java, i started with a perceptron and expanded to a multilayer network with sigmoidal activation function, which is learning through SGD and backpropagation.</p>
  
  <p>I am currently testing it, single layer and multi layer classification and regression work, linear and non-linear.</p>
  
  <p>However, if I introduce a shift in the y-direction like</p>
  
  <blockquote>
    <p>f(x)= 2*x + 40</p>
  </blockquote>
  
  <p>my network just approximates the inclination and not the + 20. If I monitor the weights it shows me that the bias(which is in charge of such shifts) is changing but very slowly and even if i let it learn for thousands of epochs it doesnt nearly reach the solution.</p>
  
  <p><a href="https://i.stack.imgur.com/hXnHi.png" rel="nofollow noreferrer">Linear function where y=2*x + 40</a></p>
  
  <p>A similar problem is with a square function where</p>
  
  <blockquote>
    <p>f(x)= x^2</p>
  </blockquote>
  
  <p>, i can make my network as big and deep as I want, it doesnt converge perfectly, just a very rough approximation, like two linear graphs. I think this also due to  bias weights not converging properly or at least very slowly.</p>
  
  <p><a href="https://i.stack.imgur.com/7C3hm.png" rel="nofollow noreferrer">neural network approximated square function, with 10 hidden neurons and one output</a></p>
  
  <p>Anyways, my question now is, wether this is normal and i should just train longer or introduce a momentum or weight decay term, or if i implemented something wrong.</p>
  
  <p>PS: I can provide code if that is necessary, I just wanted to keep this as short as possible. Didnt really work but I hope it's at least clear what i mean. Thank you all for your time in advance!</p>