title: Constant predicted value for MLPRegressor
body: <p>I want to perform a neural network regression on a data set. For testing purposes I have sample it down to 10000 rows. The input is 3 columns, the output is 1 column. I use the code below (I've replaced variable names). </p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import os
  from sklearn.neural_network import MLPRegressor
  
  
  """
  Prepare
  """  
  train = os.path.join(r'C:\Documents and Settings\', 'input.csv')
  df = pd.read_csv(train)
  
  df = df[['A', 'B', 'C','D']]
  df = df.dropna().sample(n=10000)
  
  y = df['D'].as_matrix().reshape(10000,1)
  x = df[['A', 'B','C']].as_matrix().reshape(10000,3)
  
  print x
  print y
  print "Length before regression, x: %s, y: %s" % (x.shape, y.shape)
  
  """
  Regression 
  """
  mlp = MLPRegressor(hidden_layer_sizes=(5, ), activation='relu', verbose=True, learning_rate_init=1, learning_rate='adaptive', max_iter=500,)
  
  mlp.fit(x,y)
  mlp.score(x,y)
  
  print mlp.coefs_
  print mlp.n_layers_
  print mlp.n_outputs_
  print mlp.out_activation_
  print "res: ",res
  
  res = mlp.predict(x)
  r = np.subtract(df['D'].as_matrix(), res)
  </code></pre>
  
  <p>Running this code gives the following output: </p>
  
  <pre><code>[[ 162.      9.    475.5 ]
   [ 105.      6.39  232.5 ]
   [ 141.      7.44  373.5 ]
   ..., 
   [ 120.      8.41  450.5 ]
   [ 120.      8.77  464.  ]
   [ 160.      8.77  483.  ]]
  [[ 72. ]
   [ 73. ]
   [ 74.5]
   ..., 
   [ 53. ]
   [ 52. ]
   [ 73. ]]
  Length before regression, x: (10000, 3), y: (10000, 1)
  Iteration 1, loss = 43928.72815906
  Iteration 2, loss = 3434.26257670
  Iteration 3, loss = 2393.24701752
  Iteration 4, loss = 1662.31634550
  Iteration 5, loss = 1225.37443598
  Iteration 6, loss = 997.21761203
  Iteration 7, loss = 891.10992049
  Iteration 8, loss = 847.20461842
  Iteration 9, loss = 830.60945144
  Iteration 10, loss = 825.10945455
  Iteration 11, loss = 823.39941482
  Iteration 12, loss = 822.96788084
  Iteration 13, loss = 822.85930250
  Iteration 14, loss = 822.83848702
  Iteration 15, loss = 822.84245376
  Iteration 16, loss = 822.84871312
  Iteration 17, loss = 822.83965835
  Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.
  [array([[-5.33, -5.23, -5.15, -4.86, -5.68],
         [-5.28, -5.86, -5.83, -5.98, -6.2 ],
         [-5.32, -5.79, -5.02, -4.71, -5.87]]), array([[-5.69],
         [-5.06],
         [ 4.35],
         [ 4.6 ],
         [-5.66]])]
  3
  1
  identity
  res:  [ 95.53  95.53  95.53 ...,  95.53  95.53  95.53]
  </code></pre>
  
  <p>The resulting <code>res</code> variable is then constant. </p>
  
  <p>I've played a little with the prediction and found that input values below 0.01 give a little change in result. Also I find that the out_activation_ is always <code>identity</code>, though I've set the activation function to be <code>relu</code>. </p>
  
  <p>I'm kind of lost to what might cause this behavior. Why does it seem x needs to be different (normalized?) for fit() than for predict()?</p>
  
  <p>note: as was commented below there is no cross-validation in this example. I am aware of that.</p>