title: Why do I get different weights when using TensorFlow for multiple linear regression?
body: <p>I have two implementations of multiple linear regressions, one using <code>tensorflow</code> and one using only <code>numpy</code>. I generate a dummy set of data and I try to recover the weights I used, but although the <code>numpy</code> one returns the initial weights, the <code>tensorflow</code> one always returns different weights (which also sort of work)</p>
  
  <p>The <code>numpy</code> implementation is <a href="http://pastebin.com/LMM1fju0" rel="nofollow">here</a>, and here's the TF implementation:</p>
  
  <pre><code>import numpy as np
  import tensorflow as tf
  
  x = np.array([[i, i + 10] for i in range(100)]).astype(np.float32)
  y = np.array([i * 0.4 + j * 0.9 + 1 for i, j in x]).astype(np.float32)
  
  # Add bias
  x = np.hstack((x, np.ones((x.shape[0], 1)))).astype(np.float32)
  
  # Create variable for weights
  n_features = x.shape[1]
  np.random.rand(n_features)
  w = tf.Variable(tf.random_normal([n_features, 1]))
  w = tf.Print(w, [w])
  
  # Loss function
  y_hat = tf.matmul(x, w)
  loss = tf.reduce_mean(tf.square(tf.sub(y, y_hat)))
  
  operation = tf.train.GradientDescentOptimizer(learning_rate=0.000001).minimize(loss)
  
  with tf.Session() as session:
      session.run(tf.initialize_all_variables())
      for iteration in range(5000):
          session.run(operation)
      weights = w.eval()
      print(weights)
  </code></pre>
  
  <p>Running the script gets me weights around <code>[-0.481, 1.403, 0.701]</code>, while running the <code>numpy</code> version gets me weights around <code>[0.392, 0.907, 0.9288]</code> which are much closer to the weights I used to generate the data: <code>[0.4, 0.9, 1]</code></p>
  
  <p>Both learning rates/epochs parameters are the same, and both initialise weights randomly. I don't normalize the data for either of the implementations, and I've ran them multiple times.</p>
  
  <p>Why are the results different? I also tried to initialise weights in the TF version using <code>w = tf.Variable(np.random.rand(n_features).reshape(n_features,1).astype(np.float32))</code> but that didn't fix it either. Is there something wrong with the TF implementation?</p>