title: Is this correct vector implementation of gradient descent for multiple theta values?
body: <p>This is my matlab code to predict [1;1;1] given [1;0;1] : </p>
  
  <pre><code>m = 1; 
  alpha = .00001; 
  
  x = [1;0;1;0;0;0;0;0;0]; 
  y = [1;1;1;0;0;0;0;0;0]; 
  theta1 = [4.7300;3.2800;1.4600;0;0;0;4.7300;3.2800;1.4600]; 
  theta1 = theta1 - (alpha/m .* (x .* theta1-y)' * x)' 
  theta1 = reshape(theta1(1:9) , 3 , 3) 
  sigmoid(theta1 * [1; 0; 1]) 
  
  
  x = [1;0;1;0;0;0;0;0;0]; 
  y = [1; 1; 1;0;0;0;0;0;0]; 
  theta2 = [8.892;6.167;2.745;8.892;6.167;2.745;8.892;6.167;2.745]; 
  theta2 = theta2 - (alpha/m .* (x .* theta2-y)' * x)' 
  theta2 = reshape(theta2(1:9) , 3 , 3) 
  sigmoid(theta2 * [1; 0; 1]) 
  
  
  x = [1;0;1;0;0;0;0;0;0]; 
  y = [1; 1; 1;0;0;0;0;0;0]; 
  theta3 = [9.446;6.55;2.916;9.351;6.485;2.886;8.836;6.127;2.727]; 
  theta3 = theta3 - (alpha/m .* (x .* theta3-y)' * x)' 
  theta3 = reshape(theta3(1:9) , 3 , 3) 
  sigmoid(theta3 * [1; 0; 1])
  </code></pre>
  
  <p>I'm computing theta1, theta2, theta3 individually but I think they 
  should be linked between each computation ?</p>
  
  <p>Though gradient descent appears to be working as : </p>
  
  <pre><code>sigmoid(theta1 * [1; 0; 1]) =
      0.9999
      0.9986
      0.9488
  
  sigmoid(theta2 * [1; 0; 1]) =  
      1.0000
      1.0000
      0.9959
  
  sigmoid(theta3 * [1; 0; 1]) = 
      1.0000
      1.0000
      0.9965
  </code></pre>
  
  <p>This shows for each theta value (layer in the network) the prediction is moving closer to [1;1;1]</p>
  
  <p>Update : sigmoid function : </p>
  
  <pre><code>function g = sigmoid(z)
  g = 1.0 ./ (1.0 + exp(-z));
  end
  </code></pre>
  
  <p>Update2 :</p>
  
  <p>After extended discussion with user davidhigh who provided key insights have made following changes :</p>
  
  <pre><code>x = [1;0;1]; 
  y = [1;1;1]; 
  
  theta1 =
  
      4.7300    3.2800    1.4600
           0         0         0
      4.7300    3.2800    1.4600
  
  theta2 =
  
      8.8920    8.8920    8.8920
      6.1670    6.1670    6.1670
      2.7450    2.7450    2.7450
  
  theta3 =
  
      9.4460    6.5500    2.9160
      9.3510    6.4850    2.8860
      8.8360    6.1270    2.7270
  </code></pre>
  
  <p>The crux of my issue is that I don't feed output of each layer into the next layer, once I made this change I get better result : </p>
  
  <pre><code>z1 = sigmoid(theta1 * x)
  
  z1 =
  
      0.9980
      0.5000
      0.9980
  
  z2 = sigmoid(theta2 * z1)
  
  z2 =
  
      1.0000
      1.0000
      0.9989
  
  z3 = sigmoid(theta3 * z2)
  
  z3 =
  
      1.0000
      1.0000
      1.0000
  </code></pre>
  
  <p>z3 is the predicted value which correctly is <code>[1;1;1;]</code> whereas previously it is approx <code>[1;1;1;]</code></p>