title: Basic Backpropagation Implementation Not Working
body: <p>I'm in the early stages of understanding <code>backpropagation</code> and I attempted to implement it myself. </p>
  
  <p>The dataset I attempted to work with was the iris dataset of size (150, 4). </p>
  
  <p>I'm only worried about <code>backpropagation</code> and not gradient descent, so I'm just trying my algorithm on one example to see if I can get a seemingly proper output. </p>
  
  <p>However, my issue is trying to get my gradients for my initial weight matrix, I'm getting an error with the shapes.</p>
  
  <p>I'd like my network to be something like this -
  4 inputs, 8 hidden neurons, and 1 output neuron 
  <a href="https://i.stack.imgur.com/5tDYR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5tDYR.png" alt="Neural network"></a></p>
  
  <p>My code is below. <strong>The error is with the last line because x is of size (4,1) and delta2 is of size (8,8) so I can't get the dot product</strong> I just don't understand how I am supposed to get a correct size of delta2 if I'm following the algorithm correctly according to other sources. </p>
  
  <pre><code>from sklearn.datasets import load_iris
  from keras.utils import to_categorical
  import numpy as np
  
  # LOAD DATA
  data = load_iris()
  X = data.data[:-20]
  y = to_categorical(data.target[:-20])
  # only 20 samples because we have a small dataset
  X_test = data.data[-20:]
  y_test = to_categorical(data.target[-20:])
  
  # INIT WEIGHTS  - will try to add bias later on
  w1 = np.random.rand(np.shape(X)[1], h_neurons)
  w2 = np.random.rand(h_neurons, 3)
  
  def sigmoid(x, deriv=False):
      if deriv:
          return sigmoid(x)*(1-sigmoid(x))
      else:
          return 1/(1+np.exp(-x))
  
  # Feed forward
  x = X[1].reshape(4,1)
  z1 = w1.T.dot(x) # need to transpose weight matrix
  a1 = sigmoid(z1)
  z2 = w2.T.dot(a1)
  y_hat = sigmoid(z2,deriv=True) # output
  
  
  # BACKPROP
  y_ = y[1].reshape(3,1)
  delta3 = np.multiply((y_hat - y_), sigmoid(z2, deriv=True))
  dJdW2 = a1.dot(delta3) ## ERROR !!!
  
  delta2 = np.dot(delta3, w2.T) * sigmoid(z1, deriv=True)
  dJdW1 = np.dot(x.T, delta2) ## ERROR !!!
  </code></pre>
  
  <p>I thought I implemented <code>backpropagation</code> correctly, but apparently not, can someone please point out where I went wrong? </p>
  
  <p>I'm stuck and I've looked at various sources and the code to compute dJdW (derivative of cost with respect to weights) is roughly the same. </p>