title: Difference on performance between numpy and matlab
body: <p>I am computing the <code>backpropagation</code> algorithm for a sparse autoencoder. I have implemented it in python using <code>numpy</code> and in <code>matlab</code>. The code is almost the same, but the performance is very different. The time matlab takes to complete the task is 0.252454 seconds while numpy 0.973672151566, that is almost four times more. I will call this code several times later in a minimization problem so this difference leads to several minutes of delay between the implementations. Is this a normal behaviour? How could I improve the performance in numpy?</p>
  
  <p>Numpy implementation:</p>
  
  <p>Sparse.rho is a tuning parameter, sparse.nodes are the number of nodes in the hidden layer (25), sparse.input (64) the number of nodes in the input layer, theta1 and theta2 are the weight matrices for the first and second layer respectively with dimensions 25x64 and 64x25, m is equal to 10000, rhoest has a dimension of (25,), x has a dimension of 10000x64, a3 10000x64 and a2 10000x25.</p>
  
  <p><code>UPDATE</code>: I have introduced changes in the code following some of the ideas of the responses. The performance is now numpy: 0.65 vs matlab: 0.25.</p>
  
  <pre><code>partial_j1 = np.zeros(sparse.theta1.shape)
  partial_j2 = np.zeros(sparse.theta2.shape)
  partial_b1 = np.zeros(sparse.b1.shape)
  partial_b2 = np.zeros(sparse.b2.shape)
  t = time.time()
  
  delta3t = (-(x-a3)*a3*(1-a3)).T
  
  for i in range(m):
  
      delta3 = delta3t[:,i:(i+1)]
      sum1 =  np.dot(sparse.theta2.T,delta3)
      delta2 = ( sum1 + sum2 ) * a2[i:(i+1),:].T* (1 - a2[i:(i+1),:].T)
      partial_j1 += np.dot(delta2, a1[i:(i+1),:])
      partial_j2 += np.dot(delta3, a2[i:(i+1),:])
      partial_b1 += delta2
      partial_b2 += delta3
  
  print "Backprop time:", time.time() -t
  </code></pre>
  
  <p>Matlab implementation:</p>
  
  <pre><code>tic
  for i = 1:m
  
      delta3 = -(data(i,:)-a3(i,:)).*a3(i,:).*(1 - a3(i,:));
      delta3 = delta3.';
      sum1 =  W2.'*delta3;
      sum2 = beta*(-sparsityParam./rhoest + (1 - sparsityParam) ./ (1.0 - rhoest) );
      delta2 = ( sum1 + sum2 ) .* a2(i,:).' .* (1 - a2(i,:).');
      W1grad = W1grad + delta2* a1(i,:);
      W2grad = W2grad + delta3* a2(i,:);
      b1grad = b1grad + delta2;
      b2grad = b2grad + delta3;
  end
  toc
  </code></pre>