title: Efficient manipulation of large, non-sparse arrays in matlab
body: <p>I have a row vector <code>q</code> with <code>200</code> elements, and another row vector, <code>dij</code>, which is the output of the <code>pdist</code> function with currently <code>48216200</code> elements, but I'd like to be able to go higher. The operation I want to do is essentially:</p>
  <pre><code>t=sum(q'*dij,2);
  </code></pre>
  <p>However, since this tries to allocate a <code>200x48211290</code> array, it complains that this would require 70GB of memory. Therefore I do it this way:</p>
  <pre><code>t = zeros(numel(q),1);
  for i=1:numel(q)
      qi = q(i);
      factor = qi*dij;
      t(i)=sum(factor);
  end
  </code></pre>
  <p>However, this takes too much time. By too much time, I mean it takes about <code>36s</code>, which is orders of magnitude longer than the time required by the <code>pdist</code> function. Is there a way I can speed up this operation without explicitly allocating so much memory? I'm assuming here, that if the first way could allocate the memory, (being a vector operation) it would be faster.</p>