title: Is it faster to sum the rows or columns of a matrix with CUDA?
body: <p>I want to compute a row-sum of an <code>m x n</code> matrix <code>A</code>, <strong>or equivalently</strong> the column-sum of its transpose <code>A'</code> (I have both in memory so <code>A'</code> costs me nothing extra in computation). I plan to launch <code>m</code> threads each of which can either loop over the <code>n</code> columns of <code>A</code>, or <code>n</code> rows of <code>A'</code>. Which approach will be faster if we assume the matrices are stored in <strong>column-major format</strong> (i.e. like with CUBLAS)?</p>
  
  <p><strong>My thinking so far (on coalesced memory access):</strong></p>
  
  <p>If I row-sum, then <em>threads in the same block</em> will read from adjacent memory locations <em>at each iteration</em>. Yet equally, if I column-sum instead, then <em>each thread</em> will iterate over a contiguous block of memory. So if I have threads <code>1</code>, <code>2</code> and <code>3</code> of the same block, their memory access will look like so (assuming <strong>column-major storage</strong>):</p>
  
  <pre><code>1 2 3 ... 1 2 3 ... 1 2 3 ... for row-sums
  1 1 1 ... 2 2 2 ... 3 3 3 ... for column-sums
  </code></pre>
  
  <ul>
  <li>But this doesn't tell me which will be faster.</li>
  <li>It also doesn't take into account the behavior at block-level (i.e. if the first block launched sums over rows <code>1-32</code>, will the 2nd block launched be guaranteed to sum over rows <code>33-64</code>?)</li>
  </ul>