title: Matrix multiplication poor efficiency on a 4 socket NUMA system
body: <p>I am developing dense matrix multiplication code (<a href="https://github.com/zboson/gemm" rel="nofollow">https://github.com/zboson/gemm</a>) to learn about parallel programming. I use OpenMP for the threading. My system has four sockets each with Xeon E5-1620 processor. Each processor has 10 cores/20 hyper-threads. So the total is 40 cores/80 hyper threads.  When I run my code on a single thread I get about 70% of the peak flops (13 out of 19.2 GFLOPS).  However, when I run my code using 40 threads I only get about 30% of the peak flops  (185 out of 682.56 GFLOPS). On a seperate system (Sandy Bridge) with only one socket and 4 cores I get about a 65% efficiency with four threads.</p>
  
  <p>I bind the threads to each physical core <a href="https://stackoverflow.com/questions/24862488/thread-affinity-with-windows-msvc-and-openmp">using system calls</a>. I have tried disabling this and using instead <code>export OMP_PROC_BIND=true</code> or <code>export GOMP_CPU_AFFINITY="0 4 8 12 16 20 24 28 32 36 1 5 9 13 17 21 25 29 33 37 2 6 10 14 18 22 26 30 34 38 3 7 11 15 19 23 27 31 35 39"</code> but these makes no difference. I still get about 30% efficiency (though I can get worse efficiency with other bad binding settings).</p>
  
  <p><strong>What more can I do to improve my efficiency?</strong> I understand a <a href="https://stackoverflow.com/questions/10850155/openmp-for-schedule">first touch policy</a> is used so the memory pages are allocated by the first thread that touches them. When I write out the matrix product maybe I should make a separate output for each socket and then merge the results from each socket in the end?</p>
  
  <p>I'm using GCC 4.8.0 with Linux 64-bit kernel 2.6.32</p>
  
  <p>Edit: I use the following binding for matrix size = 2048x2048</p>
  
  <pre><code>export GOMP_CPU_AFFINITY="0 4 8 12 16 20 24 28 32 36 1 5 9 13 17 21 25 29 33 37 2 6 10 14 18 22 26 30 34 38 3 7 11 15 19 23 27 31 35 39"
  </code></pre>
  
  <p>This should have threads 0-9 -> node 0, 10-19 node 1, 20-29 node 2, 30-39 node 3.</p>
  
  <p>With this binding I get:</p>
  
  <pre><code> nthread    efficiency    node
   1          77%           0
   2          76%           0
   4          74%           0
   6          62%           0
   8          64%           0
  10          52%           0
  14          50%           0+1
  16          30%           0+1
  </code></pre>