title: Problem of sorting OpenMP threads into NUMA nodes by experiment
body: <p>I'm attempting to create a <code>std::vector&lt;std::set&lt;int&gt;&gt;</code> with one set for each NUMA-node, containing the thread-ids obtained using <code>omp_get_thread_num()</code>.</p>
  <p><strong>Topo</strong>:
  <a href="https://i.stack.imgur.com/0Surw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0Surw.png" alt="enter image description here" /></a></p>
  <p><strong>Idea</strong>:</p>
  <ol>
  <li>Create data which is larger than L3 cache,</li>
  <li>set first touch using thread 0,</li>
  <li>perform multiple experiments to determine the minimum access time of each thread,</li>
  <li>extract the threads into nodes based on sorted access times and information about the topology.</li>
  </ol>
  <p><strong>Code</strong>: (Intel compiler, OpenMP)</p>
  <pre><code>    // create data which will be shared by multiple threads
      const auto part_size = std::size_t{50 * 1024 * 1024 / sizeof(double)}; // 50 MB
      const auto size = 2 * part_size;
      auto container = std::unique_ptr&lt;double&gt;(new double[size]);
      
      // open a parallel section
      auto thread_count = 0;
      auto thread_id_min_duration = std::multimap&lt;double, int&gt;{};
      #ifdef DECIDE_THREAD_COUNT
      #pragma omp parallel num_threads(std::thread::hardware_concurrency())
      #else
      #pragma omp parallel
      #endif
      {
          // perform first touch using thread 0
          const auto thread_id = omp_get_thread_num();
          if (thread_id == 0)
          {
              thread_count = omp_get_num_threads();
              for (auto index = std::size_t{}; index &lt; size; ++index)
              {
                  container.get()[index] = static_cast&lt;double&gt;(std::rand() % 10 + 1);
              }
          }
          #pragma omp barrier
          
          // access the data using all threads individually
          #pragma omp for schedule(static, 1)
          for (auto thread_counter = std::size_t{}; thread_counter &lt; thread_count; ++thread_counter)
          {
              // calculate the minimum access time of this thread
              auto this_thread_min_duration = std::numeric_limits&lt;double&gt;::max();
              for (auto experiment_counter = std::size_t{}; experiment_counter &lt; 250; ++experiment_counter)
              {
                  const auto* data = experiment_counter % 2 == 0 ? container.get() : container.get() + part_size;
                  const auto start_timestamp = omp_get_wtime();
                  for (auto index = std::size_t{}; index &lt; part_size; ++index)
                  {
                      static volatile auto exceedingly_interesting_value_wink_wink = data[index];
                  }
                  const auto end_timestamp = omp_get_wtime();
                  const auto duration = end_timestamp - start_timestamp;
                  if (duration &lt; this_thread_min_duration)
                  {
                      this_thread_min_duration = duration;
                  }
              }
              #pragma omp critical
              {
                  thread_id_min_duration.insert(std::make_pair(this_thread_min_duration, thread_id));
              }
          }
      } // #pragma omp parallel
  </code></pre>
  <p>Not shown here is code which outputs the minimum access times sorted into the multimap.</p>
  <p><strong>Env. and Output</strong></p>
  <ol>
  <li><em>How do <code>OMP_PLACES</code> and <code>OMP_PROC_BIND</code> work?</em></li>
  </ol>
  <p>I am attempting to not use SMT by using <code>export OMP_PLACES=cores OMP_PROC_BIND=spread OMP_NUM_THREADS=24</code>. However, I'm getting this output:</p>
  <p><a href="https://i.stack.imgur.com/pYH9s.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pYH9s.png" alt="enter image description here" /></a></p>
  <p>What's puzzling me is that I'm having the same access times on all threads. Since I'm trying to spread them across the 2 NUMA nodes, I expect to neatly see 12 threads with access time, say, <code>x</code> and another 12 with access time ~<code>2x</code>.</p>
  <ol start="2">
  <li><em>Why is the above happening?</em></li>
  </ol>
  <p><strong>Additional Information</strong></p>
  <p>Even more puzzling are the following environments and their outputs:</p>
  <ol>
  <li><p><code>export OMP_PLACES=cores OMP_PROC_BIND=spread OMP_NUM_THREADS=26</code>
  <a href="https://i.stack.imgur.com/SYiCP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/SYiCP.png" alt="enter image description here" /></a></p>
  </li>
  <li><p><code>export OMP_PLACES=cores OMP_PROC_BIND=spread OMP_NUM_THREADS=48</code>
  <a href="https://i.stack.imgur.com/vTrpo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vTrpo.png" alt="enter image description here" /></a></p>
  </li>
  </ol>
  <p>Any help in understanding this phenomenon would be much appreciated.</p>