title: Windows vs Linux - C++ Thread Pool Memory Usage
body: <p>I have been looking at the memory usage of some C++ REST API frameworks in Windows and Linux (Debian). In particular, I have looked at these two frameworks: <a href="https://github.com/microsoft/cpprestsdk" rel="nofollow noreferrer">cpprestsdk</a> and <a href="https://github.com/yhirose/cpp-httplib" rel="nofollow noreferrer">cpp-httplib</a>. In both, a thread pool is created and used to service requests.</p>
  <p>I took the thread pool implementation from <a href="https://github.com/yhirose/cpp-httplib" rel="nofollow noreferrer">cpp-httplib</a> and put it in a minimal working example below, to show the memory usage that I am observing on Windows and Linux.</p>
  <pre class="lang-cpp prettyprint-override"><code>#include &lt;cassert&gt;
  #include &lt;condition_variable&gt;
  #include &lt;functional&gt;
  #include &lt;iostream&gt;
  #include &lt;list&gt;
  #include &lt;map&gt;
  #include &lt;memory&gt;
  #include &lt;mutex&gt;
  #include &lt;string&gt;
  #include &lt;thread&gt;
  #include &lt;vector&gt;
  
  using namespace std;
  
  // TaskQueue and ThreadPool taken from https://github.com/yhirose/cpp-httplib
  class TaskQueue {
  public:
      TaskQueue() = default;
      virtual ~TaskQueue() = default;
  
      virtual void enqueue(std::function&lt;void()&gt; fn) = 0;
      virtual void shutdown() = 0;
  
      virtual void on_idle() {};
  };
  
  class ThreadPool : public TaskQueue {
  public:
      explicit ThreadPool(size_t n) : shutdown_(false) {
          while (n) {
              threads_.emplace_back(worker(*this));
              cout &lt;&lt; &quot;Thread number &quot; &lt;&lt; threads_.size() + 1 &lt;&lt; &quot; has ID &quot; &lt;&lt; threads_.back().get_id() &lt;&lt; endl;
              n--;
          }
      }
  
      ThreadPool(const ThreadPool&amp;) = delete;
      ~ThreadPool() override = default;
  
      void enqueue(std::function&lt;void()&gt; fn) override {
          std::unique_lock&lt;std::mutex&gt; lock(mutex_);
          jobs_.push_back(fn);
          cond_.notify_one();
      }
  
      void shutdown() override {
          // Stop all worker threads...
          {
              std::unique_lock&lt;std::mutex&gt; lock(mutex_);
              shutdown_ = true;
          }
  
          cond_.notify_all();
  
          // Join...
          for (auto&amp; t : threads_) {
              t.join();
          }
      }
  
  private:
      struct worker {
          explicit worker(ThreadPool&amp; pool) : pool_(pool) {}
  
          void operator()() {
              for (;;) {
                  std::function&lt;void()&gt; fn;
                  {
                      std::unique_lock&lt;std::mutex&gt; lock(pool_.mutex_);
  
                      pool_.cond_.wait(
                          lock, [&amp;] { return !pool_.jobs_.empty() || pool_.shutdown_; });
  
                      if (pool_.shutdown_ &amp;&amp; pool_.jobs_.empty()) { break; }
  
                      fn = pool_.jobs_.front();
                      pool_.jobs_.pop_front();
                  }
  
                  assert(true == static_cast&lt;bool&gt;(fn));
                  fn();
              }
          }
  
          ThreadPool&amp; pool_;
      };
      friend struct worker;
  
      std::vector&lt;std::thread&gt; threads_;
      std::list&lt;std::function&lt;void()&gt;&gt; jobs_;
  
      bool shutdown_;
  
      std::condition_variable cond_;
      std::mutex mutex_;
  };
  
  // MWE
  class ContainerWrapper {
  public:
      ~ContainerWrapper() {
          cout &lt;&lt; &quot;Destructor: data map is of size &quot; &lt;&lt; data.size() &lt;&lt; endl;
      }
  
      map&lt;pair&lt;string, string&gt;, double&gt; data;
  };
  
  void handle_post() {
      
      cout &lt;&lt; &quot;Start adding data, thread ID: &quot; &lt;&lt; std::this_thread::get_id() &lt;&lt; endl;
  
      ContainerWrapper cw;
      for (size_t i = 0; i &lt; 5000; ++i) {
          string date = &quot;2020-08-11&quot;;
          string id = &quot;xxxxx_&quot; + std::to_string(i);
          double value = 1.5;
          cw.data[make_pair(date, id)] = value;
      }
  
      cout &lt;&lt; &quot;Data map is now of size &quot; &lt;&lt; cw.data.size() &lt;&lt; endl;
  
      unsigned pause = 3;
      cout &lt;&lt; &quot;Sleep for &quot; &lt;&lt; pause &lt;&lt; &quot; seconds.&quot; &lt;&lt; endl;
      std::this_thread::sleep_for(std::chrono::seconds(pause));
  }
  
  int main(int argc, char* argv[]) {
  
      cout &lt;&lt; &quot;ID of main thread: &quot; &lt;&lt; std::this_thread::get_id() &lt;&lt; endl;
  
      std::unique_ptr&lt;TaskQueue&gt; task_queue(new ThreadPool(40));
  
      for (size_t i = 0; i &lt; 50; ++i) {
          
          cout &lt;&lt; &quot;Add task number: &quot; &lt;&lt; i + 1 &lt;&lt; endl;
          task_queue-&gt;enqueue([]() { handle_post(); });
  
          // Sleep enough time for the task to finish.
          std::this_thread::sleep_for(std::chrono::seconds(5));
      }
  
      task_queue-&gt;shutdown();
  
      return 0;
  }
  </code></pre>
  <p>When I run this MWE and look at the memory consumption in Windows vs Linux, I get the graph below. For Windows, I used <code>perfmon</code> to get the <em>Private Bytes</em> value. In Linux, I used <code>docker stats --no-stream --format &quot;{{.MemUsage}}</code> to log the container's memory usage. This was in line with <code>res</code> for the process from <code>top</code> running inside the container. It appears from the graph that when a thread allocates memory for the <code>map</code> variable in Windows in the <code>handle_post</code> function, that the memory is <em>given back</em> when the function exits before the next call to the function. This was the type of behaviour that I was naively expecting. I have no experience regarding how the OS deals with memory allocated by a function that is being executed in a thread when the thread stays alive i.e. like here in a thread pool. On Linux, it looks like the memory usage keeps growing and that memory is not <em>given back</em> when the function exits. When all 40 threads have been used, and there are 10 more tasks to process, the memory usage appears to stop growing. Can somebody give a high level view of what is happening here in Linux from a memory management point of view or even some pointers about where to look for some background info on this specific topic?</p>
  <p><strong>Edit 1</strong>: I have edited the graph below to show the output value of <code>rss</code> from running <code>ps -p &lt;pid&gt; -h -o etimes,pid,rss,vsz</code> every second in the Linux container where <code>&lt;pid&gt;</code> is the id of the process being tested. It is in reasonable agreement with the output of <code>docker stats --no-stream --format &quot;{{.MemUsage}}</code>.</p>
  <p><a href="https://i.stack.imgur.com/nqs1F.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nqs1F.png" alt="win_v_lin_50_seq_tasks_40_threads_rss" /></a></p>
  <p><strong>Edit 2</strong>: Based on a comment below regarding STL allocators, I removed the map from MWE by replacing the <code>handle_post</code> function with the following and adding the includes <code>#include &lt;cstdlib&gt;</code> and <code>#include &lt;cstring&gt;</code>. Now, the <code>handle_post</code> function just allocates and sets memory for 500K <code>int</code>s which is approximately 2MiB.</p>
  <pre class="lang-cpp prettyprint-override"><code>void handle_post() {
      
      size_t chunk = 500000 * sizeof(int);
      if (int* p = (int*)malloc(chunk)) {
  
          memset(p, 1, chunk);
          cout &lt;&lt; &quot;Allocated and used &quot; &lt;&lt; chunk &lt;&lt; &quot; bytes, thread ID: &quot; &lt;&lt; this_thread::get_id() &lt;&lt; endl;
          cout &lt;&lt; &quot;Memory address: &quot; &lt;&lt; p &lt;&lt; endl;
  
          unsigned pause = 3;
          cout &lt;&lt; &quot;Sleep for &quot; &lt;&lt; pause &lt;&lt; &quot; seconds.&quot; &lt;&lt; endl;
          this_thread::sleep_for(chrono::seconds(pause));
  
          free(p);
      }
  }
  </code></pre>
  <p>I get the same behaviour here. I reduced the number of threads to 8 and the number of tasks to 10 in the example. The graph below shows the results.</p>
  <p><strong>Edit 3</strong>: I have added the results from running on a Linux CentOS machine. It broadly agrees with the results from the Debian docker image result.</p>
  <p><a href="https://i.stack.imgur.com/bH86a.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bH86a.png" alt="8_threads_10_seq_tasks_e3" /></a></p>
  <p><strong>Edit 4</strong>: Based on another comment below, I ran the example under <code>valgrind</code>'s <code>massif</code> tool. The <code>massif</code> command line parameters are in the images below. I ran it with <code>--pages-as-heap=yes</code>, second image below, and without this flag, first image below. The first image would suggest that ~2MiB memory is allocated to the (shared) heap as the <code>handle_post</code> function is executed on a thread and then freed as the function exits. This is what I would expect and what I observe on Windows. I am not sure how to interpret the graph with <code>--pages-as-heap=yes</code> yet, i.e. the second image.</p>
  <p>I can't reconcile the output of <code>massif</code> in the first image with the value of <code>rss</code> from the <code>ps</code> command shown in the graphs above. If I run the Docker image and limit the container memory to 12MB using <code>docker run --rm -it --privileged --memory=&quot;12m&quot; --memory-swap=&quot;12m&quot; --name=mwe_test cpp_testing:1.0</code>, the container runs out of memory on the 7th allocation and is killed by the OS. I get <code>Killed</code> in the output and when I look at <code>dmesg</code>, I see <code>Killed process 25709 (cpp_testing) total-vm:529960kB, anon-rss:10268kB, file-rss:2904kB, shmem-rss:0kB</code>. This would suggest that the <code>rss</code> value from <code>ps</code> is accurately reflecting the (heap) memory <em>actually</em> being used by the process whereas the <code>massif</code> tool is calculating what it <em>should</em> be based on <code>malloc</code>/<code>new</code> and <code>free</code>/<code>delete</code> calls. This is just my basic assumption from this test. My question would still stand i.e. why is, or does it appear that, the heap memory is not being freed or deallocated when the <code>handle_post</code> function exits?</p>
  <p><a href="https://i.stack.imgur.com/vdSds.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vdSds.png" alt="massif_output" /></a></p>
  <p><strong>Edit 5</strong>: I have added below a graph of the memory usage as you increase the number of threads in the thread pool from 1 to 4. The pattern continues as you increase the number of threads up to 10 so I have not included 5 to 10. Note that I have added a 5 sec pause at the start of <code>main</code> which is the initial flat line in the graph for the first ~5secs. It appears that, regardless of thread count, there is a release of memory after the first task is processed but that memory is not released (kept for reuse?) after task 2 through 10. It may suggest that some memory allocation parameter is tuned during task 1 execution (just thinking out loud!)?</p>
  <p><a href="https://i.stack.imgur.com/vDq4c.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vDq4c.png" alt="increase_num_threads" /></a></p>
  <p><strong>Edit 6</strong>: Based on the suggestion from the detailed answer <a href="https://stackoverflow.com/a/63497719/1771882">below</a>, I set the environment variable <code>MALLOC_ARENA_MAX</code> to 1 and 2 before running the example. This gives the output in the following graph. This is as expected based on the explanation of the effect of this variable given in the answer.</p>
  <p><a href="https://i.stack.imgur.com/gZHEk.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gZHEk.png" alt="effect_of_malloc_arena_max" /></a></p>