title: C++ uses twice the memory when moving elements from one dequeue to another
body: <p>In my project, I use <a href="https://github.com/pybind/pybind11" rel="nofollow">pybind11</a> to bind C++ code to Python. Recently I have had to deal with very large data sets (70GB+) and encountered need to split data from one <code>std::deque</code> between multiple <code>std::deque</code>'s. Since my dataset is so large, I expect the split not to have much of memory overhead. Therefore I went for one pop - one push strategy, which in general should ensure that my requirements are met. </p>
  
  <p>That is all in theory. In practice, my process got killed. So I struggled for past two days and eventually came up with following minimal example demonstrating the problem.</p>
  
  <p>Generally the minimal example creates bunch of data in <code>deque</code> (~11GB), returns it to Python, then calls again to <code>C++</code> to move the elements. Simple as that. Moving part is done in executor.</p>
  
  <p>The interesting thing is, that if I don't use executor, memory usage is as expected and also when limits on virtual memory by ulimit are imposed, the program really respects these limits and doesn't crash.</p>
  
  <p><strong>test.py</strong></p>
  
  <pre><code>from test import _test
  import asyncio
  import concurrent
  
  async def test_main(loop, executor):
      numbers = _test.generate()
      # moved_numbers = _test.move(numbers) # This works!
      moved_numbers = await loop.run_in_executor(executor, _test.move, numbers) # This doesn't!
  
  if __name__ == '__main__':
      loop = asyncio.get_event_loop()
      executor = concurrent.futures.ThreadPoolExecutor(1)
  
      task = loop.create_task(test_main(loop, executor))
      loop.run_until_complete(task)
  
      executor.shutdown()
      loop.close()
  </code></pre>
  
  <p><strong>test.cpp</strong></p>
  
  <pre><code>#include &lt;deque&gt;
  #include &lt;iostream&gt;
  #include &lt;pybind11/pybind11.h&gt;
  #include &lt;pybind11/stl.h&gt;
  
  namespace py = pybind11;
  
  PYBIND11_MAKE_OPAQUE(std::deque&lt;uint64_t&gt;);
  PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr&lt;T&gt;);
  
  template&lt;class T&gt;
  void py_bind_opaque_deque(py::module&amp; m, const char* type_name) {
      py::class_&lt;std::deque&lt;T&gt;, std::shared_ptr&lt;std::deque&lt;T&gt;&gt;&gt;(m, type_name)
      .def(py::init&lt;&gt;())
      .def(py::init&lt;size_t, T&gt;());
  }
  
  PYBIND11_PLUGIN(_test) {
      namespace py = pybind11;
      pybind11::module m("_test");
      py_bind_opaque_deque&lt;uint64_t&gt;(m, "NumbersDequeue");
  
      // Generate ~11Gb of data.
      m.def("generate", []() {
          std::deque&lt;uint64_t&gt; numbers;
          for (uint64_t i = 0; i &lt; 1500 * 1000000; ++i) {
              numbers.push_back(i);
          }
          return numbers;
      });
  
      // Move data from one dequeue to another.
      m.def("move", [](std::deque&lt;uint64_t&gt;&amp; numbers) {
          std::deque&lt;uint64_t&gt; numbers_moved;
  
          while (!numbers.empty()) {
              numbers_moved.push_back(std::move(numbers.back()));
              numbers.pop_back();
          }
          std::cout &lt;&lt; "Done!\n";
          return numbers_moved;
      });
  
      return m.ptr();
  }
  </code></pre>
  
  <p><strong>test/__init__.py</strong></p>
  
  <pre><code>import warnings
  warnings.simplefilter("default")
  </code></pre>
  
  <p><strong>Compilation</strong>:</p>
  
  <pre><code>g++ -std=c++14 -O2 -march=native -fPIC -Iextern/pybind11 `python3.5-config --includes` `python3.5-config --ldflags` `python3.5-config --libs` -shared -o test/_test.so test.cpp
  </code></pre>
  
  <p><strong>Observations:</strong></p>
  
  <ul>
  <li>When the moving part is not done by executor, so we just call <code>moved_numbers = _test.move(numbers)</code>, all works as expected, memory usage showed by htop stays around <code>11Gb</code>, great!.</li>
  <li>When moving part is done in executor, the program takes double the memory and crashes.</li>
  <li><p>When limits on virtual memory are introduced (~15Gb), all works fine, which is probably the most interesting part.</p>
  
  <p><code>ulimit -Sv 15000000 &amp;&amp; python3.5 test.py</code> >> <code>Done!</code>.</p></li>
  <li><p>When we increase the limit the program crashes (150Gb > my RAM).</p>
  
  <p><code>ulimit -Sv 150000000 &amp;&amp; python3.5 test.py</code> >> <code>[1]    2573 killed     python3.5 test.py</code></p></li>
  <li><p>Usage of deque method <code>shrink_to_fit</code> doesn't help (And nor it should)</p></li>
  </ul>
  
  <p><strong>Used software</strong></p>
  
  <pre><code>Ubuntu 14.04
  gcc version 5.4.1 20160904 (Ubuntu 5.4.1-2ubuntu1~14.04)
  Python 3.5.2
  pybind11 latest release - v1.8.1
  </code></pre>
  
  <p><strong>Note</strong></p>
  
  <p>Please note that this example was made merely to demonstrate the problem. Usage of <code>asyncio</code> and <code>pybind</code> is necessary for problem to occur.  </p>
  
  <p>Any ideas on what might be going on are most welcomed.</p>