title: c++ boost MPI & threading -  serialize errors: Address not mapped
body: <p>I'm stumped.  <code>all_gather</code> works for primitives (e.g. <code>int</code>) but fails even for simple STL containers.  valgrind claims that the container was not allocated/initialized, but that doesn't seem right.</p>
  
  <p>In summary:</p>
  
  <ul>
  <li>I do some multi-threading with openMP, then rejoin threads.</li>
  <li>In serial, I try to <code>all_gather</code> a simple <code>std::map</code> using `boost::mpi::all_gather.  The MPI ranks are <em>not</em> the threads. (There are 2 MPI ranks, and each MPI rank has 4 threads).</li>
  <li>Then I intend to do some more (isolated) multi-threading.</li>
  </ul>
  
  <p>It seems so straightforward... what could possibly be going on here?</p>
  
  <p>main.cpp</p>
  
  <pre><code>#include &lt;openmpi/mpi.h&gt;
  #include &lt;omp.h&gt;
  #include &lt;boost/mpi.hpp&gt;    
  #include "globals.h"
  
  int main(int argc, char* argv[])
  {        
  
      int provided_MPI;
      MPI_Init_thread(&amp;argc, &amp;argv, MPI_THREAD_MULTIPLE, &amp;provided_MPI );
  
      boost::mpi::environment my_boost_mpi_env(argc, argv);
      boost::mpi::communicator world_MPI_boost;        
      world_MPI_boost_ptr = &amp;world_MPI_boost;
          // ^^^ global variable of type   boost::mpi::communicator *
  
      perform_complete_variable_elimination_schedule();
      //...
  
  }
  </code></pre>
  
  <p>Conn_Comp.cpp</p>
  
  <pre><code>#include &lt;boost/mpi.hpp&gt;    
  #include &lt;boost/mpi/collectives.hpp&gt;
  #include &lt;boost/serialization/serialization.hpp&gt;
  #include &lt;boost/serialization/vector.hpp&gt;
  #include &lt;boost/serialization/map.hpp&gt;
  
  #include "globals.h"
  
  ...
  
  void perform_complete_variable_elimination_schedule()
  {
  
      // isolated work in parallel using OpenMP
      #pragma omp parallel
      { 
      //work
      }    
  
      // SERIAL REGION (with respect to threading).
  
      std::map&lt;uint,uint&gt; my_map;
      std::vector&lt; std::map&lt;uint,uint&gt; &gt;   vec_of_my_maps;
  
      boost::mpi::all_gather&lt;    std::map&lt;uint,uint&gt;    &gt;
                       (*world_MPI_boost_ptr,
                        my_map,
                        vec_of_my_maps);  //  &lt;--- line 293 (referenced by valgrind)
  
  
      // more isolated work in parallel using OpenMP
      #pragma omp parallel
      { 
      //work
      }
  
  }
  </code></pre>
  
  <p>valgrind complains that the <code>vector</code> of <code>map</code>  results in an invalid read.  But this <code>vector</code> was created immediately preceding the <code>all_gather</code> call - so it is obviously in scope and not in parallel-threaded region.
  selected valgrind error output:</p>
  
  <pre><code>==12665== Use of uninitialised value of size 4
  ==12665==    at 0x41C8D7A: boost::archive::detail::basic_iarchive::get_library_version() const (basic_iarchive.cpp:575)
  ==12665==    by 0x41C92C6: boost::archive::detail::basic_iarchive::load_object(void*, boost::archive::detail::basic_iserializer const&amp;) (basic_iarchive.cpp:399)
  ==12665==    by 0x80F5696: void boost::mpi::all_gather&lt;std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt; &gt;(boost::mpi::communicator const&amp;, std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt; const&amp;, std::vector&lt;std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt;, std::allocator&lt;std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt; &gt; &gt;&amp;) (iserializer.hpp:387)
  ==12665==    by 0x80DEC83: Conn_Comp::perform_complete_variable_elimination_schedule() (Conn_Comp.cpp:**293**)
  ==12665==    by 0x80C840A: main (main.cpp:695)
  ==12665== 
  ==12665== Invalid read of size 2
  ==12665==    at 0x41C8D7A: boost::archive::detail::basic_iarchive::get_library_version() const (basic_iarchive.cpp:575)
  ==12665==    by 0x41C92C6: boost::archive::detail::basic_iarchive::load_object(void*, boost::archive::detail::basic_iserializer const&amp;) (basic_iarchive.cpp:399)
  ==12665==    by 0x80F5696: void boost::mpi::all_gather&lt;std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt; &gt;(boost::mpi::communicator const&amp;, std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt; const&amp;, std::vector&lt;std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt;, std::allocator&lt;std::map&lt;unsigned int, unsigned int, std::less&lt;unsigned int&gt;, std::allocator&lt;std::pair&lt;unsigned int const, unsigned int&gt; &gt; &gt; &gt; &gt;&amp;) (iserializer.hpp:387)
  ==12665==    by 0x80DEC83: Conn_Comp::perform_complete_variable_elimination_schedule() (main.cpp:**293**)
  ==12665==    by 0x80C840A: main (main.cpp:695)
  ==12665==  Address 0x3580bece is not stack'd, malloc'd or (recently) free'd
  ==12665== 
  [drosphila:12665] *** Process received signal ***
  [drosphila:12665] Signal: Segmentation fault (11)
  [drosphila:12665] Signal code: Address not mapped (1)
  [drosphila:12665] Failing at address: 0x3580bece
  [drosphila:12665] [ 0] /lib/i686/cmov/libpthread.so.0(+0xe500) [0x44f8500]
  [drosphila:12665] [ 1] /usr/lib/libboost_serialization.so.1.42.0(_ZN5boost7archive6detail14basic_iarchive11load_objectEPvRKNS1_17basic_iserializerE+0x1b7) [0x41c92c7]
  [drosphila:12665] [ 2] ./detect_NAHR(_ZN5boost3mpi10all_gatherISt3mapIjjSt4lessIjESaISt4pairIKjjEEEEEvRKNS0_12communicatorERKT_RSt6vectorISD_SaISD_EE+0x587) [0x80f5697]
  [drosphila:12665] [ 3] ./detect_NAHR(_ZN9Conn_Comp46perform_complete_variable_elimination_scheduleEv+0x534) [0x80dec84]
  [drosphila:12665] [ 4] ./detect_NAHR(main+0xf5b) [0x80c840b]
  [drosphila:12665] [ 5] /lib/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x4519ca6]
  [drosphila:12665] [ 6] ./detect_NAHR() [0x80c73e1]
  [drosphila:12665] *** End of error message ***
  </code></pre>
  
  <p>I use MPI_Init_thread based on a recommendation from a <em>boost</em> <a href="http://lists.boost.org/boost-users/2010/12/64752.php" rel="nofollow">help page</a>.</p>
  
  <p>As I said at the top, if I use a primitive (i.e. just <code>uint</code>) instead of a map, then the <code>all_gather</code> works fine.  Why should the map fail?  <code>boost serialize</code> already has methods for serializing STL containers, so that is not the problem...</p>
  
  <p>Note also that the vector which will hold all of the values is automatically resized in <code>all_gather</code> (I checked the implementation for <code>all_gather</code>) to be big enough to hold everything.  regardless, even if I initialize it myself, it still fails.</p>
  
  <p>Finally, even if I use a plain old array (properly allocated) e.g. <code>std::map&lt;uint,uint&gt; *</code>, I get the same problem.</p>