title: GDB debug output for multi-thread program
body: <p>All,</p>
  
  <p>I am debuging a 24-thread program with GDB, now I have find which line in the code the error occurs, but I cannot tell what the error is from the output of GDB. The followsing line of code leads to the error, it's just a normal insertion to a map structure.</p>
  
  <pre><code>current_node-&gt;children.insert(std::pair&lt;string, ComponentTrieNode*&gt;(comps[j], temp_node));
  </code></pre>
  
  <p>I used GDB to find out in which thread the error happens and switched to that thread, the <code>backtrace</code> command shows the function calls in the stack. (The last several lines try to print the value of some variables in a function, but failed.)</p>
  
  <p>What should I do to clear know what error is happening?</p>
  
  <pre><code>[root@localhost nameComponentEncoding]# gdb NCE_david
  GNU gdb (GDB) Fedora (7.2.90.20110429-36.fc15)
  Copyright (C) 2011 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu".
  For bug reporting instructions, please see:
  &lt;http://www.gnu.org/software/gdb/bugs/&gt;...
  Reading symbols from /mnt/disk2/experiments_BLOODMOON/two_stage_bloom_filter/programs/nameComponentEncoding/NCE_david...done.
  (gdb) r /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt
  Starting program: /mnt/disk2/experiments_BLOODMOON/two_stage_bloom_filter/programs/nameComponentEncoding/NCE_david /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt
  [Thread debugging using libthread_db enabled]
  [New Thread 0x7fffd2bf5700 (LWP 13129)]
  [New Thread 0x7fffd23f4700 (LWP 13130)]
  [New Thread 0x7fffd1bf3700 (LWP 13131)]
  [New Thread 0x7fffd13f2700 (LWP 13132)]
  [New Thread 0x7fffd0bf1700 (LWP 13133)]
  [New Thread 0x7fffd03f0700 (LWP 13134)]
  [New Thread 0x7fffcfbef700 (LWP 13135)]
  [New Thread 0x7fffcf3ee700 (LWP 13136)]
  [New Thread 0x7fffcebed700 (LWP 13137)]
  [New Thread 0x7fffce3ec700 (LWP 13138)]
  [New Thread 0x7fffcdbeb700 (LWP 13139)]
  [New Thread 0x7fffcd3ea700 (LWP 13140)]
  [New Thread 0x7fffccbe9700 (LWP 13141)]
  [New Thread 0x7fffcc3e8700 (LWP 13142)]
  [New Thread 0x7fffcbbe7700 (LWP 13143)]
  [New Thread 0x7fffcb3e6700 (LWP 13144)]
  [New Thread 0x7fffcabe5700 (LWP 13145)]
  [New Thread 0x7fffca3e4700 (LWP 13146)]
  [New Thread 0x7fffc9be3700 (LWP 13147)]
  [New Thread 0x7fffc93e2700 (LWP 13148)]
  [New Thread 0x7fffc8be1700 (LWP 13149)]
  [New Thread 0x7fffc83e0700 (LWP 13150)]
  [New Thread 0x7fffc7bdf700 (LWP 13151)]
  this is thread 1
  this is thread 7
  this is thread 14
  this is thread 18
  this is thread 2
  this is thread 19
  this is thread 6
  this is thread 8
  this is thread 24
  base: 64312646
  this is thread 11
  this is thread 5
  this is thread 12
  this is thread 13
  this is thread 3
  this is thread 15
  this is thread 16
  this is thread 17
  this is thread 4
  this is thread 20
  this is thread 21
  this is thread 22
  this is thread 23
  this is thread 9
  this is thread 10
  
  Program received signal SIGSEGV, Segmentation fault.
  [Switching to Thread 0x7fffc8be1700 (LWP 13149)]
  std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=@0x608118) at ../../../../libstdc++-v3/src/tree.cc:126
  126         __x-&gt;_M_right = __y-&gt;_M_left;
  (gdb) info threads
    Id   Target Id         Frame
    24   Thread 0x7fffc7bdf700 (LWP 13151) "NCE_david" compare (__n=&lt;optimized out&gt;, __s2=&lt;optimized out&gt;, __s1=&lt;optimized out&gt;)
      at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/char_traits.h:257
    (... other 22 threads not listed)
    2    Thread 0x7fffd2bf5700 (LWP 13129) "NCE_david" compare (__n=&lt;optimized out&gt;, __s2=&lt;optimized out&gt;, __s1=&lt;optimized out&gt;)
      at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/char_traits.h:257
    1    Thread 0x7ffff7fe57a0 (LWP 13126) "NCE_david" strtok () at ../sysdeps/x86_64/strtok.S:76
  (gdb) thread 22
  [Switching to thread 22 (Thread 0x7fffc8be1700 (LWP 13149))]
  #0  std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=@0x608118) at ../../../../libstdc++-v3/src/tree.cc:126
  126         __x-&gt;_M_right = __y-&gt;_M_left;
  
  (gdb) bt
  #0  std::local_Rb_tree_rotate_left (__x=0xa057c90, __root=@0x608118) at ../../../../libstdc++-v3/src/tree.cc:126
  #1  0x0000003cdd26e848 in std::_Rb_tree_insert_and_rebalance (__insert_left=&lt;optimized out&gt;, __x=0x7fffc0005ba0, __p=&lt;optimized out&gt;, __header=...)
      at ../../../../libstdc++-v3/src/tree.cc:266
  #2  0x00000000004029ca in std::_Rb_tree&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::pair&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, ComponentTrieNode*&gt;, std::_Select1st&lt;std::pair&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, ComponentTrieNode*&gt; &gt;, std::less&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, ComponentTrieNode*&gt; &gt; &gt;::_M_insert_ (this=0x608108, __x=&lt;optimized out&gt;, __p=0x16cd3e30, __v=...)
      at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/stl_pair.h:87
  #3  0x0000000000402b7d in std::_Rb_tree&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::pair&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, ComponentTrieNode*&gt;, std::_Select1st&lt;std::pair&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, ComponentTrieNode*&gt; &gt;, std::less&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, ComponentTrieNode*&gt; &gt; &gt;::_M_insert_unique (this=0x608108, __v=...)
      at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/stl_tree.h:1281
  #4  0x000000000040444c in insert (__x=..., this=0x608108) at /usr/lib/gcc/x86_64-redhat-linux/4.6.0/../../../../include/c++/4.6.0/bits/stl_map.h:518
  #5  ComponentTrie::add_prefix (this=0x7fffffffe2e0, prefix_input=&lt;optimized out&gt;, port=10) at ComponentTrie_david.cpp:112
  #6  0x0000000000401c3b in main._omp_fn.0 () at NameComponentEncoding_david.cpp:277
  #7  0x0000003cd2607fea in gomp_thread_start (xdata=&lt;optimized out&gt;) at ../../../libgomp/team.c:115
  #8  0x0000003cd0607cd1 in start_thread (arg=0x7fffc8be1700) at pthread_create.c:305
  #9  0x0000003cd02dfd3d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
  
  (gdb) p 'ComponentTrie::add_prefix(char*, int)'::comps[j]
  No symbol "comps" in specified context.
  (gdb) p 'ComponentTrie::add_prefix(char*, int)'::prefix
  No symbol "prefix" in specified context.
  </code></pre>
  
  <p>Edit: I have run the code with <code>valgrind --tool=memcheck</code>, the following is the result.</p>
  
  <pre><code>[root@localhost nameComponentEncoding]# valgrind --tool=memcheck ./NCE_david /mnt/disk2/FIB_with_port/10_1.txt /mnt/disk2/trace/a_10_1.trace /mnt/disk2/FIB_with_port/10_2.txt
  (... many lines omitted)
  ==13261==
  ==13261== Thread 11:
  ==13261== Invalid read of size 1
  ==13261==    at 0x3CD02849BC: strtok (strtok.S:141)
  ==13261==    by 0x40426A: ComponentTrie::add_prefix(char*, int) (ComponentTrie_david.cpp:99)
  ==13261==    by 0x40242C: main._omp_fn.0 (NameComponentEncoding_david.cpp:531)
  ==13261==    by 0x3CD2607FE9: gomp_thread_start (team.c:115)
  ==13261==    by 0x3CD0607CD0: start_thread (pthread_create.c:305)
  ==13261==    by 0x3CD02DFD3C: clone (clone.S:115)
  ==13261==  Address 0x234422c02 is not stack'd, malloc'd or (recently) free'd
  ==13261==
  ==13261== Invalid read of size 1
  ==13261==    at 0x3CD02849EC: strtok (strtok.S:167)
  ==13261==    by 0x40426A: ComponentTrie::add_prefix(char*, int) (ComponentTrie_david.cpp:99)
  ==13261==    by 0x40242C: main._omp_fn.0 (NameComponentEncoding_david.cpp:531)
  ==13261==    by 0x3CD2607FE9: gomp_thread_start (team.c:115)
  ==13261==    by 0x3CD0607CD0: start_thread (pthread_create.c:305)
  ==13261==    by 0x3CD02DFD3C: clone (clone.S:115)
  ==13261==  Address 0x234422c02 is not stack'd, malloc'd or (recently) free'd
  ==13261==
  Insertion and lookup cost time(us): 994669532   67108864        14.821731       0.067469
  component number:4849478, state number: 2545847
  Parallel threads:24
  ==13261==
  ==13261== HEAP SUMMARY:
  ==13261==     in use at exit: 4,239,081,584 bytes in 76,746,193 blocks
  ==13261==   total heap usage: 80,050,114 allocs, 3,303,921 frees, 4,323,622,103 bytes allocated
  ==13261==
  ==13261== LEAK SUMMARY:
  ==13261==    definitely lost: 0 bytes in 0 blocks
  ==13261==    indirectly lost: 0 bytes in 0 blocks
  ==13261==      possibly lost: 4,111,951,106 bytes in 74,746,429 blocks
  ==13261==    still reachable: 127,130,478 bytes in 1,999,764 blocks
  ==13261==         suppressed: 0 bytes in 0 blocks
  ==13261== Rerun with --leak-check=full to see details of leaked memory
  ==13261==
  ==13261== For counts of detected and suppressed errors, rerun with: -v
  ==13261== Use --track-origins=yes to see where uninitialised values come from
  ==13261== ERROR SUMMARY: 45 errors from 30 contexts (suppressed: 6 from 6)
  </code></pre>