title: crash in calloc
body: <p>I'm trying to debug a program I wrote. I ran it inside <code>gdb</code> and I managed to catch a <code>SIGABRT</code> from inside <code>calloc()</code>. I'm completely confused about how this can arise. Can it be a bug in <code>gcc</code> or even <code>libc</code>??</p>
  
  <p>More details: My program uses OpenMP. I ran it through <code>valgrind</code> in single-threaded mode with no errors. I also use <code>mmap()</code> to load a 40GB file, but I doubt that is relevant. Inside <code>gdb</code>, I'm running with 30 threads. Several identical runs (same input&amp;CL) finished correctly, until the problematic one that I caught. On the surface this suggests there might be a race condition of some type. However, the <code>SIGABRT</code> comes from <code>calloc()</code> which is out of my control. Here is some relevant <code>gdb</code> output:</p>
  
  <pre><code>(gdb) info threads
  [...]
    17 Thread 0x7fffec450700 (LWP 73455)  0x00007ffff6b3ce00 in __read_nocancel () from /lib64/libc.so.6
    16 Thread 0x7fffece51700 (LWP 73454)  _mm_slli_si128 (genome=&lt;value optimized out&gt;, goff=&lt;value optimized out&gt;, glen=50, read=&lt;value optimized out&gt;, rlen=36, genome_ls=&lt;value optimized out&gt;, initbp=-1, is_rna=false) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/emmintrin.h:1155
    14 Thread 0x7fffee253700 (LWP 73452)  0x000000000041183e in _mm_prefetch (re=0x7fff291c0f10, st=1, options=0x632014) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/xmmintrin.h:1193
    13 Thread 0x7fffeec54700 (LWP 73451)  0x00007ffff6ae5c84 in __memset_sse2 () from /lib64/libc.so.6
    12 Thread 0x7fffef655700 (LWP 73450)  _mm_cmpeq_epi16 (genome=&lt;value optimized out&gt;, goff=&lt;value optimized out&gt;, glen=33, read=&lt;value optimized out&gt;, rlen=24, genome_ls=&lt;value optimized out&gt;, initbp=-1, is_rna=false) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/emmintrin.h:1263
  * 11 Thread 0x7ffff0056700 (LWP 73449)  0x00007ffff6a948a5 in raise () from /lib64/libc.so.6
    10 Thread 0x7ffff0a57700 (LWP 73448)  _mm_sub_epi16 (genome=&lt;value optimized out&gt;, goff=&lt;value optimized out&gt;, glen=36, read=&lt;value optimized out&gt;, rlen=26, genome_ls=&lt;value optimized out&gt;, initbp=-1, is_rna=false) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/emmintrin.h:1046
    5 Thread 0x7ffff3c5c700 (LWP 73443)  0x000000000041183e in _mm_prefetch (re=0x7fff28615010, st=1, options=0x632014) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/xmmintrin.h:1193
    2 Thread 0x7ffff5a5f700 (LWP 73440)  0x000000000041e4cd in _mm_max_epi16 (genome=&lt;value optimized out&gt;, goff=&lt;value optimized out&gt;, glen=29, read=&lt;value optimized out&gt;, rlen=21, genome_ls=&lt;value optimized out&gt;, initbp=-1, is_rna=false) at /usr/lib/gcc/x86_64-redhat-linux/4.4.6/include/emmintrin.h:1331
    1 Thread 0x7ffff7fdcae0 (LWP 73437)  0x00007ffff6ae5cff in __memset_sse2 () from /lib64/libc.so.6
  [...]
  (gdb) thread 11
  [Switching to thread 11 (Thread 0x7ffff0056700 (LWP 73449))]#0  0x00007ffff6a948a5 in raise () from /lib64/libc.so.6
  (gdb) bt
  #0  0x00007ffff6a948a5 in raise () from /lib64/libc.so.6
  #1  0x00007ffff6a96085 in abort () from /lib64/libc.so.6
  #2  0x00007ffff6ad1fe7 in __libc_message () from /lib64/libc.so.6
  #3  0x00007ffff6ad7916 in malloc_printerr () from /lib64/libc.so.6
  #4  0x00007ffff6adb79f in _int_malloc () from /lib64/libc.so.6
  #5  0x00007ffff6adbdd6 in calloc () from /lib64/libc.so.6
  #6  0x000000000040e87f in my_calloc (re=0x7fff2867ef10, st=0, options=0x632020) at gmapper/../gmapper/../common/my-alloc.h:286
  #7  read_get_hit_list_per_strand (re=0x7fff2867ef10, st=0, options=0x632020) at gmapper/mapping.c:1046
  #8  0x000000000041308a in read_get_hit_list (re=&lt;value optimized out&gt;, options=0x632010, n_options=1) at gmapper/mapping.c:1239
  #9  handle_read (re=&lt;value optimized out&gt;, options=0x632010, n_options=1) at gmapper/mapping.c:1806
  #10 0x0000000000404f35 in launch_scan_threads (.omp_data_i=&lt;value optimized out&gt;) at gmapper/gmapper.c:557
  #11 0x00007ffff7230502 in ?? () from /usr/lib64/libgomp.so.1
  #12 0x00007ffff6dfc851 in start_thread () from /lib64/libpthread.so.0
  #13 0x00007ffff6b4a11d in clone () from /lib64/libc.so.6
  (gdb) f 6   
  #6  0x000000000040e87f in my_calloc (re=0x7fff2867ef10, st=0, options=0x632020) at gmapper/../gmapper/../common/my-alloc.h:286
  286         res = calloc(size, 1);
  (gdb) p size
  $2 = 814080
  (gdb) 
  </code></pre>
  
  <p>The function <code>my_calloc()</code> is just a wrapper, but the problem is not in there, as the real <code>calloc()</code> call looks legit. These are the limits set in the shell:</p>
  
  <pre><code>$ ulimit -a
  core file size          (blocks, -c) 0
  data seg size           (kbytes, -d) unlimited
  scheduling priority             (-e) 0
  file size               (blocks, -f) unlimited
  pending signals                 (-i) 2067285
  max locked memory       (kbytes, -l) 64
  max memory size         (kbytes, -m) unlimited
  open files                      (-n) 1024
  pipe size            (512 bytes, -p) 8
  POSIX message queues     (bytes, -q) 819200
  real-time priority              (-r) 0
  stack size              (kbytes, -s) 10240
  cpu time               (seconds, -t) unlimited
  max user processes              (-u) 1024
  virtual memory          (kbytes, -v) unlimited
  file locks                      (-x) unlimited
  </code></pre>
  
  <p>The program is not out of memory, it's using 41GB on a machine with 256GB available:</p>
  
  <pre><code>$ top -b -n 1 | grep gmapper
   73437 user      20   0 41.5g  16g  15g T  0.0  6.6  55:17.24 gmapper-ls
  $ free -m
               total       used       free     shared    buffers     cached
  Mem:        258437     195567      62869          0         82     189677
  -/+ buffers/cache:       5807     252629
  Swap:            0          0          0
  </code></pre>
  
  <p>I compiled using <code>g++ (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4)</code>, with flags <code>-g -O2 -DNDEBUG -mmmx -msse -msse2 -fopenmp -Wall -Wno-deprecated -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS</code>.</p>
  
  <p><em>Edit</em>: The detailed error message I get is below. I truncated it, lines not shown are similar to the last 5.</p>
  
  <pre><code>*** glibc detected *** /tmp/t/gmapper-ls: corrupted double-linked list: 0x0000000009447380 ***
  *** glibc detected *** /tmp/t/gmapper-ls: corrupted double-linked list: 0x0000000009447380 ***
  ======= Backtrace: =========
  ======= Backtrace: =========
  /lib64/libc.so.6(+0x75916)[0x7ffff6ad7916]
  /lib64/libc.so.6(+0x75916)[0x7ffff6ad7916]
  /lib64/libc.so.6(+0x7979f)[0x7ffff6adb79f]
  /lib64/libc.so.6(+0x7979f)[0x7ffff6adb79f]
  /lib64/libc.so.6(__libc_malloc+0x71)[0x7ffff6adc141]
  /lib64/libc.so.6(__libc_calloc+0xc6)[0x7ffff6adbdd6]
  /usr/lib64/libgomp.so.1(+0x8502)[0x7ffff7230502]
  /usr/lib64/libgomp.so.1(+0x8502)[0x7ffff7230502]
  /lib64/libc.so.6(clone+0x6d)[0x7ffff6b4a11d]
  /lib64/libpthread.so.0(+0x7851)[0x7ffff6dfc851]
  /lib64/libc.so.6(clone+0x6d)[0x7ffff6b4a11d]
  ======= Memory map: ========
  00400000-00430000 r-xp 00000000 00:14 927268870                          /tmp/t/gmapper
  00630000-00631000 rw-p 00030000 00:14 927268870                          /tmp/t/gmapper
  00631000-2f461000 rw-p 00000000 00:00 0                                  [heap]
  7f749d9be000-7f7e2053b000 r--p 00000000 00:0f 1278724                    /dev/shm/hg19-ls
  7fff24000000-7fff2727a000 rw-p 00000000 00:00 0
  7fff2727a000-7fff28000000 ---p 00000000 00:00 0
  7fff285ce000-7fff2c000000 rw-p 00000000 00:00 0
  7fff2c000000-7fff2f547000 rw-p 00000000 00:00 0
  7fff2f547000-7fff30000000 ---p 00000000 00:00 0
  [...]
  </code></pre>
  
  <p><em>Edit2</em>: I added thread info for all the threads which are stopped in what seem to be "nontrivial" places (10/30).</p>