title: dmesg shows segfault, general protection error, "blocked for more than 120 seconds", but program is still running
body: <p>i'm running some code and whilst it hasn't at any point exited or given my a visible error message, it seems to be stalling for large periods of time, and then picks up again, and then stalls, so i ran dmesg and got the following:</p>
  
  <pre><code>mount: server appcs.ra.phy.cam.ac.uk not responding, timed out
  TempoNest[951]: segfault at 7fffea89cd50 ip 0000000000417a04 sp 00007fffea89cd50 error 6 in TempoNest[400000+a9000]
  TempoNest[827]: segfault at 41 ip 000000000040ee2e sp 00007fff2b55fff0 error 4 in TempoNest[400000+a9000]
  TempoNest[850] general protection ip:40ee2e sp:7fffc92e9910 error:0 in TempoNest[400000+a9000]
  TempoNest[872]: segfault at 41 ip 000000000040ee2e sp 00007fff7c34fcf0 error 4 in TempoNest[400000+a9000]
  INFO: task TempoNest:6147 blocked for more than 120 seconds.
  "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  TempoNest     D 0000000000000001     0  6147   5924 0x00000080
   ffff88010de03c78 0000000000000086 0000000000000000 ffffffffa0799e20
   ffff88010de03be8 ffffffff81012a69 ffff88010de03c28 ffffffff8109b669
   ffff8802496e45f8 ffff88010de03fd8 000000000000fb88 ffff8802496e45f8
  Call Trace:
   [&lt;ffffffff81012a69&gt;] ? read_tsc+0x9/0x20
   [&lt;ffffffff8109b669&gt;] ? ktime_get_ts+0xa9/0xe0
   [&lt;ffffffff8109b669&gt;] ? ktime_get_ts+0xa9/0xe0
   [&lt;ffffffff8110ffe0&gt;] ? sync_page+0x0/0x50
   [&lt;ffffffff814ea7a3&gt;] io_schedule+0x73/0xc0
   [&lt;ffffffff8111001d&gt;] sync_page+0x3d/0x50
   [&lt;ffffffff814eb15f&gt;] __wait_on_bit+0x5f/0x90
   [&lt;ffffffff81110253&gt;] wait_on_page_bit+0x73/0x80
   [&lt;ffffffff81090c20&gt;] ? wake_bit_function+0x0/0x50
   [&lt;ffffffff81126665&gt;] ? pagevec_lookup_tag+0x25/0x40
   [&lt;ffffffff811106cb&gt;] wait_on_page_writeback_range+0xfb/0x190
   [&lt;ffffffff81110898&gt;] filemap_write_and_wait_range+0x78/0x90
   [&lt;ffffffff811a50be&gt;] vfs_fsync_range+0x7e/0xe0
   [&lt;ffffffff811a518d&gt;] vfs_fsync+0x1d/0x20
   [&lt;ffffffffa075f590&gt;] nfs_file_flush+0x70/0xa0 [nfs]
   [&lt;ffffffff81173abc&gt;] filp_close+0x3c/0x90
   [&lt;ffffffff81173bb5&gt;] sys_close+0xa5/0x100
   [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[1582]: segfault at 7ffff058d1f0 ip 000000000042b54d sp 00007ffff058d1f0 error 6 in TempoNest[400000+ca000]
  TempoNest[21147]: segfault at 0 ip 0000000000411b5a sp 00007fff61aa2f90 error 4 in TempoNest[400000+ca000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[6648] general protection ip:370f29d54d sp:7fff32979770 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[6669] general protection ip:370f29d54d sp:7fff7bd3c8b0 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[7787] general protection ip:370f29d54d sp:7fff94517280 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[7817] general protection ip:370f29d54d sp:7fff3bc06550 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[7842]: segfault at 4000 ip 00000037082480ac sp 00007fff53684760 error 4 in libc-2.12.so[3708200000+18a000]
  TempoNest[7861] general protection ip:370f29d54d sp:7fff418fa420 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[7971] general protection ip:370f29d54d sp:7fff76550640 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[7990] general protection ip:370f29d54d sp:7fffde5f2d60 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9004] general protection ip:370f29d54d sp:7fff4bfae000 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9025] general protection ip:370f29d54d sp:7fffb34f3c10 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9041] general protection ip:370f29d54d sp:7fff86a320b0 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9059] general protection ip:370f29d54d sp:7fff913794a0 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9074] general protection ip:370f29d54d sp:7ffff942db20 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9189] general protection ip:370f29d54d sp:7fffac340900 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[9208] general protection ip:370f29d54d sp:7fffc51ba760 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[10238] general protection ip:370f29d54d sp:7fff2aeef770 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[10293] general protection ip:370f29d25b sp:7fffadc78430 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[12364] general protection ip:370f29d54d sp:7fffd1c6b480 error:0 in libstdc++.so.6.0.13[370f200000+e8000]
  TempoNest[12424]: segfault at 0 ip 000000000040bbbf sp 00007fff673b4710 error 4 in TempoNest[400000+ce000]
  TempoNest[14531]: segfault at 0 ip 000000000040bbbc sp 00007fff8011c6d0 error 4 in TempoNest[400000+ce000]
  TempoNest[15636] trap invalid opcode ip:40bd9f sp:7fff24c35210 error:0 in TempoNest[400000+cd000]
  TempoNest[15654]: segfault at 0 ip 000000000040bbcd sp 00007fff60986920 error 4 in TempoNest[400000+ce000]
  TempoNest[15673]: segfault at 0 ip 000000000040bbca sp 00007fff76439760 error 4 in TempoNest[400000+ce000]
  TempoNest[7766] trap invalid opcode ip:40be74 sp:7fff998a57c0 error:0 in TempoNest[400000+cd000]
  TempoNest[14083]: segfault at 0 ip 000000000040bc30 sp 00007fff32bdce00 error 6 in TempoNest[400000+ce000]
  TempoNest[14107]: segfault at 0 ip 000000000040bc48 sp 00007fffe1fbfa40 error 6 in TempoNest[400000+ce000]
  TempoNest[14135]: segfault at 72 ip 00000037082480ac sp 00007fff7caaec70 error 4 in libc-2.12.so[3708200000+18a000]
  TempoNest[15883]: segfault at e70000000a ip 00002b388fcfcee0 sp 00007fff472acab0 error 4 in libtempo2.so.0.0.0[2b388fcec000+7b000]
  TempoNest[15897]: segfault at 8 ip 00002aff15244ee0 sp 00007fffaee3bfd0 error 4 in libtempo2.so.0.0.0[2aff15234000+7b000]
  TempoNest[16894]: segfault at 8 ip 00002afc90140ee0 sp 00007fff5d47cfe0 error 4 in libtempo2.so.0.0.0[2afc90130000+7b000]
  TempoNest[18342]: segfault at 0 ip 00000000004186c3 sp 00007fff2cfd31f0 error 4 in TempoNest[400000+cf000]
  TempoNest[23959]: segfault at 8 ip 00002ada3c54fee0 sp 00007ffffdf05420 error 4 in libtempo2.so.0.0.0[2ada3c53f000+7b000]
  TempoNest[19067] general protection ip:4098a5 sp:7fff31e19308 error:0 in TempoNest[400000+cf000]
  TempoNest[28229]: segfault at 0 ip 0000000000406410 sp 00007fff4c7fe9b0 error 6 in TempoNest[400000+b7000]
  TempoNest[29797]: segfault at 0 ip 0000000000418ebb sp 00007fff4da33850 error 4 in TempoNest[400000+cf000]
  TempoNest[26501] trap invalid opcode ip:4123ca sp:7fff06c29610 error:0 in TempoNest[400000+cd000]
  TempoNest[26565] trap invalid opcode ip:4123ca sp:7fff23f6a2e0 error:0 in TempoNest[400000+cd000]
  TempoNest[2642]: segfault at 21 ip 000000000040994c sp 00007fffac976ee8 error 6 in TempoNest[400000+d0000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[17172]: segfault at 0 ip 0000000000419020 sp 00007fffba1f9b20 error 4 in TempoNest[400000+d0000]
  TempoNest[17309]: segfault at 0 ip 0000000000419020 sp 00007ffffe7870d0 error 4 in TempoNest[400000+d0000]
  TempoNest[18318]: segfault at 0 ip 0000000000419020 sp 00007fff174261f0 error 4 in TempoNest[400000+d0000]
  TempoNest[18359]: segfault at 0 ip 0000000000419020 sp 00007fffd55998f0 error 4 in TempoNest[400000+d0000]
  TempoNest[18417]: segfault at 0 ip 0000000000412f6f sp 00007fff900293c0 error 4 in TempoNest[400000+d0000]
  TempoNest[18441]: segfault at 0 ip 0000000000412f6f sp 00007fffb043bbd0 error 4 in TempoNest[400000+d0000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[11774] general protection ip:4096b6 sp:7fff890e1b58 error:0 in TempoNest[400000+d0000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[15230]: segfault at 0 ip 000000000041b9c0 sp 00007fffd99d9950 error 4 in TempoNest[400000+d0000]
  TempoNest[15271]: segfault at 0 ip 000000000041b9c0 sp 00007fff46611c40 error 4 in TempoNest[400000+d0000]
  TempoNest[15317]: segfault at 60 ip 00000000004123a6 sp 00007fff5754c7e0 error 4 in TempoNest[400000+d0000]
  TempoNest[16464]: segfault at 0 ip 0000000000411908 sp 00007fff1d5d1f70 error 4 in TempoNest[400000+d0000]
  TempoNest[16586]: segfault at 0 ip 000000000041190c sp 00007fff08a3d240 error 4 in TempoNest[400000+d0000]
  TempoNest[17604]: segfault at 0 ip 0000000000411914 sp 00007ffff7aa7b80 error 4 in TempoNest[400000+d0000]
  TempoNest[17759]: segfault at 0 ip 0000000000411914 sp 00007fffd7aca430 error 4 in TempoNest[400000+d0000]
  TempoNest[17806]: segfault at 0 ip 0000000000411914 sp 00007fff33473d30 error 4 in TempoNest[400000+d0000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[31640]: segfault at 8 ip 00002b315e6cbee0 sp 00007fff34292c30 error 4 in libtempo2.so.0.0.0[2b315e6bb000+7b000]
  TempoNest[2498] general protection ip:40b512 sp:7fff19384038 error:0 in TempoNest[400000+eb000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  TempoNest[30073]: segfault at 210 ip 00000000004137e4 sp 00007fffcd2dd4f0 error 4 in TempoNest[400000+eb000]
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  NFS: v4 server apgpu04 does not accept raw uid/gids. Reenabling the idmapper.
  ib_qib IB0:1 Turning LOS on
  ib_qib IB0:1 re-enabled QDR adaptation ibclt 2
  ib_qib IB0:1 Turning LOS off
  main[7785]: segfault at 0 ip 000000000041ed7d sp 00007fff7f0bd940 error 4 in main[400000+8c000]
  main[7842]: segfault at 0 ip 00000000004205b0 sp 00007fff8ea0ba40 error 4 in main[400000+8c000]
  ib_qib IB0:1 Turning LOS on
  ib_qib IB0:1 re-enabled QDR adaptation ibclt 2
  ib_qib IB0:1 Turning LOS off
  TempoNest[4997] general protection ip:414bed sp:7fff4f50f7b0 error:0 in TempoNest[400000+d7000]
  TempoNest[6094] general protection ip:414bed sp:7fffe53d53f0 error:0 in TempoNest[400000+d7000]
  TempoNest[6106] general protection ip:414bed sp:7fff519a7930 error:0 in TempoNest[400000+d7000]
  TempoNest[11610] general protection ip:414bed sp:7fff5e48fc60 error:0 in TempoNest[400000+d7000]
  TempoNest[11666] general protection ip:414bed sp:7fff960209a0 error:0 in TempoNest[400000+d7000]
  TempoNest[12664] general protection ip:414bed sp:7fff23c978a0 error:0 in TempoNest[400000+d7000]
  TempoNest[12738] general protection ip:414bed sp:7fff25cbc7e0 error:0 in TempoNest[400000+d7000]
  TempoNest[12774] general protection ip:414bed sp:7ffff12816e0 error:0 in TempoNest[400000+d7000]
  TempoNest[13757] general protection ip:414c3d sp:7fff46c2e8e0 error:0 in TempoNest[400000+d7000]
  TempoNest[16043] general protection ip:415e0d sp:7ffff3df74a0 error:0 in TempoNest[400000+d7000]
  TempoNest[16081] general protection ip:415e0d sp:7fff1a31a650 error:0 in TempoNest[400000+d7000]
  TempoNest[16111] general protection ip:4157b1 sp:7fff78bdb960 error:0 in TempoNest[400000+d6000]
  TempoNest[16206] general protection ip:41535f sp:7fff78e06ef0 error:0 in TempoNest[400000+d6000]
  TempoNest[17285] general protection ip:4152fc sp:7fffde4b0ce0 error:0 in TempoNest[400000+d6000]
  TempoNest[6563]: segfault at 40 ip 000000000040b7a9 sp 00007fff46f20cb8 error 6 in TempoNest[400000+dd000]
  TempoNest[9782] general protection ip:40b7b0 sp:7fff1b0fe8d8 error:0 in TempoNest[400000+dd000]
  </code></pre>
  
  <p>Anyway, i'm not sure what to do with this, or how to find out where this is happening, because the program never exits.. can anyone tell me how to debug this?</p>
  
  <p>Thanks</p>