I have been trying to extract the boot and login history for a test PC running RHEL 5.8 with some kernel updates.I started by using last -f /var/log/wtmp, and noticed some strange results. Further investigation showed that there are large blocks of hex data written into the file that do not match the utmp format.These blocks appear to be ASCII values as there are several hundred bytes with ASCII values. e.g. 0a30 7831 6234 6636 3762 652c 0930 7839 3562 3037 which is 0x1b4f67be,.0x95b07 taken from the unused final 20 bytes in the structure.I have continued investigating and the ASCII text matches some of the text that is printed out during boot - it looks like the output from dmesg. Also text that is printed on the screen when my program starts up and I have seen excerts from a config file that is read during startup.I suspect that the null sections are produced by agetty on the serial port as the wtmp file grows every minute. If I comment out the line in i/etc/inittab7:2345:respawn:/sbin/agetty -h -t 60 ttyS0 115200 vt102then wtmp it stops growing.Could there be a collision on the filesystem. I have looked at the i-nodes and they do not seem to be shared. The utmp structure does contain some string values, but mostly appears to be set to zero.What would cause these entries? Does it mean that wtmp cannot be relied upon? Is there any way of decyphering these entries if they are genuine?Below are some extracts from the wtmp file pasted from a hex editor,The last sensible entry with output according to last:root     tty1                          Fri Nov 29 10:19 - crash  (00:03)00312300: 0700 0000 850b 0000 7474 7931 0000 0000  ........tty1....00312310: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312320: 0000 0000 0000 0000 3100 0000 726f 6f74  ........1...root00312330: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312340: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312350: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312360: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312370: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312380: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312390: 0000 0000 0000 0000 0000 0000 0000 0000  ................003123a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003123b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003123c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003123d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003123e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003123f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312400: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312410: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312420: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312430: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312440: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312450: 0000 0000 3e6a 9852 89b0 0000 0000 0000  ....>j.R........00312460: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312470: 0000 0000 0000 0000 0000 0000 0000 0000  ................then the entries are entirely nul and not displayed by last:00312480: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312490: 0000 0000 0000 0000 0000 0000 0000 0000  ................003124a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003124b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003124c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003124d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003124e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003124f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312500: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312510: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312520: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312530: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312540: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312550: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312560: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312570: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312580: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312590: 0000 0000 0000 0000 0000 0000 0000 0000  ................003125a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003125b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003125c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003125d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003125e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003125f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................until there appears to be a block of ASCII data written into the file. This starts part way through a utmp structure and is also not output from last00312f00: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f10: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f20: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f30: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f40: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f50: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f60: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f70: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f80: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312f90: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312fa0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312fb0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312fc0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312fd0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312fe0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00312ff0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00313000: 0930 7862 3130 3264 6131 332c 0a30 7830  .0xb102da13,.0x000313010: 3939 6264 3532 392c 0930 7863 6634 3364  99bd529,.0xcf43d00313020: 3936 312c 0930 7838 3235 6336 3331 662c  961,.0x825c631f,00313030: 0930 7863 6233 6662 3130 372c 0a30 7865  .0xcb3fb107,.0xe00313040: 3336 3265 3739 392c 0930 7866 3231 3362  362e799,.0xf213b00313050: 3730 642c 0930 7830 3133 6537 3465 622c  70d,.0x013e74eb,00313060: 0930 7832 3039 3539 6335 652c 0a30 7865  .0x20959c5e,.0xe00313070: 3166 6663 6631 642c 0930 7865 6532 6438  1ffcf1d,.0xee2d8followed by more ASCII bytes, which is interpreted by last as the following.27dba1f, 4f6fe3c,*0xe 0xed51abc6,*0xf Sun Oct  8 21:09 - crash (6626+14:13)00313080: 3232 382c 0930 7865 3466 3666 6533 632c  228,.0xe4f6fe3c,00313090: 0930 7865 6139 6531 6135 322c 0a30 7862  .0xea9e1a52,.0xb003130a0: 3035 3935 6366 372c 0930 7863 3237 6462  0595cf7,.0xc27db003130b0: 6131 662c 0930 7865 6339 3866 3162 612c  a1f,.0xec98f1ba,003130c0: 0930 7835 6135 3465 3261 632c 0a30 7865  .0x5a54e2ac,.0xe003130d0: 6435 3161 6263 362c 0930 7866 6536 3361  d51abc6,.0xfe63a003130e0: 3364 352c 0930 7866 3932 3161 3831 352c  3d5,.0xf921a815,003130f0: 0930 7837 3732 3239 6262 662c 0a30 7837  .0x77229bbf,.0x700313100: 3032 3230 3363 392c 0930 7861 6337 6166  02203c9,.0xac7af00313110: 3635 392c 0930 7834 3632 6639 3338 362c  659,.0x462f9386,00313120: 0930 7863 3562 3834 3630 622c 0a30 7862  .0xc5b8460b,.0xb00313130: 6231 6166 3862 382c 0930 7839 3238 3065  b1af8b8,.0x9280e00313140: 6161 622c 0930 7861 3361 3265 3639 332c  aab,.0xa3a2e693,00313150: 0930 7861 3565 3231 3361 362c 0a30 7832  .0xa5e213a6,.0x200313160: 6130 6262 3861 322c 0930 7865 6639 3362  a0bb8a2,.0xef93b00313170: 3061 632c 0930 7835 3936 3537 3530 352c  0ac,.0x59657505,00313180: 0930 7839 3066 6632 6366 362c 0a30 7864  .0x90ff2cf6,.0xd00313190: 3931 3564 6438 322c 0930 7864 6135 3061  915dd82,.0xda50a003131a0: 3337 332c 0930 7834 6338 3936 3137 612c  373,.0x4c89617a,003131b0: 0930 7834 6435 3465 3832 662c 0a30 7865  .0x4d54e82f,.0xe003131c0: 3237 6161 6537 322c 0930 7839 6230 6237  27aae72,.0x9b0b7003131d0: 6234 312c 0930 7830 3264 6263 3863 352c  b41,.0x02dbc8c5,003131e0: 0930 7865 3134 6532 6139 372c 0a30 7831  .0xe14e2a97,.0x1003131f0: 6234 6636 3762 652c 0930 7839 3562 3037  b4f67be,.0x95b07This block continues until the following utmp blocks are captured, which is shown by last as reboot   system boot  2.6.30.9         Fri Nov 29 10:23         (340+04:17)00315900: 0800 0000 2302 0000 0000 0000 0000 0000  ....#...........00315910: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315920: 0000 0000 0000 0000 7369 0000 0000 0000  ........si......00315930: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315940: 0000 0000 0000 0000 0000 0000 322e 362e  ............2.6.00315950: 3330 2e39 0000 0000 0000 0000 0000 0000  30.9............00315960: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315970: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315980: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315990: 0000 0000 0000 0000 0000 0000 0000 0000  ................003159a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003159b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003159c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003159d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003159e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................003159f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a00: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a10: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a20: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a30: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a40: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a50: 0000 0000 1f6b 9852 b481 0600 0000 0000  .....k.R........00315a60: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a70: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315a80: 0200 0000 0000 0000 7e00 0000 0000 0000  ........~.......00315a90: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315aa0: 0000 0000 0000 0000 7e7e 0000 7265 626f  ........~~..rebo00315ab0: 6f74 0000 0000 0000 0000 0000 0000 0000  ot..............00315ac0: 0000 0000 0000 0000 0000 0000 322e 362e  ............2.6.00315ad0: 3330 2e39 0000 0000 0000 0000 0000 0000  30.9............00315ae0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315af0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b00: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b10: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b20: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b30: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b40: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b50: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b60: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b70: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b80: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315b90: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315ba0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315bb0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315bc0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315bd0: 0000 0000 1f6b 9852 ce16 0700 0000 0000  .....k.R........00315be0: 0000 0000 0000 0000 0000 0000 0000 0000  ................00315bf0: 0000 0000 0000 0000 0000 0000 0000 0000  ................The entries are sensible again for a while, and then display similar ASCII data.
Garbage hex bytes in wtmp file
rhel;wtmp
I eventually tracked it down.The system has a watchdog that reboots the system if it is not able to contact certain processes. The problem was that my application was being killed with SIGTERM or SIGKILL and did not disconnect from the watchdog first, which it does if SIGINT is used.This reboot during shutdown writes these surious bytes into wtmp. I have now written startup and shutdown scripts in /etc/init.d and setup links in rc#.d using chkconfig and the problem has gone away/