title: Getting stack trace from Perl "Out of memory" error
body: <p>tl;dr: How to dump a perl stack trace when a Perl httpd process runs out of memory.</p>
  
  <p>We've got a mod_perl 2 server, Perl 5.8.8, RHEL 5.6, Linux 2.6.18.</p>
  
  <p>Very occasionally and unpredictably, a child httpd process starts using up all available memory at an alarming rate. We've at least used BSD::Resource::setrlimit(RLIMIT_VMEM, ...), so that the process dies with "Out of memory" before bringing down the server.</p>
  
  <p>We don't know where in the code this is happening, and it's difficult to reproduce without hours of load testing.</p>
  
  <p>What we'd really like is a way to get a <strong>Perl stack trace</strong> just before the process runs out of memory, so we know what code is causing this. Unfortunately, "Out of memory" is an <a href="http://perldoc.perl.org/perldiag.html" rel="noreferrer">untrappable error</a>.</p>
  
  <p>Here are the options I'm considering, each with their drawbacks:</p>
  
  <p>1) Use the <a href="http://perldoc.perl.org/perlvar.html" rel="noreferrer">$^M emergency memory pool</a>. Requires us to recompile perl with -DPERL_EMERGENCY_SBRK and -Dusemymalloc.</p>
  
  <p>2) Put in tons of log statements, then analyze the logs to see where the process is stopping short.</p>
  
  <p>3) Write an external script that constantly scans the pool of httpd processes, and if it sees one using a lot of memory, sends it a USR2 signal (which we've arranged to dump a stack trace).</p>
  
  <p>4) Somehow have the process monitor its own memory continuously, and dump a stack trace when memory gets high but before the "Out of Memory" error.</p>
  
  <p>Thanks!</p>
  
  <p>Jon</p>