title: APC and child pid XXXXX exit signal Segmentation fault
body: <p>First I had xCache installed on server, I had xCache on lot of my servers but in this one after couple of days you go nothing except a blank page with this error in apache error log:</p>
  
  <pre><code> child pid XXXXX exit signal Segmentation fault
  </code></pre>
  
  <p>As far as I know this mean some sort of memory corruption.</p>
  
  <p>So I removed xCache from php.ini and installed APC. But things are worse now. After one hour or less/more server become unresponsive. I can clearly see in my website's server-status that there are lot of connections in W mode. And they dont have any thing in common. Some of them are from Joomla (first page) and some from vBulletin (forums) and some other are about other custom codes. 
  If I let it for more time I will going to have this error in Apache error log:</p>
  
  <pre><code> Server reached MaxClients setting, consider raising the MaxClients setting
  </code></pre>
  
  <p>Apache is in mod_php and mod_ruid2 mode.</p>
  
  <p>Here is result of server-status when hanged:</p>
  
  <pre><code>Server uptime: 52 minutes 57 seconds
  Total accesses: 31025 - Total Traffic: 279.0 MB
  CPU Usage: u418.14 s64.28 cu0 cs0 - 15.2% CPU load
  9.77 requests/sec - 89.9 kB/second - 9.2 kB/request
  398 requests currently being processed, 27 idle workers
  
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
  WW___WW_WW_W_WWW__WWWW_W_.......................................
  _W___W___WW_W_W_W_W______.......................................
  </code></pre>
  
  <p>Here is error log for xCache (lot of these lines):</p>
  
  <pre><code>[Thu Mar 28 09:50:38 2013] [notice] child pid 1607 exit signal Aborted (6)
  httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
  [Thu Mar 28 09:51:57 2013] [notice] child pid 1643 exit signal Aborted (6)
  httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
  [Thu Mar 28 09:52:32 2013] [notice] child pid 1724 exit signal Aborted (6)
  httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
  [Thu Mar 28 09:55:21 2013] [notice] child pid 1672 exit signal Aborted (6)
  httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
  [Thu Mar 28 09:55:56 2013] [notice] child pid 1471 exit signal Aborted (6)
  httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
  [Thu Mar 28 09:56:49 2013] [notice] child pid 1753 exit signal Aborted (6)
  [Thu Mar 28 09:58:29 2013] [notice] child pid 1810 exit signal Segmentation fault (11)
  [Thu Mar 28 09:58:29 2013] [notice] child pid 1877 exit signal Segmentation fault (11)
  httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
  [Thu Mar 28 10:01:34 2013] [notice] child pid 1844 exit signal Segmentation fault (11)
  [Thu Mar 28 10:01:34 2013] [notice] child pid 1913 exit signal Aborted (6)
  [Thu Mar 28 10:01:34 2013] [notice] child pid 1941 exit signal Segmentation fault (11)
  [Thu Mar 28 10:01:36 2013] [notice] child pid 2007 exit signal Segmentation fault (11)
  [Thu Mar 28 10:01:37 2013] [notice] child pid 2035 exit signal Segmentation fault (11)
  [Thu Mar 28 10:01:37 2013] [notice] child pid 2036 exit signal Segmentation fault (11)
  </code></pre>
  
  <p>Error log for APC before reaching max client (from one restart to other restart):</p>
  
  <pre><code>[Thu Mar 28 19:17:10 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/1.0.0-fips DAV/2 PHP/5.2.17 configured -- resuming normal operations
  [Thu Mar 28 19:21:40 2013] [notice] child pid 1356 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:41 2013] [notice] child pid 1520 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:42 2013] [notice] child pid 1330 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:42 2013] [notice] child pid 1518 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:46 2013] [notice] child pid 1967 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:46 2013] [notice] child pid 1968 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:49 2013] [notice] child pid 1939 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:50 2013] [notice] child pid 2034 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:51 2013] [notice] child pid 2062 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:57 2013] [notice] child pid 2091 exit signal Segmentation fault (11)
  [Thu Mar 28 19:22:58 2013] [notice] child pid 2090 exit signal Segmentation fault (11)
  [Thu Mar 28 19:25:34 2013] [notice] caught SIGTERM, shutting down
  [Thu Mar 28 19:25:37 2013] [notice] SSL FIPS mode disabled
  [Thu Mar 28 19:25:37 2013] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
  [Thu Mar 28 19:25:37 2013] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
  [Thu Mar 28 19:25:37 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
  [Thu Mar 28 19:25:38 2013] [notice] SSL FIPS mode disabled
  [Thu Mar 28 19:25:38 2013] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
  [Thu Mar 28 19:25:38 2013] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
  [Thu Mar 28 19:25:38 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/1.0.0-fips DAV/2 PHP/5.2.17 configured -- resuming normal operations
  </code></pre>
  
  <p>Little from PHP Info:</p>
  
  <pre><code>System  Linux xxx.xxxxx.com 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 27 06:06:45 UTC 2013 x86_64
  Build Date  Mar 17 2013 12:55:14
  Configure Command   './configure' '--with-apxs2' '--with-curl=/usr/local/lib' '--with-gd' '--with-ttf' '--with-gettext' '--with-jpeg-dir=/usr/local/lib' '--with-freetype-dir=/usr/local/lib' '--with-kerberos' '--with-openssl' '--with-mcrypt' '--with-mhash' '--with-mysql=/usr' '--with-mysqli=/usr/bin/mysql_config' '--with-pdo-mysql=/usr' '--with-pcre-regex=/usr/local' '--with-pear' '--with-png-dir=/usr/local/lib' '--with-xsl' '--with-zlib' '--with-zlib-dir=/usr/local/lib' '--with-iconv=/usr/local' '--enable-bcmath' '--enable-calendar' '--enable-exif' '--enable-ftp' '--enable-gd-native-ttf' '--enable-magic-quotes' '--enable-safe-mode' '--enable-soap' '--enable-sockets' '--enable-mbstring' '--enable-zip' '--enable-wddx'
  Server API  Apache 2.0 Handler
  Virtual Directory Support   enabled
  Configuration File (php.ini) Path   /usr/local/lib
  Loaded Configuration File   /usr/local/lib/php.ini
  Scan this dir for additional .ini files     (none)
  additional .ini files parsed    (none)
  PHP API     20041225
  PHP Extension   20060613
  Zend Extension  220060519
  Debug Build     no
  Thread Safety   enabled
  Zend Memory Manager     enabled
  IPv6 Support    enabled
  Registered PHP Streams  https, ftps, compress.zlib, php, file, data, http, ftp, zip
  Registered Stream Socket Transports     tcp, udp, unix, udg, ssl, sslv3, sslv2, tls
  Registered Stream Filters   zlib.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed 
  </code></pre>
  
  <p>APC settings:</p>
  
  <pre><code>enabled
  Version     3.1.13
  APC Debugging   Disabled
  MMAP Support    Enabled
  MMAP File Mask  no value
  Locking type    pthread read/write Locks
  Serialization Support   php
  Revision    $Revision: 327136 $
  Build Date  Mar 28 2013 12:40:18
  </code></pre>
  
  <p>Php -v:</p>
  
  <pre><code>PHP 5.2.17 (cli) (built: Mar 17 2013 12:55:24) 
  Copyright (c) 1997-2010 The PHP Group
  Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
  </code></pre>
  
  <p>httpd -V:</p>
  
  <pre><code>Server version: Apache/2.2.24 (Unix)
  Server built:   Mar 10 2013 09:18:00
  Server's Module Magic Number: 20051115:31
  Server loaded:  APR 1.4.6, APR-Util 1.4.1
  Compiled using: APR 1.4.6, APR-Util 1.4.1
  Architecture:   64-bit
  Server MPM:     Worker
    threaded:     yes (fixed thread count)
      forked:     yes (variable process count)
  Server compiled with....
   -D APACHE_MPM_DIR="server/mpm/worker"
   -D APR_HAS_SENDFILE
   -D APR_HAS_MMAP
   -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
   -D APR_USE_SYSVSEM_SERIALIZE
   -D APR_USE_PTHREAD_SERIALIZE
   -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
   -D APR_HAS_OTHER_CHILD
   -D AP_HAVE_RELIABLE_PIPED_LOGS
   -D DYNAMIC_MODULE_LIMIT=128
   -D HTTPD_ROOT="/etc/httpd"
   -D SUEXEC_BIN="/usr/sbin/suexec"
   -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
   -D DEFAULT_ERRORLOG="logs/error_log"
   -D AP_TYPES_CONFIG_FILE="conf/mime.types"
   -D SERVER_CONFIG_FILE="conf/httpd.conf"
  </code></pre>
  
  <p>Any Idea?! I will reward 50 :)</p>