title: mod_fcgid: can't apply process slot for error
body: <p><strong>update</strong>: Just looked at the cache update times of long queries and they did not collide with server crash time. </p>
  
  <p><strong>update2</strong>: Found the cause of the problem. Ad server is down and the server hangs even though it seems we set socket timeouts properly. Is there any way to test timeout behaviour?</p>
  
  <p>We have a very busy server. ~3K concurrent connections
  The server has 32GB ram 2xCPUs. We have service unavailable error issues. 
  The server does not respond with 500 error and the error log shows hundreds/thousands of lines:</p>
  
  <pre><code>[warn] mod_fcgid: can't apply process slot for /var/www/fcgi-bin.d/php5-default/php-fcgi-wrapper
  </code></pre>
  
  <p>We think it can be a configuration error or a database connection/query error.
  A php process updates a cache which is a very very complex query result. 3 separate queries run each twice a day. I have enabled the slow query log. I suspect if the query exceeds the php run time limit 20 secs in our case (set in the following files). Any help is appreciated.</p>
  
  <p>We use apache worker mpm model with mod_fcgid.</p>
  
  <p>Here is fcgid.conf file:</p>
  
  <pre><code>&lt;IfModule mod_fcgid.c&gt;
  AddHandler fcgid-script .fcgi
  SocketPath /var/lib/apache2/fcgid/sock
  
  # Communication timeout: Default value is 20 seconds
  IPCCommTimeout 20
  
  # Connection timeout: Default value is 3 seconds
  
  IPCConnectTimeout 3
  </code></pre>
  
  <p></p>
  
  <p>And /etc/apache2/conf.d/php-fcgid.conf file:</p>
  
  <pre><code>&lt;IfModule !mod_php4.c&gt;
  </code></pre>
  
  <p>
  
  
  
  
  
  
  
      # Path to php.ini &lt;96> defaults to /etc/phpX/cgi
      DefaultInitEnv PHPRC=/etc/php5/cgi</p>
  
  <pre><code># Number of PHP childs that will be launched. Leave undefined to let PHP decide.
  # DefaultInitEnv PHP_FCGI_CHILDREN 8
  
  # Maximum requests before a process is stopped and a new one is launched
  DefaultInitEnv PHP_FCGI_MAX_REQUESTS 5000
  # Maximum requests a process handles before it is terminated
  MaxRequestsPerProcess 1500
  # Maximum number of PHP processes.
  MaxProcessCount       45
  
  # Define a new handler "php-fcgi" for ".php" files, plus the action that must follow
  AddHandler php-fcgi .php
  Action php-fcgi /fcgi-bin/php-fcgi-wrapper
  
  # Define the MIME-Type for ".php" files
  AddType application/x-httpd-php .php
  
  # Define alias "/fcgi-bin/". The action above is using this value, which means that
  # you could run another "php5-cgi" command by just changing this alias
  Alias /fcgi-bin/ /var/www/fcgi-bin.d/php5-default/
  
  # Turn on the fcgid-script handler for all files within the alias "/fcgi-bin/"
  &lt;Location /fcgi-bin/&gt;
      SetHandler fcgid-script
      Options +ExecCGI
  &lt;/Location&gt;
  </code></pre>
  
  <p>
  
  
  
  
  
  
  
  </p>
  
  <p>Apache2 worker mpm config:</p>
  
  <pre><code>&lt;IfModule mpm_worker_module&gt;
  StartServers         10
  MaxClients           2048
  ServerLimit          2048
  MinSpareThreads      30
  MaxSpareThreads      100
  ThreadsPerChild      64
  ThreadLimit          100
  MaxRequestsPerChild   5000
  </code></pre>
  
  <p></p>
  
  <p>We looked at the instructions on this web page and loaded high server config:
  <a href="http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html" rel="nofollow">http://2bits.com/articles/apache-fcgid-acceptable-performance-and-better-resource-utilization.html</a></p>
  
  <p><strong>update</strong>: Just looked at the cache update times of long queries and they did not collide with server crash time. </p>
  
  <p><strong>update2</strong>: Found the cause of the problem. Ad server is down and the server hangs even though it seems we set socket timeouts properly. Is there any way to test timeout behaviour?</p>