title: Apache proxy load balancing backend server failure detection
body: <p>Here's my scenario (designed by my predecessor):</p>
  
  <p>Two Apache servers serving reverse proxy duty for a number of mixed backend web servers (Apache, IIS, Tomcat, etc.).  There are some sites for which we have multiple backend web servers, and in those cases, we do something like:</p>
  
  <pre><code>&lt;Proxy balancer://www.example.com&gt;
      BalancerMember http://192.168.1.40:80
      BalancerMember http://192.168.1.41:80
  &lt;/Proxy&gt;
  &lt;VirtualHost *:80&gt;
      ServerName www.example.com:80
      CustomLog /var/log/apache2/www.example.com.log combined
      &lt;Location /&gt;
          Order allow,deny
          Allow from all
          ProxyPass balancer://www.example.com/
          ProxyPassReverse balancer://www.example.com/
      &lt;/Location&gt;
  &lt;/VirtualHost&gt;
  </code></pre>
  
  <p>So in this example, I've got one site (www.example.com) in the proxy servers' configs, and that site is proxied to one or the other of the two backend servers, 192.168.1.40 and .41.</p>
  
  <p>I'm evaluating this to make sure that we are fault tolerant on all of our web services (I've already put the two reverse proxy servers into a shared IP cluster for this reason), and I want to make sure that the load-balanced backend servers are fault tolerant as well.  But I'm having trouble figuring out if backend failure detection (and the logic to avoid the failed backend server) is built into the mod_proxy_balancer module...</p>
  
  <p>So if 192.168.202.40 goes down, will Apache detect this (I'll understand if it takes a failed request first) and automatically route all requests to the other backend, 192.168.202.41?  Or will it continue to balance requests between the failed backend and the operational backend?</p>
  
  <p>I've found some clues in the Apache documentation for <a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypass" rel="noreferrer">mod_proxy</a> and <a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html#stickyness_implementation" rel="noreferrer">mod_proxy_balancer</a> that seem to indicate that failure can be detected ("maxattempts = Maximum number of failover attempts before giving up.", "failonstatus = A single or comma-separated list of HTTP status codes. If set this will force the worker into error state when the backend returns any status code in the list."), but after a few days of searching, I've found nothing conclusive saying for sure that it <em>will</em> (or at least "should") detect backend failure and recovery.</p>
  
  <p>I will say that most of the search results reference using the AJP protocol to pass the traffic to the backend servers, and this apparently does support failure detection-- but my backends are a mixture of Apache, IIS, Tomcat and others, and I am fairly sure that many of them don't support AJP.  They are also a mixture of Windows 2k3/2k8 and Linux (mostly Ubuntu Lucid) boxes running various different applications with various different requirements, so add-on modules like Backhand and LVS aren't an option for me.</p>
  
  <p>I've also tried to empirically test this feature, by creating a new test site like this:</p>
  
  <pre><code>&lt;Proxy balancer://test.example.com&gt;
      BalancerMember http://192.168.1.40:80
      BalancerMember http://192.168.1.200:80
  &lt;/Proxy&gt;
  &lt;VirtualHost *:80&gt;
      ServerName test.example.com:80
      CustomLog /var/log/apache2/test.example.com.log combined
      LogLevel debug
      &lt;Location /&gt;
          Order allow,deny
          Allow from all
          ProxyPass balancer://test.example.com/
          ProxyPassReverse balancer://test.example.com/
      &lt;/Location&gt;
  &lt;/VirtualHost&gt;
  </code></pre>
  
  <p>Where 192.168.1.200 is a bogus address that isn't running any web server, to simulate a backend failure.  The test site was served up without a problem for a bunch of different client machines, but even with the LogLevel set to debug, I didn't see anything logged to indicate that it detected that one of the backend servers was down...  And I'd like to make 100% sure that I can take our load-balanced backends down for maintenance (one at a time, of course) without affecting production sites.</p>