title: ELB generating 504 GATEWAY_TIMEOUTS w/ 2 EC2 instances - Packets not reaching Servers
body: <p>I have one ELB facing internet and connecting to 2 EC2 instances serving a web page. The infraestructure has been deployed using AWS Opsworks.</p>
  
  <p>The configuration is as follows:</p>
  
  <ol>
  <li><p>Internet facing ELB:</p>
  
  <ul>
  <li>Present in two availability zones</li>
  <li>Cross-zone load balancing enabled</li>
  <li>Idle timeout : 60 segs</li>
  <li>Stickiness: LBCookieStickinessPolicy, expirationPeriod='0'</li>
  </ul></li>
  <li><p>EC2 instances serving a web page (demo1.virtualtoptraining.com)</p>
  
  <ul>
  <li>Configuration is exactly the same on both</li>
  <li>Running Apache 2 and PHP5 as module</li>
  </ul></li>
  <li><p>DNS Configuration:</p></li>
  </ol>
  
  <p>demo1.virtualtoptraining.com --> CNAME to DNS Name of loadbalancer</p>
  
  <p>When I run testing agains demo1.virtualtoptraining.com using JMETER, I see around 20% of requests getting errors. The error reported is 504 GATEWAY_TIMEOUT. The errors do not depend on load conditions as I can get this error navigating with a single user on a web browser. The error is 100% reproduceable.</p>
  
  <p>Tried the following:</p>
  
  <ul>
  <li><p>I have been playing with apache timers as I though this could be related to the web server not responding to the load balancer in time, but I got no improvements.</p></li>
  <li><p>I have been using servers on the same availability zone. The error is still there.</p></li>
  </ul>
  
  <p>After some investigation, I came to the conclusion that the requests where I get the 504 GATEWAY_TIMEOUT reach the load balancer, but they never get forwarded to the web servers as I can not see them in any of the web servers access logs.</p>
  
  <p>I am showing below an example of the issue with all the logs. I got this example just navigating to the web page on my browser:</p>
  
  <h2>Example Sequence</h2>
  
  <ol>
  <li>Logout from the web page (forwarded to front page)</li>
  <li>Get front page</li>
  <li>Get login page</li>
  <li>Post login credentials</li>
  <li>Follow redirection after credentials validation</li>
  <li>Get main page. ERROR --> this GET operation is not seen on any of the server access logs</li>
  </ol>
  
  <p>I am listing now, the access logs as well as the ELB log, I am highlighting the messages with ***(message #)</p>
  
  <p><strong>SERVER 1 ACCESS LOG:</strong></p>
  
  <pre><code>ubuntu@php-app1:/var/log/apache2$ tail -f moodle281-access.log
  172.31.40.186 - - [09/Jun/2015:22:44:52 +0000] "POST /login/index.php HTTP/1.1" 303 1018 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  172.31.40.186 - - [09/Jun/2015:22:44:53 +0000] "GET / HTTP/1.1" 200 39933 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  172.31.40.186 - - [09/Jun/2015:22:44:54 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /course/view.php?id=2 HTTP/1.1" 200 55453 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  172.31.40.186 - - [09/Jun/2015:22:44:55 +0000] "GET /mod/forum/discuss.php?d=1 HTTP/1.1" 200 47600 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "POST /mod/forum/post.php HTTP/1.1" 404 26456 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  172.31.40.186 - - [09/Jun/2015:22:44:56 +0000] "GET /user/index.php?id=2 HTTP/1.1" 200 82175 "-" "Apache-HttpClient/4.2.6 (java 1.5)"
  
  ****(1) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET /login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" 303 862 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
  ****(4) 172.31.19.167 - - [09/Jun/2015:22:48:55 +0000] "GET /login/index.php HTTP/1.1" 200 6088 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
  ****(5) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "GET /login/index.php?testsession=2 HTTP/1.1" 303 800 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
  </code></pre>
  
  <p><strong>SERVER 2 ACCESS LOG:</strong></p>
  
  <pre><code>....
  
  ****  (2) 172.31.19.167 - - [09/Jun/2015:22:48:52 +0000] "GET / HTTP/1.1" 200 7078 "http://demo1.virtualtoptraining.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
  ****  (3) 172.31.19.167 - - [09/Jun/2015:22:49:00 +0000] "POST /login/index.php HTTP/1.1" 303 1010 "http://demo1.virtualtoptraining.com/login/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5"
  
  ....
  </code></pre>
  
  <p><strong>LOAD BALANCER ACCESS LOG:</strong></p>
  
  <pre><code>015-06-09T22:48:52.128826Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000092 0.204629 0.000058 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/logout.php?sesskey=fXjerOYZEo HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
  2015-06-09T22:48:52.382853Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000082 0.456022 0.00007 200 200 0 6500 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
  2015-06-09T22:48:55.853509Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.000105 0.292976 0.000046 200 200 0 5617 "GET http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
  2015-06-09T22:49:00.382245Z itoptest2 151.182.141.178:52554 172.31.34.25:80 0.000105 0.545512 0.000068 303 303 36 463 "POST http://demo1.virtualtoptraining.com:80/login/index.php HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
  2015-06-09T22:49:00.980174Z itoptest2 151.182.141.178:52554 172.31.25.91:80 0.00007 0.219837 0.000044 303 303 0 434 "GET http://demo1.virtualtoptraining.com:80/login/index.php?testsession=2 HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
  2015-06-09T22:49:01.248300Z itoptest2 151.182.141.178:52554 - -1 -1 -1 504 0 0 0 "GET http://demo1.virtualtoptraining.com:80/ HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.5 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.5" - -
  </code></pre>
  
  <p>As you can see the request generating the 504 GATEWAY_TIMEOUT is never seen on any of the web servers access log.</p>
  
  <p>Any idea on of what the issue could be?</p>