upstream <%= @vhost %>-upstream {
<%- @servers.each do |server| -%>
  server <%= server -%>:<%= @port %> max_fails=3 fail_timeout=10s;
<%- end -%>
}

server {
  listen localhost:<%= @port %>;

  # Needed to do keepalives upstream
  proxy_http_version 1.1;

  # Try next upstream if one returns an error
  proxy_next_upstream error timeout http_500 http_502 http_503 http_504;

  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_redirect off;

  proxy_read_timeout <%= @read_timeout %>;

  # Don't restrict the size of queries or documents.
  client_max_body_size 4g;

  access_log /var/log/nginx/<%= @log_access %> timed_combined;
  access_log /var/log/nginx/<%= @log_json %> json_event;
  error_log /var/log/nginx/<%= @log_error %>;

  location / {
    # Only accept connnections over loopback.
    allow 127.0.0.1;
    deny all;

    proxy_pass http://<%= @vhost %>-upstream;
  }
}

upstream <%= @vhost %>-upstream {
<%- @servers.each do |server| -%>
  server <%= server -%>:<%= @port %> max_fails=3 fail_timeout=10s;
<%- end -%>
}

server {
  listen localhost:<%= @port %>;

  # Needed to do keepalives upstream
  proxy_http_version 1.1;

  # Try next upstream if one returns an error
  proxy_next_upstream error timeout http_500 http_502 http_503 http_504;

  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_redirect off;

  proxy_connect_timeout 1s;
  proxy_read_timeout <%= @read_timeout %>;

  # Don't restrict the size of queries or documents.
  client_max_body_size 4g;

  access_log /var/log/nginx/<%= @log_access %> timed_combined;
  access_log /var/log/nginx/<%= @log_json %> json_event;
  error_log /var/log/nginx/<%= @log_error %>;

  location / {
    # Only accept connnections over loopback.
    allow 127.0.0.1;
    deny all;

    proxy_pass http://<%= @vhost %>-upstream;
  }
}

Set proxy_connect_timeout for ES load balancers
Set proxy_connect_timeout for ES load balancers

The nginx load balancers in front of elasticsearch do not currently set
a connect timeout, and therefore get the default value of 60s[1]. This
means that if one node is unavailable (due to VM pauses or other
issues), then it'll take up to 60 seconds before the next node is tried.
This is substantially longer than the clients will wait for a response,
causing an error to be returned to the user.

Setting this timeout to 1 second allows for failover to happen within
the client timeout. This also matches the value used in our app proxy
vhosts[2]. A healthy server should have no trouble establishing a TCP
connection in less than a second.

[1]http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_connect_timeout
[2]https://github.gds/gds/puppet/blob/master/modules/nginx/templates/proxy-vhost.conf#L49
