title: Constant 503 errors from Elasticsearch inside a VPC when using Kinesis Firehose
body: <p>I am using the <a href="https://awslabs.github.io/amazon-kinesis-data-generator/web/producer.html" rel="nofollow noreferrer">Amazon Kinesis Data Generator</a> to send data to a test Kinesis Firehose Stream for indexing in an Elasticsearch Service cluster.</p>
  <p>The data generator sends a fairly basic json doc for processing, the stream element works fine, as does the Lambda transformation. I've verified and been able to test everything up to this point. It's only when the request is made to Elasticsearch inside my VPC that I get an error.</p>
  <p>It works fine if I switch the pipeline to use a public Elasticsearch domain, but when I use the Elasticsearch domain inside my VPC, I get a <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-503-issues" rel="nofollow noreferrer">503 error</a>. This is a consistent error on every single request so not a capacity issue.</p>
  <p>Here's an example of the error I'm seeing. Just a generic 503. I'm not sure if this is coming from the load balancer or the target (Elasticsearch itself).</p>
  <pre><code>{
   &quot;deliveryStreamARN&quot;: &quot;arn:aws:firehose:eu-west-2:xxx:deliverystream/firehose-test&quot;,
   &quot;destination&quot;: &quot;arn:aws:es:eu-west-2:xxx:domain/elasticsearch-test&quot;,
   &quot;deliveryStreamVersionId&quot;: 1,
   &quot;message&quot;: &quot;Error received from Elasticsearch cluster. &lt;html&gt;&lt;body&gt;&lt;h1&gt;503 Service 
   Unavailable&lt;/h1&gt;\nNo server is available to handle this request.\n&lt;/body&gt;&lt;/html&gt;&quot;,
   &quot;errorCode&quot;: &quot;ES.ServiceException&quot;,
   &quot;processor&quot;: &quot;arn:aws:lambda:eu-west-2:xxx:function:transform-test:$LATEST&quot;
  }
  </code></pre>
  <p>Other applications inside the VPC are able to use the Elasticsearch service without a problem. This seems to be something specific to Firehose.</p>
  <p>I have read and re-read the docs and can't figure out why the same data pipeline works fine on a public ES domain, but not on the ES domain inside my VPC. I've double checked all roles, policies, security groups and subnets. Both Firehose and Elasticsearch are using the same VPC, Security Groups and Subnets. The inbound rules on the security group allow HTTPS. It all looks correct, but still getting errors and nothing being indexed in Elastic.</p>
  <p>I've also read this AWS blog post about 15 times to no avail.
  <a href="https://aws.amazon.com/blogs/big-data/ingest-streaming-data-into-amazon-elasticsearch-service-within-the-privacy-of-your-vpc-with-amazon-kinesis-data-firehose/" rel="nofollow noreferrer">Ingest streaming data into Amazon Elasticsearch Service within the privacy of your VPC with Amazon Kinesis Data Firehose</a></p>