title: 2.1.6 Spring Boot - Elasticsearch Healthcheck failure
body: <p>***Update - I found a useful StackOverflow post where other people were having similar issues with the Healthcheck monitor failing for Elasticsearch <a href="https://stackoverflow.com/questions/54363302/springboot-elastic-search-health-management-connectexception-connection-refus">Springboot elastic search health management : ConnectException: Connection refused</a></p>
  
  <p>It seems like the Healthcheck actuator uses the Rest client, however, the Elasticsearch for mapping, getting indices and such uses the RestHighLevelClient. We have a @config file that has the variables for esPort, esHost, and esSchema (i.e. port 9200, host localhost, and schema http), the code is below, and below that the code for the "ESClient.java" class:</p>
  
  <p><strong>ESClientConfig.java</strong> class</p>
  
  <pre><code>package com.cat.digital.globalsearch.configuration;
  
          import com.cat.digital.globalsearch.component.ESClient;
          import org.apache.http.HttpHost;
          import org.springframework.beans.factory.annotation.Value;
          import org.springframework.context.annotation.Bean;
          import org.springframework.context.annotation.Configuration;
  
  @Configuration
  public class ESClientConfig {
  
      @Value("${elasticSearch.host}")
      private String esHost;
  
      @Value("${elasticSearch.port}")
      private int esPort;
  
      @Value("${elasticSearch.scheme}")
      private String esScheme;
  
      @Bean
      public ESClient esClient() {
          return new ESClient( new HttpHost(esHost, esPort, esScheme));
      }
  }
  </code></pre>
  
  <p><strong>ESClient.java</strong> class</p>
  
  <pre><code>package com.cat.digital.globalsearch.component;
  
  import com.cat.digital.globalsearch.model.IndexDocument;
  import org.apache.http.HttpHost;
  import org.elasticsearch.action.ActionListener;
  import org.elasticsearch.action.admin.indices.delete.DeleteIndexRequest;
  import org.elasticsearch.action.bulk.BulkRequest;
  import org.elasticsearch.action.bulk.BulkResponse;
  import org.elasticsearch.action.index.IndexRequest;
  import org.elasticsearch.action.search.SearchRequest;
  import org.elasticsearch.action.search.SearchResponse;
  import org.elasticsearch.action.support.master.AcknowledgedResponse;
  import org.elasticsearch.client.RestClient;
  import org.elasticsearch.client.RestClientBuilder;
  import org.elasticsearch.client.RestHighLevelClient;
  import org.elasticsearch.client.indices.CreateIndexRequest;
  import org.elasticsearch.client.indices.GetIndexRequest;
  import org.elasticsearch.client.indices.PutMappingRequest;
  
  import java.io.IOException;
  import java.util.List;
  
  import static org.elasticsearch.client.RequestOptions.DEFAULT;
  import static org.elasticsearch.common.xcontent.XContentType.JSON;
  
  /**
   * Wrapper around {@link RestHighLevelClient}
   */
  public class ESClient {
  
      private final RestClientBuilder builder;
      private final RestHighLevelClient searchClient;
  
      public ESClient(HttpHost... hosts) {
          this.builder = RestClient.builder(hosts);
          searchClient = new RestHighLevelClient(RestClient.builder(hosts));
      }
  
  /**
       * @param index String represents index name in ES
       * @return true if index exists, false if not
       */
      public boolean hasIndex(String index) throws IOException {
          final GetIndexRequest request = new GetIndexRequest(index);
          try (RestHighLevelClient client = new RestHighLevelClient(builder)) {
              return client.indices().exists(request, DEFAULT);
          }
      }
  </code></pre>
  
  <p>So now I'm thinking the "connection refused" to Elasticsearch could be because on the Dev environment its trying to use the Rest client and the correct parameters to connect are not present. But how would this explain the Healthcheck monitor working fine locally? Is the RestHighLevelClient used locally? </p>
  
  <p>I posted on an issue on the spring boot GitHub, and was referred here. I will try to make this is simple as possible so I can get some help. Its actually quite simple. </p>
  
  <p><strong>TL;DR</strong></p>
  
  <ul>
  <li><p>Used Spring Boot actuator to create custom Healthcheck monitor for Elasticsearch service</p></li>
  <li><p>Created 1 custom Java class called "IndexExists" (code below)</p></li>
  <li><p>Added the Application.yml file: rest.uri = ['our-dev-url-on-aws'] property</p></li>
  </ul>
  
  <p>I have an application that works perfectly fine both locally and remotely, however when adding a custom Healthcheck monitor using Spring Boot to monitor my Elasticsearch service, I get a "connection refused" to Elasticsearch and the health check monitor ultimately fails. Our Load Balancer on AWS attempts to hit this endpoint and because it returns a status: "DOWN" because the healthcheck cannot connect to Elasticsearch, the Load Balancer starts to create a new Container. When looking at the CloudWatch logs, this happens repeatedly in an infinite loop (the Load Balancer attempting to make more containers). I would like to add, this works perfectly fine locally - that is, when adding the healthcheck monitor, and using a HTTP GET request on the actuator/health endpoint via POSTMAN, I get the correct JSON response which is:</p>
  
  <p><strong>Local JSON response from /actuator/health endpoint (GET request)</strong></p>
  
  <pre><code>{
      "status": "UP",
      "details": {
          "indexExists": {
              "status": "UP",
              "details": {
                  "index": "exists",
                  "value": "assets"
              }
          },
          "diskSpace": {
              "status": "UP",
              "details": {
                  "total": 250790436864,
                  "free": 194987540480,
                  "threshold": 10485760
              }
          },
          "elasticsearchRest": {
              "status": "UP",
              "details": {
                  "cluster_name": "elasticsearch",
                  "status": "yellow",
                  "timed_out": false,
                  "number_of_nodes": 1,
                  "number_of_data_nodes": 1,
                  "active_primary_shards": 7,
                  "active_shards": 7,
                  "relocating_shards": 0,
                  "initializing_shards": 0,
                  "unassigned_shards": 5,
                  "delayed_unassigned_shards": 0,
                  "number_of_pending_tasks": 0,
                  "number_of_in_flight_fetch": 0,
                  "task_max_waiting_in_queue_millis": 0,
                  "active_shards_percent_as_number": 58.333333333333336
              }
          }
      }
  }
  </code></pre>
  
  <p>As you can see, the Healthcheck monitor is returning the "details": { "indexExists": etc... } portion up at the top, which checks if my Elasticsearch index is mapped to the string = "assets". If it is, then it returns "status": "UP".</p>
  
  <p>However, when pushing this code to the build pipeline in Azure, so I can test in a dev environment, this is the JSON response I get:</p>
  
  <p><strong>DEV JSON response from /actuator/health endpoint (GET request)</strong></p>
  
  <pre><code>{
      "status": "DOWN",
      "details": {
          "indexExists": {
              "status": "UP",
              "details": {
                  "index": "exists",
                  "value": "assets"
              }
          },
          "diskSpace": {
              "status": "UP",
              "details": {
                  "total": 16776032256,
                  "free": 9712218112,
                  "threshold": 10485760
              }
          },
          "elasticsearchRest": {
              "status": "DOWN",
              "details": {
                 "error": "java.net.ConnectException: Connection refused"
              }
          }
      }
  }
  </code></pre>
  
  <p>I can view the error logs on our AWS (Amazon Web Services) cluster and they look like this:
  <a href="https://i.stack.imgur.com/Xx0EK.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Xx0EK.jpg" alt="enter image description here"></a></p>
  
  <p>The java class that I created and added to our code base is <strong>IndexExists.java</strong>. It implements the HealthIndicator() interface and uses the actuator from Spring Boot:</p>
  
  <p><strong>IndexExists.java class</strong></p>
  
  <pre><code>package com.cat.digital.globalsearch.component;
  
  import org.slf4j.Logger;
  import org.slf4j.LoggerFactory;
  import org.springframework.beans.factory.annotation.Autowired;
  import org.springframework.boot.actuate.health.Health;
  import org.springframework.boot.actuate.health.HealthIndicator;
  import org.springframework.stereotype.Component;
  import com.cat.digital.globalsearch.data.Indices;
  import java.io.IOException;
  import java.util.HashMap;
  import java.util.Map;
  
  
  @Component
  public class IndexExists implements HealthIndicator {
      private final ESClient esClient;
      private static final Logger LOGGER = LoggerFactory.getLogger(IndexExists.class);
      Map&lt;String,String&gt; map = new HashMap&lt;&gt;();
  
  
      @Autowired
      public IndexExists(ESClient esClient) {
          this.esClient = esClient;
      }
  
      @Override
      public Health health() {
          try {
              if (!esClient.hasIndex(Indices.INDEX_ASSETS)) {
                  return Health.down().withDetail("index", Indices.INDEX_ASSETS + " index does not exist").build();
              }
          } catch (IOException e) {
              LOGGER.error("Error checking if Elasticsearch index {} exists , with exception", Indices.INDEX_ASSETS,  e);
          }
          map.put("index","exists");
          map.put("value", Indices.INDEX_ASSETS);
          return Health.up().withDetails(map).build();
      }
  }
  </code></pre>
  
  <p>I won't post all the code of the application.yml, but here are the bits I added. For the spring dev profile, I only added the rest uri, the rest of the code was already there:</p>
  
  <pre><code>management:
    endpoint:
      health:
        show-details: always
  
  spring:
    profiles: dev
  elasticSearch:
    host: "aws-dev-url"
    port: -1
    scheme: https
    rest:
    uris: ["aws-dev-url"]
  </code></pre>
  
  <p>I hope that information isn't too much! I really need help... If anyone needs any more information, please let me know. Thank you.</p>