We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
/healthcheck endpoint should check for Elasticsearch availability (original #487)
This issue has been migrated from the [CC Search API repository](https://github.com/creativecommons/cccatalog-api/issues/487)

```
Author: aldenstpage
Date: Wed May 06 2020
Labels: ✨ goal: improvement,🏷 status: label work required,🙅 status: discontinued
```
During deployments, our load balancer repeatedly polls the `/healthcheck` endpoint to check that the server is reachable. If this check succeeds, the newly deployed instance starts receiving production traffic. Right now, if Elasticsearch is not responsive, `/healthcheck` will still return `200 OK`.

The healthcheck endpoint should check the health of the `image` index in Elasticsearch using the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html). If it is unavailable, return error 500. Log an informative message explaining why the healthcheck failed.

Because the healthcheck endpoint may be called many times, and Elasticsearch calls are not free, we should cache the response of Elasticsearch for up to 10 seconds per call.
--------
### Original Comments: 

*madewithkode* commented on Fri May 08 2020:

>Hi Alden, this looks interesting, I'd love to work on it.
  
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625824482)


*madewithkode* commented on Fri May 08 2020:

>Hi Alden in order to check the health of the `image` index in the /healthcheck view, I'm trying to use the urllib's `urlopen()` method to make a request to Elasticsearch's  cluster API this way:

`cluster_response = urlopen('http://0.0.0.0:8000/_cluster/health/image')`

However, I keep getting a 404. Is there something I'm doing wrong?
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625854788)


*madewithkode* commented on Fri May 08 2020:

>> Hi Alden in order to check the health of the `image` index in the /healthcheck view, I'm trying to use the urllib's `urlopen()` method to make a request to Elasticsearch's cluster API this way:
> 
> `cluster_response = urlopen('http://0.0.0.0:8000/_cluster/health/image')`
> 
> However, I keep getting a 404. Is there something I'm doing wrong?

Figured this, didn't know elastic search was running on a seperate host/port :)
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625905259)


*aldenstpage* commented on Fri May 08 2020:

>That's great! 

It would be best to use the equivalent `elasticsearch-py` or `elasticsearch-dsl` query instead of making direct calls to the REST API (you can get an instance of the connection to Elasticsearch from `search_controller.py`). [Here's](https://discuss.elastic.co/t/how-to-get-cluster-health-using-python-api/25431) an example for getting the cluster health; there ought to also be a way to narrow the query to the `image` index.
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625995563)


*madewithkode* commented on Sat May 09 2020:

>Alright...would look at the suggestion.

On Fri, May 8, 2020, 21:06 Alden S Page <notifications@github.com> wrote:

> It would be best to use the equivalent elasticsearch-py query instead of
> making direct calls to the REST API. Here's
> <https://discuss.elastic.co/t/how-to-get-cluster-health-using-python-api/25431>
> an example for getting the cluster health; there ought to also be a way to
> narrow the query to the image index.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625995563>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGFLMYA5WLAQPO5GYZNX5BTRQRQ5RANCNFSM4M2V5EKA>
> .
>

[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626130991)


*madewithkode* commented on Sat May 09 2020:

>Hi Alden, I'm here again :)
I'd love to ask, what status specifically signifies the availability of the `image` index. 
`red`, `yellow` or `green` ? or should i leave out this detail in the query since I'm using an already established connection instance from `search_controller.py` which waits for a `yellow` status by default.
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626164842)


*madewithkode* commented on Sat May 09 2020:

>### Update: 
I've successfully managed to query the health of the entire cluster, using the Elasticsearch connection instance gotten from `search_controller.py`. However when i try to limit the health check to just the `image` index, the request never resolves and continues to run forever with no response. And when i try to specify a timeout for the request, i get an "Illegal argument exception" even though `timeout` is a valid kwarg referenced in the [API docs](https://elasticsearch-py.readthedocs.io/en/master/api.html?highlight=health#elasticsearch.client.ClusterClient.health).

It'd be nice to point out that as at the time of writing, I'm yet to successfully run `./load_sample_data.sh` so i don't know if this could be linked to the above problem.



[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626223763)


*madewithkode* commented on Mon May 11 2020:

>Hi Alden, Progress Report :)

Successfully got the `load_sample_data.sh` to run,  and so far every other thing is working fine. 
I've also set up the 10s response caching on the `/healthcheck` view using redis and also the error logging.

However, I figured out the reason for the unresponsiveness when querying the elastic search `image` index was that it was non-existent and that the whole cluster index was empty too.

Do I need to do a manual population or something?




[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626749591)


*aldenstpage* commented on Mon May 11 2020:

>Hi again Onyenanu – if the index doesn't exist, the healthcheck should fail. This could happen in situations where we are switching Elasticsearch clusters in production and forgot to index data into the new one (or something went wrong while we were loading data into the new cluster).

In my experience, the ES Python libs can behave in unexpected ways that you sometimes have to work around. Since it seems like querying specifically for the `image` index health hangs when the index doesn't exist, perhaps you could query for healthchecks of every index in the cluster, and fail the healthcheck if `image` is not among them and green?

It sounds like it's coming along nicely!
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626920424)


*madewithkode* commented on Tue May 12 2020:

>> Hi again Onyenanu – if the index doesn't exist, the healthcheck should fail. This could happen in situations where we are switching Elasticsearch clusters in production and forgot to index data into the new one (or something went wrong while we were loading data into the new cluster).
> 
> In my experience, the ES Python libs can behave in unexpected ways that you sometimes have to work around. Since it seems like querying specifically for the `image` index health hangs when the index doesn't exist, perhaps you could query for healthchecks of every index in the cluster, and fail the healthcheck if `image` is not among them and green?
> 
> It sounds like it's coming along nicely!

Hey Alden...Many thanks again for coming through with better insights. Suggestion sounds nice, would proceed with it.

And yes, the whole stuff  is getting more interesting, learnt a handful in the few days :)
[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626973607)


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `api/catalog/api/views/health_views.py`
Content:
```
1 from rest_framework.response import Response
2 from rest_framework.views import APIView
3 
4 
5 class HealthCheck(APIView):
6     """
7     Returns a "200 OK" response if the server is running normally.
8 
9     This endpoint is used in production to ensure that the server should receive
10     traffic. If no response is provided, the server is deregistered from the
11     load balancer and destroyed.
12     """
13 
14     swagger_schema = None
15 
16     def get(self, request, format=None):
17         return Response({"status": "200 OK"}, status=200)
18
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/api/catalog/api/views/health_views.py b/api/catalog/api/views/health_views.py
--- a/api/catalog/api/views/health_views.py
+++ b/api/catalog/api/views/health_views.py
@@ -1,10 +1,19 @@
+from django.conf import settings
+from rest_framework import status
+from rest_framework.exceptions import APIException
+from rest_framework.request import Request
 from rest_framework.response import Response
 from rest_framework.views import APIView
 
 
+class ElasticsearchHealthcheckException(APIException):
+    status_code = status.HTTP_503_SERVICE_UNAVAILABLE
+
+
 class HealthCheck(APIView):
     """
-    Returns a "200 OK" response if the server is running normally.
+    Returns a "200 OK" response if the server is running normally. Returns 503
+    otherwise.
 
     This endpoint is used in production to ensure that the server should receive
     traffic. If no response is provided, the server is deregistered from the
@@ -13,5 +22,20 @@
 
     swagger_schema = None
 
-    def get(self, request, format=None):
+    def _check_es(self) -> Response | None:
+        """
+        Checks Elasticsearch cluster health. Raises an exception if ES is not healthy.
+        """
+        es_health = settings.ES.cluster.health(timeout="5s")
+
+        if es_health["timed_out"]:
+            raise ElasticsearchHealthcheckException("es_timed_out")
+
+        if (status := es_health["status"]) != "green":
+            raise ElasticsearchHealthcheckException(f"es_status_{status}")
+
+    def get(self, request: Request):
+        if "check_es" in request.query_params:
+            self._check_es()
+
         return Response({"status": "200 OK"}, status=200)

{"golden_diff": "diff --git a/api/catalog/api/views/health_views.py b/api/catalog/api/views/health_views.py\n--- a/api/catalog/api/views/health_views.py\n+++ b/api/catalog/api/views/health_views.py\n@@ -1,10 +1,19 @@\n+from django.conf import settings\n+from rest_framework import status\n+from rest_framework.exceptions import APIException\n+from rest_framework.request import Request\n from rest_framework.response import Response\n from rest_framework.views import APIView\n \n \n+class ElasticsearchHealthcheckException(APIException):\n+    status_code = status.HTTP_503_SERVICE_UNAVAILABLE\n+\n+\n class HealthCheck(APIView):\n     \"\"\"\n-    Returns a \"200 OK\" response if the server is running normally.\n+    Returns a \"200 OK\" response if the server is running normally. Returns 503\n+    otherwise.\n \n     This endpoint is used in production to ensure that the server should receive\n     traffic. If no response is provided, the server is deregistered from the\n@@ -13,5 +22,20 @@\n \n     swagger_schema = None\n \n-    def get(self, request, format=None):\n+    def _check_es(self) -> Response | None:\n+        \"\"\"\n+        Checks Elasticsearch cluster health. Raises an exception if ES is not healthy.\n+        \"\"\"\n+        es_health = settings.ES.cluster.health(timeout=\"5s\")\n+\n+        if es_health[\"timed_out\"]:\n+            raise ElasticsearchHealthcheckException(\"es_timed_out\")\n+\n+        if (status := es_health[\"status\"]) != \"green\":\n+            raise ElasticsearchHealthcheckException(f\"es_status_{status}\")\n+\n+    def get(self, request: Request):\n+        if \"check_es\" in request.query_params:\n+            self._check_es()\n+\n         return Response({\"status\": \"200 OK\"}, status=200)\n", "issue": "/healthcheck endpoint should check for Elasticsearch availability (original #487)\nThis issue has been migrated from the [CC Search API repository](https://github.com/creativecommons/cccatalog-api/issues/487)\r\n\r\n```\r\nAuthor: aldenstpage\r\nDate: Wed May 06 2020\r\nLabels: \u2728 goal: improvement,\ud83c\udff7 status: label work required,\ud83d\ude45 status: discontinued\r\n```\r\nDuring deployments, our load balancer repeatedly polls the `/healthcheck` endpoint to check that the server is reachable. If this check succeeds, the newly deployed instance starts receiving production traffic. Right now, if Elasticsearch is not responsive, `/healthcheck` will still return `200 OK`.\r\n\r\nThe healthcheck endpoint should check the health of the `image` index in Elasticsearch using the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html). If it is unavailable, return error 500. Log an informative message explaining why the healthcheck failed.\r\n\r\nBecause the healthcheck endpoint may be called many times, and Elasticsearch calls are not free, we should cache the response of Elasticsearch for up to 10 seconds per call.\r\n--------\r\n### Original Comments: \r\n\r\n*madewithkode* commented on Fri May 08 2020:\r\n\r\n>Hi Alden, this looks interesting, I'd love to work on it.\r\n  \r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625824482)\r\n\r\n\r\n*madewithkode* commented on Fri May 08 2020:\r\n\r\n>Hi Alden in order to check the health of the `image` index in the /healthcheck view, I'm trying to use the urllib's `urlopen()` method to make a request to Elasticsearch's  cluster API this way:\r\n\r\n`cluster_response = urlopen('http://0.0.0.0:8000/_cluster/health/image')`\r\n\r\nHowever, I keep getting a 404. Is there something I'm doing wrong?\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625854788)\r\n\r\n\r\n*madewithkode* commented on Fri May 08 2020:\r\n\r\n>> Hi Alden in order to check the health of the `image` index in the /healthcheck view, I'm trying to use the urllib's `urlopen()` method to make a request to Elasticsearch's cluster API this way:\r\n> \r\n> `cluster_response = urlopen('http://0.0.0.0:8000/_cluster/health/image')`\r\n> \r\n> However, I keep getting a 404. Is there something I'm doing wrong?\r\n\r\nFigured this, didn't know elastic search was running on a seperate host/port :)\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625905259)\r\n\r\n\r\n*aldenstpage* commented on Fri May 08 2020:\r\n\r\n>That's great! \r\n\r\nIt would be best to use the equivalent `elasticsearch-py` or `elasticsearch-dsl` query instead of making direct calls to the REST API (you can get an instance of the connection to Elasticsearch from `search_controller.py`). [Here's](https://discuss.elastic.co/t/how-to-get-cluster-health-using-python-api/25431) an example for getting the cluster health; there ought to also be a way to narrow the query to the `image` index.\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625995563)\r\n\r\n\r\n*madewithkode* commented on Sat May 09 2020:\r\n\r\n>Alright...would look at the suggestion.\r\n\r\nOn Fri, May 8, 2020, 21:06 Alden S Page <notifications@github.com> wrote:\r\n\r\n> It would be best to use the equivalent elasticsearch-py query instead of\r\n> making direct calls to the REST API. Here's\r\n> <https://discuss.elastic.co/t/how-to-get-cluster-health-using-python-api/25431>\r\n> an example for getting the cluster health; there ought to also be a way to\r\n> narrow the query to the image index.\r\n>\r\n> \u2014\r\n> You are receiving this because you commented.\r\n> Reply to this email directly, view it on GitHub\r\n> <https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-625995563>,\r\n> or unsubscribe\r\n> <https://github.com/notifications/unsubscribe-auth/AGFLMYA5WLAQPO5GYZNX5BTRQRQ5RANCNFSM4M2V5EKA>\r\n> .\r\n>\r\n\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626130991)\r\n\r\n\r\n*madewithkode* commented on Sat May 09 2020:\r\n\r\n>Hi Alden, I'm here again :)\r\nI'd love to ask, what status specifically signifies the availability of the `image` index. \r\n`red`, `yellow` or `green` ? or should i leave out this detail in the query since I'm using an already established connection instance from `search_controller.py` which waits for a `yellow` status by default.\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626164842)\r\n\r\n\r\n*madewithkode* commented on Sat May 09 2020:\r\n\r\n>### Update: \r\nI've successfully managed to query the health of the entire cluster, using the Elasticsearch connection instance gotten from `search_controller.py`. However when i try to limit the health check to just the `image` index, the request never resolves and continues to run forever with no response. And when i try to specify a timeout for the request, i get an \"Illegal argument exception\" even though `timeout` is a valid kwarg referenced in the [API docs](https://elasticsearch-py.readthedocs.io/en/master/api.html?highlight=health#elasticsearch.client.ClusterClient.health).\r\n\r\nIt'd be nice to point out that as at the time of writing, I'm yet to successfully run `./load_sample_data.sh` so i don't know if this could be linked to the above problem.\r\n\r\n\r\n\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626223763)\r\n\r\n\r\n*madewithkode* commented on Mon May 11 2020:\r\n\r\n>Hi Alden, Progress Report :)\r\n\r\nSuccessfully got the `load_sample_data.sh` to run,  and so far every other thing is working fine. \r\nI've also set up the 10s response caching on the `/healthcheck` view using redis and also the error logging.\r\n\r\nHowever, I figured out the reason for the unresponsiveness when querying the elastic search `image` index was that it was non-existent and that the whole cluster index was empty too.\r\n\r\nDo I need to do a manual population or something?\r\n\r\n\r\n\r\n\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626749591)\r\n\r\n\r\n*aldenstpage* commented on Mon May 11 2020:\r\n\r\n>Hi again Onyenanu \u2013 if the index doesn't exist, the healthcheck should fail. This could happen in situations where we are switching Elasticsearch clusters in production and forgot to index data into the new one (or something went wrong while we were loading data into the new cluster).\r\n\r\nIn my experience, the ES Python libs can behave in unexpected ways that you sometimes have to work around. Since it seems like querying specifically for the `image` index health hangs when the index doesn't exist, perhaps you could query for healthchecks of every index in the cluster, and fail the healthcheck if `image` is not among them and green?\r\n\r\nIt sounds like it's coming along nicely!\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626920424)\r\n\r\n\r\n*madewithkode* commented on Tue May 12 2020:\r\n\r\n>> Hi again Onyenanu \u2013 if the index doesn't exist, the healthcheck should fail. This could happen in situations where we are switching Elasticsearch clusters in production and forgot to index data into the new one (or something went wrong while we were loading data into the new cluster).\r\n> \r\n> In my experience, the ES Python libs can behave in unexpected ways that you sometimes have to work around. Since it seems like querying specifically for the `image` index health hangs when the index doesn't exist, perhaps you could query for healthchecks of every index in the cluster, and fail the healthcheck if `image` is not among them and green?\r\n> \r\n> It sounds like it's coming along nicely!\r\n\r\nHey Alden...Many thanks again for coming through with better insights. Suggestion sounds nice, would proceed with it.\r\n\r\nAnd yes, the whole stuff  is getting more interesting, learnt a handful in the few days :)\r\n[source](https://github.com/creativecommons/cccatalog-api/issues/487#issuecomment-626973607)\r\n\n", "before_files": [{"content": "from rest_framework.response import Response\nfrom rest_framework.views import APIView\n\n\nclass HealthCheck(APIView):\n    \"\"\"\n    Returns a \"200 OK\" response if the server is running normally.\n\n    This endpoint is used in production to ensure that the server should receive\n    traffic. If no response is provided, the server is deregistered from the\n    load balancer and destroyed.\n    \"\"\"\n\n    swagger_schema = None\n\n    def get(self, request, format=None):\n        return Response({\"status\": \"200 OK\"}, status=200)\n", "path": "api/catalog/api/views/health_views.py"}], "after_files": [{"content": "from django.conf import settings\nfrom rest_framework import status\nfrom rest_framework.exceptions import APIException\nfrom rest_framework.request import Request\nfrom rest_framework.response import Response\nfrom rest_framework.views import APIView\n\n\nclass ElasticsearchHealthcheckException(APIException):\n    status_code = status.HTTP_503_SERVICE_UNAVAILABLE\n\n\nclass HealthCheck(APIView):\n    \"\"\"\n    Returns a \"200 OK\" response if the server is running normally. Returns 503\n    otherwise.\n\n    This endpoint is used in production to ensure that the server should receive\n    traffic. If no response is provided, the server is deregistered from the\n    load balancer and destroyed.\n    \"\"\"\n\n    swagger_schema = None\n\n    def _check_es(self) -> Response | None:\n        \"\"\"\n        Checks Elasticsearch cluster health. Raises an exception if ES is not healthy.\n        \"\"\"\n        es_health = settings.ES.cluster.health(timeout=\"5s\")\n\n        if es_health[\"timed_out\"]:\n            raise ElasticsearchHealthcheckException(\"es_timed_out\")\n\n        if (status := es_health[\"status\"]) != \"green\":\n            raise ElasticsearchHealthcheckException(f\"es_status_{status}\")\n\n    def get(self, request: Request):\n        if \"check_es\" in request.query_params:\n            self._check_es()\n\n        return Response({\"status\": \"200 OK\"}, status=200)\n", "path": "api/catalog/api/views/health_views.py"}]}