prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Increase the dead links cache expiration timeout
  ## Problem
  <!-- Describe a problem solved by this feature; or delete the section entirely. -->
  Currently, every front-end search results in requesting the provider image URL (the direct URL) twice: once for the API dead links filtering and once for the thumbnail generation.
  The thumbnails are cached for a period of around a year (to be confirmed), and the link validation cache expires after 24 hours for active links and 120 days for broken links. This means we're re-requesting popular valid images every 24 hours. We could increase the cache expiration timeout to reduce the number of network requests to the providers (which have often been 429-ed lately).
  
  ## Description
  <!-- Describe the feature and how it solves the problem. -->
  We could set the cache expiration timeout for active links based on an environment variable to make it easily tweakable.
  I think increasing the active link cache TTL to **one week** could be a good balance between keeping the link status up-to-date and reducing the number of requests. What do you think about the timeout period, @openverse-team?
  
  ## Alternatives
  <!-- Describe any alternative solutions or features you have considered. How is this feature better? -->
  Another option for reducing the number of requests to providers could be to request the thumbnail endpoint first, and if the request headers show that the response header comes from the Cloudflare cache, consider the link not dead. If, however, the response is coming from the proxy and does not have a 200 status, then we can consider the link dead.
  
  ## Implementation
  <!-- Replace the [ ] with [x] to check the box. -->
  - [ ] 🙋 I would be interested in implementing this feature.
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `api/catalog/api/utils/validate_images.py`
  Content:
  ```
  1 import logging
  2 import time
  3 
  4 import django_redis
  5 import grequests
  6 
  7 from catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask
  8 
  9 
  10 parent_logger = logging.getLogger(__name__)
  11 
  12 
  13 CACHE_PREFIX = "valid:"
  14 
  15 
  16 def _get_cached_statuses(redis, image_urls):
  17     cached_statuses = redis.mget([CACHE_PREFIX + url for url in image_urls])
  18     return [int(b.decode("utf-8")) if b is not None else None for b in cached_statuses]
  19 
  20 
  21 def validate_images(query_hash, start_slice, results, image_urls):
  22     """
  23     Make sure images exist before we display them. Treat redirects as broken
  24     links since 99% of the time the redirect leads to a generic "not found"
  25     placeholder.
  26 
  27     Results are cached in redis and shared amongst all API servers in the
  28     cluster.
  29     """
  30     logger = parent_logger.getChild("validate_images")
  31     if not image_urls:
  32         logger.info("no image urls to validate")
  33         return
  34 
  35     logger.debug("starting validation")
  36     start_time = time.time()
  37     # Pull matching images from the cache.
  38     redis = django_redis.get_redis_connection("default")
  39     cached_statuses = _get_cached_statuses(redis, image_urls)
  40     logger.debug(f"len(cached_statuses)={len(cached_statuses)}")
  41     # Anything that isn't in the cache needs to be validated via HEAD request.
  42     to_verify = {}
  43     for idx, url in enumerate(image_urls):
  44         if cached_statuses[idx] is None:
  45             to_verify[url] = idx
  46     logger.debug(f"len(to_verify)={len(to_verify)}")
  47     reqs = (
  48         grequests.head(u, allow_redirects=False, timeout=2, verify=False)
  49         for u in to_verify.keys()
  50     )
  51     verified = grequests.map(reqs, exception_handler=_validation_failure)
  52     # Cache newly verified image statuses.
  53     to_cache = {}
  54     for idx, url in enumerate(to_verify.keys()):
  55         cache_key = CACHE_PREFIX + url
  56         if verified[idx]:
  57             status = verified[idx].status_code
  58         # Response didn't arrive in time. Try again later.
  59         else:
  60             status = -1
  61         to_cache[cache_key] = status
  62 
  63     thirty_minutes = 60 * 30
  64     twenty_four_hours_seconds = 60 * 60 * 24
  65     pipe = redis.pipeline()
  66     if len(to_cache) > 0:
  67         pipe.mset(to_cache)
  68     for key, status in to_cache.items():
  69         # Cache successful links for a day, and broken links for 120 days.
  70         if status == 200:
  71             logger.debug("healthy link " f"key={key} ")
  72             pipe.expire(key, twenty_four_hours_seconds)
  73         elif status == -1:
  74             logger.debug("no response from provider " f"key={key}")
  75             # Content provider failed to respond; try again in a short interval
  76             pipe.expire(key, thirty_minutes)
  77         else:
  78             logger.debug("broken link " f"key={key} ")
  79             pipe.expire(key, twenty_four_hours_seconds * 120)
  80     pipe.execute()
  81 
  82     # Merge newly verified results with cached statuses
  83     for idx, url in enumerate(to_verify):
  84         cache_idx = to_verify[url]
  85         if verified[idx] is not None:
  86             cached_statuses[cache_idx] = verified[idx].status_code
  87         else:
  88             cached_statuses[cache_idx] = -1
  89 
  90     # Create a new dead link mask
  91     new_mask = [1] * len(results)
  92     # Delete broken images from the search results response.
  93     for idx, _ in enumerate(cached_statuses):
  94         del_idx = len(cached_statuses) - idx - 1
  95         status = cached_statuses[del_idx]
  96         if status == 429 or status == 403:
  97             logger.warning(
  98                 "Image validation failed due to rate limiting or blocking. "
  99                 f"url={image_urls[idx]} "
  100                 f"status={status} "
  101             )
  102         elif status != 200:
  103             logger.info(
  104                 "Deleting broken image from results "
  105                 f"id={results[del_idx]['identifier']} "
  106                 f"status={status} "
  107             )
  108             del results[del_idx]
  109             new_mask[del_idx] = 0
  110 
  111     # Merge and cache the new mask
  112     mask = get_query_mask(query_hash)
  113     if mask:
  114         new_mask = mask[:start_slice] + new_mask
  115     save_query_mask(query_hash, new_mask)
  116 
  117     end_time = time.time()
  118     logger.debug(
  119         "end validation "
  120         f"end_time={end_time} "
  121         f"start_time={start_time} "
  122         f"delta={end_time - start_time} "
  123     )
  124 
  125 
  126 def _validation_failure(request, exception):
  127     logger = parent_logger.getChild("_validation_failure")
  128     logger.warning(f"Failed to validate image! Reason: {exception}")
  129
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/api/catalog/api/utils/validate_images.py b/api/catalog/api/utils/validate_images.py
  --- a/api/catalog/api/utils/validate_images.py
  +++ b/api/catalog/api/utils/validate_images.py
  @@ -3,6 +3,7 @@
   
   import django_redis
   import grequests
  +from decouple import config
   
   from catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask
   
  @@ -18,6 +19,10 @@
       return [int(b.decode("utf-8")) if b is not None else None for b in cached_statuses]
   
   
  +def _get_expiry(status, default):
  +    return config(f"LINK_VALIDATION_CACHE_EXPIRY__{status}", default=default, cast=int)
  +
  +
   def validate_images(query_hash, start_slice, results, image_urls):
       """
       Make sure images exist before we display them. Treat redirects as broken
  @@ -69,14 +74,17 @@
           # Cache successful links for a day, and broken links for 120 days.
           if status == 200:
               logger.debug("healthy link " f"key={key} ")
  -            pipe.expire(key, twenty_four_hours_seconds)
  +            expiry = _get_expiry(200, twenty_four_hours_seconds * 30)
           elif status == -1:
               logger.debug("no response from provider " f"key={key}")
               # Content provider failed to respond; try again in a short interval
  -            pipe.expire(key, thirty_minutes)
  +            expiry = _get_expiry("_1", thirty_minutes)
           else:
               logger.debug("broken link " f"key={key} ")
  -            pipe.expire(key, twenty_four_hours_seconds * 120)
  +            expiry = _get_expiry("DEFAULT", twenty_four_hours_seconds * 120)
  +
  +        pipe.expire(key, expiry)
  +
       pipe.execute()
   
       # Merge newly verified results with cached statuses
  
verification_info: {"golden_diff": "diff --git a/api/catalog/api/utils/validate_images.py b/api/catalog/api/utils/validate_images.py\n--- a/api/catalog/api/utils/validate_images.py\n+++ b/api/catalog/api/utils/validate_images.py\n@@ -3,6 +3,7 @@\n \n import django_redis\n import grequests\n+from decouple import config\n \n from catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask\n \n@@ -18,6 +19,10 @@\n     return [int(b.decode(\"utf-8\")) if b is not None else None for b in cached_statuses]\n \n \n+def _get_expiry(status, default):\n+    return config(f\"LINK_VALIDATION_CACHE_EXPIRY__{status}\", default=default, cast=int)\n+\n+\n def validate_images(query_hash, start_slice, results, image_urls):\n     \"\"\"\n     Make sure images exist before we display them. Treat redirects as broken\n@@ -69,14 +74,17 @@\n         # Cache successful links for a day, and broken links for 120 days.\n         if status == 200:\n             logger.debug(\"healthy link \" f\"key={key} \")\n-            pipe.expire(key, twenty_four_hours_seconds)\n+            expiry = _get_expiry(200, twenty_four_hours_seconds * 30)\n         elif status == -1:\n             logger.debug(\"no response from provider \" f\"key={key}\")\n             # Content provider failed to respond; try again in a short interval\n-            pipe.expire(key, thirty_minutes)\n+            expiry = _get_expiry(\"_1\", thirty_minutes)\n         else:\n             logger.debug(\"broken link \" f\"key={key} \")\n-            pipe.expire(key, twenty_four_hours_seconds * 120)\n+            expiry = _get_expiry(\"DEFAULT\", twenty_four_hours_seconds * 120)\n+\n+        pipe.expire(key, expiry)\n+\n     pipe.execute()\n \n     # Merge newly verified results with cached statuses\n", "issue": "Increase the dead links cache expiration timeout\n## Problem\r\n<!-- Describe a problem solved by this feature; or delete the section entirely. -->\r\nCurrently, every front-end search results in requesting the provider image URL (the direct URL) twice: once for the API dead links filtering and once for the thumbnail generation.\r\nThe thumbnails are cached for a period of around a year (to be confirmed), and the link validation cache expires after 24 hours for active links and 120 days for broken links. This means we're re-requesting popular valid images every 24 hours. We could increase the cache expiration timeout to reduce the number of network requests to the providers (which have often been 429-ed lately).\r\n\r\n## Description\r\n<!-- Describe the feature and how it solves the problem. -->\r\nWe could set the cache expiration timeout for active links based on an environment variable to make it easily tweakable.\r\nI think increasing the active link cache TTL to **one week** could be a good balance between keeping the link status up-to-date and reducing the number of requests. What do you think about the timeout period, @openverse-team?\r\n\r\n## Alternatives\r\n<!-- Describe any alternative solutions or features you have considered. How is this feature better? -->\r\nAnother option for reducing the number of requests to providers could be to request the thumbnail endpoint first, and if the request headers show that the response header comes from the Cloudflare cache, consider the link not dead. If, however, the response is coming from the proxy and does not have a 200 status, then we can consider the link dead.\r\n\r\n## Implementation\r\n<!-- Replace the [ ] with [x] to check the box. -->\r\n- [ ] \ud83d\ude4b I would be interested in implementing this feature.\r\n\n", "before_files": [{"content": "import logging\nimport time\n\nimport django_redis\nimport grequests\n\nfrom catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask\n\n\nparent_logger = logging.getLogger(__name__)\n\n\nCACHE_PREFIX = \"valid:\"\n\n\ndef _get_cached_statuses(redis, image_urls):\n    cached_statuses = redis.mget([CACHE_PREFIX + url for url in image_urls])\n    return [int(b.decode(\"utf-8\")) if b is not None else None for b in cached_statuses]\n\n\ndef validate_images(query_hash, start_slice, results, image_urls):\n    \"\"\"\n    Make sure images exist before we display them. Treat redirects as broken\n    links since 99% of the time the redirect leads to a generic \"not found\"\n    placeholder.\n\n    Results are cached in redis and shared amongst all API servers in the\n    cluster.\n    \"\"\"\n    logger = parent_logger.getChild(\"validate_images\")\n    if not image_urls:\n        logger.info(\"no image urls to validate\")\n        return\n\n    logger.debug(\"starting validation\")\n    start_time = time.time()\n    # Pull matching images from the cache.\n    redis = django_redis.get_redis_connection(\"default\")\n    cached_statuses = _get_cached_statuses(redis, image_urls)\n    logger.debug(f\"len(cached_statuses)={len(cached_statuses)}\")\n    # Anything that isn't in the cache needs to be validated via HEAD request.\n    to_verify = {}\n    for idx, url in enumerate(image_urls):\n        if cached_statuses[idx] is None:\n            to_verify[url] = idx\n    logger.debug(f\"len(to_verify)={len(to_verify)}\")\n    reqs = (\n        grequests.head(u, allow_redirects=False, timeout=2, verify=False)\n        for u in to_verify.keys()\n    )\n    verified = grequests.map(reqs, exception_handler=_validation_failure)\n    # Cache newly verified image statuses.\n    to_cache = {}\n    for idx, url in enumerate(to_verify.keys()):\n        cache_key = CACHE_PREFIX + url\n        if verified[idx]:\n            status = verified[idx].status_code\n        # Response didn't arrive in time. Try again later.\n        else:\n            status = -1\n        to_cache[cache_key] = status\n\n    thirty_minutes = 60 * 30\n    twenty_four_hours_seconds = 60 * 60 * 24\n    pipe = redis.pipeline()\n    if len(to_cache) > 0:\n        pipe.mset(to_cache)\n    for key, status in to_cache.items():\n        # Cache successful links for a day, and broken links for 120 days.\n        if status == 200:\n            logger.debug(\"healthy link \" f\"key={key} \")\n            pipe.expire(key, twenty_four_hours_seconds)\n        elif status == -1:\n            logger.debug(\"no response from provider \" f\"key={key}\")\n            # Content provider failed to respond; try again in a short interval\n            pipe.expire(key, thirty_minutes)\n        else:\n            logger.debug(\"broken link \" f\"key={key} \")\n            pipe.expire(key, twenty_four_hours_seconds * 120)\n    pipe.execute()\n\n    # Merge newly verified results with cached statuses\n    for idx, url in enumerate(to_verify):\n        cache_idx = to_verify[url]\n        if verified[idx] is not None:\n            cached_statuses[cache_idx] = verified[idx].status_code\n        else:\n            cached_statuses[cache_idx] = -1\n\n    # Create a new dead link mask\n    new_mask = [1] * len(results)\n    # Delete broken images from the search results response.\n    for idx, _ in enumerate(cached_statuses):\n        del_idx = len(cached_statuses) - idx - 1\n        status = cached_statuses[del_idx]\n        if status == 429 or status == 403:\n            logger.warning(\n                \"Image validation failed due to rate limiting or blocking. \"\n                f\"url={image_urls[idx]} \"\n                f\"status={status} \"\n            )\n        elif status != 200:\n            logger.info(\n                \"Deleting broken image from results \"\n                f\"id={results[del_idx]['identifier']} \"\n                f\"status={status} \"\n            )\n            del results[del_idx]\n            new_mask[del_idx] = 0\n\n    # Merge and cache the new mask\n    mask = get_query_mask(query_hash)\n    if mask:\n        new_mask = mask[:start_slice] + new_mask\n    save_query_mask(query_hash, new_mask)\n\n    end_time = time.time()\n    logger.debug(\n        \"end validation \"\n        f\"end_time={end_time} \"\n        f\"start_time={start_time} \"\n        f\"delta={end_time - start_time} \"\n    )\n\n\ndef _validation_failure(request, exception):\n    logger = parent_logger.getChild(\"_validation_failure\")\n    logger.warning(f\"Failed to validate image! Reason: {exception}\")\n", "path": "api/catalog/api/utils/validate_images.py"}], "after_files": [{"content": "import logging\nimport time\n\nimport django_redis\nimport grequests\nfrom decouple import config\n\nfrom catalog.api.utils.dead_link_mask import get_query_mask, save_query_mask\n\n\nparent_logger = logging.getLogger(__name__)\n\n\nCACHE_PREFIX = \"valid:\"\n\n\ndef _get_cached_statuses(redis, image_urls):\n    cached_statuses = redis.mget([CACHE_PREFIX + url for url in image_urls])\n    return [int(b.decode(\"utf-8\")) if b is not None else None for b in cached_statuses]\n\n\ndef _get_expiry(status, default):\n    return config(f\"LINK_VALIDATION_CACHE_EXPIRY__{status}\", default=default, cast=int)\n\n\ndef validate_images(query_hash, start_slice, results, image_urls):\n    \"\"\"\n    Make sure images exist before we display them. Treat redirects as broken\n    links since 99% of the time the redirect leads to a generic \"not found\"\n    placeholder.\n\n    Results are cached in redis and shared amongst all API servers in the\n    cluster.\n    \"\"\"\n    logger = parent_logger.getChild(\"validate_images\")\n    if not image_urls:\n        logger.info(\"no image urls to validate\")\n        return\n\n    logger.debug(\"starting validation\")\n    start_time = time.time()\n    # Pull matching images from the cache.\n    redis = django_redis.get_redis_connection(\"default\")\n    cached_statuses = _get_cached_statuses(redis, image_urls)\n    logger.debug(f\"len(cached_statuses)={len(cached_statuses)}\")\n    # Anything that isn't in the cache needs to be validated via HEAD request.\n    to_verify = {}\n    for idx, url in enumerate(image_urls):\n        if cached_statuses[idx] is None:\n            to_verify[url] = idx\n    logger.debug(f\"len(to_verify)={len(to_verify)}\")\n    reqs = (\n        grequests.head(u, allow_redirects=False, timeout=2, verify=False)\n        for u in to_verify.keys()\n    )\n    verified = grequests.map(reqs, exception_handler=_validation_failure)\n    # Cache newly verified image statuses.\n    to_cache = {}\n    for idx, url in enumerate(to_verify.keys()):\n        cache_key = CACHE_PREFIX + url\n        if verified[idx]:\n            status = verified[idx].status_code\n        # Response didn't arrive in time. Try again later.\n        else:\n            status = -1\n        to_cache[cache_key] = status\n\n    thirty_minutes = 60 * 30\n    twenty_four_hours_seconds = 60 * 60 * 24\n    pipe = redis.pipeline()\n    if len(to_cache) > 0:\n        pipe.mset(to_cache)\n    for key, status in to_cache.items():\n        # Cache successful links for a day, and broken links for 120 days.\n        if status == 200:\n            logger.debug(\"healthy link \" f\"key={key} \")\n            expiry = _get_expiry(200, twenty_four_hours_seconds * 30)\n        elif status == -1:\n            logger.debug(\"no response from provider \" f\"key={key}\")\n            # Content provider failed to respond; try again in a short interval\n            expiry = _get_expiry(\"_1\", thirty_minutes)\n        else:\n            logger.debug(\"broken link \" f\"key={key} \")\n            expiry = _get_expiry(\"DEFAULT\", twenty_four_hours_seconds * 120)\n\n        pipe.expire(key, expiry)\n\n    pipe.execute()\n\n    # Merge newly verified results with cached statuses\n    for idx, url in enumerate(to_verify):\n        cache_idx = to_verify[url]\n        if verified[idx] is not None:\n            cached_statuses[cache_idx] = verified[idx].status_code\n        else:\n            cached_statuses[cache_idx] = -1\n\n    # Create a new dead link mask\n    new_mask = [1] * len(results)\n    # Delete broken images from the search results response.\n    for idx, _ in enumerate(cached_statuses):\n        del_idx = len(cached_statuses) - idx - 1\n        status = cached_statuses[del_idx]\n        if status == 429 or status == 403:\n            logger.warning(\n                \"Image validation failed due to rate limiting or blocking. \"\n                f\"url={image_urls[idx]} \"\n                f\"status={status} \"\n            )\n        elif status != 200:\n            logger.info(\n                \"Deleting broken image from results \"\n                f\"id={results[del_idx]['identifier']} \"\n                f\"status={status} \"\n            )\n            del results[del_idx]\n            new_mask[del_idx] = 0\n\n    # Merge and cache the new mask\n    mask = get_query_mask(query_hash)\n    if mask:\n        new_mask = mask[:start_slice] + new_mask\n    save_query_mask(query_hash, new_mask)\n\n    end_time = time.time()\n    logger.debug(\n        \"end validation \"\n        f\"end_time={end_time} \"\n        f\"start_time={start_time} \"\n        f\"delta={end_time - start_time} \"\n    )\n\n\ndef _validation_failure(request, exception):\n    logger = parent_logger.getChild(\"_validation_failure\")\n    logger.warning(f\"Failed to validate image! Reason: {exception}\")\n", "path": "api/catalog/api/utils/validate_images.py"}]}