prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  T - Update robots.tx (Warnings have spiked on Search Console)
  **Summary**
  We are seeing a large spike in warnings on Google Search Console. The spike started on December 22, 2019.
  
  <img width="952" alt="Coverage" src="https://user-images.githubusercontent.com/557852/72343521-5fb44000-36cf-11ea-8f1f-12955b98e34d.png">
  
  **Steps To Reproduce (STR)**
  1. Go to Search Console
  2. Open Coverage Report
  3. open "Indexed, though blocked by robots.txt"
  
  
  **Actual behavior**
  It looks like a bunch of URLs are blocked from indexing by Google that should not be blocked. Examples are:
  * https://developer.mozilla.org/it/docs/Web/API/GlobalEventHandlers/onresize$translate
  * https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/end%24edit
  * https://developer.mozilla.org/en-US/docs/Web/API/RTCIceCandidateInit/sdpMid$translate?tolocale=de
  
  When I test these URLs in Search Console's robots.txt tester it tells me that those URLs are allowed, and they should be. $translate, or $edit are not in robots.txt
  
  
  **Expected behavior**
  Those URLs should not be blocked, so that Google can crawl the pages, discover the no-index meta tag and remove them from its index.
  
  
  **Additional context**
  I assume that something is broken when it comes to handling the $ chracter in the URL. Maybe we just remove everything with the $url from our robots.txt. As far as I can tell, almost all of those URLs now live under wiki.developer.mozilla.org anyway.
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `kuma/landing/views.py`
  Content:
  ```
  1 from django.conf import settings
  2 from django.http import HttpResponse
  3 from django.shortcuts import redirect, render
  4 from django.views import static
  5 from django.views.decorators.cache import never_cache
  6 from django.views.generic import RedirectView
  7 
  8 from kuma.core.decorators import ensure_wiki_domain, shared_cache_control
  9 from kuma.core.utils import is_wiki
  10 from kuma.feeder.models import Bundle
  11 from kuma.feeder.sections import SECTION_HACKS
  12 from kuma.search.models import Filter
  13 
  14 from .utils import favicon_url
  15 
  16 
  17 @shared_cache_control
  18 def contribute_json(request):
  19     return static.serve(request, "contribute.json", document_root=settings.ROOT)
  20 
  21 
  22 @shared_cache_control
  23 def home(request):
  24     """Home page."""
  25     context = {}
  26     # Need for both wiki and react homepage
  27     context["updates"] = list(Bundle.objects.recent_entries(SECTION_HACKS.updates)[:5])
  28 
  29     # The default template name
  30     template_name = "landing/react_homepage.html"
  31     if is_wiki(request):
  32         template_name = "landing/homepage.html"
  33         context["default_filters"] = Filter.objects.default_filters()
  34     return render(request, template_name, context)
  35 
  36 
  37 @ensure_wiki_domain
  38 @never_cache
  39 def maintenance_mode(request):
  40     if settings.MAINTENANCE_MODE:
  41         return render(request, "landing/maintenance-mode.html")
  42     else:
  43         return redirect("home")
  44 
  45 
  46 @ensure_wiki_domain
  47 @shared_cache_control
  48 def promote_buttons(request):
  49     """Bug 646192: MDN affiliate buttons"""
  50     return render(request, "landing/promote_buttons.html")
  51 
  52 
  53 ROBOTS_ALLOWED_TXT = """\
  54 User-agent: *
  55 Sitemap: https://developer.mozilla.org/sitemap.xml
  56 
  57 Disallow: /admin/
  58 Disallow: /api/
  59 Disallow: /*/dashboards/*
  60 Disallow: /*docs/feeds
  61 Disallow: /*docs/templates
  62 Disallow: /*docs*Template:
  63 Disallow: /*docs/all
  64 Disallow: /*docs/tag*
  65 Disallow: /*docs/needs-review*
  66 Disallow: /*docs/localization-tag*
  67 Disallow: /*docs/with-errors
  68 Disallow: /*docs/without-parent
  69 Disallow: /*docs/top-level
  70 Disallow: /*docs/new
  71 Disallow: /*docs/get-documents
  72 Disallow: /*docs/submit_akismet_spam
  73 Disallow: /*docs/load*
  74 Disallow: /*docs/Experiment:*
  75 Disallow: /*$api
  76 Disallow: /*$compare
  77 Disallow: /*$revision
  78 Disallow: /*$history
  79 Disallow: /*$children
  80 Disallow: /*$flag
  81 Disallow: /*$locales
  82 Disallow: /*$toc
  83 Disallow: /*$move
  84 Disallow: /*$quick-review
  85 Disallow: /*$samples
  86 Disallow: /*$revert
  87 Disallow: /*$repair_breadcrumbs
  88 Disallow: /*$delete
  89 Disallow: /*$restore
  90 Disallow: /*$purge
  91 Disallow: /*$subscribe
  92 Disallow: /*$subscribe_to_tree
  93 Disallow: /*$vote
  94 Disallow: /*docs.json
  95 Disallow: /*docs/ckeditor_config.js
  96 Disallow: /*/files/
  97 Disallow: /media
  98 Disallow: /*move-requested
  99 Disallow: /*preview-wiki-content
  100 Disallow: /*profiles*/edit
  101 Disallow: /skins
  102 Disallow: /*type=feed
  103 Disallow: /*users/
  104 """ + "\n".join(
  105     "Disallow: /{locale}/search".format(locale=locale)
  106     for locale in settings.ENABLED_LOCALES
  107 )
  108 
  109 ROBOTS_GO_AWAY_TXT = """\
  110 User-Agent: *
  111 Disallow: /
  112 """
  113 
  114 
  115 @shared_cache_control
  116 def robots_txt(request):
  117     """Serve robots.txt that allows or forbids robots."""
  118     host = request.get_host()
  119     if host in settings.ALLOW_ROBOTS_DOMAINS:
  120         robots = ""
  121     elif host in settings.ALLOW_ROBOTS_WEB_DOMAINS:
  122         robots = ROBOTS_ALLOWED_TXT
  123     else:
  124         robots = ROBOTS_GO_AWAY_TXT
  125     return HttpResponse(robots, content_type="text/plain")
  126 
  127 
  128 class FaviconRedirect(RedirectView):
  129     """Redirect to the favicon in the static img folder (bug 1402497)"""
  130 
  131     def get_redirect_url(self, *args, **kwargs):
  132         return favicon_url()
  133
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/kuma/landing/views.py b/kuma/landing/views.py
  --- a/kuma/landing/views.py
  +++ b/kuma/landing/views.py
  @@ -50,6 +50,13 @@
       return render(request, "landing/promote_buttons.html")
   
   
  +ROBOTS_ALL_ALLOWED_TXT = """\
  +User-agent: *
  +Sitemap: https://wiki.developer.mozilla.org/sitemap.xml
  +
  +Disallow:
  +"""
  +
   ROBOTS_ALLOWED_TXT = """\
   User-agent: *
   Sitemap: https://developer.mozilla.org/sitemap.xml
  @@ -119,7 +126,10 @@
       if host in settings.ALLOW_ROBOTS_DOMAINS:
           robots = ""
       elif host in settings.ALLOW_ROBOTS_WEB_DOMAINS:
  -        robots = ROBOTS_ALLOWED_TXT
  +        if host == settings.WIKI_HOST:
  +            robots = ROBOTS_ALL_ALLOWED_TXT
  +        else:
  +            robots = ROBOTS_ALLOWED_TXT
       else:
           robots = ROBOTS_GO_AWAY_TXT
       return HttpResponse(robots, content_type="text/plain")
  
verification_info: {"golden_diff": "diff --git a/kuma/landing/views.py b/kuma/landing/views.py\n--- a/kuma/landing/views.py\n+++ b/kuma/landing/views.py\n@@ -50,6 +50,13 @@\n     return render(request, \"landing/promote_buttons.html\")\n \n \n+ROBOTS_ALL_ALLOWED_TXT = \"\"\"\\\n+User-agent: *\n+Sitemap: https://wiki.developer.mozilla.org/sitemap.xml\n+\n+Disallow:\n+\"\"\"\n+\n ROBOTS_ALLOWED_TXT = \"\"\"\\\n User-agent: *\n Sitemap: https://developer.mozilla.org/sitemap.xml\n@@ -119,7 +126,10 @@\n     if host in settings.ALLOW_ROBOTS_DOMAINS:\n         robots = \"\"\n     elif host in settings.ALLOW_ROBOTS_WEB_DOMAINS:\n-        robots = ROBOTS_ALLOWED_TXT\n+        if host == settings.WIKI_HOST:\n+            robots = ROBOTS_ALL_ALLOWED_TXT\n+        else:\n+            robots = ROBOTS_ALLOWED_TXT\n     else:\n         robots = ROBOTS_GO_AWAY_TXT\n     return HttpResponse(robots, content_type=\"text/plain\")\n", "issue": "T - Update robots.tx (Warnings have spiked on Search Console)\n**Summary**\r\nWe are seeing a large spike in warnings on Google Search Console. The spike started on December 22, 2019.\r\n\r\n<img width=\"952\" alt=\"Coverage\" src=\"https://user-images.githubusercontent.com/557852/72343521-5fb44000-36cf-11ea-8f1f-12955b98e34d.png\">\r\n\r\n**Steps To Reproduce (STR)**\r\n1. Go to Search Console\r\n2. Open Coverage Report\r\n3. open \"Indexed, though blocked by robots.txt\"\r\n\r\n\r\n**Actual behavior**\r\nIt looks like a bunch of URLs are blocked from indexing by Google that should not be blocked. Examples are:\r\n* https://developer.mozilla.org/it/docs/Web/API/GlobalEventHandlers/onresize$translate\r\n* https://developer.mozilla.org/en-US/docs/Web/SVG/Attribute/end%24edit\r\n* https://developer.mozilla.org/en-US/docs/Web/API/RTCIceCandidateInit/sdpMid$translate?tolocale=de\r\n\r\nWhen I test these URLs in Search Console's robots.txt tester it tells me that those URLs are allowed, and they should be. $translate, or $edit are not in robots.txt\r\n\r\n\r\n**Expected behavior**\r\nThose URLs should not be blocked, so that Google can crawl the pages, discover the no-index meta tag and remove them from its index.\r\n\r\n\r\n**Additional context**\r\nI assume that something is broken when it comes to handling the $ chracter in the URL. Maybe we just remove everything with the $url from our robots.txt. As far as I can tell, almost all of those URLs now live under wiki.developer.mozilla.org anyway.\n", "before_files": [{"content": "from django.conf import settings\nfrom django.http import HttpResponse\nfrom django.shortcuts import redirect, render\nfrom django.views import static\nfrom django.views.decorators.cache import never_cache\nfrom django.views.generic import RedirectView\n\nfrom kuma.core.decorators import ensure_wiki_domain, shared_cache_control\nfrom kuma.core.utils import is_wiki\nfrom kuma.feeder.models import Bundle\nfrom kuma.feeder.sections import SECTION_HACKS\nfrom kuma.search.models import Filter\n\nfrom .utils import favicon_url\n\n\n@shared_cache_control\ndef contribute_json(request):\n    return static.serve(request, \"contribute.json\", document_root=settings.ROOT)\n\n\n@shared_cache_control\ndef home(request):\n    \"\"\"Home page.\"\"\"\n    context = {}\n    # Need for both wiki and react homepage\n    context[\"updates\"] = list(Bundle.objects.recent_entries(SECTION_HACKS.updates)[:5])\n\n    # The default template name\n    template_name = \"landing/react_homepage.html\"\n    if is_wiki(request):\n        template_name = \"landing/homepage.html\"\n        context[\"default_filters\"] = Filter.objects.default_filters()\n    return render(request, template_name, context)\n\n\n@ensure_wiki_domain\n@never_cache\ndef maintenance_mode(request):\n    if settings.MAINTENANCE_MODE:\n        return render(request, \"landing/maintenance-mode.html\")\n    else:\n        return redirect(\"home\")\n\n\n@ensure_wiki_domain\n@shared_cache_control\ndef promote_buttons(request):\n    \"\"\"Bug 646192: MDN affiliate buttons\"\"\"\n    return render(request, \"landing/promote_buttons.html\")\n\n\nROBOTS_ALLOWED_TXT = \"\"\"\\\nUser-agent: *\nSitemap: https://developer.mozilla.org/sitemap.xml\n\nDisallow: /admin/\nDisallow: /api/\nDisallow: /*/dashboards/*\nDisallow: /*docs/feeds\nDisallow: /*docs/templates\nDisallow: /*docs*Template:\nDisallow: /*docs/all\nDisallow: /*docs/tag*\nDisallow: /*docs/needs-review*\nDisallow: /*docs/localization-tag*\nDisallow: /*docs/with-errors\nDisallow: /*docs/without-parent\nDisallow: /*docs/top-level\nDisallow: /*docs/new\nDisallow: /*docs/get-documents\nDisallow: /*docs/submit_akismet_spam\nDisallow: /*docs/load*\nDisallow: /*docs/Experiment:*\nDisallow: /*$api\nDisallow: /*$compare\nDisallow: /*$revision\nDisallow: /*$history\nDisallow: /*$children\nDisallow: /*$flag\nDisallow: /*$locales\nDisallow: /*$toc\nDisallow: /*$move\nDisallow: /*$quick-review\nDisallow: /*$samples\nDisallow: /*$revert\nDisallow: /*$repair_breadcrumbs\nDisallow: /*$delete\nDisallow: /*$restore\nDisallow: /*$purge\nDisallow: /*$subscribe\nDisallow: /*$subscribe_to_tree\nDisallow: /*$vote\nDisallow: /*docs.json\nDisallow: /*docs/ckeditor_config.js\nDisallow: /*/files/\nDisallow: /media\nDisallow: /*move-requested\nDisallow: /*preview-wiki-content\nDisallow: /*profiles*/edit\nDisallow: /skins\nDisallow: /*type=feed\nDisallow: /*users/\n\"\"\" + \"\\n\".join(\n    \"Disallow: /{locale}/search\".format(locale=locale)\n    for locale in settings.ENABLED_LOCALES\n)\n\nROBOTS_GO_AWAY_TXT = \"\"\"\\\nUser-Agent: *\nDisallow: /\n\"\"\"\n\n\n@shared_cache_control\ndef robots_txt(request):\n    \"\"\"Serve robots.txt that allows or forbids robots.\"\"\"\n    host = request.get_host()\n    if host in settings.ALLOW_ROBOTS_DOMAINS:\n        robots = \"\"\n    elif host in settings.ALLOW_ROBOTS_WEB_DOMAINS:\n        robots = ROBOTS_ALLOWED_TXT\n    else:\n        robots = ROBOTS_GO_AWAY_TXT\n    return HttpResponse(robots, content_type=\"text/plain\")\n\n\nclass FaviconRedirect(RedirectView):\n    \"\"\"Redirect to the favicon in the static img folder (bug 1402497)\"\"\"\n\n    def get_redirect_url(self, *args, **kwargs):\n        return favicon_url()\n", "path": "kuma/landing/views.py"}], "after_files": [{"content": "from django.conf import settings\nfrom django.http import HttpResponse\nfrom django.shortcuts import redirect, render\nfrom django.views import static\nfrom django.views.decorators.cache import never_cache\nfrom django.views.generic import RedirectView\n\nfrom kuma.core.decorators import ensure_wiki_domain, shared_cache_control\nfrom kuma.core.utils import is_wiki\nfrom kuma.feeder.models import Bundle\nfrom kuma.feeder.sections import SECTION_HACKS\nfrom kuma.search.models import Filter\n\nfrom .utils import favicon_url\n\n\n@shared_cache_control\ndef contribute_json(request):\n    return static.serve(request, \"contribute.json\", document_root=settings.ROOT)\n\n\n@shared_cache_control\ndef home(request):\n    \"\"\"Home page.\"\"\"\n    context = {}\n    # Need for both wiki and react homepage\n    context[\"updates\"] = list(Bundle.objects.recent_entries(SECTION_HACKS.updates)[:5])\n\n    # The default template name\n    template_name = \"landing/react_homepage.html\"\n    if is_wiki(request):\n        template_name = \"landing/homepage.html\"\n        context[\"default_filters\"] = Filter.objects.default_filters()\n    return render(request, template_name, context)\n\n\n@ensure_wiki_domain\n@never_cache\ndef maintenance_mode(request):\n    if settings.MAINTENANCE_MODE:\n        return render(request, \"landing/maintenance-mode.html\")\n    else:\n        return redirect(\"home\")\n\n\n@ensure_wiki_domain\n@shared_cache_control\ndef promote_buttons(request):\n    \"\"\"Bug 646192: MDN affiliate buttons\"\"\"\n    return render(request, \"landing/promote_buttons.html\")\n\n\nROBOTS_ALL_ALLOWED_TXT = \"\"\"\\\nUser-agent: *\nSitemap: https://wiki.developer.mozilla.org/sitemap.xml\n\nDisallow:\n\"\"\"\n\nROBOTS_ALLOWED_TXT = \"\"\"\\\nUser-agent: *\nSitemap: https://developer.mozilla.org/sitemap.xml\n\nDisallow: /admin/\nDisallow: /api/\nDisallow: /*/dashboards/*\nDisallow: /*docs/feeds\nDisallow: /*docs/templates\nDisallow: /*docs*Template:\nDisallow: /*docs/all\nDisallow: /*docs/tag*\nDisallow: /*docs/needs-review*\nDisallow: /*docs/localization-tag*\nDisallow: /*docs/with-errors\nDisallow: /*docs/without-parent\nDisallow: /*docs/top-level\nDisallow: /*docs/new\nDisallow: /*docs/get-documents\nDisallow: /*docs/submit_akismet_spam\nDisallow: /*docs/load*\nDisallow: /*docs/Experiment:*\nDisallow: /*$api\nDisallow: /*$compare\nDisallow: /*$revision\nDisallow: /*$history\nDisallow: /*$children\nDisallow: /*$flag\nDisallow: /*$locales\nDisallow: /*$toc\nDisallow: /*$move\nDisallow: /*$quick-review\nDisallow: /*$samples\nDisallow: /*$revert\nDisallow: /*$repair_breadcrumbs\nDisallow: /*$delete\nDisallow: /*$restore\nDisallow: /*$purge\nDisallow: /*$subscribe\nDisallow: /*$subscribe_to_tree\nDisallow: /*$vote\nDisallow: /*docs.json\nDisallow: /*docs/ckeditor_config.js\nDisallow: /*/files/\nDisallow: /media\nDisallow: /*move-requested\nDisallow: /*preview-wiki-content\nDisallow: /*profiles*/edit\nDisallow: /skins\nDisallow: /*type=feed\nDisallow: /*users/\n\"\"\" + \"\\n\".join(\n    \"Disallow: /{locale}/search\".format(locale=locale)\n    for locale in settings.ENABLED_LOCALES\n)\n\nROBOTS_GO_AWAY_TXT = \"\"\"\\\nUser-Agent: *\nDisallow: /\n\"\"\"\n\n\n@shared_cache_control\ndef robots_txt(request):\n    \"\"\"Serve robots.txt that allows or forbids robots.\"\"\"\n    host = request.get_host()\n    if host in settings.ALLOW_ROBOTS_DOMAINS:\n        robots = \"\"\n    elif host in settings.ALLOW_ROBOTS_WEB_DOMAINS:\n        if host == settings.WIKI_HOST:\n            robots = ROBOTS_ALL_ALLOWED_TXT\n        else:\n            robots = ROBOTS_ALLOWED_TXT\n    else:\n        robots = ROBOTS_GO_AWAY_TXT\n    return HttpResponse(robots, content_type=\"text/plain\")\n\n\nclass FaviconRedirect(RedirectView):\n    \"\"\"Redirect to the favicon in the static img folder (bug 1402497)\"\"\"\n\n    def get_redirect_url(self, *args, **kwargs):\n        return favicon_url()\n", "path": "kuma/landing/views.py"}]}