prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Translation for zh_TW gets ignored in some places
  In the webapp, if I try switching to the translation for "Chinese (Taiwan)", a lot of the text on the screen is still untranslated:
  ![image](https://user-images.githubusercontent.com/28173/96927079-63010700-146b-11eb-843e-ebede1e91469.png)
  That's even though many (at least) of those strings [do have translations in Transifex](https://www.transifex.com/zulip/zulip/translate/#zh_TW/$/67194598?q=text%3Asettings). Those translations have been there for months and do indeed seem to be in the repo, so it's not an issue of not having synced them.
  
  I have a suspicion that the issue is with this code in `zerver/lib/i18n.py`:
  ```py3
  def get_language_translation_data(language: str) -> Dict[str, str]:
      if language == 'en':
          return {}
      elif language == 'zh-hans':
          language = 'zh_Hans'
      elif language == 'zh-hant':
          language = 'zh_Hant'
      elif language == 'id-id':
          language = 'id_ID'
      path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')
      # …
  ```
  
  That has a handful of special cases to try to translate between two different conventions for locale names. It sure looks like it'd need another one to support `zh_TW` aka `zh-tw`, and that without that this function will fail to do its job on zh_TW.
  
  Better still, of course, would be to make this function stop being made of special cases. Here's a Django utility function that should do the job of all those cases: https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.translation.to_locale
  
  ---
  
  ~~Likely related, but possibly a separate issue: in the webapp language picker itself, the translation shouldn't be called "Chinese (Taiwan)" but rather something like "中文（台湾）" -- its name is written in English, whereas all the other languages have their names written in themselves.~~ (This other symptom is caused at least in part by #14565.)
  
  (Both issues originally reported [in chat](https://chat.zulip.org/#narrow/stream/58-translation/topic/zh-Hant.20.2F.20zh_TW/near/1045033).)
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `zerver/lib/i18n.py`
  Content:
  ```
  1 # See https://zulip.readthedocs.io/en/latest/translating/internationalization.html
  2 
  3 import logging
  4 import operator
  5 import os
  6 from functools import lru_cache
  7 from itertools import zip_longest
  8 from typing import Any, Dict, List, Optional
  9 
  10 import orjson
  11 from django.conf import settings
  12 from django.http import HttpRequest
  13 from django.utils import translation
  14 
  15 
  16 @lru_cache()
  17 def get_language_list() -> List[Dict[str, Any]]:
  18     path = os.path.join(settings.DEPLOY_ROOT, 'locale', 'language_name_map.json')
  19     with open(path, "rb") as reader:
  20         languages = orjson.loads(reader.read())
  21         return languages['name_map']
  22 
  23 def get_language_list_for_templates(default_language: str) -> List[Dict[str, Dict[str, str]]]:
  24     language_list = [lang for lang in get_language_list()
  25                      if 'percent_translated' not in lang or
  26                         lang['percent_translated'] >= 5.]
  27 
  28     formatted_list = []
  29     lang_len = len(language_list)
  30     firsts_end = (lang_len // 2) + operator.mod(lang_len, 2)
  31     firsts = list(range(0, firsts_end))
  32     seconds = list(range(firsts_end, lang_len))
  33     assert len(firsts) + len(seconds) == lang_len
  34     for row in zip_longest(firsts, seconds):
  35         item = {}
  36         for position, ind in zip(['first', 'second'], row):
  37             if ind is None:
  38                 continue
  39 
  40             lang = language_list[ind]
  41             percent = name = lang['name']
  42             if 'percent_translated' in lang:
  43                 percent = "{} ({}%)".format(name, lang['percent_translated'])
  44 
  45             selected = False
  46             if default_language in (lang['code'], lang['locale']):
  47                 selected = True
  48 
  49             item[position] = {
  50                 'name': name,
  51                 'code': lang['code'],
  52                 'percent': percent,
  53                 'selected': selected,
  54             }
  55 
  56         formatted_list.append(item)
  57 
  58     return formatted_list
  59 
  60 def get_language_name(code: str) -> str:
  61     for lang in get_language_list():
  62         if code in (lang['code'], lang['locale']):
  63             return lang['name']
  64     # Log problem, but still return a name
  65     logging.error("Unknown language code '%s'", code)
  66     return "Unknown"
  67 
  68 def get_available_language_codes() -> List[str]:
  69     language_list = get_language_list()
  70     codes = [language['code'] for language in language_list]
  71     return codes
  72 
  73 def get_language_translation_data(language: str) -> Dict[str, str]:
  74     if language == 'en':
  75         return {}
  76     elif language == 'zh-hans':
  77         language = 'zh_Hans'
  78     elif language == 'zh-hant':
  79         language = 'zh_Hant'
  80     elif language == 'id-id':
  81         language = 'id_ID'
  82     path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')
  83     try:
  84         with open(path, "rb") as reader:
  85             return orjson.loads(reader.read())
  86     except FileNotFoundError:
  87         print(f'Translation for {language} not found at {path}')
  88         return {}
  89 
  90 def get_and_set_request_language(
  91     request: HttpRequest,
  92     user_configured_language: str,
  93     testing_url_language: Optional[str]=None
  94 ) -> str:
  95     # We pick a language for the user as follows:
  96     # * First priority is the language in the URL, for debugging.
  97     # * If not in the URL, we use the language from the user's settings.
  98     request_language = testing_url_language
  99     if request_language is None:
  100         request_language = user_configured_language
  101     translation.activate(request_language)
  102 
  103     # We also save the language to the user's session, so that
  104     # something reasonable will happen in logged-in portico pages.
  105     request.session[translation.LANGUAGE_SESSION_KEY] = translation.get_language()
  106 
  107     return request_language
  108
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/zerver/lib/i18n.py b/zerver/lib/i18n.py
  --- a/zerver/lib/i18n.py
  +++ b/zerver/lib/i18n.py
  @@ -73,13 +73,8 @@
   def get_language_translation_data(language: str) -> Dict[str, str]:
       if language == 'en':
           return {}
  -    elif language == 'zh-hans':
  -        language = 'zh_Hans'
  -    elif language == 'zh-hant':
  -        language = 'zh_Hant'
  -    elif language == 'id-id':
  -        language = 'id_ID'
  -    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')
  +    locale = translation.to_locale(language)
  +    path = os.path.join(settings.DEPLOY_ROOT, 'locale', locale, 'translations.json')
       try:
           with open(path, "rb") as reader:
               return orjson.loads(reader.read())
  
verification_info: {"golden_diff": "diff --git a/zerver/lib/i18n.py b/zerver/lib/i18n.py\n--- a/zerver/lib/i18n.py\n+++ b/zerver/lib/i18n.py\n@@ -73,13 +73,8 @@\n def get_language_translation_data(language: str) -> Dict[str, str]:\n     if language == 'en':\n         return {}\n-    elif language == 'zh-hans':\n-        language = 'zh_Hans'\n-    elif language == 'zh-hant':\n-        language = 'zh_Hant'\n-    elif language == 'id-id':\n-        language = 'id_ID'\n-    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')\n+    locale = translation.to_locale(language)\n+    path = os.path.join(settings.DEPLOY_ROOT, 'locale', locale, 'translations.json')\n     try:\n         with open(path, \"rb\") as reader:\n             return orjson.loads(reader.read())\n", "issue": "Translation for zh_TW gets ignored in some places\nIn the webapp, if I try switching to the translation for \"Chinese (Taiwan)\", a lot of the text on the screen is still untranslated:\r\n![image](https://user-images.githubusercontent.com/28173/96927079-63010700-146b-11eb-843e-ebede1e91469.png)\r\nThat's even though many (at least) of those strings [do have translations in Transifex](https://www.transifex.com/zulip/zulip/translate/#zh_TW/$/67194598?q=text%3Asettings). Those translations have been there for months and do indeed seem to be in the repo, so it's not an issue of not having synced them.\r\n\r\nI have a suspicion that the issue is with this code in `zerver/lib/i18n.py`:\r\n```py3\r\ndef get_language_translation_data(language: str) -> Dict[str, str]:\r\n    if language == 'en':\r\n        return {}\r\n    elif language == 'zh-hans':\r\n        language = 'zh_Hans'\r\n    elif language == 'zh-hant':\r\n        language = 'zh_Hant'\r\n    elif language == 'id-id':\r\n        language = 'id_ID'\r\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')\r\n    # \u2026\r\n```\r\n\r\nThat has a handful of special cases to try to translate between two different conventions for locale names. It sure looks like it'd need another one to support `zh_TW` aka `zh-tw`, and that without that this function will fail to do its job on zh_TW.\r\n\r\nBetter still, of course, would be to make this function stop being made of special cases. Here's a Django utility function that should do the job of all those cases: https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.translation.to_locale\r\n\r\n---\r\n\r\n~~Likely related, but possibly a separate issue: in the webapp language picker itself, the translation shouldn't be called \"Chinese (Taiwan)\" but rather something like \"\u4e2d\u6587\uff08\u53f0\u6e7e\uff09\" -- its name is written in English, whereas all the other languages have their names written in themselves.~~ (This other symptom is caused at least in part by #14565.)\r\n\r\n(Both issues originally reported [in chat](https://chat.zulip.org/#narrow/stream/58-translation/topic/zh-Hant.20.2F.20zh_TW/near/1045033).)\r\n\n", "before_files": [{"content": "# See https://zulip.readthedocs.io/en/latest/translating/internationalization.html\n\nimport logging\nimport operator\nimport os\nfrom functools import lru_cache\nfrom itertools import zip_longest\nfrom typing import Any, Dict, List, Optional\n\nimport orjson\nfrom django.conf import settings\nfrom django.http import HttpRequest\nfrom django.utils import translation\n\n\n@lru_cache()\ndef get_language_list() -> List[Dict[str, Any]]:\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', 'language_name_map.json')\n    with open(path, \"rb\") as reader:\n        languages = orjson.loads(reader.read())\n        return languages['name_map']\n\ndef get_language_list_for_templates(default_language: str) -> List[Dict[str, Dict[str, str]]]:\n    language_list = [lang for lang in get_language_list()\n                     if 'percent_translated' not in lang or\n                        lang['percent_translated'] >= 5.]\n\n    formatted_list = []\n    lang_len = len(language_list)\n    firsts_end = (lang_len // 2) + operator.mod(lang_len, 2)\n    firsts = list(range(0, firsts_end))\n    seconds = list(range(firsts_end, lang_len))\n    assert len(firsts) + len(seconds) == lang_len\n    for row in zip_longest(firsts, seconds):\n        item = {}\n        for position, ind in zip(['first', 'second'], row):\n            if ind is None:\n                continue\n\n            lang = language_list[ind]\n            percent = name = lang['name']\n            if 'percent_translated' in lang:\n                percent = \"{} ({}%)\".format(name, lang['percent_translated'])\n\n            selected = False\n            if default_language in (lang['code'], lang['locale']):\n                selected = True\n\n            item[position] = {\n                'name': name,\n                'code': lang['code'],\n                'percent': percent,\n                'selected': selected,\n            }\n\n        formatted_list.append(item)\n\n    return formatted_list\n\ndef get_language_name(code: str) -> str:\n    for lang in get_language_list():\n        if code in (lang['code'], lang['locale']):\n            return lang['name']\n    # Log problem, but still return a name\n    logging.error(\"Unknown language code '%s'\", code)\n    return \"Unknown\"\n\ndef get_available_language_codes() -> List[str]:\n    language_list = get_language_list()\n    codes = [language['code'] for language in language_list]\n    return codes\n\ndef get_language_translation_data(language: str) -> Dict[str, str]:\n    if language == 'en':\n        return {}\n    elif language == 'zh-hans':\n        language = 'zh_Hans'\n    elif language == 'zh-hant':\n        language = 'zh_Hant'\n    elif language == 'id-id':\n        language = 'id_ID'\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')\n    try:\n        with open(path, \"rb\") as reader:\n            return orjson.loads(reader.read())\n    except FileNotFoundError:\n        print(f'Translation for {language} not found at {path}')\n        return {}\n\ndef get_and_set_request_language(\n    request: HttpRequest,\n    user_configured_language: str,\n    testing_url_language: Optional[str]=None\n) -> str:\n    # We pick a language for the user as follows:\n    # * First priority is the language in the URL, for debugging.\n    # * If not in the URL, we use the language from the user's settings.\n    request_language = testing_url_language\n    if request_language is None:\n        request_language = user_configured_language\n    translation.activate(request_language)\n\n    # We also save the language to the user's session, so that\n    # something reasonable will happen in logged-in portico pages.\n    request.session[translation.LANGUAGE_SESSION_KEY] = translation.get_language()\n\n    return request_language\n", "path": "zerver/lib/i18n.py"}], "after_files": [{"content": "# See https://zulip.readthedocs.io/en/latest/translating/internationalization.html\n\nimport logging\nimport operator\nimport os\nfrom functools import lru_cache\nfrom itertools import zip_longest\nfrom typing import Any, Dict, List, Optional\n\nimport orjson\nfrom django.conf import settings\nfrom django.http import HttpRequest\nfrom django.utils import translation\n\n\n@lru_cache()\ndef get_language_list() -> List[Dict[str, Any]]:\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', 'language_name_map.json')\n    with open(path, \"rb\") as reader:\n        languages = orjson.loads(reader.read())\n        return languages['name_map']\n\ndef get_language_list_for_templates(default_language: str) -> List[Dict[str, Dict[str, str]]]:\n    language_list = [lang for lang in get_language_list()\n                     if 'percent_translated' not in lang or\n                        lang['percent_translated'] >= 5.]\n\n    formatted_list = []\n    lang_len = len(language_list)\n    firsts_end = (lang_len // 2) + operator.mod(lang_len, 2)\n    firsts = list(range(0, firsts_end))\n    seconds = list(range(firsts_end, lang_len))\n    assert len(firsts) + len(seconds) == lang_len\n    for row in zip_longest(firsts, seconds):\n        item = {}\n        for position, ind in zip(['first', 'second'], row):\n            if ind is None:\n                continue\n\n            lang = language_list[ind]\n            percent = name = lang['name']\n            if 'percent_translated' in lang:\n                percent = \"{} ({}%)\".format(name, lang['percent_translated'])\n\n            selected = False\n            if default_language in (lang['code'], lang['locale']):\n                selected = True\n\n            item[position] = {\n                'name': name,\n                'code': lang['code'],\n                'percent': percent,\n                'selected': selected,\n            }\n\n        formatted_list.append(item)\n\n    return formatted_list\n\ndef get_language_name(code: str) -> str:\n    for lang in get_language_list():\n        if code in (lang['code'], lang['locale']):\n            return lang['name']\n    # Log problem, but still return a name\n    logging.error(\"Unknown language code '%s'\", code)\n    return \"Unknown\"\n\ndef get_available_language_codes() -> List[str]:\n    language_list = get_language_list()\n    codes = [language['code'] for language in language_list]\n    return codes\n\ndef get_language_translation_data(language: str) -> Dict[str, str]:\n    if language == 'en':\n        return {}\n    locale = translation.to_locale(language)\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', locale, 'translations.json')\n    try:\n        with open(path, \"rb\") as reader:\n            return orjson.loads(reader.read())\n    except FileNotFoundError:\n        print(f'Translation for {language} not found at {path}')\n        return {}\n\ndef get_and_set_request_language(\n    request: HttpRequest,\n    user_configured_language: str,\n    testing_url_language: Optional[str]=None\n) -> str:\n    # We pick a language for the user as follows:\n    # * First priority is the language in the URL, for debugging.\n    # * If not in the URL, we use the language from the user's settings.\n    request_language = testing_url_language\n    if request_language is None:\n        request_language = user_configured_language\n    translation.activate(request_language)\n\n    # We also save the language to the user's session, so that\n    # something reasonable will happen in logged-in portico pages.\n    request.session[translation.LANGUAGE_SESSION_KEY] = translation.get_language()\n\n    return request_language\n", "path": "zerver/lib/i18n.py"}]}