We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Translation for zh_TW gets ignored in some places
In the webapp, if I try switching to the translation for "Chinese (Taiwan)", a lot of the text on the screen is still untranslated:
![image](https://user-images.githubusercontent.com/28173/96927079-63010700-146b-11eb-843e-ebede1e91469.png)
That's even though many (at least) of those strings [do have translations in Transifex](https://www.transifex.com/zulip/zulip/translate/#zh_TW/$/67194598?q=text%3Asettings). Those translations have been there for months and do indeed seem to be in the repo, so it's not an issue of not having synced them.

I have a suspicion that the issue is with this code in `zerver/lib/i18n.py`:
```py3
def get_language_translation_data(language: str) -> Dict[str, str]:
    if language == 'en':
        return {}
    elif language == 'zh-hans':
        language = 'zh_Hans'
    elif language == 'zh-hant':
        language = 'zh_Hant'
    elif language == 'id-id':
        language = 'id_ID'
    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')
    # …
```

That has a handful of special cases to try to translate between two different conventions for locale names. It sure looks like it'd need another one to support `zh_TW` aka `zh-tw`, and that without that this function will fail to do its job on zh_TW.

Better still, of course, would be to make this function stop being made of special cases. Here's a Django utility function that should do the job of all those cases: https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.translation.to_locale

---

~~Likely related, but possibly a separate issue: in the webapp language picker itself, the translation shouldn't be called "Chinese (Taiwan)" but rather something like "中文（台湾）" -- its name is written in English, whereas all the other languages have their names written in themselves.~~ (This other symptom is caused at least in part by #14565.)

(Both issues originally reported [in chat](https://chat.zulip.org/#narrow/stream/58-translation/topic/zh-Hant.20.2F.20zh_TW/near/1045033).)


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `zerver/lib/i18n.py`
Content:
```
1 # See https://zulip.readthedocs.io/en/latest/translating/internationalization.html
2 
3 import logging
4 import operator
5 import os
6 from functools import lru_cache
7 from itertools import zip_longest
8 from typing import Any, Dict, List, Optional
9 
10 import orjson
11 from django.conf import settings
12 from django.http import HttpRequest
13 from django.utils import translation
14 
15 
16 @lru_cache()
17 def get_language_list() -> List[Dict[str, Any]]:
18     path = os.path.join(settings.DEPLOY_ROOT, 'locale', 'language_name_map.json')
19     with open(path, "rb") as reader:
20         languages = orjson.loads(reader.read())
21         return languages['name_map']
22 
23 def get_language_list_for_templates(default_language: str) -> List[Dict[str, Dict[str, str]]]:
24     language_list = [lang for lang in get_language_list()
25                      if 'percent_translated' not in lang or
26                         lang['percent_translated'] >= 5.]
27 
28     formatted_list = []
29     lang_len = len(language_list)
30     firsts_end = (lang_len // 2) + operator.mod(lang_len, 2)
31     firsts = list(range(0, firsts_end))
32     seconds = list(range(firsts_end, lang_len))
33     assert len(firsts) + len(seconds) == lang_len
34     for row in zip_longest(firsts, seconds):
35         item = {}
36         for position, ind in zip(['first', 'second'], row):
37             if ind is None:
38                 continue
39 
40             lang = language_list[ind]
41             percent = name = lang['name']
42             if 'percent_translated' in lang:
43                 percent = "{} ({}%)".format(name, lang['percent_translated'])
44 
45             selected = False
46             if default_language in (lang['code'], lang['locale']):
47                 selected = True
48 
49             item[position] = {
50                 'name': name,
51                 'code': lang['code'],
52                 'percent': percent,
53                 'selected': selected,
54             }
55 
56         formatted_list.append(item)
57 
58     return formatted_list
59 
60 def get_language_name(code: str) -> str:
61     for lang in get_language_list():
62         if code in (lang['code'], lang['locale']):
63             return lang['name']
64     # Log problem, but still return a name
65     logging.error("Unknown language code '%s'", code)
66     return "Unknown"
67 
68 def get_available_language_codes() -> List[str]:
69     language_list = get_language_list()
70     codes = [language['code'] for language in language_list]
71     return codes
72 
73 def get_language_translation_data(language: str) -> Dict[str, str]:
74     if language == 'en':
75         return {}
76     elif language == 'zh-hans':
77         language = 'zh_Hans'
78     elif language == 'zh-hant':
79         language = 'zh_Hant'
80     elif language == 'id-id':
81         language = 'id_ID'
82     path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')
83     try:
84         with open(path, "rb") as reader:
85             return orjson.loads(reader.read())
86     except FileNotFoundError:
87         print(f'Translation for {language} not found at {path}')
88         return {}
89 
90 def get_and_set_request_language(
91     request: HttpRequest,
92     user_configured_language: str,
93     testing_url_language: Optional[str]=None
94 ) -> str:
95     # We pick a language for the user as follows:
96     # * First priority is the language in the URL, for debugging.
97     # * If not in the URL, we use the language from the user's settings.
98     request_language = testing_url_language
99     if request_language is None:
100         request_language = user_configured_language
101     translation.activate(request_language)
102 
103     # We also save the language to the user's session, so that
104     # something reasonable will happen in logged-in portico pages.
105     request.session[translation.LANGUAGE_SESSION_KEY] = translation.get_language()
106 
107     return request_language
108
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/zerver/lib/i18n.py b/zerver/lib/i18n.py
--- a/zerver/lib/i18n.py
+++ b/zerver/lib/i18n.py
@@ -73,13 +73,8 @@
 def get_language_translation_data(language: str) -> Dict[str, str]:
     if language == 'en':
         return {}
-    elif language == 'zh-hans':
-        language = 'zh_Hans'
-    elif language == 'zh-hant':
-        language = 'zh_Hant'
-    elif language == 'id-id':
-        language = 'id_ID'
-    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')
+    locale = translation.to_locale(language)
+    path = os.path.join(settings.DEPLOY_ROOT, 'locale', locale, 'translations.json')
     try:
         with open(path, "rb") as reader:
             return orjson.loads(reader.read())

{"golden_diff": "diff --git a/zerver/lib/i18n.py b/zerver/lib/i18n.py\n--- a/zerver/lib/i18n.py\n+++ b/zerver/lib/i18n.py\n@@ -73,13 +73,8 @@\n def get_language_translation_data(language: str) -> Dict[str, str]:\n     if language == 'en':\n         return {}\n-    elif language == 'zh-hans':\n-        language = 'zh_Hans'\n-    elif language == 'zh-hant':\n-        language = 'zh_Hant'\n-    elif language == 'id-id':\n-        language = 'id_ID'\n-    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')\n+    locale = translation.to_locale(language)\n+    path = os.path.join(settings.DEPLOY_ROOT, 'locale', locale, 'translations.json')\n     try:\n         with open(path, \"rb\") as reader:\n             return orjson.loads(reader.read())\n", "issue": "Translation for zh_TW gets ignored in some places\nIn the webapp, if I try switching to the translation for \"Chinese (Taiwan)\", a lot of the text on the screen is still untranslated:\r\n![image](https://user-images.githubusercontent.com/28173/96927079-63010700-146b-11eb-843e-ebede1e91469.png)\r\nThat's even though many (at least) of those strings [do have translations in Transifex](https://www.transifex.com/zulip/zulip/translate/#zh_TW/$/67194598?q=text%3Asettings). Those translations have been there for months and do indeed seem to be in the repo, so it's not an issue of not having synced them.\r\n\r\nI have a suspicion that the issue is with this code in `zerver/lib/i18n.py`:\r\n```py3\r\ndef get_language_translation_data(language: str) -> Dict[str, str]:\r\n    if language == 'en':\r\n        return {}\r\n    elif language == 'zh-hans':\r\n        language = 'zh_Hans'\r\n    elif language == 'zh-hant':\r\n        language = 'zh_Hant'\r\n    elif language == 'id-id':\r\n        language = 'id_ID'\r\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')\r\n    # \u2026\r\n```\r\n\r\nThat has a handful of special cases to try to translate between two different conventions for locale names. It sure looks like it'd need another one to support `zh_TW` aka `zh-tw`, and that without that this function will fail to do its job on zh_TW.\r\n\r\nBetter still, of course, would be to make this function stop being made of special cases. Here's a Django utility function that should do the job of all those cases: https://docs.djangoproject.com/en/2.2/ref/utils/#django.utils.translation.to_locale\r\n\r\n---\r\n\r\n~~Likely related, but possibly a separate issue: in the webapp language picker itself, the translation shouldn't be called \"Chinese (Taiwan)\" but rather something like \"\u4e2d\u6587\uff08\u53f0\u6e7e\uff09\" -- its name is written in English, whereas all the other languages have their names written in themselves.~~ (This other symptom is caused at least in part by #14565.)\r\n\r\n(Both issues originally reported [in chat](https://chat.zulip.org/#narrow/stream/58-translation/topic/zh-Hant.20.2F.20zh_TW/near/1045033).)\r\n\n", "before_files": [{"content": "# See https://zulip.readthedocs.io/en/latest/translating/internationalization.html\n\nimport logging\nimport operator\nimport os\nfrom functools import lru_cache\nfrom itertools import zip_longest\nfrom typing import Any, Dict, List, Optional\n\nimport orjson\nfrom django.conf import settings\nfrom django.http import HttpRequest\nfrom django.utils import translation\n\n\n@lru_cache()\ndef get_language_list() -> List[Dict[str, Any]]:\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', 'language_name_map.json')\n    with open(path, \"rb\") as reader:\n        languages = orjson.loads(reader.read())\n        return languages['name_map']\n\ndef get_language_list_for_templates(default_language: str) -> List[Dict[str, Dict[str, str]]]:\n    language_list = [lang for lang in get_language_list()\n                     if 'percent_translated' not in lang or\n                        lang['percent_translated'] >= 5.]\n\n    formatted_list = []\n    lang_len = len(language_list)\n    firsts_end = (lang_len // 2) + operator.mod(lang_len, 2)\n    firsts = list(range(0, firsts_end))\n    seconds = list(range(firsts_end, lang_len))\n    assert len(firsts) + len(seconds) == lang_len\n    for row in zip_longest(firsts, seconds):\n        item = {}\n        for position, ind in zip(['first', 'second'], row):\n            if ind is None:\n                continue\n\n            lang = language_list[ind]\n            percent = name = lang['name']\n            if 'percent_translated' in lang:\n                percent = \"{} ({}%)\".format(name, lang['percent_translated'])\n\n            selected = False\n            if default_language in (lang['code'], lang['locale']):\n                selected = True\n\n            item[position] = {\n                'name': name,\n                'code': lang['code'],\n                'percent': percent,\n                'selected': selected,\n            }\n\n        formatted_list.append(item)\n\n    return formatted_list\n\ndef get_language_name(code: str) -> str:\n    for lang in get_language_list():\n        if code in (lang['code'], lang['locale']):\n            return lang['name']\n    # Log problem, but still return a name\n    logging.error(\"Unknown language code '%s'\", code)\n    return \"Unknown\"\n\ndef get_available_language_codes() -> List[str]:\n    language_list = get_language_list()\n    codes = [language['code'] for language in language_list]\n    return codes\n\ndef get_language_translation_data(language: str) -> Dict[str, str]:\n    if language == 'en':\n        return {}\n    elif language == 'zh-hans':\n        language = 'zh_Hans'\n    elif language == 'zh-hant':\n        language = 'zh_Hant'\n    elif language == 'id-id':\n        language = 'id_ID'\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', language, 'translations.json')\n    try:\n        with open(path, \"rb\") as reader:\n            return orjson.loads(reader.read())\n    except FileNotFoundError:\n        print(f'Translation for {language} not found at {path}')\n        return {}\n\ndef get_and_set_request_language(\n    request: HttpRequest,\n    user_configured_language: str,\n    testing_url_language: Optional[str]=None\n) -> str:\n    # We pick a language for the user as follows:\n    # * First priority is the language in the URL, for debugging.\n    # * If not in the URL, we use the language from the user's settings.\n    request_language = testing_url_language\n    if request_language is None:\n        request_language = user_configured_language\n    translation.activate(request_language)\n\n    # We also save the language to the user's session, so that\n    # something reasonable will happen in logged-in portico pages.\n    request.session[translation.LANGUAGE_SESSION_KEY] = translation.get_language()\n\n    return request_language\n", "path": "zerver/lib/i18n.py"}], "after_files": [{"content": "# See https://zulip.readthedocs.io/en/latest/translating/internationalization.html\n\nimport logging\nimport operator\nimport os\nfrom functools import lru_cache\nfrom itertools import zip_longest\nfrom typing import Any, Dict, List, Optional\n\nimport orjson\nfrom django.conf import settings\nfrom django.http import HttpRequest\nfrom django.utils import translation\n\n\n@lru_cache()\ndef get_language_list() -> List[Dict[str, Any]]:\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', 'language_name_map.json')\n    with open(path, \"rb\") as reader:\n        languages = orjson.loads(reader.read())\n        return languages['name_map']\n\ndef get_language_list_for_templates(default_language: str) -> List[Dict[str, Dict[str, str]]]:\n    language_list = [lang for lang in get_language_list()\n                     if 'percent_translated' not in lang or\n                        lang['percent_translated'] >= 5.]\n\n    formatted_list = []\n    lang_len = len(language_list)\n    firsts_end = (lang_len // 2) + operator.mod(lang_len, 2)\n    firsts = list(range(0, firsts_end))\n    seconds = list(range(firsts_end, lang_len))\n    assert len(firsts) + len(seconds) == lang_len\n    for row in zip_longest(firsts, seconds):\n        item = {}\n        for position, ind in zip(['first', 'second'], row):\n            if ind is None:\n                continue\n\n            lang = language_list[ind]\n            percent = name = lang['name']\n            if 'percent_translated' in lang:\n                percent = \"{} ({}%)\".format(name, lang['percent_translated'])\n\n            selected = False\n            if default_language in (lang['code'], lang['locale']):\n                selected = True\n\n            item[position] = {\n                'name': name,\n                'code': lang['code'],\n                'percent': percent,\n                'selected': selected,\n            }\n\n        formatted_list.append(item)\n\n    return formatted_list\n\ndef get_language_name(code: str) -> str:\n    for lang in get_language_list():\n        if code in (lang['code'], lang['locale']):\n            return lang['name']\n    # Log problem, but still return a name\n    logging.error(\"Unknown language code '%s'\", code)\n    return \"Unknown\"\n\ndef get_available_language_codes() -> List[str]:\n    language_list = get_language_list()\n    codes = [language['code'] for language in language_list]\n    return codes\n\ndef get_language_translation_data(language: str) -> Dict[str, str]:\n    if language == 'en':\n        return {}\n    locale = translation.to_locale(language)\n    path = os.path.join(settings.DEPLOY_ROOT, 'locale', locale, 'translations.json')\n    try:\n        with open(path, \"rb\") as reader:\n            return orjson.loads(reader.read())\n    except FileNotFoundError:\n        print(f'Translation for {language} not found at {path}')\n        return {}\n\ndef get_and_set_request_language(\n    request: HttpRequest,\n    user_configured_language: str,\n    testing_url_language: Optional[str]=None\n) -> str:\n    # We pick a language for the user as follows:\n    # * First priority is the language in the URL, for debugging.\n    # * If not in the URL, we use the language from the user's settings.\n    request_language = testing_url_language\n    if request_language is None:\n        request_language = user_configured_language\n    translation.activate(request_language)\n\n    # We also save the language to the user's session, so that\n    # something reasonable will happen in logged-in portico pages.\n    request.session[translation.LANGUAGE_SESSION_KEY] = translation.get_language()\n\n    return request_language\n", "path": "zerver/lib/i18n.py"}]}