We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
color_sentence fails in tutorial notebook
<!-- Briefly summarize the issue. -->

In the notebook, `display_issues` highlights all token issues with a call to `color_sentence`:

https://github.com/cleanlab/cleanlab/blob/1a239922fe195d2a6104d6dc3552d53da16380ce/docs/source/tutorials/token_classification.ipynb?short_path=2ebceca#L369-L379

One of the examples trips everything up with the following error:
```
missing ), unterminated subpattern at position 2
```


# Stack trace

From [failed CI job](https://github.com/cleanlab/cleanlab/actions/runs/2996555945):
<details><summary> Click to toggle stack trace</summary>

```bash
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
Input In [12], in <module>
----> 1 display_issues(issues,given_words,pred_probs=pred_probs,given_labels=labels,
      2 exclude=[(0,1),(1,0)],class_names=merged_entities)

File ~/work/cleanlab/cleanlab/cleanlab/token_classification/summary.py:81, in display_issues(issues, given_words, pred_probs, given_labels, exclude, class_names, top)
     78     given = class_names[given]
     80 shown += 1
---> 81 print("Sentence %d, token %d: \n%s" % (i, j, color_sentence(sentence,word)))
     82 if given_labels and not pred_probs:
     83     print("Given label: %s\n" % str(given))

File ~/work/cleanlab/cleanlab/cleanlab/internal/token_classification_utils.py:175, in color_sentence(sentence, word)
    158 """
    159 Searches for a given token in the sentence and returns the sentence where the given token is colored red
    160 
   (...)
    172 
    173 """
    174 colored_word = colored(word, "red")
--> 175 colored_sentence, number_of_substitions = re.subn(
    176 r"\b{}\b".format(word),colored_word,sentence
    177 )
    178 if number_of_substitions == 0:
    179     # Use basic string manipulation if regex fails
    180     colored_sentence = sentence.replace(word, colored_word)

File /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/re.py:221, in subn(pattern, repl, string, count, flags)
    212 def subn(pattern, repl, string, count=0, flags=0):
    213     """Return a 2-tuple containing (new_string, number).
    214     new_string is the string obtained by replacing the leftmost
    215     non-overlapping occurrences of the pattern in the source
   (...)
    219     If it is a callable, it's passed the Match object and must
    220     return a replacement string to be used."""
--> 221     return _compile(pattern,flags).subn(repl, string, count)

File /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/re.py:304, in _compile(pattern, flags)
    302 if not sre_compile.isstring(pattern):
    303     raise TypeError("first argument must be string or compiled pattern")
--> 304 p = sre_compile.compile(pattern,flags)
    305 if not (flags & DEBUG):
    306     if len(_cache) >= _MAXCACHE:
    307         # Drop the oldest item

File /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_compile.py:764, in compile(p, flags)
    762 if isstring(p):
    763     pattern = p
--> 764     p = sre_parse.parse(p,flags)
    765 else:
    766     pattern = None

File /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_parse.py:948, in parse(str, flags, state)
    945 state.str = str
    947 try:
--> 948     p = _parse_sub(source,state,flags&SRE_FLAG_VERBOSE,0)
    949 except Verbose:
    950     # the VERBOSE flag was switched on inside the pattern.  to be
    951     # on the safe side, we'll parse the whole thing again...
    952     state = State()

File /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_parse.py:443, in _parse_sub(source, state, verbose, nested)
    441 start = source.tell()
    442 while True:
--> 443     itemsappend(_parse(source,state,verbose,nested+1,
    444 notnestedandnotitems))
    445     if not sourcematch("|"):
    446         break

File /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_parse.py:836, in _parse(source, state, verbose, nested, first)
    834 p = _parse_sub(source, state, sub_verbose, nested + 1)
    835 if not source.match(")"):
--> 836     raise source.error("missing ), unterminated subpattern",
    837                        source.tell() - start)
    838 if group is not None:
    839     state.closegroup(group, p)

error: missing ), unterminated subpattern at position 2
```

</details>


--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `cleanlab/internal/token_classification_utils.py`
Content:
```
1 import re
2 import string
3 import numpy as np
4 from termcolor import colored
5 from typing import List, Optional, Callable, Tuple
6 
7 
8 def get_sentence(words: List[str]) -> str:
9     """
10     Get sentence formed by a list of words with minor processing for readability
11 
12     Parameters
13     ----------
14     words: List[str]
15         list of word-level tokens
16 
17     Returns
18     ----------
19     sentence: string
20         sentence formed by list of word-level tokens
21 
22     """
23     sentence = ""
24     for word in words:
25         if word not in string.punctuation or word in ["-", "("]:
26             word = " " + word
27         sentence += word
28     sentence = sentence.replace(" '", "'").replace("( ", "(").strip()
29     return sentence
30 
31 
32 def filter_sentence(
33     sentences: List[str],
34     condition: Optional[Callable[[str], bool]] = None,
35 ) -> Tuple[List[str], List[bool]]:
36     """
37     Filter sentence based on some condition, and returns filter mask
38 
39     Parameters
40     ----------
41         sentences: List[str]
42             list of sentences
43 
44         condition: Optional[Callable[[str], bool]]
45             sentence filtering condition
46 
47     Returns
48     ---------
49         sentences: List[str]
50             list of sentences filtered
51 
52         mask: List[bool]
53             boolean mask such that `mask[i] == True` if the i'th sentence is included in the
54             filtered sentence, otherwise `mask[i] == False`
55 
56     """
57     if not condition:
58         condition = lambda sentence: len(sentence) > 1 and "#" not in sentence
59     mask = list(map(condition, sentences))
60     sentences = [sentence for m, sentence in zip(mask, sentences) if m]
61     return sentences, mask
62 
63 
64 def process_token(token: str, replace: List[Tuple[str, str]] = [("#", "")]) -> str:
65     """
66     Replaces special characters in the tokens
67 
68     Parameters
69     ----------
70         token: str
71             token which potentially contains special characters
72 
73         replace: List[Tuple[str, str]]
74             list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2
75 
76     Returns
77     ---------
78         processed_token: str
79             processed token whose special character has been replaced
80 
81     Note
82     ----
83         Only applies to characters in the original input token.
84     """
85     replace_dict = {re.escape(k): v for (k, v) in replace}
86     pattern = "|".join(replace_dict.keys())
87     compiled_pattern = re.compile(pattern)
88     replacement = lambda match: replace_dict[re.escape(match.group(0))]
89     processed_token = compiled_pattern.sub(replacement, token)
90     return processed_token
91 
92 
93 def mapping(entities: List[int], maps: List[int]) -> List[int]:
94     """
95     Map a list of entities to its corresponding entities
96 
97     Parameters
98     ----------
99         entities: List[int]
100             a list of given entities
101 
102         maps: List[int]
103             a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`
104 
105     Returns
106     ---------
107         mapped_entities: List[int]
108             a list of mapped entities
109 
110     Examples
111     --------
112         >>> unique_identities = [0, 1, 2, 3, 4]  # ["O", "B-PER", "I-PER", "B-LOC", "I-LOC"]
113         >>> maps = [0, 1, 1, 2, 2]  # ["O", "PER", "PER", "LOC", "LOC"]
114         >>> mapping(unique_identities, maps)
115         [0, 1, 1, 2, 2]  # ["O", "PER", "PER", "LOC", "LOC"]
116         >>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)
117         [0, 0, 2, 2, 2, 2, 0, 1]  # ["O", "O", "LOC", "LOC", "LOC", "LOC", "O", "PER"]
118     """
119     f = lambda x: maps[x]
120     return list(map(f, entities))
121 
122 
123 def merge_probs(probs: np.ndarray, maps: List[int]) -> np.ndarray:
124     """
125     Merges model-predictive probabilities with desired mapping
126 
127     Parameters
128     ----------
129         probs:
130             np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model
131 
132         maps: List[int]
133             a list of mapped index, such that the probability of the token being in the i'th class is mapped to the
134             `maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the
135             returned probability is re-normalized.
136 
137     Returns
138     ---------
139         probs_merged:
140             np.array of shape `(N, K')`, where K' is the number of new classes. Probablities are merged and
141             re-normalized if necessary.
142 
143     """
144     old_classes = probs.shape[1]
145     map_size = np.max(maps) + 1
146     probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)
147 
148     for i in range(old_classes):
149         if maps[i] >= 0:
150             probs_merged[:, maps[i]] += probs[:, i]
151     if -1 in maps:
152         row_sums = probs_merged.sum(axis=1)
153         probs_merged /= row_sums[:, np.newaxis]
154     return probs_merged
155 
156 
157 def color_sentence(sentence: str, word: str) -> str:
158     """
159     Searches for a given token in the sentence and returns the sentence where the given token is colored red
160 
161     Parameters
162     ----------
163         sentence:
164             a sentence where the word is searched
165 
166         word:
167             keyword to find in `sentence`. Assumes the word exists in the sentence.
168     Returns
169     ---------
170         colored_sentence:
171             `sentence` where the every occurance of the word is colored red, using `termcolor.colored`
172 
173     """
174     colored_word = colored(word, "red")
175     colored_sentence, number_of_substitions = re.subn(
176         r"\b{}\b".format(word), colored_word, sentence
177     )
178     if number_of_substitions == 0:
179         # Use basic string manipulation if regex fails
180         colored_sentence = sentence.replace(word, colored_word)
181     return colored_sentence
182
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/cleanlab/internal/token_classification_utils.py b/cleanlab/internal/token_classification_utils.py
--- a/cleanlab/internal/token_classification_utils.py
+++ b/cleanlab/internal/token_classification_utils.py
@@ -173,7 +173,7 @@
     """
     colored_word = colored(word, "red")
     colored_sentence, number_of_substitions = re.subn(
-        r"\b{}\b".format(word), colored_word, sentence
+        r"\b{}\b".format(re.escape(word)), colored_word, sentence
     )
     if number_of_substitions == 0:
         # Use basic string manipulation if regex fails

{"golden_diff": "diff --git a/cleanlab/internal/token_classification_utils.py b/cleanlab/internal/token_classification_utils.py\n--- a/cleanlab/internal/token_classification_utils.py\n+++ b/cleanlab/internal/token_classification_utils.py\n@@ -173,7 +173,7 @@\n     \"\"\"\n     colored_word = colored(word, \"red\")\n     colored_sentence, number_of_substitions = re.subn(\n-        r\"\\b{}\\b\".format(word), colored_word, sentence\n+        r\"\\b{}\\b\".format(re.escape(word)), colored_word, sentence\n     )\n     if number_of_substitions == 0:\n         # Use basic string manipulation if regex fails\n", "issue": "color_sentence fails in tutorial notebook\n<!-- Briefly summarize the issue. -->\r\n\r\nIn the notebook, `display_issues` highlights all token issues with a call to `color_sentence`:\r\n\r\nhttps://github.com/cleanlab/cleanlab/blob/1a239922fe195d2a6104d6dc3552d53da16380ce/docs/source/tutorials/token_classification.ipynb?short_path=2ebceca#L369-L379\r\n\r\nOne of the examples trips everything up with the following error:\r\n```\r\nmissing ), unterminated subpattern at position 2\r\n```\r\n\r\n\r\n# Stack trace\r\n\r\nFrom [failed CI job](https://github.com/cleanlab/cleanlab/actions/runs/2996555945):\r\n<details><summary> Click to toggle stack trace</summary>\r\n\r\n```bash\r\n---------------------------------------------------------------------------\r\nerror                                     Traceback (most recent call last)\r\nInput In [12], in <module>\r\n----> 1 display_issues(issues,given_words,pred_probs=pred_probs,given_labels=labels,\r\n      2 exclude=[(0,1),(1,0)],class_names=merged_entities)\r\n\r\nFile ~/work/cleanlab/cleanlab/cleanlab/token_classification/summary.py:81, in display_issues(issues, given_words, pred_probs, given_labels, exclude, class_names, top)\r\n     78     given = class_names[given]\r\n     80 shown += 1\r\n---> 81 print(\"Sentence %d, token %d: \\n%s\" % (i, j, color_sentence(sentence,word)))\r\n     82 if given_labels and not pred_probs:\r\n     83     print(\"Given label: %s\\n\" % str(given))\r\n\r\nFile ~/work/cleanlab/cleanlab/cleanlab/internal/token_classification_utils.py:175, in color_sentence(sentence, word)\r\n    158 \"\"\"\r\n    159 Searches for a given token in the sentence and returns the sentence where the given token is colored red\r\n    160 \r\n   (...)\r\n    172 \r\n    173 \"\"\"\r\n    174 colored_word = colored(word, \"red\")\r\n--> 175 colored_sentence, number_of_substitions = re.subn(\r\n    176 r\"\\b{}\\b\".format(word),colored_word,sentence\r\n    177 )\r\n    178 if number_of_substitions == 0:\r\n    179     # Use basic string manipulation if regex fails\r\n    180     colored_sentence = sentence.replace(word, colored_word)\r\n\r\nFile /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/re.py:221, in subn(pattern, repl, string, count, flags)\r\n    212 def subn(pattern, repl, string, count=0, flags=0):\r\n    213     \"\"\"Return a 2-tuple containing (new_string, number).\r\n    214     new_string is the string obtained by replacing the leftmost\r\n    215     non-overlapping occurrences of the pattern in the source\r\n   (...)\r\n    219     If it is a callable, it's passed the Match object and must\r\n    220     return a replacement string to be used.\"\"\"\r\n--> 221     return _compile(pattern,flags).subn(repl, string, count)\r\n\r\nFile /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/re.py:304, in _compile(pattern, flags)\r\n    302 if not sre_compile.isstring(pattern):\r\n    303     raise TypeError(\"first argument must be string or compiled pattern\")\r\n--> 304 p = sre_compile.compile(pattern,flags)\r\n    305 if not (flags & DEBUG):\r\n    306     if len(_cache) >= _MAXCACHE:\r\n    307         # Drop the oldest item\r\n\r\nFile /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_compile.py:764, in compile(p, flags)\r\n    762 if isstring(p):\r\n    763     pattern = p\r\n--> 764     p = sre_parse.parse(p,flags)\r\n    765 else:\r\n    766     pattern = None\r\n\r\nFile /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_parse.py:948, in parse(str, flags, state)\r\n    945 state.str = str\r\n    947 try:\r\n--> 948     p = _parse_sub(source,state,flags&SRE_FLAG_VERBOSE,0)\r\n    949 except Verbose:\r\n    950     # the VERBOSE flag was switched on inside the pattern.  to be\r\n    951     # on the safe side, we'll parse the whole thing again...\r\n    952     state = State()\r\n\r\nFile /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_parse.py:443, in _parse_sub(source, state, verbose, nested)\r\n    441 start = source.tell()\r\n    442 while True:\r\n--> 443     itemsappend(_parse(source,state,verbose,nested+1,\r\n    444 notnestedandnotitems))\r\n    445     if not sourcematch(\"|\"):\r\n    446         break\r\n\r\nFile /opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/sre_parse.py:836, in _parse(source, state, verbose, nested, first)\r\n    834 p = _parse_sub(source, state, sub_verbose, nested + 1)\r\n    835 if not source.match(\")\"):\r\n--> 836     raise source.error(\"missing ), unterminated subpattern\",\r\n    837                        source.tell() - start)\r\n    838 if group is not None:\r\n    839     state.closegroup(group, p)\r\n\r\nerror: missing ), unterminated subpattern at position 2\r\n```\r\n\r\n</details>\r\n\n", "before_files": [{"content": "import re\nimport string\nimport numpy as np\nfrom termcolor import colored\nfrom typing import List, Optional, Callable, Tuple\n\n\ndef get_sentence(words: List[str]) -> str:\n    \"\"\"\n    Get sentence formed by a list of words with minor processing for readability\n\n    Parameters\n    ----------\n    words: List[str]\n        list of word-level tokens\n\n    Returns\n    ----------\n    sentence: string\n        sentence formed by list of word-level tokens\n\n    \"\"\"\n    sentence = \"\"\n    for word in words:\n        if word not in string.punctuation or word in [\"-\", \"(\"]:\n            word = \" \" + word\n        sentence += word\n    sentence = sentence.replace(\" '\", \"'\").replace(\"( \", \"(\").strip()\n    return sentence\n\n\ndef filter_sentence(\n    sentences: List[str],\n    condition: Optional[Callable[[str], bool]] = None,\n) -> Tuple[List[str], List[bool]]:\n    \"\"\"\n    Filter sentence based on some condition, and returns filter mask\n\n    Parameters\n    ----------\n        sentences: List[str]\n            list of sentences\n\n        condition: Optional[Callable[[str], bool]]\n            sentence filtering condition\n\n    Returns\n    ---------\n        sentences: List[str]\n            list of sentences filtered\n\n        mask: List[bool]\n            boolean mask such that `mask[i] == True` if the i'th sentence is included in the\n            filtered sentence, otherwise `mask[i] == False`\n\n    \"\"\"\n    if not condition:\n        condition = lambda sentence: len(sentence) > 1 and \"#\" not in sentence\n    mask = list(map(condition, sentences))\n    sentences = [sentence for m, sentence in zip(mask, sentences) if m]\n    return sentences, mask\n\n\ndef process_token(token: str, replace: List[Tuple[str, str]] = [(\"#\", \"\")]) -> str:\n    \"\"\"\n    Replaces special characters in the tokens\n\n    Parameters\n    ----------\n        token: str\n            token which potentially contains special characters\n\n        replace: List[Tuple[str, str]]\n            list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2\n\n    Returns\n    ---------\n        processed_token: str\n            processed token whose special character has been replaced\n\n    Note\n    ----\n        Only applies to characters in the original input token.\n    \"\"\"\n    replace_dict = {re.escape(k): v for (k, v) in replace}\n    pattern = \"|\".join(replace_dict.keys())\n    compiled_pattern = re.compile(pattern)\n    replacement = lambda match: replace_dict[re.escape(match.group(0))]\n    processed_token = compiled_pattern.sub(replacement, token)\n    return processed_token\n\n\ndef mapping(entities: List[int], maps: List[int]) -> List[int]:\n    \"\"\"\n    Map a list of entities to its corresponding entities\n\n    Parameters\n    ----------\n        entities: List[int]\n            a list of given entities\n\n        maps: List[int]\n            a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`\n\n    Returns\n    ---------\n        mapped_entities: List[int]\n            a list of mapped entities\n\n    Examples\n    --------\n        >>> unique_identities = [0, 1, 2, 3, 4]  # [\"O\", \"B-PER\", \"I-PER\", \"B-LOC\", \"I-LOC\"]\n        >>> maps = [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping(unique_identities, maps)\n        [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)\n        [0, 0, 2, 2, 2, 2, 0, 1]  # [\"O\", \"O\", \"LOC\", \"LOC\", \"LOC\", \"LOC\", \"O\", \"PER\"]\n    \"\"\"\n    f = lambda x: maps[x]\n    return list(map(f, entities))\n\n\ndef merge_probs(probs: np.ndarray, maps: List[int]) -> np.ndarray:\n    \"\"\"\n    Merges model-predictive probabilities with desired mapping\n\n    Parameters\n    ----------\n        probs:\n            np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model\n\n        maps: List[int]\n            a list of mapped index, such that the probability of the token being in the i'th class is mapped to the\n            `maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the\n            returned probability is re-normalized.\n\n    Returns\n    ---------\n        probs_merged:\n            np.array of shape `(N, K')`, where K' is the number of new classes. Probablities are merged and\n            re-normalized if necessary.\n\n    \"\"\"\n    old_classes = probs.shape[1]\n    map_size = np.max(maps) + 1\n    probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)\n\n    for i in range(old_classes):\n        if maps[i] >= 0:\n            probs_merged[:, maps[i]] += probs[:, i]\n    if -1 in maps:\n        row_sums = probs_merged.sum(axis=1)\n        probs_merged /= row_sums[:, np.newaxis]\n    return probs_merged\n\n\ndef color_sentence(sentence: str, word: str) -> str:\n    \"\"\"\n    Searches for a given token in the sentence and returns the sentence where the given token is colored red\n\n    Parameters\n    ----------\n        sentence:\n            a sentence where the word is searched\n\n        word:\n            keyword to find in `sentence`. Assumes the word exists in the sentence.\n    Returns\n    ---------\n        colored_sentence:\n            `sentence` where the every occurance of the word is colored red, using `termcolor.colored`\n\n    \"\"\"\n    colored_word = colored(word, \"red\")\n    colored_sentence, number_of_substitions = re.subn(\n        r\"\\b{}\\b\".format(word), colored_word, sentence\n    )\n    if number_of_substitions == 0:\n        # Use basic string manipulation if regex fails\n        colored_sentence = sentence.replace(word, colored_word)\n    return colored_sentence\n", "path": "cleanlab/internal/token_classification_utils.py"}], "after_files": [{"content": "import re\nimport string\nimport numpy as np\nfrom termcolor import colored\nfrom typing import List, Optional, Callable, Tuple\n\n\ndef get_sentence(words: List[str]) -> str:\n    \"\"\"\n    Get sentence formed by a list of words with minor processing for readability\n\n    Parameters\n    ----------\n    words: List[str]\n        list of word-level tokens\n\n    Returns\n    ----------\n    sentence: string\n        sentence formed by list of word-level tokens\n\n    \"\"\"\n    sentence = \"\"\n    for word in words:\n        if word not in string.punctuation or word in [\"-\", \"(\"]:\n            word = \" \" + word\n        sentence += word\n    sentence = sentence.replace(\" '\", \"'\").replace(\"( \", \"(\").strip()\n    return sentence\n\n\ndef filter_sentence(\n    sentences: List[str],\n    condition: Optional[Callable[[str], bool]] = None,\n) -> Tuple[List[str], List[bool]]:\n    \"\"\"\n    Filter sentence based on some condition, and returns filter mask\n\n    Parameters\n    ----------\n        sentences: List[str]\n            list of sentences\n\n        condition: Optional[Callable[[str], bool]]\n            sentence filtering condition\n\n    Returns\n    ---------\n        sentences: List[str]\n            list of sentences filtered\n\n        mask: List[bool]\n            boolean mask such that `mask[i] == True` if the i'th sentence is included in the\n            filtered sentence, otherwise `mask[i] == False`\n\n    \"\"\"\n    if not condition:\n        condition = lambda sentence: len(sentence) > 1 and \"#\" not in sentence\n    mask = list(map(condition, sentences))\n    sentences = [sentence for m, sentence in zip(mask, sentences) if m]\n    return sentences, mask\n\n\ndef process_token(token: str, replace: List[Tuple[str, str]] = [(\"#\", \"\")]) -> str:\n    \"\"\"\n    Replaces special characters in the tokens\n\n    Parameters\n    ----------\n        token: str\n            token which potentially contains special characters\n\n        replace: List[Tuple[str, str]]\n            list of tuples `(s1, s2)`, where all occurances of s1 are replaced by s2\n\n    Returns\n    ---------\n        processed_token: str\n            processed token whose special character has been replaced\n\n    Note\n    ----\n        Only applies to characters in the original input token.\n    \"\"\"\n    replace_dict = {re.escape(k): v for (k, v) in replace}\n    pattern = \"|\".join(replace_dict.keys())\n    compiled_pattern = re.compile(pattern)\n    replacement = lambda match: replace_dict[re.escape(match.group(0))]\n    processed_token = compiled_pattern.sub(replacement, token)\n    return processed_token\n\n\ndef mapping(entities: List[int], maps: List[int]) -> List[int]:\n    \"\"\"\n    Map a list of entities to its corresponding entities\n\n    Parameters\n    ----------\n        entities: List[int]\n            a list of given entities\n\n        maps: List[int]\n            a list of mapped entities, such that the i'th indexed token should be mapped to `maps[i]`\n\n    Returns\n    ---------\n        mapped_entities: List[int]\n            a list of mapped entities\n\n    Examples\n    --------\n        >>> unique_identities = [0, 1, 2, 3, 4]  # [\"O\", \"B-PER\", \"I-PER\", \"B-LOC\", \"I-LOC\"]\n        >>> maps = [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping(unique_identities, maps)\n        [0, 1, 1, 2, 2]  # [\"O\", \"PER\", \"PER\", \"LOC\", \"LOC\"]\n        >>> mapping([0, 0, 4, 4, 3, 4, 0, 2], maps)\n        [0, 0, 2, 2, 2, 2, 0, 1]  # [\"O\", \"O\", \"LOC\", \"LOC\", \"LOC\", \"LOC\", \"O\", \"PER\"]\n    \"\"\"\n    f = lambda x: maps[x]\n    return list(map(f, entities))\n\n\ndef merge_probs(probs: np.ndarray, maps: List[int]) -> np.ndarray:\n    \"\"\"\n    Merges model-predictive probabilities with desired mapping\n\n    Parameters\n    ----------\n        probs:\n            np.array of shape `(N, K)`, where N is the number of tokens, and K is the number of classes for the model\n\n        maps: List[int]\n            a list of mapped index, such that the probability of the token being in the i'th class is mapped to the\n            `maps[i]` index. If `maps[i] == -1`, the i'th column of `probs` is ignored. If `np.any(maps == -1)`, the\n            returned probability is re-normalized.\n\n    Returns\n    ---------\n        probs_merged:\n            np.array of shape `(N, K')`, where K' is the number of new classes. Probablities are merged and\n            re-normalized if necessary.\n\n    \"\"\"\n    old_classes = probs.shape[1]\n    map_size = np.max(maps) + 1\n    probs_merged = np.zeros([len(probs), map_size], dtype=probs.dtype.type)\n\n    for i in range(old_classes):\n        if maps[i] >= 0:\n            probs_merged[:, maps[i]] += probs[:, i]\n    if -1 in maps:\n        row_sums = probs_merged.sum(axis=1)\n        probs_merged /= row_sums[:, np.newaxis]\n    return probs_merged\n\n\ndef color_sentence(sentence: str, word: str) -> str:\n    \"\"\"\n    Searches for a given token in the sentence and returns the sentence where the given token is colored red\n\n    Parameters\n    ----------\n        sentence:\n            a sentence where the word is searched\n\n        word:\n            keyword to find in `sentence`. Assumes the word exists in the sentence.\n    Returns\n    ---------\n        colored_sentence:\n            `sentence` where the every occurance of the word is colored red, using `termcolor.colored`\n\n    \"\"\"\n    colored_word = colored(word, \"red\")\n    colored_sentence, number_of_substitions = re.subn(\n        r\"\\b{}\\b\".format(re.escape(word)), colored_word, sentence\n    )\n    if number_of_substitions == 0:\n        # Use basic string manipulation if regex fails\n        colored_sentence = sentence.replace(word, colored_word)\n    return colored_sentence\n", "path": "cleanlab/internal/token_classification_utils.py"}]}