prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Subword tokenisation spacer can mark the beginning of word
  Certain sequence noising operations need to retrieve a list of words from the raw list of subword tokens. For example:
  
  * Decoding with word removal/reordering to produce noisy back-translations as in [Scaling BT paper](https://arxiv.org/abs/1808.09381)
  
  * Word omission to support the new contrastive learning feature as in the [contrastive learning paper](https://www.aclweb.org/anthology/P19-1623.pdf)
  
  * Presumably more features relying on word level noise might come up in the future
  
  In these cases the user should specify some details for the sub-tokenisation process: 
  1. What subword tokens was used? (`decoding_subword_token`)
  2. Was that token a joiner or a spacer? (`decoding_subword_token_is_spacer`)
  
  When the user specifies (explicitly or implicitly) a spacer, the framework assumes that the spacer symbol appears at the beginning of each word, similar to what SentencePiece does. However this does not have to be the case, the spacer could also appear at the end of each word - for example [this one does](https://github.com/kovalevfm/SubTokenizer). If that extra sub-tokenisation flexibility is desired, we can add this configuration parameter. A sample implementation could look like [this](https://github.com/steremma/OpenNMT-tf/commit/d109af49911431e424b28def575fb94f07bfec47).
  
  I realise that most user's rely on standard tools that are covered by the current implementation. If there is a user base for which the extra flexibility is desired, I can submit a PR that reads this option from the YAML.
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `opennmt/data/text.py`
  Content:
  ```
  1 # -*- coding: utf-8 -*-
  2 
  3 """Text manipulation."""
  4 
  5 import tensorflow as tf
  6 
  7 
  8 def tokens_to_chars(tokens):
  9   """Splits tokens into unicode characters.
  10 
  11   Args:
  12     tokens: A string ``tf.Tensor`` of shape :math:`[T]`.
  13 
  14   Returns:
  15     The characters as a 2D string ``tf.RaggedTensor``.
  16   """
  17   return tf.strings.unicode_split(tokens, "UTF-8")
  18 
  19 def tokens_to_words(tokens, subword_token="￭", is_spacer=None):
  20   """Converts a sequence of tokens to a sequence of words.
  21 
  22   For example, if a BPE tokenization produces this sequence:
  23 
  24       ["He@@", "llo", "W@@", "orld", "@@!"]
  25 
  26   this function will return the tensor:
  27 
  28       [["He@@", "llo", ""], ["W@@", "orld", "@@!"]]
  29 
  30   Args:
  31     tokens: A 1D string ``tf.Tensor``.
  32     subword_token: The special token used by the subword tokenizer.
  33     is_spacer: Whether :obj:`subword_token` is used as a spacer (as in
  34       SentencePiece) or a joiner (as in BPE). If ``None``, will infer
  35       directly from :obj:`subword_token`.
  36 
  37   Returns:
  38     The words as a 2D string ``tf.RaggedTensor``.
  39   """
  40   if is_spacer is None:
  41     is_spacer = subword_token == "▁"
  42   if is_spacer:
  43     subword = tf.strings.regex_full_match(tokens, "[^%s].*" % subword_token)
  44   else:
  45     right = tf.strings.regex_full_match(tokens, ".*%s" % subword_token)
  46     left = tf.strings.regex_full_match(tokens, "%s.*" % subword_token)
  47     subword = tf.logical_or(tf.roll(right, shift=1, axis=0), left)
  48   start = tf.logical_not(subword)
  49   start_indices = tf.squeeze(tf.where(start), -1)
  50   return tf.RaggedTensor.from_row_starts(tokens, start_indices)
  51 
  52 def alignment_matrix_from_pharaoh(alignment_line,
  53                                   source_length,
  54                                   target_length,
  55                                   dtype=tf.float32):
  56   """Parse Pharaoh alignments into an alignment matrix.
  57 
  58   Args:
  59     alignment_line: A string ``tf.Tensor`` in the Pharaoh format.
  60     source_length: The length of the source sentence, without special symbols.
  61     target_length: The length of the target sentence, without special symbols.
  62     dtype: The output matrix dtype. Defaults to ``tf.float32`` for convenience
  63       when computing the guided alignment loss.
  64 
  65   Returns:
  66     The alignment matrix as a 2-D ``tf.Tensor`` of type :obj:`dtype` and shape
  67     ``[target_length, source_length]``, where ``[i, j] = 1`` if the ``i`` th
  68     target word is aligned with the ``j`` th source word.
  69   """
  70   align_pairs_str = tf.strings.split([alignment_line]).values
  71   align_pairs_flat_str = tf.strings.split(align_pairs_str, sep="-").values
  72   align_pairs_flat = tf.strings.to_number(align_pairs_flat_str, out_type=tf.int64)
  73   sparse_indices = tf.reshape(align_pairs_flat, [-1, 2])
  74   sparse_values = tf.ones([tf.shape(sparse_indices)[0]], dtype=dtype)
  75   source_length = tf.cast(source_length, tf.int64)
  76   target_length = tf.cast(target_length, tf.int64)
  77   alignment_matrix_sparse = tf.sparse.SparseTensor(
  78       sparse_indices, sparse_values, [source_length, target_length])
  79   alignment_matrix = tf.sparse.to_dense(alignment_matrix_sparse, validate_indices=False)
  80   return tf.transpose(alignment_matrix)
  81
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/opennmt/data/text.py b/opennmt/data/text.py
  --- a/opennmt/data/text.py
  +++ b/opennmt/data/text.py
  @@ -40,13 +40,18 @@
     if is_spacer is None:
       is_spacer = subword_token == "▁"
     if is_spacer:
  -    subword = tf.strings.regex_full_match(tokens, "[^%s].*" % subword_token)
  +    # First token implicitly starts with a spacer.
  +    left_and_single = tf.logical_or(
  +        tf.strings.regex_full_match(tokens, "%s.*" % subword_token),
  +        tf.one_hot(0, tf.shape(tokens)[0], on_value=True, off_value=False))
  +    right = tf.strings.regex_full_match(tokens, ".+%s" % subword_token)
  +    word_start = tf.logical_or(tf.roll(right, shift=1, axis=0), left_and_single)
     else:
       right = tf.strings.regex_full_match(tokens, ".*%s" % subword_token)
       left = tf.strings.regex_full_match(tokens, "%s.*" % subword_token)
       subword = tf.logical_or(tf.roll(right, shift=1, axis=0), left)
  -  start = tf.logical_not(subword)
  -  start_indices = tf.squeeze(tf.where(start), -1)
  +    word_start = tf.logical_not(subword)
  +  start_indices = tf.squeeze(tf.where(word_start), -1)
     return tf.RaggedTensor.from_row_starts(tokens, start_indices)
   
   def alignment_matrix_from_pharaoh(alignment_line,
  
verification_info: {"golden_diff": "diff --git a/opennmt/data/text.py b/opennmt/data/text.py\n--- a/opennmt/data/text.py\n+++ b/opennmt/data/text.py\n@@ -40,13 +40,18 @@\n   if is_spacer is None:\n     is_spacer = subword_token == \"\u2581\"\n   if is_spacer:\n-    subword = tf.strings.regex_full_match(tokens, \"[^%s].*\" % subword_token)\n+    # First token implicitly starts with a spacer.\n+    left_and_single = tf.logical_or(\n+        tf.strings.regex_full_match(tokens, \"%s.*\" % subword_token),\n+        tf.one_hot(0, tf.shape(tokens)[0], on_value=True, off_value=False))\n+    right = tf.strings.regex_full_match(tokens, \".+%s\" % subword_token)\n+    word_start = tf.logical_or(tf.roll(right, shift=1, axis=0), left_and_single)\n   else:\n     right = tf.strings.regex_full_match(tokens, \".*%s\" % subword_token)\n     left = tf.strings.regex_full_match(tokens, \"%s.*\" % subword_token)\n     subword = tf.logical_or(tf.roll(right, shift=1, axis=0), left)\n-  start = tf.logical_not(subword)\n-  start_indices = tf.squeeze(tf.where(start), -1)\n+    word_start = tf.logical_not(subword)\n+  start_indices = tf.squeeze(tf.where(word_start), -1)\n   return tf.RaggedTensor.from_row_starts(tokens, start_indices)\n \n def alignment_matrix_from_pharaoh(alignment_line,\n", "issue": "Subword tokenisation spacer can mark the beginning of word\nCertain sequence noising operations need to retrieve a list of words from the raw list of subword tokens. For example:\r\n\r\n* Decoding with word removal/reordering to produce noisy back-translations as in [Scaling BT paper](https://arxiv.org/abs/1808.09381)\r\n\r\n* Word omission to support the new contrastive learning feature as in the [contrastive learning paper](https://www.aclweb.org/anthology/P19-1623.pdf)\r\n\r\n* Presumably more features relying on word level noise might come up in the future\r\n\r\nIn these cases the user should specify some details for the sub-tokenisation process: \r\n1. What subword tokens was used? (`decoding_subword_token`)\r\n2. Was that token a joiner or a spacer? (`decoding_subword_token_is_spacer`)\r\n\r\nWhen the user specifies (explicitly or implicitly) a spacer, the framework assumes that the spacer symbol appears at the beginning of each word, similar to what SentencePiece does. However this does not have to be the case, the spacer could also appear at the end of each word - for example [this one does](https://github.com/kovalevfm/SubTokenizer). If that extra sub-tokenisation flexibility is desired, we can add this configuration parameter. A sample implementation could look like [this](https://github.com/steremma/OpenNMT-tf/commit/d109af49911431e424b28def575fb94f07bfec47).\r\n\r\nI realise that most user's rely on standard tools that are covered by the current implementation. If there is a user base for which the extra flexibility is desired, I can submit a PR that reads this option from the YAML.\n", "before_files": [{"content": "# -*- coding: utf-8 -*-\n\n\"\"\"Text manipulation.\"\"\"\n\nimport tensorflow as tf\n\n\ndef tokens_to_chars(tokens):\n  \"\"\"Splits tokens into unicode characters.\n\n  Args:\n    tokens: A string ``tf.Tensor`` of shape :math:`[T]`.\n\n  Returns:\n    The characters as a 2D string ``tf.RaggedTensor``.\n  \"\"\"\n  return tf.strings.unicode_split(tokens, \"UTF-8\")\n\ndef tokens_to_words(tokens, subword_token=\"\uffed\", is_spacer=None):\n  \"\"\"Converts a sequence of tokens to a sequence of words.\n\n  For example, if a BPE tokenization produces this sequence:\n\n      [\"He@@\", \"llo\", \"W@@\", \"orld\", \"@@!\"]\n\n  this function will return the tensor:\n\n      [[\"He@@\", \"llo\", \"\"], [\"W@@\", \"orld\", \"@@!\"]]\n\n  Args:\n    tokens: A 1D string ``tf.Tensor``.\n    subword_token: The special token used by the subword tokenizer.\n    is_spacer: Whether :obj:`subword_token` is used as a spacer (as in\n      SentencePiece) or a joiner (as in BPE). If ``None``, will infer\n      directly from :obj:`subword_token`.\n\n  Returns:\n    The words as a 2D string ``tf.RaggedTensor``.\n  \"\"\"\n  if is_spacer is None:\n    is_spacer = subword_token == \"\u2581\"\n  if is_spacer:\n    subword = tf.strings.regex_full_match(tokens, \"[^%s].*\" % subword_token)\n  else:\n    right = tf.strings.regex_full_match(tokens, \".*%s\" % subword_token)\n    left = tf.strings.regex_full_match(tokens, \"%s.*\" % subword_token)\n    subword = tf.logical_or(tf.roll(right, shift=1, axis=0), left)\n  start = tf.logical_not(subword)\n  start_indices = tf.squeeze(tf.where(start), -1)\n  return tf.RaggedTensor.from_row_starts(tokens, start_indices)\n\ndef alignment_matrix_from_pharaoh(alignment_line,\n                                  source_length,\n                                  target_length,\n                                  dtype=tf.float32):\n  \"\"\"Parse Pharaoh alignments into an alignment matrix.\n\n  Args:\n    alignment_line: A string ``tf.Tensor`` in the Pharaoh format.\n    source_length: The length of the source sentence, without special symbols.\n    target_length: The length of the target sentence, without special symbols.\n    dtype: The output matrix dtype. Defaults to ``tf.float32`` for convenience\n      when computing the guided alignment loss.\n\n  Returns:\n    The alignment matrix as a 2-D ``tf.Tensor`` of type :obj:`dtype` and shape\n    ``[target_length, source_length]``, where ``[i, j] = 1`` if the ``i`` th\n    target word is aligned with the ``j`` th source word.\n  \"\"\"\n  align_pairs_str = tf.strings.split([alignment_line]).values\n  align_pairs_flat_str = tf.strings.split(align_pairs_str, sep=\"-\").values\n  align_pairs_flat = tf.strings.to_number(align_pairs_flat_str, out_type=tf.int64)\n  sparse_indices = tf.reshape(align_pairs_flat, [-1, 2])\n  sparse_values = tf.ones([tf.shape(sparse_indices)[0]], dtype=dtype)\n  source_length = tf.cast(source_length, tf.int64)\n  target_length = tf.cast(target_length, tf.int64)\n  alignment_matrix_sparse = tf.sparse.SparseTensor(\n      sparse_indices, sparse_values, [source_length, target_length])\n  alignment_matrix = tf.sparse.to_dense(alignment_matrix_sparse, validate_indices=False)\n  return tf.transpose(alignment_matrix)\n", "path": "opennmt/data/text.py"}], "after_files": [{"content": "# -*- coding: utf-8 -*-\n\n\"\"\"Text manipulation.\"\"\"\n\nimport tensorflow as tf\n\n\ndef tokens_to_chars(tokens):\n  \"\"\"Splits tokens into unicode characters.\n\n  Args:\n    tokens: A string ``tf.Tensor`` of shape :math:`[T]`.\n\n  Returns:\n    The characters as a 2D string ``tf.RaggedTensor``.\n  \"\"\"\n  return tf.strings.unicode_split(tokens, \"UTF-8\")\n\ndef tokens_to_words(tokens, subword_token=\"\uffed\", is_spacer=None):\n  \"\"\"Converts a sequence of tokens to a sequence of words.\n\n  For example, if a BPE tokenization produces this sequence:\n\n      [\"He@@\", \"llo\", \"W@@\", \"orld\", \"@@!\"]\n\n  this function will return the tensor:\n\n      [[\"He@@\", \"llo\", \"\"], [\"W@@\", \"orld\", \"@@!\"]]\n\n  Args:\n    tokens: A 1D string ``tf.Tensor``.\n    subword_token: The special token used by the subword tokenizer.\n    is_spacer: Whether :obj:`subword_token` is used as a spacer (as in\n      SentencePiece) or a joiner (as in BPE). If ``None``, will infer\n      directly from :obj:`subword_token`.\n\n  Returns:\n    The words as a 2D string ``tf.RaggedTensor``.\n  \"\"\"\n  if is_spacer is None:\n    is_spacer = subword_token == \"\u2581\"\n  if is_spacer:\n    # First token implicitly starts with a spacer.\n    left_and_single = tf.logical_or(\n        tf.strings.regex_full_match(tokens, \"%s.*\" % subword_token),\n        tf.one_hot(0, tf.shape(tokens)[0], on_value=True, off_value=False))\n    right = tf.strings.regex_full_match(tokens, \".+%s\" % subword_token)\n    word_start = tf.logical_or(tf.roll(right, shift=1, axis=0), left_and_single)\n  else:\n    right = tf.strings.regex_full_match(tokens, \".*%s\" % subword_token)\n    left = tf.strings.regex_full_match(tokens, \"%s.*\" % subword_token)\n    subword = tf.logical_or(tf.roll(right, shift=1, axis=0), left)\n    word_start = tf.logical_not(subword)\n  start_indices = tf.squeeze(tf.where(word_start), -1)\n  return tf.RaggedTensor.from_row_starts(tokens, start_indices)\n\ndef alignment_matrix_from_pharaoh(alignment_line,\n                                  source_length,\n                                  target_length,\n                                  dtype=tf.float32):\n  \"\"\"Parse Pharaoh alignments into an alignment matrix.\n\n  Args:\n    alignment_line: A string ``tf.Tensor`` in the Pharaoh format.\n    source_length: The length of the source sentence, without special symbols.\n    target_length: The length of the target sentence, without special symbols.\n    dtype: The output matrix dtype. Defaults to ``tf.float32`` for convenience\n      when computing the guided alignment loss.\n\n  Returns:\n    The alignment matrix as a 2-D ``tf.Tensor`` of type :obj:`dtype` and shape\n    ``[target_length, source_length]``, where ``[i, j] = 1`` if the ``i`` th\n    target word is aligned with the ``j`` th source word.\n  \"\"\"\n  align_pairs_str = tf.strings.split([alignment_line]).values\n  align_pairs_flat_str = tf.strings.split(align_pairs_str, sep=\"-\").values\n  align_pairs_flat = tf.strings.to_number(align_pairs_flat_str, out_type=tf.int64)\n  sparse_indices = tf.reshape(align_pairs_flat, [-1, 2])\n  sparse_values = tf.ones([tf.shape(sparse_indices)[0]], dtype=dtype)\n  source_length = tf.cast(source_length, tf.int64)\n  target_length = tf.cast(target_length, tf.int64)\n  alignment_matrix_sparse = tf.sparse.SparseTensor(\n      sparse_indices, sparse_values, [source_length, target_length])\n  alignment_matrix = tf.sparse.to_dense(alignment_matrix_sparse, validate_indices=False)\n  return tf.transpose(alignment_matrix)\n", "path": "opennmt/data/text.py"}]}