title: How does the `tfds.features.text.SubwordTextEncoder` create word encoding?
body: <p>I'm currently doing a tensorflow transformer <a href="https://www.tensorflow.org/tutorials/text/transformer" rel="nofollow noreferrer">tutorial</a> for sequence to sequence translation. At the beginning of the tutorial the class <a href="https://www.tensorflow.org/datasets/api_docs/python/tfds/features/text/SubwordTextEncoder" rel="nofollow noreferrer">tfds.features.text.SubwordTextEncoder</a> is called. This class can be used to convert a string to a list with integers, each representing a word.</p>
  <p>After using the class <code>SubwordTextEncoder</code> to train an english tokenizer as follows:</p>
  <pre><code>tokenizer_en = tfds.features.text.SubwordTextEncoder.build_from_corpus(
  (en.numpy() for pt, en in train_examples), target_vocab_size=2**13)
  </code></pre>
  <p>the tutorial shows how this tokenizer can now be used to convert strings to lists with integers. This code snippet</p>
  <pre><code>sample_string = 'Transformer is awesome.'
  
  tokenized_string = tokenizer_en.encode(sample_string)
  print ('Tokenized string is {}'.format(tokenized_string))
  </code></pre>
  <p>gives the following result:</p>
  <pre><code>[7915, 1248, 7946, 7194, 13, 2799]
  </code></pre>
  <p>where the integer to word mapping can be shown as follows:</p>
  <pre><code>for ts in tokenized_string:
    print ('{} ----&gt; {}'.format(ts, tokenizer_en.decode([ts])))
  </code></pre>
  <p>returns</p>
  <pre><code>7915 ----&gt; T
  1248 ----&gt; ran
  7946 ----&gt; s
  7194 ----&gt; former 
  13 ----&gt; is 
  2799 ----&gt; awesome
  </code></pre>
  <p>This all makes sense to me. The tokenizer recognises the words 'is' and 'awesome' from its training set and assigns the corresponding integers. The word 'Transformer' which was not in its training set is being split up into parts as is mentioned in the documentation.</p>
  <p>After some experimenting with the tokenizer however, I got confused. Please consider the following code snippets</p>
  <pre><code>sample_string2 = 'the best there is'
  tokenized_string2 = tokenizer_en.encode(sample_string2)
  print(tokenized_string2)
  </code></pre>
  <p>which returns</p>
  <pre><code>[3, 332, 64, 156]
  </code></pre>
  <p>and</p>
  <pre><code>for ts in tokenized_string2:
    print ('{} ----&gt; {}'.format(ts, tokenizer_en.decode([ts])))
  </code></pre>
  <p>which returns</p>
  <pre><code>3 ----&gt; the 
  332 ----&gt; best 
  64 ----&gt; there 
  156 ----&gt; is
  </code></pre>
  <p><strong>Question:</strong> Why does the tokenizer return different integers for the same word if they are in a different part of the sentence? The word 'is' maps to 156 in the second example, where in the first example it is mapped to the integer 13, using the same tokenizer.</p>