class_docstr: Encoder based on a user-supplied vocabulary (file or list).
  
  This encoder extends over TokenTextEncoder by additionally assigning distinct
  temporary IDs to OOV tokens appearing in the source sequence. This facilitates
  decoding with the pointer-generator mechanism using word-based tokenization.
  
  NOTE: TokenTextEncoderOov does not conform to the TextEncoder API; it changes
  the signature of encode and decode.
class_docstr_tok: ["Encoder","based","on","a","user-supplied","vocabulary","(","file","or","list",")",".","This","encoder","extends","over","TokenTextEncoder","by","additionally","assigning","distinct","temporary","IDs","to","OOV","tokens","appearing","in","the","source","sequence",".","This","facilitates","decoding","with","the","pointer-generator","mechanism","using","word-based","tokenization",".","NOTE",":","TokenTextEncoderOov","does","not","conform","to","the","TextEncoder","API",";","it","changes","the","signature","of","encode","and","decode","."]
docstr: Converts a space-separated string of tokens to lists of ids.
  
  Also store temporary vocabulary IDs for source OOV tokens. OOVs are
  represented by their temporary OOV number. E.g., if the vocabulary size
  is 50k and the source has 3 OOVs, then these temporary OOV numbers will
  be 50000, 50001, 50002.
  
  Args:
    s: human-readable string to be converted.
  
  Returns:
    ids: list of integers
    ids_extend: list of integers including extended temporary vocab IDs for
    source OOVs.
    oovs: A dict storing source OOV words, used for the decoder to copy. The
    key is OOV word, and the value is the order they appear in the source,
    starting from 0.
    source_oov_id_to_token: a list of source OOV tokens, in the same order as
    they appear in the source.
docstr_tok: ["Converts","a","space-separated","string","of","tokens","to","lists","of","ids",".","Also","store","temporary","vocabulary","IDs","for","source","OOV","tokens",".","OOVs","are","represented","by","their","temporary","OOV","number",".","E.g.",",","if","the","vocabulary","size","is","50k","and","the","source","has","3","OOVs",",","then","these","temporary","OOV","numbers","will","be","50000",",","50001",",","50002",".","Args",":","s",":","human-readable","string","to","be","converted",".","Returns",":","ids",":","list","of","integers","ids_extend",":","list","of","integers","including","extended","temporary","vocab","IDs","for","source","OOVs",".","oovs",":","A","dict","storing","source","OOV","words",",","used","for","the","decoder","to","copy",".","The","key","is","OOV","word",",","and","the","value","is","the","order","they","appear","in","the","source",",","starting","from","0.","source_oov_id_to_token",":","a","list","of","source","OOV","tokens",",","in","the","same","order","as","they","appear","in","the","source","."]
code: def encode(self, s):
      """Converts a space-separated string of tokens to lists of ids.
  
      Also store temporary vocabulary IDs for source OOV tokens. OOVs are
      represented by their temporary OOV number. E.g., if the vocabulary size
      is 50k and the source has 3 OOVs, then these temporary OOV numbers will
      be 50000, 50001, 50002.
  
      Args:
        s: human-readable string to be converted.
  
      Returns:
        ids: list of integers
        ids_extend: list of integers including extended temporary vocab IDs for
        source OOVs.
        oovs: A dict storing source OOV words, used for the decoder to copy. The
        key is OOV word, and the value is the order they appear in the source,
        starting from 0.
        source_oov_id_to_token: a list of source OOV tokens, in the same order as
        they appear in the source.
      """
      sentence = s
      tokens = sentence.strip().split()
      ids = []
      ids_extend = []
      oovs = {}
      for t in tokens:
          if t in self._token_to_id:
              ids.append(self._token_to_id[t])
              ids_extend.append(self._token_to_id[t])
          else:
              next_oov_id = len(oovs)
              oov_num = oovs.get(t, next_oov_id)
              if oov_num == next_oov_id:
                  oovs[t] = oov_num
              ids_extend.append(self.vocab_size + oov_num)
              ids.append(self._token_to_id[self._replace_oov])
      source_oov_id_to_token = [""] * len(oovs)
      for oov in oovs:
          source_oov_id_to_token[oovs[oov]] = oov
      if self._reverse:
          return (
              ids[::-1],
              ids_extend[::-1],
              oovs,
              source_oov_id_to_token,
          )
      else:
          return ids, ids_extend, oovs, source_oov_id_to_token
  
file_path: tensor2tensor/tensor2tensor/data_generators/pointer_generator_word.py
imports: from __future__ import absolute_import
  from __future__ import division
  from __future__ import print_function
  import os
  from tensor2tensor.data_generators import text_encoder
  from tensor2tensor.data_generators import text_problems
  from tensor2tensor.utils import registry
  import tensorflow.compat.v1
label_desc: Use image node_id 1 for calling the TokenTextEncoderOov obj's underlying member method code with example usage: obj.encode(s) and returns: ids,  ids_extend,  oovs,  source_oov_id_to_token, ids,  ids_extend,  oovs,  source_oov_id_to_token