utils.py
def tokens_to_ids(tokens, token_vocab):
  """ Convert tokens to integer IDs.

  `token_vocab` is a list for convenience. This isn't too inefficient because
  we build a dictionary for any specific `token_vocab` and cache it for future
  calls.

  Args:
    tokens: A list of tokens.
    token_vocab: A list of unique tokens serving as the vocabulary.

  Returns:
    A list of integers with the same length as `tokens`. The integers range
    from `0` to `len(token_vocab) - 1`.
  """
  token_vocab = tuple(token_vocab)
  if not hasattr(tokens_to_ids, 'token_to_id_dicts'):
    tokens_to_ids.token_to_id_dicts = {}
  if token_vocab not in tokens_to_ids.token_to_id_dicts:
    tokens_to_ids.token_to_id_dicts[token_vocab] = {token: i for (i, token) in enumerate(token_vocab)}
  token_to_id_dict = tokens_to_ids.token_to_id_dicts[token_vocab]
  ids = [token_to_id_dict[token] for token in tokens]
  return ids
 Convert tokens to integer IDs.

  `token_vocab` is a list for convenience. This isn't too inefficient because
  we build a dictionary for any specific `token_vocab` and cache it for future
  calls.

  Args:
    tokens: A list of tokens.
    token_vocab: A list of unique tokens serving as the vocabulary.

  Returns:
    A list of integers with the same length as `tokens`. The integers range
    from `0` to `len(token_vocab) - 1`.
  
Convert tokens to integer IDs.
`token_vocab` is a list for convenience. This isn't too inefficient because
we build a dictionary for any specific `token_vocab` and cache it for future
calls.
Convert tokens to integer IDs . ` token_vocab ` is a list for convenience . This isn ' t too inefficient because we build a dictionary for any specific ` token_vocab ` and cache it for future calls .
def tokens_to_ids(tokens, token_vocab):
  token_vocab = tuple(token_vocab)
  if not hasattr(tokens_to_ids, 'token_to_id_dicts'):
    tokens_to_ids.token_to_id_dicts = {}
  if token_vocab not in tokens_to_ids.token_to_id_dicts:
    tokens_to_ids.token_to_id_dicts[token_vocab] = {token: i for (i, token) in enumerate(token_vocab)}
  token_to_id_dict = tokens_to_ids.token_to_id_dicts[token_vocab]
  ids = [token_to_id_dict[token] for token in tokens]
  return ids
def tokens_to_ids ( tokens , token_vocab ) : token_vocab = tuple ( token_vocab ) if not hasattr ( tokens_to_ids , 'token_to_id_dicts' ) : tokens_to_ids . token_to_id_dicts = { } if token_vocab not in tokens_to_ids . token_to_id_dicts : tokens_to_ids . token_to_id_dicts [ token_vocab ] = { token : i for ( i , token ) in enumerate ( token_vocab ) } token_to_id_dict = tokens_to_ids . token_to_id_dicts [ token_vocab ] ids = [ token_to_id_dict [ token ] for token in tokens ] return ids
Convert tokens to integer IDs.
Convert tokens to integer IDs .
""" Convert tokens to integer IDs.

  `token_vocab` is a list for convenience. This isn't too inefficient because
  we build a dictionary for any specific `token_vocab` and cache it for future
  calls.

  Args:
    tokens: A list of tokens.
    token_vocab: A list of unique tokens serving as the vocabulary.

  Returns:
    A list of integers with the same length as `tokens`. The integers range
    from `0` to `len(token_vocab) - 1`.
  """
A list of integers with the same length as `tokens`. The integers range
from `0` to `len(token_vocab) - 1`.
A list of integers with the same length as ` tokens ` . The integers range from ` 0 ` to ` len ( token_vocab ) - 1 ` .
tokens
A list of tokens.
A list of tokens .
token_vocab
A list of unique tokens serving as the vocabulary.
A list of unique tokens serving as the vocabulary .