class_docstr: This :class:`TokenIndexer` represents tokens as lists of character indices.
  
  Registered as a `TokenIndexer` with name "characters".
  
  # Parameters
  
  namespace : `str`, optional (default=`token_characters`)
      We will use this namespace in the :class:`Vocabulary` to map the characters in each token
      to indices.
  character_tokenizer : `CharacterTokenizer`, optional (default=`CharacterTokenizer()`)
      We use a :class:`CharacterTokenizer` to handle splitting tokens into characters, as it has
      options for byte encoding and other things.  The default here is to instantiate a
      `CharacterTokenizer` with its default parameters, which uses unicode characters and
      retains casing.
  start_tokens : `List[str]`, optional (default=`None`)
      These are prepended to the tokens provided to `tokens_to_indices`.
  end_tokens : `List[str]`, optional (default=`None`)
      These are appended to the tokens provided to `tokens_to_indices`.
  min_padding_length : `int`, optional (default=`0`)
      We use this value as the minimum length of padding. Usually used with :class:`CnnEncoder`, its
      value should be set to the maximum value of `ngram_filter_sizes` correspondingly.
  token_min_padding_length : `int`, optional (default=`0`)
      See :class:`TokenIndexer`.
class_docstr_tok: ["This",":","class",":","`","TokenIndexer","`","represents","tokens","as","lists","of","character","indices",".","Registered","as","a","`","TokenIndexer","`","with","name","``","characters","''",".","#","Parameters","namespace",":","`","str","`",",","optional","(","default=","`","token_characters","`",")","We","will","use","this","namespace","in","the",":","class",":","`","Vocabulary","`","to","map","the","characters","in","each","token","to","indices",".","character_tokenizer",":","`","CharacterTokenizer","`",",","optional","(","default=","`","CharacterTokenizer","(",")","`",")","We","use","a",":","class",":","`","CharacterTokenizer","`","to","handle","splitting","tokens","into","characters",",","as","it","has","options","for","byte","encoding","and","other","things",".","The","default","here","is","to","instantiate","a","`","CharacterTokenizer","`","with","its","default","parameters",",","which","uses","unicode","characters","and","retains","casing",".","start_tokens",":","`","List","[","str","]","`",",","optional","(","default=","`","None","`",")","These","are","prepended","to","the","tokens","provided","to","`","tokens_to_indices","`",".","end_tokens",":","`","List","[","str","]","`",",","optional","(","default=","`","None","`",")","These","are","appended","to","the","tokens","provided","to","`","tokens_to_indices","`",".","min_padding_length",":","`","int","`",",","optional","(","default=","`","0","`",")","We","use","this","value","as","the","minimum","length","of","padding",".","Usually","used","with",":","class",":","`","CnnEncoder","`",",","its","value","should","be","set","to","the","maximum","value","of","`","ngram_filter_sizes","`","correspondingly",".","token_min_padding_length",":","`","int","`",",","optional","(","default=","`","0","`",")","See",":","class",":","`","TokenIndexer","`","."]
code: def tokens_to_indices(
      self, tokens: List[Token], vocabulary: Vocabulary
  ) -> Dict[str, List[List[int]]]:
      indices: List[List[int]] = []
      for token in itertools.chain(
          self._start_tokens, tokens, self._end_tokens
      ):
          token_indices: List[int] = []
          if token.text is None:
              raise ConfigurationError(
                  "TokenCharactersIndexer needs a tokenizer that retains text"
              )
          for character in self._character_tokenizer.tokenize(
              token.text
          ):
              if getattr(character, "text_id", None) is not None:
                  # `text_id` being set on the token means that we aren't using the vocab, we just
                  # use this id instead.
                  index = character.text_id
              else:
                  assert character.text is not None
                  index = vocabulary.get_token_index(
                      character.text, self._namespace
                  )
              assert index is not None
              token_indices.append(index)
          indices.append(token_indices)
      return {"token_characters": indices}
  
file_path: allennlp/allennlp/data/token_indexers/token_characters_indexer.py
imports: from typing import Dict, List
  import itertools
  import warnings
  import torch
  from allennlp.common.checks import ConfigurationError
  from allennlp.common.util import pad_sequence_to_length
  from allennlp.data.token_indexers.token_indexer import TokenIndexer, IndexedTokenList
  from allennlp.data.tokenizers import Token, CharacterTokenizer
  from allennlp.data.vocabulary import Vocabulary
label_desc: Use image node_id 3 for calling the TokenCharactersIndexer obj's underlying member method code with example usage: obj.tokens_to_indices(tokens, vocabulary) and returns: dict