class_docstr: A `Tokenizer` splits strings of text into tokens.  Typically, this either splits text into
  word tokens or character tokens, and those are the two tokenizer subclasses we have implemented
  here, though you could imagine wanting to do other kinds of tokenization for structured or
  other inputs.
  
  See the parameters to, e.g., :class:`~.SpacyTokenizer`, or whichever tokenizer
  you want to use.
  
  If the base input to your model is words, you should use a :class:`~.SpacyTokenizer`, even if
  you also want to have a character-level encoder to get an additional vector for each word
  token.  Splitting word tokens into character arrays is handled separately, in the
  :class:`..token_representations.TokenRepresentation` class.
docstr: Adds special tokens to tokenized text. These are tokens like [CLS] or [SEP].
  
  Not all tokenizers do this. The default is to just return the tokens unchanged.
  
  # Parameters
  
  tokens1 : `List[Token]`
      The list of tokens to add special tokens to.
  tokens2 : `Optional[List[Token]]`
      An optional second list of tokens. This will be concatenated with `tokens1`. Special tokens will be
      added as appropriate.
  
  # Returns
  tokens : `List[Token]`
      The combined list of tokens, with special tokens added.
docstr_tok: ["Adds","special","tokens","to","tokenized","text",".","These","are","tokens","like","[","CLS","]","or","[","SEP","]",".","Not","all","tokenizers","do","this",".","The","default","is","to","just","return","the","tokens","unchanged",".","#","Parameters","tokens1",":","`","List","[","Token","]","`","The","list","of","tokens","to","add","special","tokens","to",".","tokens2",":","`","Optional","[","List","[","Token","]","]","`","An","optional","second","list","of","tokens",".","This","will","be","concatenated","with","`","tokens1","`",".","Special","tokens","will","be","added","as","appropriate",".","#","Returns","tokens",":","`","List","[","Token","]","`","The","combined","list","of","tokens",",","with","special","tokens","added","."]
code: def add_special_tokens(
      self, tokens1: List[Token], tokens2: Optional[List[Token]] = None
  ) -> List[Token]:
      """
      Adds special tokens to tokenized text. These are tokens like [CLS] or [SEP].
  
      Not all tokenizers do this. The default is to just return the tokens unchanged.
  
      # Parameters
  
      tokens1 : `List[Token]`
          The list of tokens to add special tokens to.
      tokens2 : `Optional[List[Token]]`
          An optional second list of tokens. This will be concatenated with `tokens1`. Special tokens will be
          added as appropriate.
  
      # Returns
      tokens : `List[Token]`
          The combined list of tokens, with special tokens added.
      """
      return tokens1 + (tokens2 or [])
  
file_path: allennlp/allennlp/data/tokenizers/tokenizer.py
imports: from typing import List, Optional
  import logging
  from allennlp.common import Registrable
  from allennlp.data.tokenizers.token_class import Token
label_desc: Use image node_id 3 for calling the Tokenizer obj's underlying member method code with example usage: obj.add_special_tokens(tokens1, tokens2) and returns: unknown