class_docstr: A class that regroups important test to make sure that we properly handle the special tokens.
class_docstr_tok: ["A","class","that","regroups","important","test","to","make","sure","that","we","properly","handle","the","special","tokens","."]
code: def test_character_after_special_token(self):
      # Make sure that `tokenizer.tokenize` is similar to
      # adding the equivalent special token to the vocab
      input_ids = self.tokenizer.encode("Hey <extra_id_0>I")
      self.assertEqual(input_ids, [156, 30, 1001, 100, 2])
      tokens = self.tokenizer.tokenize("Hey <extra_id_0>I")
      self.assertEqual(tokens, ["▁He", "y", "<extra_id_0>", "I"])
  
      input_ids = self.tokenizer.encode("Hello, <extra_id_0>,")
      self.assertEqual(input_ids, [156, 86, 20, 3, 1001, 3, 2])
      tokens = self.tokenizer.tokenize("Hello, <extra_id_0>,")
      self.assertEqual(
          tokens, ["▁He", "ll", "o", ",", "<extra_id_0>", ","]
      )
  
imports: import json
  import os
  import re
  import tempfile
  import unittest
  from transformers import SPIECE_UNDERLINE, AddedToken, BatchEncoding, T5Tokenizer, T5TokenizerFast
  from transformers.testing_utils import get_tests_dir, require_sentencepiece, require_seqio, require_tokenizers, slow
  from transformers.utils import cached_property, is_tf_available, is_torch_available
  from ...test_tokenization_common import TokenizerTesterMixin
label_desc: Use image node_id 4 for calling the CommonSpmIntegrationTests obj's underlying member method code with example usage: obj.test_character_after_special_token() without return types