class_docstr: A class containing all functions for auto-regressive text generation
  
  Pass custom parameter values to 'generate' .
docstr: Decode the id to words
  
  :param text: The text or batch of text to be tokenized
  :param add_bos:
  
  :return: list of ids that indicates the tokens
docstr_tok: ["Decode","the","id","to","words",":","param","text",":","The","text","or","batch","of","text","to","be","tokenized",":","param","add_bos",":",":","return",":","list","of","ids","that","indicates","the","tokens"]
code: def tokenize(
      self, text: Union[str, List[str]], add_bos: bool = True
  ) -> List[int]:
      """
      Decode the id to words
  
      :param text: The text or batch of text to be tokenized
      :param add_bos:
  
      :return: list of ids that indicates the tokens
      """
      is_batched = True if isinstance(text, (list, tuple)) else False
      if not is_batched:
          text = [text]
  
      result = []
      for t in text:
          if isinstance(t, str):
              bstr = t.encode()
          else:
              bstr = t
          result.append(self._tokenize(bstr, add_bos))
  
      if not is_batched:
          result = result[0]
      return result
  
imports: from typing import Optional, Union, Sequence, List
  from bigdl.llm.utils.common import invalidInputError
  import torch
label_desc: Use image node_id 1 for calling the GenerationMixin obj's underlying member method code with example usage: obj.tokenize(text, add_bos) and returns: result