class_docstr: Base class for TextGeneration Strategy
class_docstr_tok: ["Base","class","for","TextGeneration","Strategy"]
docstr: convert the sentences into lists of tokens, pad them to the same length, add bos tokens if it is needed
  Args:
      sentences (List[str]): list of input sentences in str format.
      max_len (int): max number of tokens to generate.
      add_BOS (bool): whether to add the BOS token at the beginning
  Returns:
      Tuple[torch.Tensor], the tokenized and padded torch tensor and the token context length tensor.
code: def tokenize_batch(self, sentences, max_len, add_BOS):
      """
      convert the sentences into lists of tokens, pad them to the same length, add bos tokens if it is needed
      Args:
          sentences (List[str]): list of input sentences in str format.
          max_len (int): max number of tokens to generate.
          add_BOS (bool): whether to add the BOS token at the beginning
      Returns:
          Tuple[torch.Tensor], the tokenized and padded torch tensor and the token context length tensor.
      """
      tokenizer = self.model.tokenizer
      if add_BOS:
          context_tokens = [
              [tokenizer.bos_id] + tokenizer.text_to_ids(s)
              for s in sentences
          ]
      else:
          context_tokens = [tokenizer.text_to_ids(s) for s in sentences]
      context_tokens, context_lengths = pad_batch(
          context_tokens, tokenizer.eos_id, max_len
      )
      context_tokens_tensor = torch.cuda.LongTensor(context_tokens)
      context_length_tensor = torch.cuda.LongTensor(context_lengths)
      return context_tokens_tensor, context_length_tensor
  
imports: import abc
  import copy
  import os
  import re
  import warnings
  from typing import List, Set, Tuple
  import torch
  from transformers import CLIPImageProcessor
  from nemo.collections.nlp.modules.common.lm_utils import pad_batch
  from nemo.collections.nlp.modules.common.megatron.utils import get_ltor_masks_and_position_ids
label_desc: Use image node_id 3 for calling the TextGenerationStrategy obj's underlying member method code with example usage: obj.tokenize_batch(sentences, max_len, add_BOS) and returns: context_tokens_tensor,  context_length_tensor