class_docstr: Term Frequency - Inverse Document Frequency (TF-IDF) Recommender
  
  This class provides content-based recommendations using TF-IDF vectorization in combination with cosine similarity.
docstr: Tokenize the input text.
  For more details on the TfidfVectorizer, see https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
  
  Args:
      df_clean (pandas.DataFrame): Dataframe with cleaned text in the new column.
      text_col (str): Name of column containing the cleaned text.
      ngram_range (tuple of int): The lower and upper boundary of the range of n-values for different n-grams to be extracted.
      min_df (int): When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold.
  
  Returns:
      TfidfVectorizer, pandas.Series:
      - Scikit-learn TfidfVectorizer object defined in `.tokenize_text()`.
      - Each row contains tokens for respective documents separated by spaces.
docstr_tok: ["Tokenize","the","input","text",".","For","more","details","on","the","TfidfVectorizer",",","see","https",":","\/\/scikit-learn.org\/stable\/modules\/generated\/sklearn.feature_extraction.text.TfidfVectorizer.html","Args",":","df_clean","(","pandas.DataFrame",")",":","Dataframe","with","cleaned","text","in","the","new","column",".","text_col","(","str",")",":","Name","of","column","containing","the","cleaned","text",".","ngram_range","(","tuple","of","int",")",":","The","lower","and","upper","boundary","of","the","range","of","n-values","for","different","n-grams","to","be","extracted",".","min_df","(","int",")",":","When","building","the","vocabulary","ignore","terms","that","have","a","document","frequency","strictly","lower","than","the","given","threshold",".","Returns",":","TfidfVectorizer",",","pandas.Series",":","-","Scikit-learn","TfidfVectorizer","object","defined","in","`",".tokenize_text","(",")","`",".","-","Each","row","contains","tokens","for","respective","documents","separated","by","spaces","."]
code: def tokenize_text(
      self,
      df_clean,
      text_col="cleaned_text",
      ngram_range=(1, 3),
      min_df=0,
  ):
      """Tokenize the input text.
      For more details on the TfidfVectorizer, see https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html
  
      Args:
          df_clean (pandas.DataFrame): Dataframe with cleaned text in the new column.
          text_col (str): Name of column containing the cleaned text.
          ngram_range (tuple of int): The lower and upper boundary of the range of n-values for different n-grams to be extracted.
          min_df (int): When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold.
  
      Returns:
          TfidfVectorizer, pandas.Series:
          - Scikit-learn TfidfVectorizer object defined in `.tokenize_text()`.
          - Each row contains tokens for respective documents separated by spaces.
      """
      vectors = df_clean[text_col]
  
      # If a HuggingFace BERT word tokenization method
      if self.tokenization_method in ["bert", "scibert"]:
          # Set vectorizer
          tf = TfidfVectorizer(
              analyzer="word",
              ngram_range=ngram_range,
              min_df=min_df,
              stop_words="english",
          )
  
          # Get appropriate transformer name
          if self.tokenization_method == "bert":
              bert_method = "bert-base-cased"
          elif self.tokenization_method == "scibert":
              bert_method = "allenai/scibert_scivocab_cased"
  
          # Load pre-trained model tokenizer (vocabulary)
          tokenizer = BertTokenizer.from_pretrained(bert_method)
  
          # Loop through each item
          vectors_tokenized = vectors.copy()
          for i in range(0, len(vectors)):
              vectors_tokenized[i] = " ".join(
                  tokenizer.tokenize(vectors[i])
              )
  
      elif self.tokenization_method == "nltk":
          # NLTK Stemming
          token_dict = {}  # noqa: F841
          stemmer = PorterStemmer()
  
          def stem_tokens(tokens, stemmer):
              stemmed = []
              for item in tokens:
                  stemmed.append(stemmer.stem(item))
              return stemmed
  
          def tokenize(text):
              tokens = nltk.word_tokenize(text)
              stems = stem_tokens(tokens, stemmer)
              return stems
  
          # When defining a custome tokenizer with TfidfVectorizer, the tokenization is applied in the fit function
          tf = TfidfVectorizer(
              tokenizer=tokenize,
              analyzer="word",
              ngram_range=ngram_range,
              min_df=min_df,
              stop_words="english",
          )
          vectors_tokenized = vectors
  
      elif self.tokenization_method == "none":
          # No tokenization applied
          tf = TfidfVectorizer(
              analyzer="word",
              ngram_range=ngram_range,
              min_df=min_df,
              stop_words="english",
          )
          vectors_tokenized = vectors
  
      # Save to class variable
      self.tf = tf
  
      return tf, vectors_tokenized
  
imports: from sklearn.feature_extraction.text import TfidfVectorizer
  from sklearn.metrics.pairwise import linear_kernel
  from transformers import BertTokenizer
  import re
  import unicodedata
  import pandas
  import numpy
  import nltk
  from nltk.stem.porter import PorterStemmer
label_desc: Use image node_id 4 for calling the TfidfRecommender obj's underlying member method code with example usage: obj.tokenize_text(df_clean, text_col, ngram_range, min_df) and returns: tf,  vectors_tokenized, stemmed, stems