class_docstr: Checks for frequent substrings in the dataset.
  
  Substrings of varying lengths (n-grams) are extracted from the dataset text samples.
  The frequencies of these n-grams are calculated and only substrings exceeding a defined minimum length are retained.
  The substrings are then sorted by their frequencies and the most frequent substrings are identified.
  Finally, the substrings with the highest frequency and those surpassing a significance level are displayed.
  
  Parameters
  ----------
  n_to_show : int, default: 5
      Number of most frequent substrings to show.
  n_samples : int, default: 10_000
      Number of samples to use for this check.
  random_state : int, default: 42
      Random seed for all check internals.
  n_sentences : int, default: 5
      The number of sentences to extract from the beginning and end of the text content.
  min_ngram_length: int, default: 4
      Minimum amount of words for a substring to be considered a frequent substring.
  min_substring_ratio: float, default: 0.05
      Minimum frequency required for a substring to be considered "frequent".
  significant_substring_ratio: float, default: 0.3
      Frequency above which samples are considered significant. Substrings
      meeting or exceeding this ratio will always be returned, regardless of
      other parameters and conditions.
  frequency_margin: float, default: 0.02
      Defines the tolerance level for selecting longer overlapping substrings. If a longer substring
      has a frequency that's less than a shorter overlapping substring but the difference is within
      the specified frequency_margin, the longer substring is still preferred.
  min_relative_change : float, optional, default=0.05
      Defines the threshold for relative change. If the computed relative
      change falls below this specified threshold, it is
      considered insignificant and is thus set to zero.
class_docstr_tok: ["Checks","for","frequent","substrings","in","the","dataset",".","Substrings","of","varying","lengths","(","n-grams",")","are","extracted","from","the","dataset","text","samples",".","The","frequencies","of","these","n-grams","are","calculated","and","only","substrings","exceeding","a","defined","minimum","length","are","retained",".","The","substrings","are","then","sorted","by","their","frequencies","and","the","most","frequent","substrings","are","identified",".","Finally",",","the","substrings","with","the","highest","frequency","and","those","surpassing","a","significance","level","are","displayed",".","Parameters","--","--","--","--","--","n_to_show",":","int",",","default",":","5","Number","of","most","frequent","substrings","to","show",".","n_samples",":","int",",","default",":","10_000","Number","of","samples","to","use","for","this","check",".","random_state",":","int",",","default",":","42","Random","seed","for","all","check","internals",".","n_sentences",":","int",",","default",":","5","The","number","of","sentences","to","extract","from","the","beginning","and","end","of","the","text","content",".","min_ngram_length",":","int",",","default",":","4","Minimum","amount","of","words","for","a","substring","to","be","considered","a","frequent","substring",".","min_substring_ratio",":","float",",","default",":","0.05","Minimum","frequency","required","for","a","substring","to","be","considered","``","frequent","''",".","significant_substring_ratio",":","float",",","default",":","0.3","Frequency","above","which","samples","are","considered","significant",".","Substrings","meeting","or","exceeding","this","ratio","will","always","be","returned",",","regardless","of","other","parameters","and","conditions",".","frequency_margin",":","float",",","default",":","0.02","Defines","the","tolerance","level","for","selecting","longer","overlapping","substrings",".","If","a","longer","substring","has","a","frequency","that","'s","less","than","a","shorter","overlapping","substring","but","the","difference","is","within","the","specified","frequency_margin",",","the","longer","substring","is","still","preferred",".","min_relative_change",":","float",",","optional",",","default=0.05","Defines","the","threshold","for","relative","change",".","If","the","computed","relative","change","falls","below","this","specified","threshold",",","it","is","considered","insignificant","and","is","thus","set","to","zero","."]
docstr: Calculate the frequencies of n-grams in the provided data.
  
  For each n-gram extracted from the dataset, the method computes its frequency
  and keeps track of the original and filtered indexes where the n-gram occurs.
  Only n-grams that have a frequency greater than or equal to `self.min_substring_ratio`
  are retained in the results.
  
  Parameters:
  -----------
  data : list of tuple
      The dataset from which to extract n-grams. Each tuple consists of
      an original index and a text string.
  n : int
      The length of the n-grams to be extracted.
  num_samples : int
      The total number of samples in the dataset.
  
  Returns:
  --------
  tuple
      A tuple containing two items:
      1. A dictionary where keys are n-grams and values are another dictionary
         containing frequency ('freq') and original indexes ('original_indexes') of the n-gram.
      2. A set containing indexes of filtered samples that have qualifying n-grams.
  
  Notes:
  ------
  The method uses `self._get_ngrams` to extract n-grams from text strings
  and `self.min_substring_ratio` as the threshold for deciding which n-grams
  are frequent enough to be included in the results.
docstr_tok: ["Calculate","the","frequencies","of","n-grams","in","the","provided","data",".","For","each","n-gram","extracted","from","the","dataset",",","the","method","computes","its","frequency","and","keeps","track","of","the","original","and","filtered","indexes","where","the","n-gram","occurs",".","Only","n-grams","that","have","a","frequency","greater","than","or","equal","to","`","self.min_substring_ratio","`","are","retained","in","the","results",".","Parameters",":","--","--","--","--","--","-","data",":","list","of","tuple","The","dataset","from","which","to","extract","n-grams",".","Each","tuple","consists","of","an","original","index","and","a","text","string",".","n",":","int","The","length","of","the","n-grams","to","be","extracted",".","num_samples",":","int","The","total","number","of","samples","in","the","dataset",".","Returns",":","--","--","--","--","tuple","A","tuple","containing","two","items",":","1",".","A","dictionary","where","keys","are","n-grams","and","values","are","another","dictionary","containing","frequency","(","'freq","'",")","and","original","indexes","(","'original_indexes","'",")","of","the","n-gram",".","2",".","A","set","containing","indexes","of","filtered","samples","that","have","qualifying","n-grams",".","Notes",":","--","--","--","The","method","uses","`","self._get_ngrams","`","to","extract","n-grams","from","text","strings","and","`","self.min_substring_ratio","`","as","the","threshold","for","deciding","which","n-grams","are","frequent","enough","to","be","included","in","the","results","."]
code: def _calculate_ngram_frequencies(self, data, n, num_samples):
      """
      Calculate the frequencies of n-grams in the provided data.
  
      For each n-gram extracted from the dataset, the method computes its frequency
      and keeps track of the original and filtered indexes where the n-gram occurs.
      Only n-grams that have a frequency greater than or equal to `self.min_substring_ratio`
      are retained in the results.
  
      Parameters:
      -----------
      data : list of tuple
          The dataset from which to extract n-grams. Each tuple consists of
          an original index and a text string.
      n : int
          The length of the n-grams to be extracted.
      num_samples : int
          The total number of samples in the dataset.
  
      Returns:
      --------
      tuple
          A tuple containing two items:
          1. A dictionary where keys are n-grams and values are another dictionary
             containing frequency ('freq') and original indexes ('original_indexes') of the n-gram.
          2. A set containing indexes of filtered samples that have qualifying n-grams.
  
      Notes:
      ------
      The method uses `self._get_ngrams` to extract n-grams from text strings
      and `self.min_substring_ratio` as the threshold for deciding which n-grams
      are frequent enough to be included in the results.
      """
      ngram_info = defaultdict(
          lambda: {
              "freq": 0,
              "original_indexes": [],
              "filtered_indexes": [],
          }
      )
      filtered_samples = set()
  
      for index, item in enumerate(data):
          for ngram in self._get_ngrams(item[1], n):
              ngram_info[ngram]["original_indexes"].append(item[0])
              ngram_info[ngram]["filtered_indexes"].append(index)
  
      ngrams = list(ngram_info.keys())
      for ngram in ngrams:
          ngram_freq = (
              len(ngram_info[ngram]["original_indexes"]) / num_samples
          )
          if ngram_freq >= self.min_substring_ratio:
              ngram_info[ngram]["freq"] = ngram_freq
              for index in ngram_info[ngram]["filtered_indexes"]:
                  filtered_samples.add(index)
              del ngram_info[ngram]["filtered_indexes"]
          else:
              del ngram_info[ngram]
  
      return ngram_info, filtered_samples
  
imports: import re
  from collections import defaultdict
  from typing import Dict
  import pandas
  from deepchecks.core import CheckResult, ConditionCategory, ConditionResult
  from deepchecks.core.errors import DeepchecksValueError
  from deepchecks.nlp import Context, SingleDatasetCheck
  from deepchecks.utils.strings import format_percent
label_desc: Use image node_id 5 for calling the FrequentSubstrings obj's underlying member method code with example usage: obj._calculate_ngram_frequencies(data, n, num_samples) and returns: ngram_info,  filtered_samples