class_docstr: Implements the *TopicRank* algorithm described by
  [[bougouin-etal-2013-topicrank]](https://derwen.ai/docs/ptr/biblio/#bougouin-etal-2013-topicrank)
  deployed as a `spaCy` pipeline component.
  
  This class does not get called directly; instantiate its factory
  instead.
  
  Algorithm Overview:
  
  1. Preprocessing: Sentence segmentation, word tokenization, POS tagging.
     After this stage, we have preprocessed text.
  2. Candidate extraction: Extract sequences of nouns and adjectives (i.e. noun chunks)
     After this stage, we have a list of keyphrases that may be topics.
  3. Candidate clustering: Hierarchical Agglomerative Clustering algorithm with average
     linking using simple set-based overlap of lemmas. Similarity is achieved at > 25%
     overlap. **Note**: PyTextRank deviates from the original algorithm here, which uses
     stems rather than lemmas.
     After this stage, we have a list of topics.
  4. Candidate ranking: Apply *TextRank* on a complete graph, with topics as nodes
     (i.e. clusters derived in the last step), where edge weights are higher between
     topics that appear closer together within the document.
     After this stage, we have a ranked list of topics.
  5. Candidate selection: Select the first occurring keyphrase from each topic to
     represent that topic.
     After this stage, we have a ranked list of topics, with a keyphrase to represent
     the topic.
      
docstr: Cluster candidates using Hierarchical Agglomerative Clustering (HAC),
  where similarity is defined by overlap of lemmas in candidates.
  
  The `threshold` and `method` parameters influence the clustering
  behaviour. A `threshold` of 0.25 means that there must at least be
  a 25% overlap of lemmas between two candidates for them to be allowed
  in the same cluster.
  
      candidates:
  list of spans, where each span is a candidate topic.
  
      returns:
  list of clusters of candidates.
          
docstr_tok: ["Cluster","candidates","using","Hierarchical","Agglomerative","Clustering","(","HAC",")",",","where","similarity","is","defined","by","overlap","of","lemmas","in","candidates",".","The","`","threshold","`","and","`","method","`","parameters","influence","the","clustering","behaviour",".","A","`","threshold","`","of","0.25","means","that","there","must","at","least","be","a","25","%","overlap","of","lemmas","between","two","candidates","for","them","to","be","allowed","in","the","same","cluster",".","candidates",":","list","of","spans",",","where","each","span","is","a","candidate","topic",".","returns",":","list","of","clusters","of","candidates","."]
code: def _cluster(
      self,
      candidates: typing.List[Span],
  ) -> typing.List[typing.List[Span]]:
      """
      Cluster candidates using Hierarchical Agglomerative Clustering (HAC),
      where similarity is defined by overlap of lemmas in candidates.
  
      The `threshold` and `method` parameters influence the clustering
      behaviour. A `threshold` of 0.25 means that there must at least be
      a 25% overlap of lemmas between two candidates for them to be allowed
      in the same cluster.
  
          candidates:
      list of spans, where each span is a candidate topic.
  
          returns:
      list of clusters of candidates.
      """
      if not candidates:
          return []
  
      bag_of_words = list(
          {
              word.text
              for candidate in candidates
              for word in candidate
              if not word.is_stop
          }
      )
  
      # Create a bag-of-words representation with a
      # |candidates|x|bag_of_words| matrix
      # matrix = [[0] * len(bag_of_words) for _ in candidates]
      matrix = []
  
      for candidate in candidates:
          matrix.append([0] * len(bag_of_words))
  
          for term in candidate:
              if not term.is_stop:
                  try:
                      matrix[-1][bag_of_words.index(term.text)] += 1
                  except IndexError:
                      pass
  
      # Apply average clustering on pairwise distance,
      # using a threshold of 0.01 below 1 - threshold.
      # So, 0.74 for the default 0.25 threshold.
      pairwise_dist = pdist(matrix, "jaccard")
  
      if not pairwise_dist.size:
          return [[candidate] for candidate in candidates]
  
      raw_clusters = linkage(pairwise_dist, method=self.method)
      cluster_ids = fcluster(
          raw_clusters, t=0.99 - self.threshold, criterion="distance"
      )
  
      # Map cluster_ids to the corresponding candidates, and then
      # ignore the cluster id keys.
      clusters = defaultdict(list)
  
      for cluster_id, candidate in zip(cluster_ids, candidates):
          clusters[cluster_id].append(candidate)
  
      return list(clusters.values())
  
imports: from collections import defaultdict
  from functools import lru_cache
  import time
  import typing
  from icecream import ic
  from scipy.cluster.hierarchy import fcluster, linkage
  from scipy.spatial.distance import pdist
  from spacy.tokens import Doc, Span
  import networkx
  from .base import BaseTextRank, BaseTextRankFactory, Phrase, StopWordsLike
label_desc: Use image node_id 2 for calling the TopicRank obj's underlying member method code with example usage: obj._cluster(candidates) and returns: list, list, unknown