class_docstr: Implements the *TextRank* algorithm defined by
  [[mihalcea04textrank]](https://derwen.ai/docs/ptr/biblio/#mihalcea04textrank),
  deployed as a `spaCy` pipeline component.
  
  This class does not get called directly; instantiate its factory
  instead.
      
class_docstr_tok: ["Implements","the","*","TextRank","*","algorithm","defined","by","[","[","mihalcea04textrank","]","]","(","https",":","\/\/derwen.ai\/docs\/ptr\/biblio\/","#","mihalcea04textrank",")",",","deployed","as","a","`","spaCy","`","pipeline","component",".","This","class","does","not","get","called","directly",";","instantiate","its","factory","instead","."]
docstr: Since the noun chunking is greedy, we discount the ranks using a point
  estimate based on the number of non-lemma tokens within a phrase.
  
      span:
  a span representing one phrase
  
      sum_rank:
  sum of the ranks for each token within this span
  
      returns:
  normalized rank metric
          
code: def _calc_discounted_normalised_rank(
      self, span: Span, sum_rank: float
  ) -> float:
      """
      Since the noun chunking is greedy, we discount the ranks using a point
      estimate based on the number of non-lemma tokens within a phrase.
  
          span:
      a span representing one phrase
  
          sum_rank:
      sum of the ranks for each token within this span
  
          returns:
      normalized rank metric
      """
      non_lemma = len(
          [tok for tok in span if tok.pos_ not in self.pos_kept]
      )
      non_lemma_discount = len(span) / (
          len(span) + (2.0 * non_lemma) + 1.0
      )
  
      # NB:
      # This implements a *point estimate* for the ratio of the span
      # of an extracted phrase divided by the number of non-lemma
      # tokens within that span.
      #
      # The algorithm defined as in the original paper
      # [[mihalcea04textrank]](https://derwen.ai/docs/ptr/biblio/#mihalcea04textrank)
      # only considered multi-word phrases which had adjacent lemmas.
      # Early use cases in industry (circa 2009) required better recall
      # of phrases so this approach of using a point estimate here to
      # "normalize" the rank metric causes the longer phrases to get
      # discounted when they included too many non-lemma tokens.
      # In other words, we allow some non-lemmas, but avoid having
      # the algorithm become too "greedy" as it builds phrases.
      #
      # Kudos to @debraj135 who asked for an explanation about this
      # section of the code.
      phrase_rank = math.sqrt(sum_rank / (len(span) + non_lemma))
  
      return phrase_rank * non_lemma_discount
  
imports: from collections import Counter, defaultdict, OrderedDict
  from dataclasses import dataclass
  import json
  import math
  import pathlib
  import time
  import typing
  from icecream import ic
  from spacy.tokens import Doc, Span, Token
  import graphviz
  import networkx
  from .util import groupby_apply, default_scrubber
label_desc: Use image node_id 11 for calling the BaseTextRank obj's underlying member method code with example usage: obj._calc_discounted_normalised_rank(span, sum_rank) and returns: unknown