class_docstr: Manages issues related to non-iid data distributions.
  
  Parameters
  ----------
  datalab :
      The Datalab instance that this issue manager searches for issues in.
  
  metric :
      The distance metric used to compute the KNN graph of the examples in the dataset.
      If set to `None`, the metric will be automatically selected based on the dimensionality
      of the features used to represent the examples in the dataset.
  
  k :
      The number of nearest neighbors to consider when computing the KNN graph of the examples.
  
  num_permutations :
      The number of trials to run when performing permutation testing to determine whether
      the distribution of index-distances between neighbors in the dataset is IID or not.
  
  Note
  ----
  This class will only flag a single example as an issue if the dataset is considered non-IID. This type of issue
  is more relevant to the entire dataset as a whole, rather than to individual examples.
class_docstr_tok: ["Manages","issues","related","to","non-iid","data","distributions",".","Parameters","--","--","--","--","--","datalab",":","The","Datalab","instance","that","this","issue","manager","searches","for","issues","in",".","metric",":","The","distance","metric","used","to","compute","the","KNN","graph","of","the","examples","in","the","dataset",".","If","set","to","`","None","`",",","the","metric","will","be","automatically","selected","based","on","the","dimensionality","of","the","features","used","to","represent","the","examples","in","the","dataset",".","k",":","The","number","of","nearest","neighbors","to","consider","when","computing","the","KNN","graph","of","the","examples",".","num_permutations",":","The","number","of","trials","to","run","when","performing","permutation","testing","to","determine","whether","the","distribution","of","index-distances","between","neighbors","in","the","dataset","is","IID","or","not",".","Note","--","--","This","class","will","only","flag","a","single","example","as","an","issue","if","the","dataset","is","considered","non-IID",".","This","type","of","issue","is","more","relevant","to","the","entire","dataset","as","a","whole",",","rather","than","to","individual","examples","."]
docstr: This function computes a variant of the KS statistic for each
  datapoint. Rather than computing the maximum difference
  between the CDF of the neighbor distances (foreground
  distribution) and the CDF of the all index distances
  (background distribution), we compute the absolute difference
  in area-under-the-curve of the two CDFs.
  
  The foreground distribution is computed by sampling the
  neighbor distances from the KNN graph, but the background
  distribution is computed analytically. The background CDF for
  a datapoint i can be split up into three parts. Let d = min(i,
  N - i - 1).
  
  1. For 0 < j <= d, the slope of the CDF is 2 / (N - 1) since
  there are two datapoints in the dataset that are distance j
  from datapoint i. We call this threshold the 'double distance
  threshold'
  
  2. For d < j <= N - d - 1, the slope of the CDF is
  1 / (N - 1) since there is only one datapoint in the dataset
  that is distance j from datapoint i.
  
  3. For j > N - d - 1, the slope of the CDF is 0 and is
  constant at 1.0 since there are no datapoints in the dataset
  that are distance j from datapoint i.
  
  We compute the area differences on each of the k intervals for
  which the foreground CDF is constant which allows for the
  possibility that the background CDF may intersect the
  foreground CDF on this interval. We do not account for these
  cases when computing absolute AUC difference.
  
  Our algorithm is simple, sort the k sampled neighbor
  distances. Then, for each of the k neighbor distances sampled,
  compute the AUC for each CDF up to that point. Then, subtract
  from each area the previous area in the sorted order to get
  the AUC of the CDF on the interval between those two
  points. Subtract the background interval AUCs from the
  foreground interval AUCs, take the absolute value, and
  sum. The algorithm is vectorized such that this statistic is
  computed for each of the N datapoints simultaneously.
  
  The statistics are then normalized by their respective maximum
  possible distance (N - d - 1) and then mapped to [0,1] via
  tanh.
docstr_tok: ["This","function","computes","a","variant","of","the","KS","statistic","for","each","datapoint",".","Rather","than","computing","the","maximum","difference","between","the","CDF","of","the","neighbor","distances","(","foreground","distribution",")","and","the","CDF","of","the","all","index","distances","(","background","distribution",")",",","we","compute","the","absolute","difference","in","area-under-the-curve","of","the","two","CDFs",".","The","foreground","distribution","is","computed","by","sampling","the","neighbor","distances","from","the","KNN","graph",",","but","the","background","distribution","is","computed","analytically",".","The","background","CDF","for","a","datapoint","i","can","be","split","up","into","three","parts",".","Let","d","=","min","(","i",",","N","-","i","-","1",")",".","1",".","For","0","<","j","<","=","d",",","the","slope","of","the","CDF","is","2","\/","(","N","-","1",")","since","there","are","two","datapoints","in","the","dataset","that","are","distance","j","from","datapoint","i",".","We","call","this","threshold","the","'double","distance","threshold'","2",".","For","d","<","j","<","=","N","-","d","-","1",",","the","slope","of","the","CDF","is","1","\/","(","N","-","1",")","since","there","is","only","one","datapoint","in","the","dataset","that","is","distance","j","from","datapoint","i",".","3",".","For","j",">","N","-","d","-","1",",","the","slope","of","the","CDF","is","0","and","is","constant","at","1.0","since","there","are","no","datapoints","in","the","dataset","that","are","distance","j","from","datapoint","i",".","We","compute","the","area","differences","on","each","of","the","k","intervals","for","which","the","foreground","CDF","is","constant","which","allows","for","the","possibility","that","the","background","CDF","may","intersect","the","foreground","CDF","on","this","interval",".","We","do","not","account","for","these","cases","when","computing","absolute","AUC","difference",".","Our","algorithm","is","simple",",","sort","the","k","sampled","neighbor","distances",".","Then",",","for","each","of","the","k","neighbor","distances","sampled",",","compute","the","AUC","for","each","CDF","up","to","that","point",".","Then",",","subtract","from","each","area","the","previous","area","in","the","sorted","order","to","get","the","AUC","of","the","CDF","on","the","interval","between","those","two","points",".","Subtract","the","background","interval","AUCs","from","the","foreground","interval","AUCs",",","take","the","absolute","value",",","and","sum",".","The","algorithm","is","vectorized","such","that","this","statistic","is","computed","for","each","of","the","N","datapoints","simultaneously",".","The","statistics","are","then","normalized","by","their","respective","maximum","possible","distance","(","N","-","d","-","1",")","and","then","mapped","to","[","0,1","]","via","tanh","."]
code: def _score_dataset(self) -> npt.NDArray[np.float64]:
      """This function computes a variant of the KS statistic for each
      datapoint. Rather than computing the maximum difference
      between the CDF of the neighbor distances (foreground
      distribution) and the CDF of the all index distances
      (background distribution), we compute the absolute difference
      in area-under-the-curve of the two CDFs.
  
      The foreground distribution is computed by sampling the
      neighbor distances from the KNN graph, but the background
      distribution is computed analytically. The background CDF for
      a datapoint i can be split up into three parts. Let d = min(i,
      N - i - 1).
  
      1. For 0 < j <= d, the slope of the CDF is 2 / (N - 1) since
      there are two datapoints in the dataset that are distance j
      from datapoint i. We call this threshold the 'double distance
      threshold'
  
      2. For d < j <= N - d - 1, the slope of the CDF is
      1 / (N - 1) since there is only one datapoint in the dataset
      that is distance j from datapoint i.
  
      3. For j > N - d - 1, the slope of the CDF is 0 and is
      constant at 1.0 since there are no datapoints in the dataset
      that are distance j from datapoint i.
  
      We compute the area differences on each of the k intervals for
      which the foreground CDF is constant which allows for the
      possibility that the background CDF may intersect the
      foreground CDF on this interval. We do not account for these
      cases when computing absolute AUC difference.
  
      Our algorithm is simple, sort the k sampled neighbor
      distances. Then, for each of the k neighbor distances sampled,
      compute the AUC for each CDF up to that point. Then, subtract
      from each area the previous area in the sorted order to get
      the AUC of the CDF on the interval between those two
      points. Subtract the background interval AUCs from the
      foreground interval AUCs, take the absolute value, and
      sum. The algorithm is vectorized such that this statistic is
      computed for each of the N datapoints simultaneously.
  
      The statistics are then normalized by their respective maximum
      possible distance (N - d - 1) and then mapped to [0,1] via
      tanh.
      """
      N = self.N
  
      sorted_neighbors = np.sort(self.neighbor_index_distances, axis=1)
  
      # find the maximum distance that occurs with double probability
      middle_idx = np.floor((N - 1) / 2).astype(int)
      double_distances = np.arange(N).reshape(N, 1)
      double_distances[double_distances > middle_idx] -= N - 1
      double_distances = np.abs(double_distances)
  
      sorted_neighbors = np.hstack(
          [sorted_neighbors, np.ones((N, 1)) * (N - 1)]
      ).astype(int)
  
      # the set of distances that are less than the double distance threshold
      set_beginning = sorted_neighbors <= double_distances
      # the set of distances that are greater than the double distance threshold but have nonzero probability
      set_middle = (sorted_neighbors > double_distances) & (
          sorted_neighbors <= (N - double_distances - 1)
      )
      # the set of distances that occur with 0 probability
      set_end = sorted_neighbors > (N - double_distances - 1)
  
      shifted_neighbors = np.zeros(sorted_neighbors.shape)
      shifted_neighbors[:, 1:] = sorted_neighbors[:, :-1]
      diffs = (
          sorted_neighbors - shifted_neighbors
      )  # the distances between the sorted indices
  
      area_beginning = (double_distances**2) / (N - 1)
      length = N - 2 * double_distances - 1
      a = 2 * double_distances / (N - 1)
      area_middle = 0.5 * (a + 1) * length
  
      # compute the area under the CDF for each of the indices in sorted_neighbors
      background_area = np.zeros(diffs.shape)
      background_diffs = np.zeros(diffs.shape)
      background_area[set_beginning] = (
          (sorted_neighbors**2) / (N - 1)
      )[set_beginning]
      background_area[set_middle] = (
          area_beginning
          + 0.5
          * (
              (sorted_neighbors + 3 * double_distances)
              * (sorted_neighbors - double_distances)
              / (N - 1)
          )
      )[set_middle]
      background_area[set_end] = (
          area_beginning
          + area_middle
          + (sorted_neighbors - (N - double_distances - 1) * 1.0)
      )[set_end]
  
      # compute the area under the CDF between indices in sorted_neighbors
      shifted_background = np.zeros(background_area.shape)
      shifted_background[:, 1:] = background_area[:, :-1]
      background_diffs = background_area - shifted_background
  
      # compute the foreground CDF and AUC between indices in sorted_neighbors
      foreground_cdf = np.arange(sorted_neighbors.shape[1]) / (
          sorted_neighbors.shape[1] - 1
      )
      foreground_diffs = foreground_cdf.reshape(1, -1) * diffs
  
      # compute the differences between foreground and background area intervals
      area_diffs = np.abs(foreground_diffs - background_diffs)
      stats = np.sum(area_diffs, axis=1)
  
      # normalize scores by the index and transform to [0, 1]
      indices = np.arange(N)
      reverse = N - indices
      normalizer = np.where(indices > reverse, indices, reverse)
  
      scores = stats / normalizer
      scores = np.tanh(-1 * scores) + 1
      return scores
  
imports: from __future__ import annotations
  from typing import TYPE_CHECKING, Any, ClassVar, Dict, Optional, Union, cast
  import warnings
  import itertools
  from scipy.stats import gaussian_kde
  import numpy
  import pandas
  from scipy.sparse import csr_matrix
  from sklearn.neighbors import NearestNeighbors
  from sklearn.exceptions import NotFittedError
  from sklearn.utils.validation import check_is_fitted
  from cleanlab.datalab.internal.issue_manager import IssueManager
label_desc: Use image node_id 9 for calling the NonIIDIssueManager obj's underlying member method code with example usage: obj._score_dataset() and returns: scores