class_docstr: Analyze results of machine learning experiments.
docstr: Non-parametric test for consistent differences between observation pairs.
  
  See `<https://en.wikipedia.org/wiki/Sign_test>`_ for details about
  the test and
  `<https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy
  .stats.binom_test.html>`_
  for details about the scipy implementation.
docstr_tok: ["Non-parametric","test","for","consistent","differences","between","observation","pairs",".","See","`","<","https",":","\/\/en.wikipedia.org\/wiki\/Sign_test",">","`","_","for","details","about","the","test","and","`","<","https",":","\/\/docs.scipy.org\/doc\/scipy-0.14.0\/reference\/generated\/scipy",".stats.binom_test.html",">","`","_","for","details","about","the","scipy","implementation","."]
code: def sign_test(self, metric_name=None):
      """Non-parametric test for consistent differences between observation pairs.
  
      See `<https://en.wikipedia.org/wiki/Sign_test>`_ for details about
      the test and
      `<https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy
      .stats.binom_test.html>`_
      for details about the scipy implementation.
      """
      self._check_is_evaluated()
      metric_name = self._validate_metric_name(metric_name)
      metrics_per_estimator_dataset = (
          self._get_metrics_per_estimator_dataset(metric_name)
      )
  
      sign_df = pd.DataFrame()
      perms = itertools.product(
          metrics_per_estimator_dataset.keys(), repeat=2
      )
  
      for perm in perms:
          x = np.array(metrics_per_estimator_dataset[perm[0]])
          y = np.array(metrics_per_estimator_dataset[perm[1]])
          signs = np.sum([i[0] > i[1] for i in zip(x, y)])
          n = len(x)
  
          # this if/else is for compatibility with scipy < 0.15.0
          if hasattr(stats, "binomtest"):
              binom = stats.binomtest
          else:
              binom = stats.binom_test
  
          p_val = binom(signs, n).pvalue
          sign_test = {
              "estimator_1": perm[0],
              "estimator_2": perm[1],
              "p_val": p_val,
          }
  
          sign_df = pd.concat(
              [sign_df, pd.DataFrame(sign_test, index=[0])],
              ignore_index=True,
          )
          sign_df_pivot = sign_df.pivot(
              index="estimator_1", columns="estimator_2", values="p_val"
          )
  
      return sign_df, sign_df_pivot
  
imports: import itertools
  import numpy
  import pandas
  from scipy import stats
  from scipy.stats import ranksums, ttest_ind
  from sktime.benchmarking.base import BaseResults
  from sktime.exceptions import NotEvaluatedError
  from sktime.utils.validation._dependencies import _check_soft_dependencies
label_desc: Use image node_id 10 for calling the Evaluator obj's underlying member method code with example usage: obj.sign_test(metric_name) and returns: sign_df,  sign_df_pivot