cytometer/utils.py
def compare_ecdfs(x, y, alpha=0.05, num_quantiles=101, num_perms=1000, rng_seed=0,
                  resampling_method='bootstrap', multitest_method=None):
    """
    Compute p-values for the difference between each percentile point of the empirical cumulative distribution
    functions (ECDFs) of two samples x, y.

    This function allows multiple test adjustment of p-values using statsmodels.stats.multitest.multipletests.

    This function is basically an implementation of permutation testing (see overview in
    http://rasbt.github.io/mlxtend/user_guide/evaluate/permutation_test/#overview), but generating one p-value
    for each percentile.

    For example, if the median(x)=10, median(y)=7, this function computes the p-value, or probability
    that the observed difference is due to mere chance. The median is the 50 percentile. This function
    computes p-values for the differences between the 0, 1, 2, ..., 100 percentiles.

    It also has an experimental implementation, where instead of permutations, it uses bootstrap samples with
    replacement.

    :param x: numpy.ndarray vector, data set 1.
    :param y: numpy.ndarray vector, data set 2.
    :param alpha: (def 0.05) float. Error rate. Confidence is 1-alpha.
    :param num_quantiles: (def 101) Number of points the quantile range is split into. For 101, the
    quantiles are the percentiles 0%, 1%, 2%, ..., 100%.
    :param num_perms: (def 1000) Number of bootstrap permutations the p-values are estimated from.
    :param resampling_method: (def 'bootstrap') String with the name used for resampling. Note that it's 'bootstrap'
    for historical reasons, but the orthodox approach is 'permutation'. Because we haven't validated the bootstrap
    approach, using it produces a warning message.
    :param rng_seed: (def 0) Seed for the random generator of permutations or bootstrap samples. This allows to get
    the same results in different runs.
    :param multitest_method: (def None) Adjustment of p-values due to multitesting. These are the same
    values as for statsmodels.stats.multitest.multipletests. Currently
    (https://www.statsmodels.org/dev/_modules/statsmodels/stats/multitest.html),
        - `bonferroni` : one-step correction
        - `sidak` : one-step correction
        - `holm-sidak` : step down method using Sidak adjustments
        - `holm` : step-down method using Bonferroni adjustments
        - `simes-hochberg` : step-up method  (independent)
        - `hommel` : closed method based on Simes tests (non-negative)
        - `fdr_bh` : Benjamini/Hochberg  (non-negative)
        - `fdr_by` : Benjamini/Yekutieli (negative)
        - `fdr_tsbh` : two stage fdr correction (non-negative)
        - `fdr_tsbky` : two stage fdr correction (non-negative)
    :return:
    quantiles: numpy.ndarray vector with quantile values in [0.0, 1.0].
    pval: corresponding p-values for each quantile, whether adjusted or not.
    reject_h0: boolean vector, whether the null-hypothesis is rejected for each percentile, i.e. there's a significant
    effect, pval < alpha_c (where alpha_c is the corrected alpha value due to multiple test adjustment).
    """

    def compute_test_statistics(x, y, quantiles):
        """
        Compute test statistic for each quantile.

        :param x: see compare_ecdfs().
        :param y: see compare_ecdfs().
        :param quantiles: vector of quantile values, e.g. [0.00, 0.01, 0.02, ..., 1.00].
        :return: ts (numpy.ndarray vector) with test statistic values, one per quantile.
        """

        # compute ECDF functions for each bootstrap sampling
        x_ecdf_func = ECDF(x)
        y_ecdf_func = ECDF(y)

        # inverse of the ECDF functions
        xu = np.unique(x)
        yu = np.unique(y)
        x_ecdf_func_inv = monotone_fn_inverter(x_ecdf_func, xu, vectorized=True)
        y_ecdf_func_inv = monotone_fn_inverter(y_ecdf_func, yu, vectorized=True)

        # small quantile values are outside the interpolation range, so they cannot be used for computations
        idx = quantiles >= np.max([x_ecdf_func(xu[0]), y_ecdf_func(yu[0])])

        # data values that correspond to the quantiles
        x_data = x_ecdf_func_inv(quantiles[idx])
        y_data = y_ecdf_func_inv(quantiles[idx])

        # compute test statistic for each quantile
        ts = np.full(shape=(1, len(quantiles)), dtype=np.float32, fill_value=np.nan)
        ts[0, idx] = np.abs(x_data - y_data)  # two-sided test

        return ts

    if resampling_method == 'bootstrap':
        warnings.warn('resampling_method=\'bootstrap\' is an experimental approach. Select \'permutation\' instead')

    # number of samples
    nx = len(x)
    ny = len(y)

    # equispaced points in the quantile axis
    quantiles = np.linspace(0.0, 1.0, num_quantiles)

    # merge data samples
    xy = np.concatenate((x, y))

    # compute test statistic for each quantile
    t = compute_test_statistics(x, y, quantiles)

    # allocate vector for the output
    pval = np.zeros(shape=t.shape, dtype=np.float32)

    # init random generator
    rng = np.random.RandomState(rng_seed)

    if resampling_method == 'bootstrap':

        # bootstrap loop
        for i in range(num_perms):

            # sample with replacement from the merged data vectors
            xs = np.random.choice(xy, size=(nx,), replace=True)
            ys = np.random.choice(xy, size=(ny,), replace=True)

            # compute ECDF statistic and check whether the null-distribution statistics are greater than the sample's
            ts = compute_test_statistics(xs, ys, quantiles)
            pval += (ts > t).astype(np.float32)

    elif resampling_method == 'permutation':

        # permutation loop
        for i in range(num_perms):

            # split combined dataset into two groups according to the permutation
            rng.shuffle(xy)
            xs = xy[:nx]
            ys = xy[nx:]

            # compute ECDF statistic and check whether the null-distribution statistics are greater than the sample's
            ts = compute_test_statistics(xs, ys, quantiles)
            pval += (ts > t).astype(np.float32)

    else:

        raise ValueError('Invalid resampling_method value')

    # p-value = N_(ts >= t) / N_ts
    pval /= num_perms

    # p-values under error rate
    reject_h0 = pval < alpha

    # apply multiple test correction
    if multitest_method is not None:
        idx = np.logical_not(np.isnan(pval))
        reject_h0[idx], pval[idx], _, _ = multipletests(pval[idx], alpha=alpha, method=multitest_method,
                                                     is_sorted=False, returnsorted=False)

    return quantiles, pval.flatten(), reject_h0.flatten()

    Compute p-values for the difference between each percentile point of the empirical cumulative distribution
    functions (ECDFs) of two samples x, y.

    This function allows multiple test adjustment of p-values using statsmodels.stats.multitest.multipletests.

    This function is basically an implementation of permutation testing (see overview in
    http://rasbt.github.io/mlxtend/user_guide/evaluate/permutation_test/#overview), but generating one p-value
    for each percentile.

    For example, if the median(x)=10, median(y)=7, this function computes the p-value, or probability
    that the observed difference is due to mere chance. The median is the 50 percentile. This function
    computes p-values for the differences between the 0, 1, 2, ..., 100 percentiles.

    It also has an experimental implementation, where instead of permutations, it uses bootstrap samples with
    replacement.

    :param x: numpy.ndarray vector, data set 1.
    :param y: numpy.ndarray vector, data set 2.
    :param alpha: (def 0.05) float. Error rate. Confidence is 1-alpha.
    :param num_quantiles: (def 101) Number of points the quantile range is split into. For 101, the
    quantiles are the percentiles 0%, 1%, 2%, ..., 100%.
    :param num_perms: (def 1000) Number of bootstrap permutations the p-values are estimated from.
    :param resampling_method: (def 'bootstrap') String with the name used for resampling. Note that it's 'bootstrap'
    for historical reasons, but the orthodox approach is 'permutation'. Because we haven't validated the bootstrap
    approach, using it produces a warning message.
    :param rng_seed: (def 0) Seed for the random generator of permutations or bootstrap samples. This allows to get
    the same results in different runs.
    :param multitest_method: (def None) Adjustment of p-values due to multitesting. These are the same
    values as for statsmodels.stats.multitest.multipletests. Currently
    (https://www.statsmodels.org/dev/_modules/statsmodels/stats/multitest.html),
        - `bonferroni` : one-step correction
        - `sidak` : one-step correction
        - `holm-sidak` : step down method using Sidak adjustments
        - `holm` : step-down method using Bonferroni adjustments
        - `simes-hochberg` : step-up method  (independent)
        - `hommel` : closed method based on Simes tests (non-negative)
        - `fdr_bh` : Benjamini/Hochberg  (non-negative)
        - `fdr_by` : Benjamini/Yekutieli (negative)
        - `fdr_tsbh` : two stage fdr correction (non-negative)
        - `fdr_tsbky` : two stage fdr correction (non-negative)
    :return:
    quantiles: numpy.ndarray vector with quantile values in [0.0, 1.0].
    pval: corresponding p-values for each quantile, whether adjusted or not.
    reject_h0: boolean vector, whether the null-hypothesis is rejected for each percentile, i.e. there's a significant
    effect, pval < alpha_c (where alpha_c is the corrected alpha value due to multiple test adjustment).
    
Compute p-values for the difference between each percentile point of the empirical cumulative distribution
functions (ECDFs) of two samples x, y.

This function allows multiple test adjustment of p-values using statsmodels.stats.multitest.multipletests.



For example, if the median(x)=10, median(y)=7, this function computes the p-value, or probability
that the observed difference is due to mere chance. The median is the 50 percentile. This function
computes p-values for the differences between the 0, 1, 2, ..., 100 percentiles.

It also has an experimental implementation, where instead of permutations, it uses bootstrap samples with
replacement.
Compute p - values for the difference between each percentile point of the empirical cumulative distribution functions ( ECDFs ) of two samples x y . This function allows multiple test adjustment of p - values using statsmodels . stats . multitest . multipletests . For example if the median ( x ) = 10 median ( y ) = 7 this function computes the p - value or probability that the observed difference is due to mere chance . The median is the 50 percentile . This function computes p - values for the differences between the 0 1 2 ... 100 percentiles . It also has an experimental implementation where instead of permutations it uses bootstrap samples with replacement .
def compare_ecdfs(x, y, alpha=0.05, num_quantiles=101, num_perms=1000, rng_seed=0,
                  resampling_method='bootstrap', multitest_method=None):
    def compute_test_statistics(x, y, quantiles):
        x_ecdf_func = ECDF(x)
        y_ecdf_func = ECDF(y)
        xu = np.unique(x)
        yu = np.unique(y)
        x_ecdf_func_inv = monotone_fn_inverter(x_ecdf_func, xu, vectorized=True)
        y_ecdf_func_inv = monotone_fn_inverter(y_ecdf_func, yu, vectorized=True)
        idx = quantiles >= np.max([x_ecdf_func(xu[0]), y_ecdf_func(yu[0])])
        x_data = x_ecdf_func_inv(quantiles[idx])
        y_data = y_ecdf_func_inv(quantiles[idx])
        ts = np.full(shape=(1, len(quantiles)), dtype=np.float32, fill_value=np.nan)
        ts[0, idx] = np.abs(x_data - y_data)  
        return ts
    if resampling_method == 'bootstrap':
        warnings.warn('resampling_method=\'bootstrap\' is an experimental approach. Select \'permutation\' instead')
    nx = len(x)
    ny = len(y)
    quantiles = np.linspace(0.0, 1.0, num_quantiles)
    xy = np.concatenate((x, y))
    t = compute_test_statistics(x, y, quantiles)
    pval = np.zeros(shape=t.shape, dtype=np.float32)
    rng = np.random.RandomState(rng_seed)
    if resampling_method == 'bootstrap':
        for i in range(num_perms):
            xs = np.random.choice(xy, size=(nx,), replace=True)
            ys = np.random.choice(xy, size=(ny,), replace=True)
            ts = compute_test_statistics(xs, ys, quantiles)
            pval += (ts > t).astype(np.float32)
    elif resampling_method == 'permutation':
        for i in range(num_perms):
            rng.shuffle(xy)
            xs = xy[:nx]
            ys = xy[nx:]
            ts = compute_test_statistics(xs, ys, quantiles)
            pval += (ts > t).astype(np.float32)
    else:
        raise ValueError('Invalid resampling_method value')
    pval /= num_perms
    reject_h0 = pval < alpha
    if multitest_method is not None:
        idx = np.logical_not(np.isnan(pval))
        reject_h0[idx], pval[idx], _, _ = multipletests(pval[idx], alpha=alpha, method=multitest_method,
                                                     is_sorted=False, returnsorted=False)
    return quantiles, pval.flatten(), reject_h0.flatten()
def compare_ecdfs ( x , y , alpha = 0.05 , num_quantiles = 101 , num_perms = 1000 , rng_seed = 0 , resampling_method = 'bootstrap' , multitest_method = None ) : def compute_test_statistics ( x , y , quantiles ) : """
        Compute test statistic for each quantile.

        :param x: see compare_ecdfs().
        :param y: see compare_ecdfs().
        :param quantiles: vector of quantile values, e.g. [0.00, 0.01, 0.02, ..., 1.00].
        :return: ts (numpy.ndarray vector) with test statistic values, one per quantile.
        """ x_ecdf_func = ECDF ( x ) y_ecdf_func = ECDF ( y ) xu = np . unique ( x ) yu = np . unique ( y ) x_ecdf_func_inv = monotone_fn_inverter ( x_ecdf_func , xu , vectorized = True ) y_ecdf_func_inv = monotone_fn_inverter ( y_ecdf_func , yu , vectorized = True ) idx = quantiles >= np . max ( [ x_ecdf_func ( xu [ 0 ] ) , y_ecdf_func ( yu [ 0 ] ) ] ) x_data = x_ecdf_func_inv ( quantiles [ idx ] ) y_data = y_ecdf_func_inv ( quantiles [ idx ] ) ts = np . full ( shape = ( 1 , len ( quantiles ) ) , dtype = np . float32 , fill_value = np . nan ) ts [ 0 , idx ] = np . abs ( x_data - y_data ) return ts if resampling_method == 'bootstrap' : warnings . warn ( 'resampling_method=\'bootstrap\' is an experimental approach. Select \'permutation\' instead' ) nx = len ( x ) ny = len ( y ) quantiles = np . linspace ( 0.0 , 1.0 , num_quantiles ) xy = np . concatenate ( ( x , y ) ) t = compute_test_statistics ( x , y , quantiles ) pval = np . zeros ( shape = t . shape , dtype = np . float32 ) rng = np . random . RandomState ( rng_seed ) if resampling_method == 'bootstrap' : for i in range ( num_perms ) : xs = np . random . choice ( xy , size = ( nx , ) , replace = True ) ys = np . random . choice ( xy , size = ( ny , ) , replace = True ) ts = compute_test_statistics ( xs , ys , quantiles ) pval += ( ts > t ) . astype ( np . float32 ) elif resampling_method == 'permutation' : for i in range ( num_perms ) : rng . shuffle ( xy ) xs = xy [ : nx ] ys = xy [ nx : ] ts = compute_test_statistics ( xs , ys , quantiles ) pval += ( ts > t ) . astype ( np . float32 ) else : raise ValueError ( 'Invalid resampling_method value' ) pval /= num_perms reject_h0 = pval < alpha if multitest_method is not None : idx = np . logical_not ( np . isnan ( pval ) ) reject_h0 [ idx ] , pval [ idx ] , _ , _ = multipletests ( pval [ idx ] , alpha = alpha , method = multitest_method , is_sorted = False , returnsorted = False ) return quantiles , pval . flatten ( ) , reject_h0 . flatten ( )
Compute p-values for the difference between each percentile point of the empirical cumulative distribution
functions (ECDFs) of two samples x, y.
Compute p - values for the difference between each percentile point of the empirical cumulative distribution functions ( ECDFs ) of two samples x y .
"""
    Compute p-values for the difference between each percentile point of the empirical cumulative distribution
    functions (ECDFs) of two samples x, y.

    This function allows multiple test adjustment of p-values using statsmodels.stats.multitest.multipletests.

    This function is basically an implementation of permutation testing (see overview in
    http://rasbt.github.io/mlxtend/user_guide/evaluate/permutation_test/#overview), but generating one p-value
    for each percentile.

    For example, if the median(x)=10, median(y)=7, this function computes the p-value, or probability
    that the observed difference is due to mere chance. The median is the 50 percentile. This function
    computes p-values for the differences between the 0, 1, 2, ..., 100 percentiles.

    It also has an experimental implementation, where instead of permutations, it uses bootstrap samples with
    replacement.

    :param x: numpy.ndarray vector, data set 1.
    :param y: numpy.ndarray vector, data set 2.
    :param alpha: (def 0.05) float. Error rate. Confidence is 1-alpha.
    :param num_quantiles: (def 101) Number of points the quantile range is split into. For 101, the
    quantiles are the percentiles 0%, 1%, 2%, ..., 100%.
    :param num_perms: (def 1000) Number of bootstrap permutations the p-values are estimated from.
    :param resampling_method: (def 'bootstrap') String with the name used for resampling. Note that it's 'bootstrap'
    for historical reasons, but the orthodox approach is 'permutation'. Because we haven't validated the bootstrap
    approach, using it produces a warning message.
    :param rng_seed: (def 0) Seed for the random generator of permutations or bootstrap samples. This allows to get
    the same results in different runs.
    :param multitest_method: (def None) Adjustment of p-values due to multitesting. These are the same
    values as for statsmodels.stats.multitest.multipletests. Currently
    (https://www.statsmodels.org/dev/_modules/statsmodels/stats/multitest.html),
        - `bonferroni` : one-step correction
        - `sidak` : one-step correction
        - `holm-sidak` : step down method using Sidak adjustments
        - `holm` : step-down method using Bonferroni adjustments
        - `simes-hochberg` : step-up method  (independent)
        - `hommel` : closed method based on Simes tests (non-negative)
        - `fdr_bh` : Benjamini/Hochberg  (non-negative)
        - `fdr_by` : Benjamini/Yekutieli (negative)
        - `fdr_tsbh` : two stage fdr correction (non-negative)
        - `fdr_tsbky` : two stage fdr correction (non-negative)
    :return:
    quantiles: numpy.ndarray vector with quantile values in [0.0, 1.0].
    pval: corresponding p-values for each quantile, whether adjusted or not.
    reject_h0: boolean vector, whether the null-hypothesis is rejected for each percentile, i.e. there's a significant
    effect, pval < alpha_c (where alpha_c is the corrected alpha value due to multiple test adjustment).
    """ """
        Compute test statistic for each quantile.

        :param x: see compare_ecdfs().
        :param y: see compare_ecdfs().
        :param quantiles: vector of quantile values, e.g. [0.00, 0.01, 0.02, ..., 1.00].
        :return: ts (numpy.ndarray vector) with test statistic values, one per quantile.
        """ # compute ECDF functions for each bootstrap sampling # inverse of the ECDF functions # small quantile values are outside the interpolation range, so they cannot be used for computations # data values that correspond to the quantiles # compute test statistic for each quantile # two-sided test # number of samples # equispaced points in the quantile axis # merge data samples # compute test statistic for each quantile # allocate vector for the output # init random generator # bootstrap loop # sample with replacement from the merged data vectors # compute ECDF statistic and check whether the null-distribution statistics are greater than the sample's # permutation loop # split combined dataset into two groups according to the permutation # compute ECDF statistic and check whether the null-distribution statistics are greater than the sample's # p-value = N_(ts >= t) / N_ts # p-values under error rate # apply multiple test correction
numpy.ndarray vector with quantile values in [0.0, 1.0].
pval: corresponding p-values for each quantile, whether adjusted or not.
reject_h0: boolean vector, whether the null-hypothesis is rejected for each percentile, i.e. there's a significant
effect, pval < alpha_c (where alpha_c is the corrected alpha value due to multiple test adjustment).
numpy . ndarray vector with quantile values in [ 0 . 0 1 . 0 ] . pval : corresponding p - values for each quantile whether adjusted or not . reject_h0 : boolean vector whether the null - hypothesis is rejected for each percentile i . e . there ' s a significant effect pval < alpha_c ( where alpha_c is the corrected alpha value due to multiple test adjustment ) .
x
numpy.ndarray vector, data set 1.
numpy . ndarray vector data set 1 .
y
numpy.ndarray vector, data set 2.
numpy . ndarray vector data set 2 .
alpha
None
num_quantiles
(def 101) Number of points the quantile range is split into.
( def 101 ) Number of points the quantile range is split into .
num_perms
(def 1000) Number of bootstrap permutations the p-values are estimated from.
( def 1000 ) Number of bootstrap permutations the p - values are estimated from .
rng_seed
(def 0) Seed for the random generator of permutations or bootstrap samples. This allows to get
the same results in different runs.
( def 0 ) Seed for the random generator of permutations or bootstrap samples . This allows to get the same results in different runs .
resampling_method
(def 'bootstrap') String with the name used for resampling. Note that it's 'bootstrap'
for historical reasons, but the orthodox approach is 'permutation'. Because we haven't validated the bootstrap
approach, using it produces a warning message.
( def ' bootstrap ' ) String with the name used for resampling . Note that it ' s ' bootstrap ' for historical reasons but the orthodox approach is ' permutation ' . Because we haven ' t validated the bootstrap approach using it produces a warning message .
multitest_method
(def None) Adjustment of p-values due to multitesting. These are the same
values as for statsmodels.stats.multitest.multipletests. Currently
,
`bonferroni` : one-step correction
`sidak` : one-step correction
`holm-sidak` : step down method using Sidak adjustments
`holm` : step-down method using Bonferroni adjustments
`simes-hochberg` : step-up method  (independent)
`hommel` : closed method based on Simes tests (non-negative)
`fdr_bh` : Benjamini/Hochberg  (non-negative)
`fdr_by` : Benjamini/Yekutieli (negative)
`fdr_tsbh` : two stage fdr correction (non-negative)
`fdr_tsbky` : two stage fdr correction (non-negative)
( def None ) Adjustment of p - values due to multitesting . These are the same values as for statsmodels . stats . multitest . multipletests . Currently ` bonferroni ` : one - step correction ` sidak ` : one - step correction ` holm - sidak ` : step down method using Sidak adjustments ` holm ` : step - down method using Bonferroni adjustments ` simes - hochberg ` : step - up method ( independent ) ` hommel ` : closed method based on Simes tests ( non - negative ) ` fdr_bh ` : Benjamini / Hochberg ( non - negative ) ` fdr_by ` : Benjamini / Yekutieli ( negative ) ` fdr_tsbh ` : two stage fdr correction ( non - negative ) ` fdr_tsbky ` : two stage fdr correction ( non - negative )