<h3 dir="auto">Describe your issue.</h3>
<p dir="auto">I'm running a pre-post hypothesis test on a small dataset using the Wilcoxon signed rank test (scipy.stats.wilcoxon), and noticed when I validated the results the p-value did not match. It looks like it is not using the normal approximation even though ties exist.</p>
<p dir="auto">Per the documentation the expected approach is:</p>
<p dir="auto">"To derive the p-value, the exact distribution (mode == 'exact') can be used for sample sizes of up to 25. The default mode == 'auto' uses the exact distribution if there are <strong>at most 25 observations and no ties</strong>, otherwise a normal approximation is used (mode == 'approx')."</p>
<p dir="auto">The example demonstrates this by running the out of the box wilcoxon then testing for ties and running the "mode = 'approx'" if they exist.</p>
<p dir="auto">You will find the first result when running the regular test is <code class="notranslate">pvalue=0.01220703125</code> and the second with logic built in to test for ties is <code class="notranslate">pvalue=0.014944844395687155</code></p>
<p dir="auto">In this dataset they exist in both the pre and post period.</p>
<h3 dir="auto">Reproducing Code Example</h3>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="dfgh = pd.DataFrame({'ID':[1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,13,13],
                    'Period':['0 Month','3 Month','0 Month','3 Month','0 Month','3 Month',
                            '0 Month','3 Month','0 Month','3 Month','0 Month','3 Month',
                            '0 Month','3 Month','0 Month','3 Month','0 Month','3 Month',
                            '0 Month','3 Month','0 Month','3 Month','0 Month','3 Month',],
                    'Scale':['Something','Something','Something','Something','Something',
                            'Something','Something','Something','Something','Something',
                            'Something','Something','Something','Something','Something',
                            'Something','Something','Something','Something','Something',
                            'Something','Something','Something','Something',],
                    'Value':[18,26,23,4,24,3,27,26,9,0,40,3,17,10,33,9,6,7,8,1,9,4,26,9,]
                            })

dfhb = dfgh.query('Period == &quot;0 Month&quot;')
dfh3 = dfgh.query('Period == &quot;3 Month&quot;')

print('run standard wilcoxon:')
print(wilcoxon(dfhb.Value, dfh3.Value,alternative='two-sided', correction=True,))
print('\ntest for ties and selct normal approximation if the exist')
if (dfhb['Value'].nunique() &lt; len(dfhb['Value'])) or (dfh3['Value'].nunique() &lt; len(dfh3['Value'])):
    print('ties identified - use approx:')
    pval = wilcoxon(dfhb.Value, dfh3.Value, alternative='two-sided', correction=True,mode ='approx',)
else:
    print('no ties')
    plav = wilcoxon(dfhb.Value, dfh3.Value,alternative='two-sided', correction=True,mode ='exact',)
print(pval)
print('SciPy Version: ' + scipy.__version__)"><pre class="notranslate"><span class="pl-s1">dfgh</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'ID'</span>:[<span class="pl-c1">1</span>,<span class="pl-c1">1</span>,<span class="pl-c1">2</span>,<span class="pl-c1">2</span>,<span class="pl-c1">3</span>,<span class="pl-c1">3</span>,<span class="pl-c1">4</span>,<span class="pl-c1">4</span>,<span class="pl-c1">5</span>,<span class="pl-c1">5</span>,<span class="pl-c1">6</span>,<span class="pl-c1">6</span>,<span class="pl-c1">7</span>,<span class="pl-c1">7</span>,<span class="pl-c1">8</span>,<span class="pl-c1">8</span>,<span class="pl-c1">9</span>,<span class="pl-c1">9</span>,<span class="pl-c1">10</span>,<span class="pl-c1">10</span>,<span class="pl-c1">11</span>,<span class="pl-c1">11</span>,<span class="pl-c1">13</span>,<span class="pl-c1">13</span>],
                    <span class="pl-s">'Period'</span>:[<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,
                            <span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,
                            <span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,
                            <span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,<span class="pl-s">'0 Month'</span>,<span class="pl-s">'3 Month'</span>,],
                    <span class="pl-s">'Scale'</span>:[<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,
                            <span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,
                            <span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,
                            <span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,
                            <span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,<span class="pl-s">'Something'</span>,],
                    <span class="pl-s">'Value'</span>:[<span class="pl-c1">18</span>,<span class="pl-c1">26</span>,<span class="pl-c1">23</span>,<span class="pl-c1">4</span>,<span class="pl-c1">24</span>,<span class="pl-c1">3</span>,<span class="pl-c1">27</span>,<span class="pl-c1">26</span>,<span class="pl-c1">9</span>,<span class="pl-c1">0</span>,<span class="pl-c1">40</span>,<span class="pl-c1">3</span>,<span class="pl-c1">17</span>,<span class="pl-c1">10</span>,<span class="pl-c1">33</span>,<span class="pl-c1">9</span>,<span class="pl-c1">6</span>,<span class="pl-c1">7</span>,<span class="pl-c1">8</span>,<span class="pl-c1">1</span>,<span class="pl-c1">9</span>,<span class="pl-c1">4</span>,<span class="pl-c1">26</span>,<span class="pl-c1">9</span>,]
                            })

<span class="pl-s1">dfhb</span> <span class="pl-c1">=</span> <span class="pl-s1">dfgh</span>.<span class="pl-en">query</span>(<span class="pl-s">'Period == "0 Month"'</span>)
<span class="pl-s1">dfh3</span> <span class="pl-c1">=</span> <span class="pl-s1">dfgh</span>.<span class="pl-en">query</span>(<span class="pl-s">'Period == "3 Month"'</span>)

<span class="pl-en">print</span>(<span class="pl-s">'run standard wilcoxon:'</span>)
<span class="pl-en">print</span>(<span class="pl-en">wilcoxon</span>(<span class="pl-s1">dfhb</span>.<span class="pl-v">Value</span>, <span class="pl-s1">dfh3</span>.<span class="pl-v">Value</span>,<span class="pl-s1">alternative</span><span class="pl-c1">=</span><span class="pl-s">'two-sided'</span>, <span class="pl-s1">correction</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,))
<span class="pl-en">print</span>(<span class="pl-s">'<span class="pl-cce">\n</span>test for ties and selct normal approximation if the exist'</span>)
<span class="pl-k">if</span> (<span class="pl-s1">dfhb</span>[<span class="pl-s">'Value'</span>].<span class="pl-en">nunique</span>() <span class="pl-c1">&lt;</span> <span class="pl-en">len</span>(<span class="pl-s1">dfhb</span>[<span class="pl-s">'Value'</span>])) <span class="pl-c1">or</span> (<span class="pl-s1">dfh3</span>[<span class="pl-s">'Value'</span>].<span class="pl-en">nunique</span>() <span class="pl-c1">&lt;</span> <span class="pl-en">len</span>(<span class="pl-s1">dfh3</span>[<span class="pl-s">'Value'</span>])):
    <span class="pl-en">print</span>(<span class="pl-s">'ties identified - use approx:'</span>)
    <span class="pl-s1">pval</span> <span class="pl-c1">=</span> <span class="pl-en">wilcoxon</span>(<span class="pl-s1">dfhb</span>.<span class="pl-v">Value</span>, <span class="pl-s1">dfh3</span>.<span class="pl-v">Value</span>, <span class="pl-s1">alternative</span><span class="pl-c1">=</span><span class="pl-s">'two-sided'</span>, <span class="pl-s1">correction</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,<span class="pl-s1">mode</span> <span class="pl-c1">=</span><span class="pl-s">'approx'</span>,)
<span class="pl-k">else</span>:
    <span class="pl-en">print</span>(<span class="pl-s">'no ties'</span>)
    <span class="pl-s1">plav</span> <span class="pl-c1">=</span> <span class="pl-en">wilcoxon</span>(<span class="pl-s1">dfhb</span>.<span class="pl-v">Value</span>, <span class="pl-s1">dfh3</span>.<span class="pl-v">Value</span>,<span class="pl-s1">alternative</span><span class="pl-c1">=</span><span class="pl-s">'two-sided'</span>, <span class="pl-s1">correction</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,<span class="pl-s1">mode</span> <span class="pl-c1">=</span><span class="pl-s">'exact'</span>,)
<span class="pl-en">print</span>(<span class="pl-s1">pval</span>)
<span class="pl-en">print</span>(<span class="pl-s">'SciPy Version: '</span> <span class="pl-c1">+</span> <span class="pl-s1">scipy</span>.<span class="pl-s1">__version__</span>)</pre></div>
<h3 dir="auto">Error message</h3>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="There is no associated error."><pre class="notranslate">There is no associated error.</pre></div>
<h3 dir="auto">SciPy/NumPy/Python version information</h3>
<p dir="auto">1.6.2</p>
<p dir="auto">My issue is about <code class="notranslate">stats.wilcoxon()</code>. <code class="notranslate">stats.wilcoxon</code> detects when d== 0, and appropriately warns that 'exact' will not work under such a circumstance. However, it reports that this is because of 'ties', and not because of the presence of 0s.</p>
<p dir="auto">Exact calculations will also not be accurate when there are ties in rank, but the function does not appropriately detect such scenarios.</p>
<p dir="auto">R produces appropriate warnings in both circumstances</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="wilcox.test(0:4)

	Wilcoxon signed rank test with continuity correction

data:  0:4
V = 10, p-value = 0.1003
alternative hypothesis: true location is not equal to 0

Warning message:
In wilcox.test.default(0:4) : cannot compute exact p-value with zeroes
&gt; wilcox.test(c(1,1:4))

	Wilcoxon signed rank test with continuity correction

data:  c(1, 1:4)
V = 15, p-value = 0.05791
alternative hypothesis: true location is not equal to 0

Warning message:
In wilcox.test.default(c(1, 1:4)) : cannot compute exact p-value with ties"><pre class="notranslate"><code class="notranslate">wilcox.test(0:4)

	Wilcoxon signed rank test with continuity correction

data:  0:4
V = 10, p-value = 0.1003
alternative hypothesis: true location is not equal to 0

Warning message:
In wilcox.test.default(0:4) : cannot compute exact p-value with zeroes
&gt; wilcox.test(c(1,1:4))

	Wilcoxon signed rank test with continuity correction

data:  c(1, 1:4)
V = 15, p-value = 0.05791
alternative hypothesis: true location is not equal to 0

Warning message:
In wilcox.test.default(c(1, 1:4)) : cannot compute exact p-value with ties
</code></pre></div>
<h4 dir="auto">Reproducing code example:</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from scipy import stats as stats
&gt;&gt;&gt; x = [1, 1, 2, 3, 4]
&gt;&gt;&gt; stats.wilcoxon(x, correction=True, mode='exact') ## should produce a warning similar to R and should fall back to approx
WilcoxonResult(statistic=0.0, pvalue=0.0625)
&gt;&gt;&gt; stats.wilcoxon(x, correction=True, mode='approx')
/home/jrr/miniconda3/lib/python3.8/site-packages/scipy/stats/morestats.py:2981: UserWarning: Sample size too small for normal approximation.
  warnings.warn(&quot;Sample size too small for normal approximation.&quot;)
WilcoxonResult(statistic=0.0, pvalue=0.05790726541729728)
&gt;&gt;&gt; x = [0, 1, 2, 3, 4]
&gt;&gt;&gt; stats.wilcoxon(x, correction=True) ## warns, but calls ties.
/home/jrr/miniconda3/lib/python3.8/site-packages/scipy/stats/morestats.py:2967: UserWarning: Exact p-value calculation does not work if there are ties. Switching to normal approximation.
  warnings.warn(&quot;Exact p-value calculation does not work if there are &quot;
WilcoxonResult(statistic=0.0, pvalue=0.10034824646229075)"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; from scipy import stats as stats
&gt;&gt;&gt; x = [1, 1, 2, 3, 4]
&gt;&gt;&gt; stats.wilcoxon(x, correction=True, mode='exact') ## should produce a warning similar to R and should fall back to approx
WilcoxonResult(statistic=0.0, pvalue=0.0625)
&gt;&gt;&gt; stats.wilcoxon(x, correction=True, mode='approx')
/home/jrr/miniconda3/lib/python3.8/site-packages/scipy/stats/morestats.py:2981: UserWarning: Sample size too small for normal approximation.
  warnings.warn("Sample size too small for normal approximation.")
WilcoxonResult(statistic=0.0, pvalue=0.05790726541729728)
&gt;&gt;&gt; x = [0, 1, 2, 3, 4]
&gt;&gt;&gt; stats.wilcoxon(x, correction=True) ## warns, but calls ties.
/home/jrr/miniconda3/lib/python3.8/site-packages/scipy/stats/morestats.py:2967: UserWarning: Exact p-value calculation does not work if there are ties. Switching to normal approximation.
  warnings.warn("Exact p-value calculation does not work if there are "
WilcoxonResult(statistic=0.0, pvalue=0.10034824646229075)
</code></pre></div>
<h4 dir="auto">Scipy/Numpy/Python version information:</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)

1.6.1 1.19.2 sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)"><pre class="notranslate"><code class="notranslate">import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info)

1.6.1 1.19.2 sys.version_info(major=3, minor=8, micro=8, releaselevel='final', serial=0)
</code></pre></div>