<h4 dir="auto">Description</h4>
<p dir="auto">The average_precision_score() function in sklearn doesn't return a correct AUC value.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<p dir="auto">Example:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
&quot;&quot;&quot;
    Desc: average_precision_score returns overestimated AUC of precision-recall curve
&quot;&quot;&quot;
# pathological example
p = [0.833, 0.800] # precision
r = [0.294, 0.235] # recall

# computation of average_precision_score()
print(&quot;AUC       = {:3f}&quot;.format(-np.sum(np.diff(r) * np.array(p)[:-1]))) # _binary_uninterpolated_average_precision()

# computation of auc() with trapezoid interpolation
print(&quot;AUC TRAP. = {:3f}&quot;.format(-np.trapz(p, r)))

# possible fix in _binary_uninterpolated_average_precision() **(edited)**
print(&quot;AUC FIX   = {:3f}&quot;.format(-np.sum(np.diff(r) * np.minimum(p[:-1], p[1:])))

#&gt;&gt; AUC       = 0.049147
#&gt;&gt; AUC TRAP. = 0.048174
#&gt;&gt; AUC FIX   = 0.047200"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-s">"""</span>
<span class="pl-s">    Desc: average_precision_score returns overestimated AUC of precision-recall curve</span>
<span class="pl-s">"""</span>
<span class="pl-c"># pathological example</span>
<span class="pl-s1">p</span> <span class="pl-c1">=</span> [<span class="pl-c1">0.833</span>, <span class="pl-c1">0.800</span>] <span class="pl-c"># precision</span>
<span class="pl-s1">r</span> <span class="pl-c1">=</span> [<span class="pl-c1">0.294</span>, <span class="pl-c1">0.235</span>] <span class="pl-c"># recall</span>

<span class="pl-c"># computation of average_precision_score()</span>
<span class="pl-en">print</span>(<span class="pl-s">"AUC       = {:3f}"</span>.<span class="pl-en">format</span>(<span class="pl-c1">-</span><span class="pl-s1">np</span>.<span class="pl-en">sum</span>(<span class="pl-s1">np</span>.<span class="pl-en">diff</span>(<span class="pl-s1">r</span>) <span class="pl-c1">*</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>(<span class="pl-s1">p</span>)[:<span class="pl-c1">-</span><span class="pl-c1">1</span>]))) <span class="pl-c"># _binary_uninterpolated_average_precision()</span>

<span class="pl-c"># computation of auc() with trapezoid interpolation</span>
<span class="pl-en">print</span>(<span class="pl-s">"AUC TRAP. = {:3f}"</span>.<span class="pl-en">format</span>(<span class="pl-c1">-</span><span class="pl-s1">np</span>.<span class="pl-en">trapz</span>(<span class="pl-s1">p</span>, <span class="pl-s1">r</span>)))

<span class="pl-c"># possible fix in _binary_uninterpolated_average_precision() **(edited)**</span>
<span class="pl-s1">print</span>(<span class="pl-s">"AUC FIX   = {:3f}"</span>.<span class="pl-en">format</span>(<span class="pl-c1">-</span><span class="pl-s1">np</span>.<span class="pl-en">sum</span>(<span class="pl-s1">np</span>.<span class="pl-en">diff</span>(<span class="pl-s1">r</span>) <span class="pl-c1">*</span> <span class="pl-s1">np</span>.<span class="pl-en">minimum</span>(<span class="pl-s1">p</span>[:<span class="pl-c1">-</span><span class="pl-c1">1</span>], <span class="pl-s1">p</span>[<span class="pl-c1">1</span>:])))

<span class="pl-c">#&gt;&gt; AUC       = 0.049147</span>
<span class="pl-c">#&gt;&gt; AUC TRAP. = 0.048174</span>
<span class="pl-c">#&gt;&gt; AUC FIX   = 0.047200</span></pre></div>
<h4 dir="auto">Expected Results</h4>
<p dir="auto">AUC without interpolation = (0.294 - 0.235) * 0.800 = 0.472<br>
AUC with trapezoidal interpolation = 0.472 + (0.294 - 0.235) * (0.833 - 0.800) / 2 = 0.0482</p>
<h4 dir="auto">Actual Results</h4>
<p dir="auto">This is what sklearn implements for AUC without interpolation (<a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html" rel="nofollow">https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html</a>):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sum((r[i] - r[i+1]) * p[i] for i in range(len(p)-1))
&gt;&gt; 0.049147"><pre class="notranslate"><span class="pl-en">sum</span>((<span class="pl-s1">r</span>[<span class="pl-s1">i</span>] <span class="pl-c1">-</span> <span class="pl-s1">r</span>[<span class="pl-s1">i</span><span class="pl-c1">+</span><span class="pl-c1">1</span>]) <span class="pl-c1">*</span> <span class="pl-s1">p</span>[<span class="pl-s1">i</span>] <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-en">len</span>(<span class="pl-s1">p</span>)<span class="pl-c1">-</span><span class="pl-c1">1</span>))
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.049147</span></pre></div>
<p dir="auto">This is what I think is correct (<strong>no longer; see edit</strong>):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sum((r[i] - r[i+1]) * p[i+1] for i in range(len(p)-1))
&gt;&gt; 0.047200"><pre class="notranslate"><span class="pl-en">sum</span>((<span class="pl-s1">r</span>[<span class="pl-s1">i</span>] <span class="pl-c1">-</span> <span class="pl-s1">r</span>[<span class="pl-s1">i</span><span class="pl-c1">+</span><span class="pl-c1">1</span>]) <span class="pl-c1">*</span> <span class="pl-s1">p</span>[<span class="pl-s1">i</span><span class="pl-c1">+</span><span class="pl-c1">1</span>] <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-en">len</span>(<span class="pl-s1">p</span>)<span class="pl-c1">-</span><span class="pl-c1">1</span>))
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.047200</span></pre></div>
<p dir="auto"><strong>EDIT:</strong> I found that the above 'correct' implementation doesn't always underestimate. It depends on the input. Therefore I have revised the uninterpolated AUC calculation to this:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="sum((r[i] - r[i+1]) * min(p[i] + p[i+1]) for i in range(len(p)-1)) 
&gt;&gt; 0.047200"><pre class="notranslate"><span class="pl-en">sum</span>((<span class="pl-s1">r</span>[<span class="pl-s1">i</span>] <span class="pl-c1">-</span> <span class="pl-s1">r</span>[<span class="pl-s1">i</span><span class="pl-c1">+</span><span class="pl-c1">1</span>]) <span class="pl-c1">*</span> <span class="pl-en">min</span>(<span class="pl-s1">p</span>[<span class="pl-s1">i</span>] <span class="pl-c1">+</span> <span class="pl-s1">p</span>[<span class="pl-s1">i</span><span class="pl-c1">+</span><span class="pl-c1">1</span>]) <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-en">len</span>(<span class="pl-s1">p</span>)<span class="pl-c1">-</span><span class="pl-c1">1</span>)) 
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.047200</span></pre></div>
<p dir="auto">This has the advantage that the AUC calculation is more consistent; it is either equal or underestimated, but never overestimated (compared to the current uninterpolated AUC function). Below I show some examples on what it does:</p>
<ul dir="auto">
<li>Example 1: all work fine</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="p = [0.3, 1.0]
r = [1.0, 0.0]

#Results:
&gt;&gt; 0.30    # sklearn's _binary_uninterpolated_average_precision()
&gt;&gt; 0.30    # my consistent _binary_uninterpolated_average_precision()
&gt;&gt; 0.65    # np.trapz() (trapezoidal interpolation)"><pre class="notranslate"><span class="pl-s1">p</span> <span class="pl-c1">=</span> [<span class="pl-c1">0.3</span>, <span class="pl-c1">1.0</span>]
<span class="pl-s1">r</span> <span class="pl-c1">=</span> [<span class="pl-c1">1.0</span>, <span class="pl-c1">0.0</span>]

<span class="pl-c">#Results:</span><span class="pl-s1"></span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.30</span>    <span class="pl-c"># sklearn's _binary_uninterpolated_average_precision()</span><span class="pl-s1"></span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.30</span>    <span class="pl-c"># my consistent _binary_uninterpolated_average_precision()</span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.65</span>    <span class="pl-c"># np.trapz() (trapezoidal interpolation)</span></pre></div>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/36004944/52123847-9a8ca780-2627-11e9-8abb-a313102e74ab.png"><img src="https://user-images.githubusercontent.com/36004944/52123847-9a8ca780-2627-11e9-8abb-a313102e74ab.png" alt="pr_curve1" style="max-width: 100%;"></a></p>
<ul dir="auto">
<li>Example 2: sklearn's _binary_uninterpolated_average_precision returns inaccurate number</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="p = [1.0, 0.3]
r = [1.0, 0.0]

#Results:
&gt;&gt; 1.00    # sklearn's _binary_uninterpolated_average_precision()
&gt;&gt; 0.30    # my consistent _binary_uninterpolated_average_precision()
&gt;&gt; 0.65    # np.trapz() (trapezoidal interpolation)"><pre class="notranslate"><span class="pl-s1">p</span> <span class="pl-c1">=</span> [<span class="pl-c1">1.0</span>, <span class="pl-c1">0.3</span>]
<span class="pl-s1">r</span> <span class="pl-c1">=</span> [<span class="pl-c1">1.0</span>, <span class="pl-c1">0.0</span>]

<span class="pl-c">#Results:</span><span class="pl-s1"></span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">1.00</span>    <span class="pl-c"># sklearn's _binary_uninterpolated_average_precision()</span><span class="pl-s1"></span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.30</span>    <span class="pl-c"># my consistent _binary_uninterpolated_average_precision()</span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.65</span>    <span class="pl-c"># np.trapz() (trapezoidal interpolation)</span></pre></div>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/36004944/52123845-99f41100-2627-11e9-8b55-f0957980ecd5.png"><img src="https://user-images.githubusercontent.com/36004944/52123845-99f41100-2627-11e9-8b55-f0957980ecd5.png" alt="pr_curve2" style="max-width: 100%;"></a></p>
<ul dir="auto">
<li>Example 3: extra example</li>
</ul>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="p = [0.4, 0.1, 1.0]
r = [1.0, 0.9, 0.0]

#Results:
&gt;&gt; 0.13      # sklearn's _binary_uninterpolated_average_precision()
&gt;&gt; 0.10      # my consistent _binary_uninterpolated_average_precision()
&gt;&gt; 0.52      # np.trapz() (trapezoidal interpolation)"><pre class="notranslate"><span class="pl-s1">p</span> <span class="pl-c1">=</span> [<span class="pl-c1">0.4</span>, <span class="pl-c1">0.1</span>, <span class="pl-c1">1.0</span>]
<span class="pl-s1">r</span> <span class="pl-c1">=</span> [<span class="pl-c1">1.0</span>, <span class="pl-c1">0.9</span>, <span class="pl-c1">0.0</span>]

<span class="pl-c">#Results:</span><span class="pl-s1"></span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.13</span>      <span class="pl-c"># sklearn's _binary_uninterpolated_average_precision()</span><span class="pl-s1"></span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.10</span>      <span class="pl-c"># my consistent _binary_uninterpolated_average_precision()</span>
<span class="pl-c1">&gt;&gt;</span> <span class="pl-c1">0.52</span>      <span class="pl-c"># np.trapz() (trapezoidal interpolation)</span></pre></div>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/36004944/52123846-9a8ca780-2627-11e9-8460-2ccb398dfa12.png"><img src="https://user-images.githubusercontent.com/36004944/52123846-9a8ca780-2627-11e9-8460-2ccb398dfa12.png" alt="pr_curve3" style="max-width: 100%;"></a></p>
<h4 dir="auto">Versions</h4>
<p dir="auto">Windows-10-10.0.17134-SP0<br>
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 10:22:32) [MSC v.1900 64 bit (AMD64)]<br>
NumPy 1.14.0<br>
SciPy 1.0.0<br>
Scikit-Learn 0.19.1</p>
<p dir="auto">Hi,</p>
<p dir="auto">Scikit Learn seems to implement Precision Recall curves (and Average Precision values/AUC under PR curve) in a non-standard way, without documenting the discrepancy. The standard way of computing Precision Recall numbers is by interpolating the curve, as described here:<br>
<a href="http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html" rel="nofollow">http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html</a><br>
the motivation is to</p>
<ol dir="auto">
<li>Smooth out the kinks and reduce noise contribution to the score</li>
<li>In any practical application, if your PR curve ever went up, then you would strictly prefer to set your threshold there rather than at the original place (achieving both more precision and recall). Hence, people prefer to interpolate the curve, which better integrates out the threshold parameter and gives a more sensible estimate of the real performance.</li>
</ol>
<p dir="auto">This is also what standard code for Pascal VOC does and explain this in their writeup:<br>
<a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.157.5766&amp;rep=rep1&amp;type=pdf" rel="nofollow">http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.157.5766&amp;rep=rep1&amp;type=pdf</a></p>
<p dir="auto">VL_FEAT also has options for interpolation:<br>
<a href="http://www.vlfeat.org/matlab/vl_pr.html" rel="nofollow">http://www.vlfeat.org/matlab/vl_pr.html</a><br>
and as shown in their code here: <a href="https://github.com/vlfeat/vlfeat/blob/edc378a722ea0d79e29f4648a54bb62f32b22568/toolbox/plotop/vl_pr.m">https://github.com/vlfeat/vlfeat/blob/edc378a722ea0d79e29f4648a54bb62f32b22568/toolbox/plotop/vl_pr.m</a></p>
<p dir="auto">The concern is that people using the scikit version will see incorrectly reported LOWER performance, than what they might see reported in other papers.</p>