<h4 dir="auto">Description</h4>
<p dir="auto">The ROC functionality reduces its workload by first extracting forward differences between scores and then discarding potential thresholds that are too close together by calling <a href="https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/utils/fixes.py#L221"><code class="notranslate">utils.isclose.fixes</code></a>. The problem is that this happens without notification of the user (or any chance to intervene, setting <code class="notranslate">drop_intermediate</code> argument to <code class="notranslate">False</code> does not change anything) and, moreover, without scaling the tolerance w.r.t. to the variance of the score vector.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<p dir="auto">This is the (slightly simplified) example given in the documentation of <a href="https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/metrics/ranking.py#L421"><code class="notranslate">metrics.roc_curve</code></a></p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
from sklearn import metrics
y = np.array([False, False, True, True])
original_scores = np.array([0.1, 0.4, 0.35, 0.8])

scores = original_scores
fpr, tpr, thresholds = metrics.roc_curve(y, scores)
print fpr, tpr, thresholds, metrics.roc_auc_score(y, scores)

scores = 1e-6*original_scores
fpr, tpr, thresholds = metrics.roc_curve(y, scores)
print fpr, tpr, thresholds, metrics.roc_auc_score(y, scores)

scores = 1e-7*original_scores
fpr, tpr, thresholds = metrics.roc_curve(y, scores)
print fpr, tpr, thresholds, metrics.roc_auc_score(y, scores)

scores = 1e-8*original_scores
fpr, tpr, thresholds = metrics.roc_curve(y, scores)
print fpr, tpr, thresholds, metrics.roc_auc_score(y, scores)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span> <span class="pl-k">import</span> <span class="pl-s1">metrics</span>
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">False</span>, <span class="pl-c1">False</span>, <span class="pl-c1">True</span>, <span class="pl-c1">True</span>])
<span class="pl-s1">original_scores</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-c1">0.1</span>, <span class="pl-c1">0.4</span>, <span class="pl-c1">0.35</span>, <span class="pl-c1">0.8</span>])

<span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-s1">original_scores</span>
<span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span> <span class="pl-c1">=</span> <span class="pl-s1">metrics</span>.<span class="pl-en">roc_curve</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)
<span class="pl-k">print</span> <span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span>, <span class="pl-s1">metrics</span>.<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)

<span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-c1">1e-6</span><span class="pl-c1">*</span><span class="pl-s1">original_scores</span>
<span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span> <span class="pl-c1">=</span> <span class="pl-s1">metrics</span>.<span class="pl-en">roc_curve</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)
<span class="pl-k">print</span> <span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span>, <span class="pl-s1">metrics</span>.<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)

<span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-c1">1e-7</span><span class="pl-c1">*</span><span class="pl-s1">original_scores</span>
<span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span> <span class="pl-c1">=</span> <span class="pl-s1">metrics</span>.<span class="pl-en">roc_curve</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)
<span class="pl-k">print</span> <span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span>, <span class="pl-s1">metrics</span>.<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)

<span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-c1">1e-8</span><span class="pl-c1">*</span><span class="pl-s1">original_scores</span>
<span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span> <span class="pl-c1">=</span> <span class="pl-s1">metrics</span>.<span class="pl-en">roc_curve</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)
<span class="pl-k">print</span> <span class="pl-s1">fpr</span>, <span class="pl-s1">tpr</span>, <span class="pl-s1">thresholds</span>, <span class="pl-s1">metrics</span>.<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">y</span>, <span class="pl-s1">scores</span>)</pre></div>
<h4 dir="auto">Expected Results</h4>
<p dir="auto">All printed lines should yield the same output.</p>
<h4 dir="auto">Versions</h4>
<p dir="auto">Windows-10-10.0.10586<br>
('Python', '2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Jan 29 2016, 14:26:21) [MSC v.1500 64 bit (AMD64)]')<br>
('NumPy', '1.10.4')<br>
('SciPy', '0.17.0')<br>
('Scikit-Learn', '0.17')</p>
<p dir="auto">pred=[1e-10, 0, 0]<br>
sol=[1, 0, 0]<br>
metrics.roc_auc_score(sol, pred) # 0.5, wrong, 1 is correct</p>
<p dir="auto">pred=[1, 0, 0]<br>
sol=[1, 0, 0]<br>
metrics.roc_auc_score(sol, pred) # 1 correct</p>