<p dir="auto">We should expect this to work, but currently we get an inappropriate error message.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from sklearn import *
&gt;&gt;&gt; model_selection.cross_validate(linear_model.LogisticRegression(), np.random.rand(10, 2), ['no'] * 3 + ['yes'] * 7, scoring='roc_auc')
...
ValueError: Data is not binary and pos_label is not specified"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; from sklearn import *
&gt;&gt;&gt; model_selection.cross_validate(linear_model.LogisticRegression(), np.random.rand(10, 2), ['no'] * 3 + ['yes'] * 7, scoring='roc_auc')
...
ValueError: Data is not binary and pos_label is not specified
</code></pre></div>
<p dir="auto"><code class="notranslate">_binary_clf_curve</code> is assuming that <code class="notranslate">pos_label=1</code> if not specified. But it should be assuming the same ordering as the classifier, i.e. the greater string is the positive class label.</p>
<p dir="auto">Ping <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/qinhanmin2014/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/qinhanmin2014">@qinhanmin2014</a></p>
<p dir="auto">to reproduce:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
np.random.seed(13)
classes = np.array(['yes', 'no'])
y = classes[np.random.randint(2, size=10)]
t = classes[np.random.randint(2, size=10)]
from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score
roc_auc_score(t, y)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">13</span>)
<span class="pl-s1">classes</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-s">'yes'</span>, <span class="pl-s">'no'</span>])
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">classes</span>[<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">2</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)]
<span class="pl-s1">t</span> <span class="pl-c1">=</span> <span class="pl-s1">classes</span>[<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">2</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)]
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">roc_auc_score</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">accuracy_score</span>
<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">t</span>, <span class="pl-s1">y</span>)</pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ValueError: Data is not binary and pos_label is not specified"><pre class="notranslate"><code class="notranslate">ValueError: Data is not binary and pos_label is not specified
</code></pre></div>