<p dir="auto">I think we should add support for that.<br>
Though it's still tricky when using it in grid-search because that requires a custom scorer (via make_scorer) every time :-/</p>
<p dir="auto">to reproduce:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import numpy as np
np.random.seed(13)
classes = np.array(['yes', 'no'])
y = classes[np.random.randint(2, size=10)]
t = classes[np.random.randint(2, size=10)]
from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score
roc_auc_score(t, y)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">seed</span>(<span class="pl-c1">13</span>)
<span class="pl-s1">classes</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">array</span>([<span class="pl-s">'yes'</span>, <span class="pl-s">'no'</span>])
<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">classes</span>[<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">2</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)]
<span class="pl-s1">t</span> <span class="pl-c1">=</span> <span class="pl-s1">classes</span>[<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randint</span>(<span class="pl-c1">2</span>, <span class="pl-s1">size</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)]
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">roc_auc_score</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">metrics</span> <span class="pl-k">import</span> <span class="pl-s1">accuracy_score</span>
<span class="pl-en">roc_auc_score</span>(<span class="pl-s1">t</span>, <span class="pl-s1">y</span>)</pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ValueError: Data is not binary and pos_label is not specified"><pre class="notranslate"><code class="notranslate">ValueError: Data is not binary and pos_label is not specified
</code></pre></div>