<p dir="auto">Suppose we wish to classify some examples where one class is relatively rare.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd,numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from random import choice

df0=pd.DataFrame(np.random.normal(size=(100,3)))
df1=pd.Series([choice([-1,1]) for i in xrange(df0.shape[0])])
df1.iloc[-1]=0
df2=pd.Series(np.ones(df0.shape[0])/df0.shape[0])

clf=RandomForestClassifier(n_estimators=10)                               
score=cross_val_score(clf,X=df0,y=df1,cv=10,scoring='neg_log_loss',fit_params={'sample_weight':df2.values}).mean()"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>,<span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">ensemble</span> <span class="pl-k">import</span> <span class="pl-v">RandomForestClassifier</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">model_selection</span> <span class="pl-k">import</span> <span class="pl-s1">cross_val_score</span>
<span class="pl-k">from</span> <span class="pl-s1">random</span> <span class="pl-k">import</span> <span class="pl-s1">choice</span>

<span class="pl-s1">df0</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">normal</span>(<span class="pl-s1">size</span><span class="pl-c1">=</span>(<span class="pl-c1">100</span>,<span class="pl-c1">3</span>)))
<span class="pl-s1">df1</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-v">Series</span>([<span class="pl-en">choice</span>([<span class="pl-c1">-</span><span class="pl-c1">1</span>,<span class="pl-c1">1</span>]) <span class="pl-k">for</span> <span class="pl-s1">i</span> <span class="pl-c1">in</span> <span class="pl-en">xrange</span>(<span class="pl-s1">df0</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>])])
<span class="pl-s1">df1</span>.<span class="pl-s1">iloc</span>[<span class="pl-c1">-</span><span class="pl-c1">1</span>]<span class="pl-c1">=</span><span class="pl-c1">0</span>
<span class="pl-s1">df2</span><span class="pl-c1">=</span><span class="pl-s1">pd</span>.<span class="pl-v">Series</span>(<span class="pl-s1">np</span>.<span class="pl-en">ones</span>(<span class="pl-s1">df0</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>])<span class="pl-c1">/</span><span class="pl-s1">df0</span>.<span class="pl-s1">shape</span>[<span class="pl-c1">0</span>])

<span class="pl-s1">clf</span><span class="pl-c1">=</span><span class="pl-v">RandomForestClassifier</span>(<span class="pl-s1">n_estimators</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)                               
<span class="pl-s1">score</span><span class="pl-c1">=</span><span class="pl-en">cross_val_score</span>(<span class="pl-s1">clf</span>,<span class="pl-v">X</span><span class="pl-c1">=</span><span class="pl-s1">df0</span>,<span class="pl-s1">y</span><span class="pl-c1">=</span><span class="pl-s1">df1</span>,<span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">10</span>,<span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-s">'neg_log_loss'</span>,<span class="pl-s1">fit_params</span><span class="pl-c1">=</span>{<span class="pl-s">'sample_weight'</span>:<span class="pl-s1">df2</span>.<span class="pl-s1">values</span>}).<span class="pl-en">mean</span>()</pre></div>
<p dir="auto">This will spit the error:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="*** ValueError: y_true and y_pred contain different number of classes 3, 2. Please provide the true labels explicitly through the labels argument. Classes found in y_true: [-1  0  1]"><pre class="notranslate"><span class="pl-c1">**</span><span class="pl-c1">*</span> <span class="pl-v">ValueError</span>: <span class="pl-s1">y_true</span> <span class="pl-c1">and</span> <span class="pl-s1">y_pred</span> <span class="pl-s1">contain</span> <span class="pl-s1">different</span> <span class="pl-s1">number</span> <span class="pl-s1">of</span> <span class="pl-s1">classes</span> <span class="pl-c1">3</span>, <span class="pl-c1">2.</span> <span class="pl-v">Please</span> <span class="pl-s1">provide</span> <span class="pl-s1">the</span> <span class="pl-s1">true</span> <span class="pl-s1">labels</span> <span class="pl-s1">explicitly</span> <span class="pl-s1">through</span> <span class="pl-s1">the</span> <span class="pl-s1">labels</span> <span class="pl-s1">argument</span>. <span class="pl-v">Classes</span> <span class="pl-s1">found</span> <span class="pl-c1">in</span> <span class="pl-s1">y_true</span>: [<span class="pl-c1">-</span><span class="pl-c1">1</span>  <span class="pl-c1">0</span>  <span class="pl-c1">1</span>]</pre></div>
<p dir="auto">This is a known problem for <em>log_loss</em>, which was fixed by adding an argument <strong>labels</strong> to that function: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="53133263" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/4033" data-hovercard-type="issue" data-hovercard-url="/scikit-learn/scikit-learn/issues/4033/hovercard" href="https://github.com/scikit-learn/scikit-learn/issues/4033">#4033</a></p>
<p dir="auto">The problem above is, <em>cross_val_score</em> invokes that function, but does not allow us to pass the labels. Ideally, cross_val_score should be able to infer the true labels from y=df1.</p>
<p dir="auto">sklearn version '0.18.1'</p>
<p dir="auto">Moving the discussion with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/amueller/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/amueller">@amueller</a> from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="116300904" data-permission-text="Title is private" data-url="https://github.com/pydata/patsy/issues/77" data-hovercard-type="issue" data-hovercard-url="/pydata/patsy/issues/77/hovercard?comment_id=174335341&amp;comment_type=issue_comment" href="https://github.com/pydata/patsy/issues/77#issuecomment-174335341">pydata/patsy#77 (comment)</a>.</p>
<p dir="auto">Proposing to:</p>
<ul dir="auto">
<li>add an optional 'labels' argument to <code class="notranslate">log_loss</code></li>
<li>add an argument to <code class="notranslate">make_scorer</code> to enable passing labels argument to the custom loss function</li>
</ul>
<p dir="auto">Related to this - perhaps also allow <code class="notranslate">make_scorer</code>'s <code class="notranslate">need_threshold = True</code> to be used for non-binary classification problems. Not sure why it's limited to binary. Totally meaningful for multi-class situations, where the metric might use the decision function to evaluate based on the rank order or something (for example, <code class="notranslate">SVC</code> doesn't have a cheap <code class="notranslate">predict_proba</code>, but has a cheap <code class="notranslate">decision_function</code>, so it would allow it to be used with rank-based metris). From <code class="notranslate">make_scorer</code> perspective, it just seems like a completely arbitrary restriction because it shouldn't care about what the custom loss function does with the <code class="notranslate">decision_function</code> output.</p>