<p dir="auto">Use case: for classifiers with predict_proba I like to see precision/recall data across different probability values. This would be really easy to return from current cross_validation.py master if not for<br>
if not isinstance(score, numbers.Number):<br>
raise ValueError<br>
in _cross_val_score<br>
with this changed to a warning, cross_val_score can do the check before converting scores to np.array, and return plain list for incompatible types.</p>
<p dir="auto">I'd like to use cross_validation.cross_val_score with metrics.precision_recall_fscore_support so that I can get all relevant cross-validation metrics without having to run my cross-validation once for accuracy, once for precision, once for recall, and once for f1. But when I try this I get a ValueError:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from sklearn.datasets import fetch_20newsgroups

from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import metrics
from sklearn import cross_validation
import numpy as np

data_train = fetch_20newsgroups(subset='train', #categories=categories,
                                shuffle=True, random_state=42)
clf = LinearSVC(loss='l1', penalty='l2')
vectorizer = TfidfVectorizer(
  sublinear_tf=False, 
  max_df=0.5,
  min_df=2, 
  ngram_range = (1,1),
  use_idf=False,
  stop_words='english')

X_train = vectorizer.fit_transform(data_train.data)

# Cross-validate:
scores = cross_validation.cross_val_score(
  clf, X_train, data_train.target, cv=5, 
  scoring=metrics.precision_recall_fscore_support)"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">datasets</span> <span class="pl-k">import</span> <span class="pl-s1">fetch_20newsgroups</span>

<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">svm</span> <span class="pl-k">import</span> <span class="pl-v">LinearSVC</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">feature_extraction</span>.<span class="pl-s1">text</span> <span class="pl-k">import</span> <span class="pl-v">TfidfVectorizer</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span> <span class="pl-k">import</span> <span class="pl-s1">metrics</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span> <span class="pl-k">import</span> <span class="pl-s1">cross_validation</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>

<span class="pl-s1">data_train</span> <span class="pl-c1">=</span> <span class="pl-en">fetch_20newsgroups</span>(<span class="pl-s1">subset</span><span class="pl-c1">=</span><span class="pl-s">'train'</span>, <span class="pl-c">#categories=categories,</span>
                                <span class="pl-s1">shuffle</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">random_state</span><span class="pl-c1">=</span><span class="pl-c1">42</span>)
<span class="pl-s1">clf</span> <span class="pl-c1">=</span> <span class="pl-v">LinearSVC</span>(<span class="pl-s1">loss</span><span class="pl-c1">=</span><span class="pl-s">'l1'</span>, <span class="pl-s1">penalty</span><span class="pl-c1">=</span><span class="pl-s">'l2'</span>)
<span class="pl-s1">vectorizer</span> <span class="pl-c1">=</span> <span class="pl-v">TfidfVectorizer</span>(
  <span class="pl-s1">sublinear_tf</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, 
  <span class="pl-s1">max_df</span><span class="pl-c1">=</span><span class="pl-c1">0.5</span>,
  <span class="pl-s1">min_df</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, 
  <span class="pl-s1">ngram_range</span> <span class="pl-c1">=</span> (<span class="pl-c1">1</span>,<span class="pl-c1">1</span>),
  <span class="pl-s1">use_idf</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
  <span class="pl-s1">stop_words</span><span class="pl-c1">=</span><span class="pl-s">'english'</span>)

<span class="pl-v">X_train</span> <span class="pl-c1">=</span> <span class="pl-s1">vectorizer</span>.<span class="pl-en">fit_transform</span>(<span class="pl-s1">data_train</span>.<span class="pl-s1">data</span>)

<span class="pl-c"># Cross-validate:</span>
<span class="pl-s1">scores</span> <span class="pl-c1">=</span> <span class="pl-s1">cross_validation</span>.<span class="pl-en">cross_val_score</span>(
  <span class="pl-s1">clf</span>, <span class="pl-v">X_train</span>, <span class="pl-s1">data_train</span>.<span class="pl-s1">target</span>, <span class="pl-s1">cv</span><span class="pl-c1">=</span><span class="pl-c1">5</span>, 
  <span class="pl-s1">scoring</span><span class="pl-c1">=</span><span class="pl-s1">metrics</span>.<span class="pl-s1">precision_recall_fscore_support</span>)</pre></div>