<p dir="auto"><a href="https://scikit-learn.org/stable/modules/model_evaluation.html#multimetric-scoring" rel="nofollow">Using multiple metric evaluation</a> has the following example:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; def tn(y_true, y_pred): return confusion_matrix(y_true, y_pred)[0, 0]
&gt;&gt;&gt; def fp(y_true, y_pred): return confusion_matrix(y_true, y_pred)[0, 1]
&gt;&gt;&gt; def fn(y_true, y_pred): return confusion_matrix(y_true, y_pred)[1, 0]
&gt;&gt;&gt; def tp(y_true, y_pred): return confusion_matrix(y_true, y_pred)[1, 1]
&gt;&gt;&gt; scoring = {'tp': make_scorer(tp), 'tn': make_scorer(tn),
...            'fp': make_scorer(fp), 'fn': make_scorer(fn)}"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; def tn(y_true, y_pred): return confusion_matrix(y_true, y_pred)[0, 0]
&gt;&gt;&gt; def fp(y_true, y_pred): return confusion_matrix(y_true, y_pred)[0, 1]
&gt;&gt;&gt; def fn(y_true, y_pred): return confusion_matrix(y_true, y_pred)[1, 0]
&gt;&gt;&gt; def tp(y_true, y_pred): return confusion_matrix(y_true, y_pred)[1, 1]
&gt;&gt;&gt; scoring = {'tp': make_scorer(tp), 'tn': make_scorer(tn),
...            'fp': make_scorer(fp), 'fn': make_scorer(fn)}
</code></pre></div>
<p dir="auto">This means that <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html" rel="nofollow"><code class="notranslate">confusion_matrix</code></a> will be called <em><strong>4</strong></em> times for each <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html" rel="nofollow"><code class="notranslate">cross_validate</code></a> fold.</p>
<p dir="auto">This is probably not a performance issue because the cost of <code class="notranslate">confusion_matrix</code> is dwarfed by the cost of fitting and applying the model, but it is still clearly suboptimal.</p>
<p dir="auto">It would seem natural to allow scorers to return <em>dictionaries</em> rather than numbers.</p>
<p dir="auto">E.g., the previous example would be</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="def cm(y_true, y_pred):
    mx = confusion_matrix(y_true, y_pred)
    return {&quot;tn:mx[0,0], &quot;fp&quot;:mx[0,1], &quot;fn&quot;:mx[1,0], &quot;tp&quot;:mx[1,1]}
scoring = make_scorer(cm)"><pre class="notranslate"><code class="notranslate">def cm(y_true, y_pred):
    mx = confusion_matrix(y_true, y_pred)
    return {"tn:mx[0,0], "fp":mx[0,1], "fn":mx[1,0], "tp":mx[1,1]}
scoring = make_scorer(cm)
</code></pre></div>
<p dir="auto"><code class="notranslate">sklearn: 0.20.0</code></p>
<h4 dir="auto">Description</h4>
<p dir="auto">The implementation of <code class="notranslate">_multimetric_score</code> will call every scorer individually. <a href="https://github.com/scikit-learn/scikit-learn/blob/0.19.1/sklearn/model_selection/_validation.py#L545">HERE</a></p>
<p dir="auto">Unfortunately, these scorers are typically generated via <code class="notranslate">make_scorer</code> and each individual metric will end up repeatedly calling <code class="notranslate">predict</code>, <code class="notranslate">proba</code>, etc. <a href="https://github.com/scikit-learn/scikit-learn/blob/a24c8b464d094d2c468a16ea9f8bf8d42d949f84/sklearn/metrics/scorer.py#L75">HERE</a></p>
<p dir="auto">For a recent exploratory GridSearch where I was generating lots of metrics (multi-output regression and wanted individual statistics for each output), My scoring time was 75% as long as my fit time which is bonkers and I know that my scoring functions are nowhere near that slow.</p>
<h4 dir="auto">Suggested Change</h4>
<p dir="auto">This code should really just be calling predict once and feeding the same predictions into each of the scorers.</p>