<p dir="auto">REF: <a href="https://github.com/scikit-learn/scikit-learn/pull/18183/files#r472079672">https://github.com/scikit-learn/scikit-learn/pull/18183/files#r472079672</a></p>
<p dir="auto">Currently, the following list of metrics will raise an error if <code class="notranslate">str</code> are provided as labels and that <code class="notranslate">pos_label</code> is not defined and needed to compute the metrics:</p>
<ul dir="auto">
<li><code class="notranslate">average_precision_score</code></li>
<li><code class="notranslate">f1_score</code></li>
<li><code class="notranslate">fbeta_score</code></li>
<li><code class="notranslate">jaccard_score</code></li>
<li><code class="notranslate">precision_recall_curve</code></li>
<li><code class="notranslate">precision_score</code></li>
<li><code class="notranslate">recall_score</code></li>
<li><code class="notranslate">roc_curve</code></li>
</ul>
<p dir="auto">We try to make sure that the error message is consistent in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="681646578" data-permission-text="Title is private" data-url="https://github.com/scikit-learn/scikit-learn/issues/18192" data-hovercard-type="pull_request" data-hovercard-url="/scikit-learn/scikit-learn/pull/18192/hovercard" href="https://github.com/scikit-learn/scikit-learn/pull/18192">#18192</a></p>
<p dir="auto"><code class="notranslate">brier_score_loss</code> should supposedly follow the behaviour than the other metrics. However, it seems that up-to-now, it does some inference when labels are string.</p>
<p dir="auto">Thus, 2 questions can be raised:</p>
<ul dir="auto">
<li>Is it normal that <code class="notranslate">brier_score_loss</code> can do such inference or should it raise a similar error as other metrics?</li>
<li>If it should raise an error, shall we open consider as a bug fix or make a deprecation cycle?</li>
</ul>