<h3 dir="auto">Describe the workflow you want to enable</h3>
<p dir="auto">Measuring calibration error in deep learning is a big issue now-a-days. However, we do not find any suitable package available to measure the calibration error. I would like to add a function that can measure calibration error with adaptive bin size as described here: <a href="https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf" rel="nofollow">https://openaccess.thecvf.com/content_CVPRW_2019/papers/Uncertainty%20and%20Robustness%20in%20Deep%20Visual%20Learning/Nixon_Measuring_Calibration_in_Deep_Learning_CVPRW_2019_paper.pdf</a>.  Note that there are many ECE algorithms that measures the calibration error with fixed bin size that is major problem for inhomogeneous data distribution.</p>
<h3 dir="auto">Describe your proposed solution</h3>
<p dir="auto">We already have a tentative implementation here: <a href="https://github.com/neurodata/kdg/blob/cc88886819676020402f19ec8b224ed5aafdc440/kdg/utils.py#L8">https://github.com/neurodata/kdg/blob/cc88886819676020402f19ec8b224ed5aafdc440/kdg/utils.py#L8</a>. We would like to adapt the code for sklearn so that people can use it.</p>
<h3 dir="auto">Describe alternatives you've considered, if relevant</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Additional context</h3>
<p dir="auto"><em>No response</em></p>
<h4 dir="auto">Describe the workflow you want to enable</h4>
<p dir="auto">I would like to add the ability to calculate Expected Calibration Error (ECE) within scikit-learn. ECE is defined in equation (3) from <a href="https://arxiv.org/abs/1706.04599" rel="nofollow">Guo et al. <em>On Calibration of Modern Neural Networks.</em> (2017).</a> This is a well-cited paper (over 700 citations as of Aug. 26, 2020), and the ECE metric is now widely used in academic papers on model calibration. There is even a method now in TensorFlow Probability for computing ECE: <a href="https://www.tensorflow.org/probability/api_docs/python/tfp/stats/expected_calibration_error" rel="nofollow">tfp.stats.expected_calibration_error()</a>.</p>
<h4 dir="auto">Describe your proposed solution</h4>
<p dir="auto">I can see several possibilities for how the ECE calculation could be added to scikit-learn, with pros and cons for each:</p>
<ol dir="auto">
<li>
<p dir="auto">make ECE an extra return value of <code class="notranslate">sklearn.calibration.calibration_curve()</code>. This method currently already does 99% of the work towards calculating ECE. Computing ECE is simply a one-liner added to the end of that method:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="def calibration_curve(...):
    ...
    ece = np.sum(np.abs(prob_true - prob_pred) * (bin_total[nonzero] / len(y_true)))
    return prob_true, prob_pred, ece"><pre class="notranslate"><span class="pl-k">def</span> <span class="pl-en">calibration_curve</span>(...):
    ...
    <span class="pl-s1">ece</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">sum</span>(<span class="pl-s1">np</span>.<span class="pl-en">abs</span>(<span class="pl-s1">prob_true</span> <span class="pl-c1">-</span> <span class="pl-s1">prob_pred</span>) <span class="pl-c1">*</span> (<span class="pl-s1">bin_total</span>[<span class="pl-s1">nonzero</span>] <span class="pl-c1">/</span> <span class="pl-en">len</span>(<span class="pl-s1">y_true</span>)))
    <span class="pl-k">return</span> <span class="pl-s1">prob_true</span>, <span class="pl-s1">prob_pred</span>, <span class="pl-s1">ece</span></pre></div>
<p dir="auto">The downside is that introducing a 3rd return value is a breaking change. One possible mitigation is to add a boolean <code class="notranslate">return_ece</code> parameter to the function definition with a default value of <code class="notranslate">False</code>, and only return <code class="notranslate">ece</code> if <code class="notranslate">return_ece=True</code>.</p>
</li>
<li>
<p dir="auto">add a dedicated function to <code class="notranslate">sklearn.metrics</code>, i.e., <code class="notranslate">sklearn.metrics.expected_calibration_error(y_true, y_pred)</code>. Downside here is that it would need to re-compute everything that calibration_curve() already computes, so there is a performance penalty for a user who wants to both calculate ECE and plot a calibration curve.</p>
</li>
<li>
<p dir="auto">add a dedicated function to <code class="notranslate">sklearn.calibration</code>, i.e., <code class="notranslate">sklearn.calibration.expected_calibration_error(y_true, y_pred)</code>. This would keep the ECE calibration within the calibration subpackage. Same downside as in option 2.</p>
</li>
</ol>
<h4 dir="auto">Additional context</h4>
<p dir="auto">I am happy to write the code and tests to add ECE calculation to scikit-learn. After all, the code to calculate ECE is just a one line addition to <code class="notranslate">calibration_curve()</code>. However, as I am not a regular contributor to scikit-learn, I am unfamiliar with the best place within the scikit-learn library to add such a feature. Please advise!</p>