---
abstract: 'In many applications, accurate class probability estimates are required, but many types of models produce poor quality probability estimates despite achieving acceptable classification accuracy. Even though probability calibration has been a hot topic of research in recent times, the majority of this has investigated non-sequential data. In this paper, we consider calibrating models that produce class probability estimates from sequences of data, focusing on the case where predictions are obtained from incomplete sequences. We show that traditional calibration techniques are not sufficiently expressive for this task, and propose methods that adapt calibration schemes depending on the length of an input sequence. Experimental evaluation shows that the proposed methods are often substantially more effective at calibrating probability estimates from modern sequential architectures for incomplete sequences across a range of application domains.'
bibliography:
- 'example\_paper.bib'
---

=1

\[

0.3in \]

Introduction {#sec:introduction}
============

Sequential data is abundant in the modern world, commonly seen in forms such as natural language @harper2016movielens [@rajpurkar2016squad], video streams @Cordts2016Cityscapes and financial trading patterns @brown2013dynamic. Modern approaches to making predictions from these kinds of data typically involves using model architectures such as deep averaging networks @iyyer2015deep, recurrent neural networks @hochreiter1997long [@cho2014learning] and more recently, transformers @vaswani2017attention [@devlin2018bert; @radford2019language].

In many domains, predicting the most likely class label $\hat{y}_i$ for an instance $i$ with features $\mathbf{x}_i$ and label $y_i$ is sufficient for classification tasks. However, it is often the case in real applications that an estimated probability distribution $\hat{\mathbf{p}}_i$ over the labels can improve the quality or usefulness of the system. For example, an automated loan approval system can be used to minimise expected monetary losses to the lender if the probability of defaulting is accurately estimated. In semi-autonomous vehicles, low estimated confidence for the most likely class for some object may indicate that extra caution is required, possibly alerting the driver to intervene. Sometimes, an accurate probability distribution, rather than a hard classification, is required to have a functioning jsimmons@example.com. For example, if a model is used to predict the winner of a sports game in order to automatically set betting odds, accurate probability estimates are necessary.

\[ width=8cm, height=7.5cm, ymin=-0.015, ymax=0.365, xmin=3.3, xmax=625, ytick=<span>0,0.05,...,0.4</span>, ytick align=inside, ytick pos=left, xtick=<span>0,2,4,8,16,32,64,128,256,512,1024</span>, xtick align=inside, xtick pos=left, xmode=log, log basis x=2, grid=major, axis line style=thick, yticklabel style=<span>/pgf/number format/fixed</span>, xlabel=Sequence Length, ylabel=Expected Calibration Error, legend pos=north east, legend style=<span>nodes=<span>scale=0.65, transform shape</span></span>, legend cell align=<span>left</span> \] table \[x=x, y=y, col sep=tab\] ; ; table \[x=x, y=y, col sep=tab\] ; ; table \[x=x, y=y, col sep=tab\] ; ;

Even though most modern machine learning algorithms natively produce an estimated probability distribution over the class labels for a given instance, it is not always the case that they closely reflect the true probabilities of each class @niculescu2005predicting [@guo2017calibration; @leathart2019calibration; @kumar2019calibration]. Models for which this is true are said to be poorly *calibrated*. Probability calibration is an additional step one can apply when training a model $f$, where its class probability estimates (or logits) are used as inputs for another model $\pi$ that scales them appropriately to better match the true probabilities.

This work considers situations in which we wish to obtain class probability estimates for a sequence at any time during the formation of the sequence. This situation is fairly common, e.g., offering a "help" article to a website user while they are typing a description of the issue they are facing, predicting if an investor should buy or sell an option up until the expiration date, or predicting the outcome of a sports game given information about the current state of the game. For these types of problems, the prediction task typically gets easier as the end of the full sequence draws nearer—if the score in a football game is 1-0 with one minute remaining, we should be much more confident in our prediction of the winner than if the score is jsimmons@example.com. Figure \[fig:intro\_ece\] shows how expected calibration error, a commonly used calibration metric described in Section \[sec:evaluating\_calibration\], changes for the Large Movie Review dataset @imdb as the sequence length increases for several different models. In this example, deep average networks and transformers have poorer calibration for shorter sequences than longer, and vice-versa for the recurrent network. Intuitively, a global calibration strategy that applies the same calibration to sequences of any length will not be suitable for these models.

In this paper, we propose several simple strategies to adapt calibration schemes to better handle incomplete sequences, and evaluate them against traditional, global calibration methods. The paper is structured as follows. First, an introduction to probability calibration is provided, where definitions, existing approaches and evaluation methods are discussed. Then, our proposed temporal probability calibration techniques are described, considering both discrete and continuous sequences of fixed or variable length. Experiments are described and their results discussed. Finally, we go over conclusions and future work.

Probability Calibration
=======================

A probabilistic classifier is said to be perfectly calibrated when the probability estimates for each example exactly match the true class probabilities of the example. For instance, for those examples that are assigned a confidence of $75\%$ by a perfectly calibrated classifier, $75\%$ of them should actually be classified correctly.

More formally, for a probabilistic classifier $f$ for an $M$-class classification task and predicted probability distribution $\mathbf{\hat{p}} = [\hat{p}_1, \dots, \hat{p}_M]$, the proportions of classes for all possible instances that would be assigned the prediction $\mathbf{\hat{p}}$ by $f$ are equal to $\mathbf{\hat{p}}$ @kull2019beyond: $$\begin{aligned}
    \mathbb{P}(Y = m~|~f(\mathbf{X}) = \mathbf{\hat{p}}) = \hat{p}_m \quad \text{for }m \in \{1 \dots M\}.\label{eq:perfect_calibration}\end{aligned}$$

Evaluating Calibration\[sec:evaluating\_calibration\]
-----------------------------------------------------

Without an infinite number of samples, the condition in (\[eq:perfect\_calibration\]) is not possible to achieve. However, there exist several proxy metrics that aim to emulate this intuition. Common metrics for evaluating the quality of probability estimates include negative log likelihood (NLL) $$\begin{aligned}
    \text{NLL} = - \frac{1}{n} \sum_{i=1}^n \mathbf{y}_i \log \mathbf{\hat{p}}_i\end{aligned}$$ and the Brier score @brier1950verification, also known as mean squared error (MSE) $$\begin{aligned}
    \text{MSE} = \frac{1}{n} \sum_{i=1}^n (\mathbf{y}_i - \mathbf{\hat{p}}_i)^2.\end{aligned}$$ Technically speaking, neither of these metrics directly measure calibration, as for each example $i$, the estimated probability $\mathbf{\hat{p}}_i$ is compared to its label $\mathbf{y}_i$ rather than the true probability distribution over the label space. However, they are good proxy metrics, and possess convenient properties for machine learning such as being differentiable, applicable to individual examples, and easy to compute.

Of course, obtaining a true probability distribution over the label space for a single example is usually not possible in practice, as typically, only labels are supplied. Expected calibration error (ECE) @naeini2015obtaining is a binning-based approach that attempts to approximate this distribution empirically. The probability space $[0, 1]$ is split into $K$ bins, and the test examples are grouped into the bins based on their estimated probabilities. The accuracy and average confidence of each bin $B_k$ are computed as $$\begin{aligned}
    \text{acc}(B_k) &= \frac{1}{n_k} \sum_i^{n_k} \mathbb{I}(\hat{y}_i = y_i) \\
    \text{conf}(B_k) &= \frac{1}{n_k} \sum_i^{n_k} \hat{p}_i.\end{aligned}$$ ECE is then defined as $$\begin{aligned}
    \text{ECE} = \sum_{k=1}^K \frac{n_k}{n} \big| \text{acc}(B_k) - \text{conf}(B_k) \big|.\end{aligned}$$ For a perfectly calibrated model, the accuracy and confidence of each bin should be equal. Even though ECE measures (average) calibration directly, it is not without problems—the choice for number of bins is arbitrary, probabilities for individual examples are discarded in favour of aggregated bins, and its applicability to multiclass problems is debated @nixon2019measuring [@leathart2019tree]. Strategies such as classwise-ECE @kull2019beyond have been proposed to better handle the multiclass case.

Accuracy and confidence are often compared visually in reliability diagrams, where they are plotted against each other @degroot1983comparison. In reliability diagrams, perfect calibration is shown by a straight diagonal line. Regions where the curve sits above the diagonal represent underconfidence, and regions where the curve sits under the diagonal represent overconfidence.

Parametric Calibration Methods
------------------------------

One of the most well-known approaches to probability calibration is Platt scaling @platt1999probabilistic, in which a univariate logistic regression model with parameters $(\alpha, \beta)$ is learned to minimise NLL between a binary model's outputs $f(\mathbf{x}_i)$ and the labels $y_i$. Calibrated probabilities $\hat{p}_i$ for an instance $i$ can be obtained by $$\begin{aligned}
    \hat{p}_i = \pi(f(\mathbf{x}_i); \alpha, \beta) = \sigma\left(\alpha f(\mathbf{x}_i) + \beta\right)\end{aligned}$$ where $\sigma$ is the sigmoid function. Platt scaling can be applied to multiclass problems using the widely known one-vs-rest method @zadrozny2002transforming. Note that $f(\mathbf{x})$, the input to the calibration model, should be a logit rather than a probability. This is because logistic regression assumes a linear relationship between the inputs $f(\mathbf{x})$ and the output logits.

@guo2017calibration proposed several variants of Platt scaling—vector scaling, matrix scaling and temperature scaling—that can be applied to multiclass problems. Matrix and vector scaling are identical to multinomial logistic regression where a weights matrix $\mathbf{W}$ and bias vector $\mathbf{b}$ is learned $$\begin{aligned}
    \mathbf{\hat{p}}_i = \pi(f(\mathbf{x}_i); \mathbf{W}, \mathbf{b}) = \sigma \left(\mathbf{W} f(\mathbf{x}_i) + \mathbf{b}\right).\end{aligned}$$ The difference between them is that in vector scaling, $\mathbf{W}$ is restricted to be a diagonal matrix, while matrix scaling imposes no constraints on $\mathbf{W}$. Vector scaling can be seen as applying Platt scaling in a one-vs-rest strategy, but with jointly optimised weights. Finally, temperature scaling is the most simple strategy which learns a single parameter $\tau$ (referred to as the temperature) that scales the logits for all classes in the same way: $$\begin{aligned}
    \mathbf{\hat{p}}_i = \pi(f(\mathbf{x}_i); \tau) = \sigma \left(\frac{f(\mathbf{x}_i)}{\tau}\right)\end{aligned}$$ An interesting property of temperature scaling is that because there is no bias term applied, it simply makes probabilistic predictions more or less extreme without changing their classification, and hence does not affect the classification accuracy @guo2017calibration.

Beta scaling @kull2017beyond and Dirichlet scaling @kull2019beyond are, in a practical sense, very similar to Platt scaling and matrix scaling respectively. They model beta and Dirichlet distributions on probabilities $f(\mathbf{x}) \in [0, 1]$, but convert them to logits and log-probabilities respectively and use logistic regression to fit them. A regularisation scheme that penalises off-diagonal and bias parameters was also proposed which lead to strong results for both Dirichlet scaling and matrix scaling @kull2019beyond.

Nonparametric Calibration Methods
---------------------------------

Histogram binning @zadrozny2001obtaining is a simple nonparametric approach to probability calibration. In histogram binning, the model's output space is split into $K$ bins, typically by equal-width or equal-frequency strategies. A calibrated probability per bin is assigned such that the MSE for each bin is minimised, which turns out to be equal to the percentage of positive examples in each bin respectively. The calibrated probability estimate for a test example is given by the assigned value for the bin that it lands in. @naeini2015obtaining proposed an extension of histogram binning called Bayesian binning into quantiles, which performs Bayesian model averaging over all possible equal-frequency binning schemes.

Isotonic regression @zadrozny2002transforming learns a piecewise constant function that minimises MSE between the uncalibrated probabilities and labels, with the constraint that it must be monotonically increasing. This can be seen as an extension of histogram binning in that it optimises the bin widths and predictions jointly. @naeini2016binary proposed an extension of this where monotonicity is not constrained, but encouraged through regularisation. The removal of the monotonicity constraint leads to overfitting if training is performed to completion; to combat this, the collection of so called near-isotonic regression models produced at each step of training are used in an ensemble, with their predictions weighted by the Bayesian information criterion @schwarz1978estimating.

Temporal Probability Calibration
================================

Temporal probability calibration is motivated by the idea that a model ought to be more or less confident about its predictions at different stages of completion of a sequence. @leathart2017probability showed that overall calibration can be improved by applying different calibration models in different regions of the input space; this work takes a similar approach in the temporal dimension.

Discrete Fixed-Length Sequences\[sec:discrete\_calibration\]
------------------------------------------------------------

For classification problems where predictions are made at discrete timesteps from $t \in \{1, \dots, T\}$, it is simple and highly effective to produce a series of $T$ calibration models each parametrised by $\theta_t$, $\pi_1(f; \theta_1), \dots, \pi_T (f; \theta_T)$, corresponding to sequences of each possible length. Calibration of a model output $f(\mathbf{X})$ can be performed by $$\begin{aligned}
    \pi(f(\mathbf{X}), t; \theta) = \pi_t(f(\mathbf{X}); \theta_t)\end{aligned}$$ These calibration models $\pi_1, \dots, \pi_T$ are straightforward to fit by optimising the NLL for a held-out calibration set: $$\begin{aligned}
    \text{arg}\min_{\theta_t}~\text{NLL}\big(\mathbf{y}_t, \pi_{t}(f(\mathbf{X}_t); \theta_t)\big) \quad t \in \{1, \dots, T\}\end{aligned}$$ where $\mathbf{X}_t$ and $\mathbf{y}_t$ are the sequential features and labels respectively of a sample of examples from the calibration set of length $t$.

Variable-Length and Continuous Sequences
----------------------------------------

For classification problems where predictions are made at any real-valued or integer time $t \in (0, T]$ with possibly infinite $T$, a suitable function must be chosen to continuously evolve the parameters $\theta$ of a calibration model $\pi : \mathbb{R} \rightarrow [0, 1]$. Throughout this section, we consider temperature scaling as the main calibration method for simplicity of notation; however, the ideas presented are quite general and could, in theory, be applied to other (parametric) calibration methods.

A suitable type of function for modelling an adaptive temperature used in calibration for many problems is a saturating function, such as an exponentially decaying function: $$\begin{aligned}
    g(t; \alpha, \beta, \gamma, s) = \gamma - \alpha e^{(-\beta t - s)} \label{eq:exp_temperature}\end{aligned}$$ which can then be applied in a calibration function like so (parameters of $g$ have been written as $\theta$): $$\begin{aligned}
    \pi(f(\mathbf{X}), t; g, \theta) = \sigma \left( g(t; \theta) f(\mathbf{X})\right).\label{eq:exponential_decay}\end{aligned}$$ The coefficient of $f(\mathbf{X})$ in (\[eq:exponential\_decay\]) can be interpreted as the inverse of $\tau$ in temperature scaling. A function of this form is flexible enough to allow many different temporal calibration schemes, e.g., continuously decaying toward an upper or lower bound from any starting position.[^1] Figure \[fig:exponential\_examples\] shows some examples of possible temperature functions of this form. A convenient feature of applying temperature scaling in a temporal fashion in this way is that, like in the global calibration case, the total accuracy is left unchanged. Similarly to the discrete case, the parameters can be fit to minimise NLL on a held-out calibration set: $$\begin{aligned}
    \text{arg}\min_{\theta}~\text{NLL}\big(\mathbf{y}, \pi(f(\mathbf{X}); g, \theta)\big) \label{eq:fit_continuous}\end{aligned}$$ except that in this case, $\mathbf{X}$ and $\mathbf{y}$ must be the sequential features and labels respectively of a held-out set containing sequences that have been artificially truncated to produce a range of sequence lengths. We found that (\[eq:fit\_continuous\]) can be optimised reliably using typical off-the-shelf optimisers when $t$ is normalised to the range $[0, 1]$, based on the maximum length of sequences in the calibration set.

Alternative Temporal Measures\[sec:alternative\_measures\]
----------------------------------------------------------

\[ width=8.5cm, height=6.5cm, ymin=0.35, ymax=3.15, ytick=<span>0.5,1,...,3.5</span>, xmin=-1, xmax=25, xtick=<span>0,3,...,25</span>, xticklabels=<span>,,</span>, ylabel=Inverse Temperature, xlabel=Sequence Length, grid=major, axis line style=thick, yticklabel style=<span>/pgf/number format/fixed</span>, \] table \[x=x, y=y1, col sep=tab\] ; table \[x=x, y=y2, col sep=tab\] ; table \[x=x, y=y3, col sep=tab\] ; table \[x=x, y=y4, col sep=tab\] ;

In some problems, it may be more appropriate to use a different indicator of sequence completion than the time $t$ directly. Consider a game played between two players, where the goal is to win five rounds, and a draw is not possible in each round. Clearly, there are a maximum of nine rounds. Intuitively, one might think that as the number of rounds passed increases, we should be more confident of the outcome. However, if a game reaches the ninth round, then the players are likely to be of a similar skill level; thus the probabilities of each outcome becomes more difficult to predict accurately.

A superior measure of sequence completion than the round number $t$ for a game like this may be to fit calibration models to subsets of historical games with equal absolute score difference $s$. Sequences from other domains may have ways of representing an estimation of completion other than directly using the time, and it is left to the practitioner to decide the best approach for their specific situation.

Experimental Results
====================

In this section, our experimental methods are described and results discussed. As explained in the introduction, there are many situations in which accurate probability estimates from incomplete sequences are useful. However, no datasets for any of these specific tasks exist in the public domain. Nevertheless, we artificially create incomplete sequences from natural language datasets, as well as introduce a sequential dataset for esports outcome prediction, and show that temporal calibration works well across these domains.

\[t\]

  ------ ---------------------- --------------------------------- --------------------------------- -- --------------------------------- ---------------------------------
                                                                                                                                         
                                $\quad$    NLL                    $\quad$    ECE                       $\quad$    NLL                    $\quad$    ECE
                 No Calibration 0.646 $\pm$ 0.008                 0.107 $\pm$ 0.077                    0.412 $\pm$ 0.005                 0.048 $\pm$ 0.027 $\bullet$
   DAN       Global Calibration 0.548 $\pm$ 0.005 $\bullet$       0.110 $\pm$ 0.048                    0.386 $\pm$ 0.032                 0.152 $\pm$ 0.046
           Temporal Calibration **0.497 $\pm$ 0.004** $\bullet$   **0.034 $\pm$ 0.012** $\bullet$      **0.340 $\pm$ 0.007** $\bullet$   **0.037 $\pm$ 0.023** $\bullet$
                 No Calibration 0.464 $\pm$ 0.011                 0.076 $\pm$ 0.018 $\bullet$          0.233 $\pm$ 0.009                 0.039 $\pm$ 0.055 $\bullet$
   GRU       Global Calibration 0.458 $\pm$ 0.010                 0.089 $\pm$ 0.011                    0.229 $\pm$ 0.008 $\bullet$       0.047 $\pm$ 0.061
           Temporal Calibration **0.450 $\pm$ 0.008** $\bullet$   **0.072 $\pm$ 0.010** $\bullet$      **0.226 $\pm$ 0.008** $\bullet$   **0.038 $\pm$ 0.058** $\bullet$
                 No Calibration 0.593 $\pm$ 0.012                 0.098 $\pm$ 0.029 $\bullet$          0.533 $\pm$ 0.045                 0.172 $\pm$ 0.068
   BERT      Global Calibration 0.592 $\pm$ 0.011                 0.114 $\pm$ 0.024                    0.534 $\pm$ 0.047                 **0.157 $\pm$ 0.075** $\bullet$
           Temporal Calibration **0.577 $\pm$ 0.009** $\bullet$   **0.085 $\pm$ 0.011** $\bullet$      **0.521 $\pm$ 0.038** $\bullet$   0.162 $\pm$ 0.074 $\bullet$
  ------ ---------------------- --------------------------------- --------------------------------- -- --------------------------------- ---------------------------------

For all results, we present the mean of ten runs with ten different random seeds. In our results tables, the method with the best mean score for each metric is bolded. Additionally, we also compute statistical significance by the Friedman test followed by the Nemenyi post-hoc test at $p=0.05$ @demvsar2006statistical. This is a nonparametric test for multiple classifier comparison that compares average ranks of each method across the ten runs. If the average ranks of two methods are less than a critical difference (a function of number of classifiers, number of runs and desired $p$-value), then they are considered statistically indistinguishable from each other. In our tables, a bullet ($\bullet$) indicates that a method is in the best-performing group.

Experiments were conducted using the PyTorch neural networks framework @paszke2019pytorch and Huggingface's transformers library @Wolf2019HuggingFacesTS on a Google Cloud Service instance equipped with 16 vCPUs, 60GB of memory and an NVIDIA P100 GPU.

Natural Language Datasets
-------------------------

We use two natural language datasets: Large Movie Review @imdb and Amazon Fine Food Review @mcauley2013amateurs. Large Movie Review is a collection of film reviews taken from the Internet Movie Database (IMDB) @imdb. The classification task is to predict if reviews speak positively or negatively about the film. The dataset is split into 20,000 train, 5,000 calibration and 25,000 test examples. The average length of samples in this dataset is 233 words. Amazon Fine Food Review is a larger dataset of food reviews taken from Amazon. The original dataset has classes from one to five stars; in this investigation, we combine the positive reviews (four and five stars) and the neutral/negative reviews (one to three stars) to form a binary classification problem for simplicity. There are 450,000 train, 50,000 calibration, and 68,484 test examples, with an average length of 85 words.

### Experimental Setup

\[t\]\

For these datasets, we compare no calibration, global temperature scaling and temporal temperature scaling for three simple natural language processing (NLP) models: a deep averaging network (DAN) @iyyer2015deep, recurrent network using gated recurrent units (GRU) @cho2014learning and a BERT-based classifier [@devlin2018bert]. Even though transformers are quickly becoming the de-facto models for NLP research, we decided to include DANs and GRUs in this investigation because they have much smaller computational and memory requirements in order to be effective, and are still commonly used in industrial applications. We use 300-dimensional GloVe embeddings to represent the words for DAN and GRU models @pennington-etal-2014-glove.

The DAN had two fully-connected layers after the embedding layer, of 512 and 256 units respectively, before the output layer. The recurrent network contained a GRU followed by one fully-connected layer, each of 256 units, before the output layer. The sum of hidden outputs from the GRU at each time step was passed to the fully-connected. In both of these networks, ReLU activations @krizhevsky2012learning and dropout @srivastava2014dropout with $p=0.25$ were used between the fully-connected layers. For the BERT-based classifier, we truncate sequences to a maximum length of 512 tokens and use fixed pre-trained weights, only learning the final layer for prediction. Adam @kingma2014adam with default settings was used to optimise each network and temporal calibration parameters.

As the sequence length of the examples in these datasets is not uniform, we optimised a continuously decaying exponential function for temporal calibration as in (\[eq:exponential\_decay\]) in each experiment. The test and calibration sets were constructed by truncating each sequence in the original test set at five random uniformly-sampled points in order to obtain datasets containing a range of sequence completenesses.

### Results and Discussion

Table \[tab:nlp\_results\] lists the global NLL and ECE for each calibration method and model for both NLP datasets. Temporal calibration is in the top performing group of results at $p=0.05$ in every comparison.[^2] In all but one experiment, the temporal calibration approach achieves the lowest NLL and ECE on the test sets. The effect of temporal calibration is most pronounced for DANs. Interestingly, applying a global calibration scheme often resulted in degraded overall ECE compared to the baseline, despite an improvement in NLL.

A more enlightening approach to the evaluation of temporal calibration is to visualise calibration for incomplete sequences of different lengths. Figure \[fig:nlp\_ece\] shows how ECE evolves as the sequence length grows. The altered test sets with truncated examples were binned by sequence length such that each bin contains approximately the same number of samples, and ECE computed for each bin. For all dataset-model pairs, temporal calibration gives the lowest (or tied lowest) ECE for the majority of the time.

For both datasets, DANs saw marked improvements when using temporal calibration, compared to the baseline as well as global calibration (Figures \[fig:dan\_imdb\], \[fig:dan\_amazon\]). The effect of a fine-grained calibration strategy is most obvious for shorter sequences for both datasets. As the sequence length reaches the higher end, temporal calibration matches the performance of the baseline. This may be because a DAN has no notion of sequence length, so it is not able to learn behaviour for specifically dealing with very short sequences, and is often overconfident without the full context available.

The global calibration scheme baseline had interesting behaviour—usually it matches the performance of temporal scaling for one bin with ECE rising on either side, especially for the smaller dataset, Large Movie Review. It is intuitive that global calibration should behave in this manner, if our assumption that different calibration strategy is required at different sequence lengths is correct, as a global calibration scheme will be unable to adapt to the needs of different regions of temporal space. This result sheds light on how overall ECE can be worse after applying global calibration, despite slight improvements in overall NLL.

In general, calibration appears to be more impactful for smaller datasets when considering GRUs (Figures \[fig:gru\_imdb\], \[fig:gru\_amazon\]) and the BERT-based classifier (Figures \[fig:bert\_imdb\], \[fig:bert\_amazon\]). We did not see substantial improvements for these models with the larger Amazon Fine Food Review dataset. On the other hand, calibration of these models is greatly improved for Large Movie Reviews. The BERT-based classifier, like the DAN, sees the most improvement for shorter sequences. The calibration curve for BERT (Figure \[fig:bert\_imdb\]) appears to share a mixture of the behaviours of DAN and GRU (Figures \[fig:dan\_imdb\], \[fig:gru\_imdb\]). This may be because the bidirectional self-attention of BERT acts like a weighted average over the input tokens at each timestep similarly to DANs. However, unlike DANs, BERT retains temporal information through its positional encodings.

The GRU has a stronger sequential assumption than the DAN and BERT built into its architecture. Temporal calibration of GRUs matches the performance of the baseline for shorter sequences, only seeing a reduction in ECE for longer sequences. The performance drop for longer sequences is likely due to the GRU "forgetting" about the first tokens as the sequences grow. However, the network has learned to make low-confidence predictions at the beginning of sequences, so there is little to gain from calibration here.

CS:GO Round Sequences
---------------------

   Calibration Method  $\qquad$ NLL                       $\qquad$ ECE
  -------------------- ---------------------------------- ----------------------------------
     No Calibration    0.3830 $\pm$ 0.006                 0.0316 $\pm$ 0.013
   Global Calibration  0.3815 $\pm$ 0.006                 0.0290 $\pm$ 0.012 $\bullet$
    Temporal (Round)   **0.3793 $\pm$ 0.006** $\bullet$   **0.0235 $\pm$ 0.012** $\bullet$
    Temporal (Score)   0.4062 $\pm$ 0.004                 0.0450 $\pm$ 0.018

  : Global metrics for CS:GO dataset. Values presented are the means and standard deviations of ten independent runs. Test examples have been randomly truncated to simulate incomplete sequences. \[tab:csgo\]

Counter-Strike: Global Offensive (CS:GO) is a first-person shooter multiplayer video game in which two teams of five players play against each other. A game is comprised of two halves of fifteen rounds, where the winning team is the first team to win sixteen rounds. In the event of a draw, a six-round overtime is added to the match until a winner is decided.

We introduce the first public dataset for CS:GO game winner prediction. It contains sequences of round statistics where each sequence element includes information such as the round winner, number of surviving players and current score. There are 12,362 games in the dataset, which we split into 8,751 training samples, 2,907 calibration samples and 704 test samples. More information about the dataset can be found in the supplementary material.

### Experimental Setup

\

For this dataset, we only use a recurrent neural network with gated recurrent units @cho2014learning that uses the raw features as inputs. The GRU has 96 units, followed by two feed-forward fully-connected layers of 96 units each. ReLU activations @krizhevsky2012learning are applied between the linear fully-connected layers, as well as dropout @srivastava2014dropout with $p=0.25$.

As before, no calibration and global temperature scaling are used as baselines. These sequences have a maximum length and discrete timesteps, so we use discrete temporal calibration as described in Section \[sec:discrete\_calibration\]. Temperature scaling is used to calibrate each timestep. We experiment with using the round number and the absolute score difference as measures of time. As with the NLP datasets, we produce an augmented test and calibration set where examples are randomly truncated.

### Results and Discussion

Table \[tab:csgo\] shows global NLL and ECE of each calibration method for the CS:GO dataset. For this data, global temperature scaling shows slight improvements overall compared to the baseline, and temporal scaling based on round number achieves the best results. Interestingly, temporal scaling based on score difference has the worst calibration overall with a degradation compared to the baseline.

Figure \[fig:csgo\_ece\] shows how ECE evolves with the length of the sequence. While global calibration does achieve slight ECE improvements between rounds five and sixteen, as well as improvements at the end of the game and during overtime, the calibration degrades in-between these areas. Again, it is clear that global calibration strategies are suboptimal. Temporal calibration by round number results in lower ECE than global calibration in these areas, while matching the performance of the baseline during the mid-game rounds. Temporal calibration by score difference, hypothesised in Section \[sec:alternative\_measures\] to be more appropriate than round number for this type of sequence, turned out to have relatively poor performance, with ECE rising fairly steadily above that of the baseline after round sixteen. This may be due to the score difference not being able to differentiate between, for example, an absolute score difference of two at round two as opposed to at the penultimate round. The truncation strategy employed results in comparatively more short sequences than long ones, which may explain why calibration degrades near the end of the games.

Figure \[fig:csgo\_reliability\] shows reliability diagrams comparing the baseline to temporal calibration (by round number). Each plot shows the calibration at a different round number, illustrating the progression of the predictions across the length of a whole game. In reliability diagrams, perfect calibration is shown by a perfect diagonal line. We use ten equal-frequency bins in these plots. Even though the calibration of the baseline at each step is reasonably good, visible improvements are made at rounds six, ten, and thirty, while the calibration of the middle rounds stays virtually the same. As equal-frequency binning and temperature scaling are employed for each round number, the accuracy of each bin (on the $y$-axis) does not change, but the confidence (on the $x$-axis) moves closer to the diagonal line after calibration is applied.

Conclusion
==========

In this paper, we investigated probability calibration for sequential data; specifically looking at how calibration of the classification of a sequence that is being produced changes over time. Two simple methods for calibrating different types of sequences—exponentially-decaying temperature for continuous sequences and timestep-based binning for discrete sequences—were proposed, each showing an improvement in NLL and ECE in their respective areas of application. Especially high performance is obtained when a DAN is used for natural language data, or a relatively low amount of training data is available.

This paper touches on the calibration of predictions made by transformer-based classifiers, specifically BERT; an interesting avenue of future research would be to perform a thorough investigation on this topic. Some work has been done in this area for neural machine translation @kumar2019calibration [@muller2019does], but further investigation for standard classification tasks would be a valuable research contribution.

Acknowledgements {#acknowledgements .unnumbered}
================

The authors thank Christopher Laing and Chris Herrmann for useful discussions. The authors also thank Google for providing cloud credits that were used to run the experiments using Google Cloud Platform.

[^1]: This assumes that $\beta$ is positive. One can square this term to enforce this property.

[^2]: In fact, temporal calibration achieved an average rank of exactly one for most comparisons, but the Nemenyi test at $n=10, k=3$ and $p=0.05$ only finds two methods to be statistically indistinguishable if the difference in average ranks is greater than $1.04$. This is why in some cases, pairs of methods with seemingly large NLL and ECE differences are reported as being statistically indistinguishable.

{'source': 'arXiv_out/2002'}