Quantifying Past Error Matters: Conformal Inference for Time Series
Time Series; Uncertainty Quantification; Conformal Prediction; Distribution Shift
Yes
NA
No ethics review needed.
For Table 3: Did the authors try to use a similar scorecaster term within the ECI update and check if that could improve over conformal PID in the Prophet model? 

Also, is there any intuition for why decay-OGD baseline beats ECI in Table 5 (Prophet model)?
The idea of a smoothed version of quantile loss function to generate a more informative feedback for online gradient descent is simple and intuitive. It contributes meaningfully to improve uncertainty quantification in the online setting. 

The mathematical intuition for the idea is well-presented and contextualized within the literature on online conformal inference. 

The proposed technique produces consistent improvement in the width of prediction sets while maintaining coverage guarantees in a number of time series prediction tasks.
This paper focuses on uncertainty quantification for time-series prediction, where exchangeability does not hold due to distribution shift and sequential dependence of data-points. Online conformal inference methods can be utilized for this task. Recent body of work on online conformal inference methods uses online gradient descent on the quantile loss function to update the thresholds of prediction sets. However, the subgradients of quantile loss function, that update the threshold at every time step, only leverage information on whether the outcome at that time-step is contained in the prediction set. The update does not include information on the magnitude of miscoverage. This paper introduces a smoothed version of quantile loss function such that the subgradient carries this information. Thus, 'Error-quantified Conformal Inference' incorporates a smooth version of feedback to update the threshold that defines prediction sets. The paper also introduces variants of smoothed ECI feedback for uncertainty quantification in time-series. Theoretical results on coverage guarantees and averaged miscoverage error are obtained without any assumptions on the data-generating distribution. Experimental results demonstrate superior performance of the ECI method in terms of reduced width of prediction sets while maintaining good coverage in comparison with other important online conformal inference baselines.
The authors mention difficulties in calibrating the outcomes of complex machine learning models like transformer, along with other models like deep Gaussian processes. However, the proposed techniques were evaluated on simpler time series models. It would be helpful to have some results with more complex models or discuss how the method would impact them. 

It would be helpful to discuss the tradeoffs between different ECI variants in the context of experimental results. 

For the place where the conformal PID baseline beats the proposed method, it would be helpful to explain the scorecaster term in a couple of lines so that it is more obvious why this baseline could produce a superior performance.
Yes
No ethics review needed.
See weaknesses.
- While the methods are relatively straightforward, they are explained clearly and precisely.
- The use of smoothing feedback to actively incorporate the degree of miscoverage during the update is a compelling idea, allowing the prediction sets to adapt well to, e.g., potential distributional shifts in time series data.
- Theoretical results are based on reasonable assumptions.
This paper proposes Error-quantified Conformal Inference (ECI), based on the ACI approach, for online conformal inference with time series data. The key idea is to augment the online gradient descent with a smoothing term that reflects the extent of the gap between the non-conformity score and the threshold at each time step, moving beyond simple binary feedback. The main results include demonstrating coverage guarantees under a fixed learning rate, given bounded conditions for the smoothing function and non-conformity scores, and presenting a reasonable bound on the coverage gap in the case of well-designed adaptive learning rates.
- The authors primarily select the Sigmoid function for smoothing. However, as illustrated in Figure 2, when the extent of miscoverage exceeds a certain level, the EQ term actually decreases (which is also the case for the Gaussian error function). While the authors justify this with the goal of ensuring robustness, the rationale seems somewhat unconvincing.
- In connection with the above, if the Sigmoid function is used, the parameter $c$ can be considered a hyperparameter. Also given that Theorem 1 imposes constraints on $c$ for coverage guarantees, a more in-depth discussion on its selection, beyond the brief mention in Appendix D, would be appreciated.
- The interpretation of the miscoverage bound presented in Theorem 2 is rather ambiguous. The choice of the learning rate is a crucial aspect, yet theoretical discussion on this remains superficial.
- Similarly, the experiments lack detailed discussion on how the learning rate, which is critical for the performance of each method, was chosen. The statement on Line 330, “we select the most appropriate range of learning rates $\eta$ for the respective datasets and present the best results in the tables,” is not only vague but could potentially raise concerns about data leakage. More transparency in the tuning process is needed.
- Additionally, the choices of $h$ and $w$ in the ECI-cutoff method are not discussed.

Overall, while the proposed method is straightforward and built on a solid idea, there is room for more detailed discussion regarding the rationales, interpretations of the results, and specific hyperparameter tuning.

Minor:
- Line 119: Please use \citep for Barber et al. (2021).
- Line 238: incorrect interval order
- Line 653: non-differential -> non-differentiable
- Are "coverage"s in, e.g., Figure 3, rolling averages?
Yes
No ethics review needed.
- Do the proofs for the validity of EQ-integral and EQ-cutoff need to be adjusted from the proof of EQI?
- For Assumption 1 should it also be "$\forall t$"
- Line 159: "$(1-\alpha) (1 + t^{-1})$" is not necessarily in $[0, 1]$. For instance if $t=3$ and $\alpha = 0.001$. How can this be a sample quantile?
- The EQI intervals visually are much more jagged. Could this ever result in dangerous overfitting?
- The significance of Figure 2 is unclear to me and the caption does not provide much detail. Why is this important and what is going on in this figure?
- Would error magnitude information be more useful for better or worse performing base models $f$ relative to other methods (ACI, OGD, etc)?
- General presentation of the problem is clear and this is an approach could be of interest to an active community
- Mathematical derivations are included
- Strong empirical results on synthetic and real-world datasets
This work introduces Error-quantified Conformal Inference (ECI), a scheme for providing prediction intervals for time series data. Standard conformal inference requires exchangeability for statistical coverage guarantees to hold but this is generally violated in the context of time series data. Other methods to provide prediction intervals exist but many do not incorporate the magnitude of miscoverage when providing updates for future intervals. ECI utilizes this information for its updates. The authors derive a coverage guarantee for their methodology and  provide empirical results on synthetic and real-world datasets to support their claims. Compared to baselines that do not incorporate the magnitude of error they find that they are able to provide narrower intervals.
- Derivation on line 266 is difficult to read because it is inline
- Minor grammatical errors and occasional typos scattered throughout the manuscript
- See questions
Yes
No ethics review needed.
- The "adaptive" CI school is only one of the possible approache to deal with the absence of indipendence between observations in Conformal Inference, in fact, https://proceedings.mlr.press/v75/chernozhukov18a.html provide a fairly interesting approach, which would be interesting to see compared to the authors proposal.

- A very recent contribution by Oliveira et al. https://jmlr.org/papers/v25/23-1553.html shows that the contribution of adaptivity procedures to be negligible. How do the authors comment about this?
The proposed extension is rather interesting, seems to provide reasonable results, and puts itself in a rather interesting area of conformal prediction.
The author propose an extension over the Adaptive Conformal Inference method of Gibbs and Candes, then used in the time-series realm by Zaffran et al. by incorporating in the adaptation procedure information about the magnitude of the error in the miscoverage rate.
After having stated the method and analysed its performance with some real-world applications.
- "Conformal Inference for Time Series" is an extremely broad title, which gives the impression that the authors are the first in doing something in this which is, in fact, not true.

- I found the article written in a very rough and unrefined way: I have counted at least 10 instances of missing articles (the...). Moreover, the introduction contains claims that are either too bold, or require a deeper explanation. What does it mean, for instance, that "Bayesian recurrent neural networks or deep gaussian processes are difficult to calibrate?", or, why is it meaningful to say that "quantile regression models may overfit when estimating uncertainty"

- I am seriously concerned about the "stock market" application. It seems to me that the authors aim at forecasting log stock prices, which are known to be nonstationary. In fact, the usual approach in the econometric literature is to forecast returns (i.e. log differences) which instead, tend to be a stationary time series.
We propose a new online conformal inference method ECI by quantifying the extent of under/over coverage, which can react quickly to distribution shift in time series.
@inproceedings{
anonymous2024quantifying,
title={Quantifying Past Error Matters: Conformal Inference for Time Series},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=RD9q5vEe1Q},
note={under review}
}
Uncertainty quantification in time series prediction is challenging due to the temporal dependence and distribution shift on sequential data. Conformal prediction provides a pivotal and flexible instrument for assessing the uncertainty of machine learning models through prediction sets. Recently, a series of online conformal inference methods updated thresholds of prediction sets by performing online gradient descent on a sequence of quantile loss functions. A drawback of such methods is that they only use the information of revealed non-conformity scores via miscoverage indicators but ignore error quantification, namely the distance between the non-conformity score and the current threshold. To accurately leverage the dynamic of miscoverage error, we propose Error-quantified Conformal Inference (ECI) by smoothing the quantile loss function. ECI introduces a continuous and adaptive feedback scale with the miscoverage error, rather than simple binary feedback in existing methods. We establish a long-term coverage guarantee for ECI under arbitrary dependence and distribution shift. The extensive experimental results show that ECI can achieve valid miscoverage control and output tighter prediction sets than other baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Time Series; Uncertainty Quantification; Conformal Prediction; Distribution Shift
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c18e1f891dc702270ef6774e6910a44d93afa089.pdf
learning on time series and dynamical systems
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Quantifying Past Error Matters: Conformal Inference for Time Series
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission