Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint
evaluation;deep neural network;stochasticity;complex systems;forecasting
Yes
No ethics review needed.
- I don't understand the second part of the critical question, "is it encountering different stochastic behaviours" (different from what)? how is the "differentness" relevant?
- While it's pretty clear to me how to use this immediately in my work, I think anyone who wasn't already aware they wanted exactly this might struggle. Could you provide something like a "practical users guide" for non-domain experts?
 - if the clarity of the plots can be improved, the naming of the stat/metric you're introducing, and improve it's "usability" to the community, I would be happy to upgrade my score. You've done great work and this would bring the paper to the level it deserves.
Great paper, wonderfully practical and insightful; I've been looking for something like this for 5+ years! Nice eval on real-world data.
I started writing a thing I would like to you add and then discovered it was already in the paper (long horizon behaviour)
the metric of expected calibration error is introduced and studied as a way to capture fidelity of a learned representation to an underlying stochastic process (rather than a single realization of that process, as with typical metrics like AUC or MSE).
While overall the paper is very clear, some of the captions and explanations of the experiments/insights from them and how they tie to the figures could be improved. 
Some specifics:
- first fig should say what you mean by realization, and F2R and F2SP should be bolded (not ital) to make them easy to find in the text. Observed GT should be explained a bit more, or maybe it would be enough to move the sentence currently after F2R to be the second sentence of the paragraph.
 - Fig 5 is unclear to me. What is the data, what is S-level, why is it "good" that the 20 vs 10 lines are far apart? All of this should be clear from the caption
 - the clarity wanes a bit as the paper goes on, and it's a bit confusing that you call it ECE vs. F2SP vs Statistic-GP. Do these different namings really serve something? It could be a lot more clear if you just have one naming.
Yes
No ethics review needed.
A question arises regarding Figure 1: Can ECE be an effective metric for measuring F2R compared to other available metrics? Figure 1 suggests that the answer may be *no*.

An important indicator that ECE is a reliable measure is its diagonal pattern, showing low scores only when training and test S-Levels align, as illustrated in Figure 4. Could the authors provide theoretical insights to support this indicator?
The topic of evaluating DNNs within stochastic complex systems is both intriguing and important.

In the primary evaluations, the author conducted experiments across various settings, including different DNN architectures, comparisons with multiple evaluation metrics, and diverse simulation tasks.

The main text clearly explains the difference between ECE in classical assessment and stochastic process settings.
This paper presents a study evaluating deep neural networks (DNNs) within stochastic complex systems, emphasizing the importance of Expected Calibration Error (ECE) in measuring fidelity to stochastic processes. The findings are validated through multiple experiments and comparisons.
The paper is somewhat difficult to follow. For example, providing a brief introduction to the structure of each section would enhance clarity, particularly in Sections 2 and 3. Additionally, it is difficult to grasp the main messages conveyed by the table in Figure 2(b). Furthermore, in lines 229–240, the macro-level concept is introduced abruptly, which may disrupt the clarity and readability of the main text.

The main findings' practical applicability appears limited. In real-world scenarios, data generally provides only a single observed outcome centered on observable ground truth (line 117). Since the primary evaluation is simulation-based, the controlled stochasticity falls short of capturing real-world complexity. The Statistic-GT is basically derived by normalizing the frequency of target state occurrences across multiple Monte Carlo simulations.

Minors:

M1. The original text for the abbreviation RV is not given.

M2. In Table 1, what about the possibility of recovery in the Host-Pathogen problem?

M3. In line 152, maybe consider using an alternative symbol for Moore neighborhood, instead of $\mathcal{N}$ (normally representing Gaussian distribution).
Yes
No ethics review needed.
L50: Is --> is (lowercase)
Fig1: no need to write the whole name, you can use acronyms because they're already defined in the text, however MSE is not defined at this point.
L88: fidelity to realization --> F2R (it was already defined previously, so you can use the acronym)
L99: the notation of the dimension of the real vector O_t is confusing, what is (R^n)^(H x W), is n = H x W? If so, make that explicit.
Table 1: some rows end with full stop, other don't. Please make it consistent. Either all with or all without.
I find it odd to place Figures in columns as Figure 1 (which has a large top white margin) and Figure 3. I would suggest column figures into one row figure with multiple subfigures as you did with Figure 2. 
L201: Isn't the indicator variable already defined as B_t in L99? Why defining again with different notation?
L298: MSE already defined in text previously, no need to write the whole name again.
L516: ECE already defined in text previously, no need to write the whole name again.
Table 2 and Table 7: highlight the best performing DNNs.
The paper makes a significant contribution by introducing the concept of Fidelity to Stochastic Process (F2SP), a novel evaluation criterion specifically designed to assess a DNN's ability to learn the underlying stochastic interactions in complex systems.

The authors provide a rigorous formalization of F2SP within a stochastic framework, establishing clear criteria for its valid measurement. The use of Expected Calibration Error (ECE) as an evaluation metric is well-justified.
This paper presents a study on evaluating deep neural networks designed to forecast the evolution of stochastic complex systems. The authors identify a gap in traditional evaluation methods—such as threshold-based classification metrics and error-based scoring rules—which focus on a model's ability to replicate observed ground truth but fail to assess how well the model has learned the underlying stochastic process. To address this issue, they introduce a new property called Fidelity to Stochastic Process, representing the DNN's ability to predict the statistical ground truth of the stochastic process.

The paper proposes using the Expected Calibration Error (ECE) as an evaluation metric that satisfies the necessary conditions for assessing fidelity to statistical ground truth. This work underscores the importance of capturing the underlying stochastic processes in deep neural networks  evaluations for complex systems.
I found it hard to read the paper because there was a lack of consistency in the acronyms, the authors would redefine them in several parts of the text again and again. I addressed my comments on text in the questions section. 

In the tables, the best neural networks based on each criterion are not highlighted, which makes it difficult to the reader to infer and correlate the arguments in the text. I addressed my comments on text in the questions section. 

The focus of the paper is primarily on binary or discrete prediction tasks, leaving out regression tasks where the definition of calibration is more complex. While the authors acknowledge this and suggest it as an area for future work, the current scope limits the immediate applicability of the findings to a broader range of problems involving continuous outcomes.

Additionally, the use of the NDWS dataset, which is restricted to next-day predictions, prevents the assessment of ECE over longer time horizons, which are common in many complex systems. Could you elaborate on how future work might address this limitation? 

The paper highlights the lack of open-source complex system datasets as a barrier to broader validation. Are there any ongoing initiatives or plans to develop, collect, or standardize such datasets?
Yes
No ethics review needed.
As mentioned in the "Weaknesses" part.
The paper offers a new perspective on evaluating DNNs by considering DNNs as stochastic processes and uses a widely used criteria in Bayesian Deep Learning application to assess the fidelity to stochastic process. This work clearly explains the Expected Calibration Error is used to assess DNN modes in three synthetic cases and one real world case.
This work offers a new perspective on evaluating DNNs in stochastic complex systems by emphasizing  the importance of capturing underlying the stochastic process. Traditional evaluation methods assess the DNN’s ability to replicate the observed ground truth but fail to measure the DNN’s learning of the underlying stochastic process. This paper proposes a new property called Fidelity to Stochastic Process, representing the DNN’s ability to predict the ground truth of the stochastic process, and introduces an evaluation metric that exclusively assesses fidelity to  the ground truth of the stochastic process. The Expected Calibration Error is used to evaluate the fidelity to ground truth of statistic process. Empirical experiments on synthetic datasets (including wildfire, host-pathogen, and stock market models) and real-world wildfire data are used to show the measurement of fidelity to stochastic process by Expected Calibration Error.
This paper is well organized and well written, several minor issues should be addressed: (1) The explaination of figures is not sufficient, e.g., in Figure 2 (1), the label for x-axis is not specified (I guess it is time?), either add a label or explain it in the captions. Same problems also exist in Figure 4. (2) This work examines ECE on three synthetic environments (forest fire, host-pathogen and stock market models) and a real world wildfire spread dataset. I can tell that these datasets are all multivariate either for classification or regression. Maybe due to the limit of pages, the authors didn't include the experiments on images. I suggest the authors add some discussions or comments in the paper.
Yes
No ethics review needed.
I'm curious about how the author evaluates ECE at time $t$ based on Statistic-GT $P_{t}$. Do we have to simulate it again from $t=0$ for $N$ times or we can sample states from $t-1$ and go forward $N$ times (the system is Markov)? Can we still apply ECE on Statistic-GT when the system is not Markov?
1. Evaluating model fidelity on the stochastic system is significant and has wide applications.
2. The paper is well-motivated and both the dataset and experiments are thorough.
This paper introduces a novel stochasticity-compatible evaluation strategy for assessing existing models in the context of complex systems. The author justifies the Expected Calibration Error (ECE) as suitable for assessing the model fidelity of stochastic systems through both simulation environments and real-world data.
1. Although the author attempts to explain the difference between their work and ECE in deep learning in Lines 282-288, it appears to me this work is still a direct application of using ECE to evaluate the model performance on a stochastic system. The author is encouraged to discuss more in-depth about the distinction between ECE in the proposed method (stochasticity comes from evolving in the environment, aka, Statistic-GT) and ECE in previous works (stochasticity comes from the output distribution).
2. In Lines 243-244, the author claims that Statistic-GT is more stable than classification-based metrics, but I could not find any evidence related to calculating ECE on Statistic-GT is less sensitive to the system variance than MSE. Is there any theoretical support for using ECE over MSE on stochastic systems with different noise levels and could the author clarify it a bit more?
A novel evaluation criterion to assess whether DNNs modeling stochastic complex systems have learnt the underlying stochastic process
@inproceedings{
anonymous2024has,
title={Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2U8owdruSQ},
note={under review}
}
This paper presents the first systematic study of evaluating Deep Neural Networks (DNNs) designed to forecast the evolution of stochastic complex systems. We show that traditional evaluation methods, such as threshold-based classification metrics and error-based scoring rules, assess a DNN's ability to replicate the observed ground truth (Observed-GT) but fail to measure the DNN's learning of the underlying stochastic process. To address this gap, we propose a new property called *Fidelity to Stochastic Process (F2SP)*, representing the DNN's ability to predict the *Statistic-GT*—the ground truth of the stochastic process—and introduce an evaluation metric that exclusively assesses fidelity to Statistic-GT. We formalize F2SP within a stochastic framework and establish criteria for validly measuring it. We demonstrate that the Expected Calibration Error (ECE) satisfies the necessary conditions for evaluating fidelity to Statistic-GT. Empirical experiments on synthetic datasets—including wildfire, host-pathogen, and stock market models—show that ECE exclusively measures F2SP. We further extend our study to real-world wildfire data, highlighting the limitations of conventional evaluation and discussing the practical utility of incorporating F2SP into model assessment. This work offers a new perspective on evaluating DNNs in stochastic complex systems by emphasizing the importance of capturing the underlying stochastic process.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
evaluation deep neural network stochasticity complex systems forecasting
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3d15ba9c12c76f9d777203be175c2d81d2094d2b.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Has the Deep Neural Network learned the Stochastic Process? An Evaluation Viewpoint
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission