Combining Denoised Neural Network and Genetic Symbolic Regression for Memory Behavior Modeling via Dynamic Asynchronous Optimization
Memory behavior;asynchronous optimization;neural networks;genetic symbolic regression
Yes
No ethics review needed.
In the ablation, what does it mean to have asynchronous training but not "dynamic optimization"? And is the opposite of that (having dynamic optimization but not asynchronous training) the same as the Waiting Optimization Strategy SPsyINN-W?

How well does symbolic regression alone do, and is this equivalent to the first line of Table 3? If the neural network is trained without symbolic regression, how does it do, and is that equivalent to the fourth line of Table 3? Or is the fourth line equivalent to training a neural network without the noise addition? If either of those are the case, stating them in the text would be really useful.

Is the log in the function set a natural log? In table 8, the result of ACT-R is presented as a natural log, but the function set just says "log", which could be assumed to be log10. If it is log10, why not include ln if it is used in ACT-R's results?
The proposed methodology and the application domain are both interesting. The idea of training a neural network and a symbolic regression model simultaneously, and aligning them in their respective optimization processes, is worthwhile and potentially novel. It could be used in a number of applications where physical laws are known or, as is the case here, there are established equations that map the relationship that should be approximated by machine learning. The interpretability of the end result, combined with the performance of a neural network training, motivates this approach. It would be interesting to see the method applied to the discovery of known physical laws, like in the following work:

Cranmer, Miles, et al. "Discovering symbolic models from deep learning with inductive biases." Advances in neural information processing systems 33 (2020): 17429-17442.

The application of memory modeling is also interesting; I am not aware of the application of PINNs to such psychological modeling problems. Most of the knowledge-informed literature is on physics-informed, including in symbolic regression, but the methods can be applied to other domains where existing relationships have been expressed as equations, even if they are not physical laws. There is a clear application to economics here, and expanding the perspectives to include similar applications could be helpful.
This article presents a hybrid symbolic neural network learning approach to model user interactions in memory-based learning tasks, specifically language learning. The approach uses interpretable models based on memory theory and integrates their optimization with a neural network training process. A comparison with other methods for knowledge modeling demonstrate that this hybrid approach has performance benefits, beyond the creation of interpretable results.
The main weakness of this paper is in its presentation. This is a mix of methods and an application readers might not be familiar with, so everything, from deep learning to symbolic regression to memory theory, need to be made clear to a reader. Even a reader who is an expert in some of those things might not know the others.

First, the mathematical notation is highly verbose, with subscripts for almost all variables, even when certain information is redundant or clear. For example, all tasks map user data U to word sets W. The inclusion of this mapping for every variable is unnecessary and makes the loss equations, like Equations 8 and 9, very hard to parse. Some equations are maybe not even necessary, like the definition of MSE. Simplifying the notation and reducing redundancies in the math would greatly increase clarity.

More explanation of the baseline methods would also help. DKT-F and FIFKT aren't fully explained, nor is the way that symbolic regression is integrated into their methods. The one-sentence explanations in appendix B are not sufficiently nor sufficiently referenced in the main text. For example, the description "DKT-F: An improved version of DKT that incorporates students’ forgetting behaviors. (Piech et al., 2015)" assumes that the reader understands that DKT refers to "Deep Knowledge Tracing" (it was not defined), and that the reader is familiar with Deep Knowledge Tracing's standard mechanisms, which are not described. In the Background, a short explanation of at least DKT could easily replace the sentence "The superiority of deep learning techniques in knowledge tracing and cognitive modeling has been well-established (Abdelrahman et al., 2023)," which doesn't give much information to the reader and is rather subjective.

Greater clarity in the text on the background methods and the problem domain would really help. Acronyms are rarely defined before use, and some acronyms are defined never to be used again (eg, NODS). So, I'm left with a number of questions despite a thorough reading of the paper, which is a shame because it is a very interesting method and application.
Yes
No ethics review needed.
1. How does SPsyINN perform on existing equation discovery benchmark datasets like SRBench and SRSD?

1. How do state-of-the-art SR algorithms perform in comparison on SPsyINN? The equations these state-of-the-art SR algorithms discover can be used to expand Table 2.

1. Where are the error bars for all the empirical results (i.e., standard deviation or inter-quartile range)?

1. Table 2 is given but not referenced to. Can the paper include a description and discussion of the results in Table 2?

Some of these questions may simply not be relevant because of the scope that the authors have set for the paper. If that is the case, I hope the justification for the limited scope can be addressed in the rebuttal.
1. The idea of jointly optimizing an equation with a neural network is novel among symbolic regression (SR) algorithms to the best of my knowledge.
1. Figure 2 gives a clear overview of the algorithm.
This paper proposes a new algorithm that jointly optimizes a neural network and an equation (that acts as an interpretable surrogate). The paper also explores update strategies of varying update rules. The approach proposed is then evaluated on real-world memory behavior datasets. The prediction performance of the neural network and discovered equation are reported separately.
1. It is unclear whether the main aim of the paper is to discover memory equations or to propose a new SR-based algorithm. If the objective is to introduce both at the same time, then the paper is in an awkward position because it is not mentioned or made obvious in the paper why this specific task "to discover memory equations" require the proposed method (e.g., the paper should explain why the joint optimization with a neural network method is particularly effective for discovery memory equations and not applicable to other domains such as Physics). If the method proposed is indeed not specifically tailored to "discover memory equations", then evaluation on other datasets would provide a stronger case for this paper.

1. Missing comparisons to state-of-the-art SR algorithms, that do not use joint optimization with a neural network, should be included as comparisons in the paper (expand Table 2).

1. Existing SR benchmark datasets such as SRBench and SRSD should be used to evaluate the proposed algorithm's equation discovery ability to improve the quality of experiments.

1. Missing error bars for empirical results, unable to tell if the difference in performance is significant (apart from Table 1 which performs t-test).

1. Missing details on the selection of MLP architecture and tuning.

1. In line 308, PySR was selected among SR algorithms but this choice was not justified. Several recent state-of-the-art SR algorithms (e.g., DSR [1], TPSR [2]) should be considered as well. Otherwise, the paper should justify why these methods were not considered.

[1] Petersen, B. K., Larma, M. L., Mundhenk, T. N., Santiago, C. P., Kim, S. K., & Kim, J. T. (2020, October). Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations.

[2] Shojaee, P., Meidani, K., Barati Farimani, A., & Reddy, C. (2023). Transformer-based planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 45907-45919.
Yes
No ethics review needed.
The actual task is not fully described. Fig 1 mentions answers being "correct" or "incorrect", but about what? What was the actual question being asked for each word?

What is the final overall output of the model (i.e. the one used to generate results in the tables)? Is it the output computed from the generated equation, or the output of the neural network?

The notation is confusing and seems to vary. I'd recommend using always 1:m to denote multiple timesteps and m to denote one single time steps (in the last sentence of Problem statement, apparently just 'm' is used to denote a whole sequence?)

There are many undefined acronyms. E.g. in l. 269 What does KT stand for? Where does this "KT-based framework" come from?

L. 276, where do the Beta_t come from? The noise schedule equations look very much like the ones used in diffusion model, which should warrant some kind of citation!

In the results, particularly the ablations, the various alternative methods are not described. As a result it is not at all easy to understand what each alternative version represents. In particular: do you report results based on training a neural network alone (with or without denoising), and a symbolic regressor alone?
The problem is interesting, as far as I can tell (I am not an expert in this area).

The method seems novel and the proposed interplay between deep learning and symbolic regression is interesting.

The model seems successful, judging from reported results.
The author propose a method for modelling memory behavior. This method combines symbolic regression and deep learning. A deep network and a symbolic regressor based on genetic algorithms are both jointly trained to predict memory performance from data. The neural network component also includes a loss based on noisy data to improve noise tolerance. Importantly, the two components are also trained to match each other through an alignment loss on their respective output, allowing both of them to train each other.

The resulting model seems to outperform existing approaches in predicting memory behaviors on various datasets.
I am mostly concerned about clarity, as the paper often uses confusing notation and undefined acronyms. While the overall method is reasonably clear, it is not easy to understand the details or reported comparisons in performance. See Questions below.
Yes
No ethics review needed.
The Authors state that the equations, that the model converges to, vary depending on the initial conditions and, it seems, on the model’s waiting strategies. Thus, which of the equations should neuroscientists / cognitive scientists use in their research as a result of this project? Are these equations similar or locally similar? Should they be distilled or approximated? How sensitive are these equations to the numerical coefficients? As one of the work’s stated goals is the interpretability of the results, it is important to know what results to use and to what degree to trust them. It would be great to hear the Authors’ thoughts on this topic. Separately, it would be highly interesting to see an analysis of the final equation once it’s established. How similar or dissimilar would it be to/from the existing models? What are the additional terms and what do we learn from them? Does it help us to ground the memory dynamics in neural circuits? An analysis like that has the potential to further increase the impact of this work.
The originality of the model lies in the novel combination of existing techniques (deep networks, denoising, and symbolic regression) and its application to the new domain (memory). I would like to especially highlight the new alignment algorithms, proposed here to account for the CPU – GPU interaction in the model, and domain priors that, first, kept the symbolic regression equations within the realm of memory model equations, and, second, accounted for the noise specific to memory-related data.

The significance of this model is in providing the equations (e.g. in Table 2) that are concise and at the same time have a high explanatory power for the memory-related data. Further analysis of such equations looks plausible and may be beneficial for the fields of psychology, cognitive science, and neuroscience.

The quality of this work is in the thorough empirical evaluation of the proposed model. While the model itself is rooted in literature and thus alone holds the potential of being useful for the task at hand, the evaluation on a series of datasets and the ablation study shows that the model indeed leads to the improvement of the performance and that all model components are necessary for such an improvement.
In this work, the Authors combine deep neural networks with genetic symbolic regression to model human memory in an efficient and interpretable manner. They proposed multiple ways to combine these two models, aiming at both compute efficiency and accuracy. The proposed model was tested on a panel of benchmarks where it showed an improved performance compared to a panel of baseline models.
I would suggest working on the text a little bit more to enhance its clarity to make it more accessible to the broad ICLR community. While the work introduces a relatively straightforward idea – (i) merge an efficient deep learning model with an interpretable symbolic regression model and a denoising module; (ii) use auxiliary losses that would match the outputs of the three models; (iii) compute auxiliary losses on an intermittently-generated proxy dataset to smoothly synchronize CPU and GPU computations – the text itself is often unnecessarily complicated. For example, I’d either simplify Figure 2, remove the equations from it, or move it down the text to serve as a summary. I’d then simplify the equations and remove some of them because, while they introduce straightforward concepts – like, the mean square error loss – they end up being pretty lengthy. This is best exemplified by Equations 8 and 9 which say: “compute the loss on a batch” but somehow occupy nearly half a page. I would also draw attention to some of the results that are present in the paper but may be overlooked, e.g. the equations in Table 2. These results also could use further discussion. Besides, even though the code is provided, I couldn’t find the Methods section that would describe the model in sufficient detail to reproduce it (e.g. the parameters of the neural network and the training schedule).

Minor:

The background section mostly repeats the introduction. I’d suggest shortening one of these sections and either expanding the other with the details of the models or using the vacated space for an additional discussion of the results.

‘1 + 1 greater than 2’ effect -> synergy effect

KT-based framework: KT is not defined in the main text

MAE is not defined in the main text

Table 1: second best models: clarify that those do not include SPsyINN models

Table 2 is not referenced in the results.

Table 3: provide the statistical significance test data (ideally with the false discovery rate correction)

Overall, I found this work interesting and relevant, and I am happy to recommend it for acceptance at ICLR but I believe that the text needs to be further edited to enhance the accessibility of this work.
The self-evolving psychological knowledge-informed neural network model (SPsyINN) uses Genetic Symbolic Regression to improve classical memory equations and enhances predictive accuracy and interpretability through neural networks.
@inproceedings{
anonymous2024combining,
title={Combining Denoised Neural Network and Genetic Symbolic Regression for Memory Behavior Modeling via Dynamic Asynchronous Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=DpOQwOzTc2},
note={under review}
}
Memory behavior modeling is a key topic in cognitive psychology and education. Traditional psychology describes the dynamic nature of memory through memory equations derived from experimental data, but these models often lack accuracy and their forms are subject to considerable debate. In recent years, data-driven modeling approaches have improved the predictive accuracy of models but often lack interpretability, making it difficult to provide deeper cognitive insights. Although knowledge-driven neural network models have achieved remarkable success in fields such as physics, their application in behavior modeling has been limited. To address this, this paper proposes a Self-evolving Psychological knowledge informed neural network (SPsyINN), which leverages genetic symbolic regression to improve classical memory equations and integrates a neural network model to enhance both prediction accuracy and interpretability. Specifically, this work first constructs a framework that integrates genetic symbolic regression with neural networks and establishes an asynchronous dynamic loss update strategy between them. Secondly, it constructs an interaction mechanism mediated by proxy data, promoting effective communication between genetic symbolic regression and the neural network to achieve efficient joint optimization of the model. Lastly, a denoising module is introduced into the neural network model to improve its robustness and generalization ability to noisy data. SPsyINN outperforms state-of-the-art methods across all key performance metrics on four large-scale real-world memory datasets. Additional experiments demonstrate that the proposed dynamic asynchronous optimization strategy is key to achieving joint model optimization. Moreover, the findings suggest that the method can effectively improve or discover memory equations, highlighting its potential application in psychological research. Our code is released at: https://anonymous.4open.science/status/SPsyINN-3F18
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Memory behavior asynchronous optimization neural networks genetic symbolic regression
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/729940f37b36c6eb43306879d56bbb740b13c168.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Combining Denoised Neural Network and Genetic Symbolic Regression for Memory Behavior Modeling via Dynamic Asynchronous Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission