COPU: Recognizing Time Series' Heterogeneity In Stacked Neural Network
Neural Network;Time Series;Regression;Nonlinear Modeling;Natural Gradient
**Response to Reviewer 1q18**

We are grateful for the time and effort you have invested in reviewing our manuscript. We take each of your concerns seriously and are confident that we can address all issues raised to your satisfaction.

**Response to Weaknesses 1**

We respectfully point out that this could be a misunderstanding. 
Eq. (6) is implicitly referenced by Eq. (7), Eq. (9), as well as in the text from line 245 to line 249. The $\mathcal{L}$ in Eq. (9) specifics the loss computation with respect to the dual parameter $m$ in Eq. (7), the $\nabla_m$ specifics taking the derivative with respect to $m$, and the following textual explanation,
> Specifically, NGD has been applied to the trackable parameter $\theta_r$ as shown in Figure 4.

All of these jointly describe how Eq. (6) has been applied to COPU.

The paragraph from line 251 to line 257 should be under Eq. (8). This is a mistake in formatting. We apologize for the inconvenience caused by our carelessness.


**Response to Weaknesses 2**

We respectfully point out that this could be a misunderstanding.
Innovation is an important concept in the field of stochastic/statistical signal processing. It represents the information of newly observed data that cannot be expressed through orthogonal projections (Orthogonal Projection Unit, OPU) from previously seen data. Innovation has significant applications in areas such as Kalman filtering, control, and estimation. In our work, we similarly use innovation to denote the part of the information that is not easily represented by orthogonal projection from other samples. This filtering process is realized through the analysis of cross-correlation, which is why the model is termed COPU.

**Response to Weaknesses 3**

Thank you for pointing out this typo, the accurate form is "module".

**Response to Weaknesses 4**

Thank you for pointing out this issue. The citation should have been used in parenthesis. This is indeed a wrong reference type. Thank you for your meticulous examination.


**Response to Weaknesses 5**

Thank you for pointing out this issue. We will include additional clarifications regarding this metric.


**Response to Weaknesses 6**

Thank you for pointing out this issue. We apologize for the carelessness. S4 refers to the Structured State Space model proposed by Albert Gu in his paper "EFFICIENTLY MODELING LONG SEQUENCES WITH STRUCTURED STATE SPACES".


**Response to Weaknesses 7**

Thank you for your valuable suggestion. Reviewer bobw and Reviewer VW2o have also suggested expanding the scope of our experiments by incorporating additional datasets and comparative models to further validate the generality and superiority of COPU. We have initiated the preparations and hope to complete the additional experimental work within the deadline of the rebuttal period.

**Response to Weaknesses 8**

We respectfully point out that this could be a misunderstanding.
The MLP's final layer is constructed as a temporal-wise fully connected neural network to enable temporal dependencies modeling, as described in line 410. This configuration is equivalent to the DLinear or NLinear approaches discussed in "Are Transformers Effective for Time Series Forecasting?", with the distinction that we do not perform additional series decomposition or normalization operations. Therefore, the experimental results observed by the reviewer corroborate the issues raised in that paper, which is also one of the motivations behind our study. Rather than naively applying architectures from CV/NLP or designing overly intricate modules, it is more productive to deepen our understanding of the modality. Such insight into RR is more practically significant and can guide future research in time series analysis.

**Conclusion**

We are sincere and candid about our work. We believe in our innovativeness and contributions and are willing to take critical suggestions. COPU is not flashy or avant-garde research; it addresses a very down-to-earth issue: Why do complex models often underperform in time series forecasting problems? Unlike CV/NLP, this field lacks a dominant network architecture akin to the Transformer. We have proposed the hypothesis that the time-series modality is less informative compared to CV/NLP, making models from those fields less adaptable. Subsequently, we introduced CEP and COPU to address this issue. Table I demonstrates that COPU's performance indeed significantly surpasses the compared SOTA and basic models. Table II shows that COPU-CEP remains robust to changes in batch size, further validating the necessity of CEP.

The reviewer can refer to ["Response to Weaknesses 1 / VW2o"]() and ["Response to Weaknesses 1 / bobw"]() for further explanation of our motivation if interested. We hope the reviewer can consider raising COPU's rating. We believe that recognizing these modality differences is of significant importance and aspire to contribute this discovery to the community.
Response #1 to Reviewer 1q18
**Response to Weaknesses 3**

We also find this phenomenon intriguing. Traditionally, one would expect that during the training process, networks continuously learn conducive features and construct nonlinear relationships, causing fluctuations in RR, i.e., RR might decrease or increase as the network converges to a few important features or learns numerous nonlinear characteristics. However, intriguingly, in all models except COPU, RR almost remains constant. We specifically conjectured about this phenomenon in our manuscript:

> “Examining Figures 7 and 8 together uncovers intriguing characteristics. Notably, the interval during which the model's validation loss decreases rapidly closely coincides with the sharp rise in RR. The initial 20 iterations mark the rapid convergence phase for each model, during which most models exhibit a significant upward trend in RR. The observations suggest that in the early stages of training, NNs actively search across the wide parameter space, eagerly exploring various feature representation methods relevant to the task at hand. As training progresses and begins to stabilize, the RR settles down; however, the validation loss continues its steady decline. This indicates that the NNs are now fine-tuning the intricate mapping from latent variables to outputs, building upon the feature representation methods they previously discovered. Figure 7 and 8 imply that the window for effective learning, especially the acquisition of feature representation, is remarkably brief. Consequently, the ability of nonlinear models to extract diverse and rich information from inputs becomes critical.”

In summary, we find this phenomenon novel and fascinating, and we believe that the community will also be eager to explore questions such as, **"Why does the network training process not align with our previous assumptions? How can we control the training process to improve performance?"**

**Response to Weaknesses 4**

Thank you for your reminder; we find there lacks a detailed explanation of the lines in Figure 8. Figure 8 serves as a direct evaluation of the performance of each base module in end-to-end experiments. The curves representing the different base modules are indicated in the legend of the figure. The output dimension of the network's final layer is 1.

**Response to Weaknesses 5**

Thank you for your suggestion. Since the author's team focuses on AI applications in industry, we habitually choose experiments from that field. We have conducted extensive experiments on hyperparameters, ablation studies, and significance properties, and we initially believe that the existing experiments are quite sufficient. We will strive to incorporate more datasets to provide a more comprehensive experimental analysis for COPU.

We sincerely hope that our response has addressed the reviewer's concerns and cleared up any misunderstandings. We would greatly appreciate it if the reviewer could consider raising COPU's rating. If the reviewer has any further concerns, we are more than happy to help address them.
Response #2 to Reviewer VW2o
**Response to Reviewer VW2o**

We thank the reviewer for taking the time to assess our paper and giving the valuable feedback. We will meticulously address the reviewer's concerns in the following. Thank you for your comprehensive examination and critical suggestions.

**Response to Weaknesses 1**

We understand the reviewer's confusion. We have endeavored to use concise language to describe the phenomena we have observed.

The term **"ambiguous discriminative patterns"** specifically refers to Figure 1, where two sequences that overlap by nearly two-thirds yield vastly different outputs after a simple convolution, while two entirely unrelated sequences produce similar outputs. Therefore, in the context of time-series analysis, it becomes challenging to discern the influence of input on the system's output. This contrasts with other data modalities. Even a simple convolution system exhibits such behavior in time-series data, let alone complex nonlinear dynamic systems. This is why we stated at the end of the first paragraph:

>“Thus, from the perspective of input, the discriminative patterns among different time series are not only difficult to express mathematically but also inherently ambiguous. Figure 1 vividly illustrates this process using a simple kernel convolution.”

**Time-series data, especially in industrial settings, are less informative compared to other modalities.** Images and sentences (through embedding techniques) can easily have variable dimensions reaching thousands or even tens of thousands, not to mention their rich semantic information. In contrast, time-series data are often recorded by sensors, with variable dimensions typically ranging from tens to hundreds (less than one hundred in industrial contexts). These sensors may even measure the same object at different time delays. Consequently, when the batch size is too large, the number of effective samples (i.e., samples that cannot be linearly represented by other samples) within a mini-batch may be smaller than the batch size. This characteristic is not present in other modalities such as computer vision and natural language processing.

**Modeling time-series data is non-intuitive.** With images and sentences, we can intuitively accomplish downstream tasks through their semantic information, such as image classification, object detection, named entity recognition, machine translation, and so on—even though it is difficult to mathematically represent this process—hence the introduction of neural networks as modeling tools. In contrast, performing tasks like system identification, fault detection, and regression analysis through time-series data is extremely non-intuitive, but mathematical modeling is possible under specific structural assumptions (e.g., subspace identification, Gaussian process regression).

Therefore, when the time-series modeling object is too complex to be represented using traditional mathematical tools, trivially applying networks designed for CV and NLP to extract features may be inefficient. This is because these architectures may struggle to effectively construct nonlinear relationships on less informative data, as shown in Figure 2. We should deepen our understanding of the time-series domain, recognizing that its data are less informative, as depicted in Figure 5. We should not let the scarce and valuable heterogeneous information be overwhelmed by abundant homogeneous information in stacked neural networks, i.e., to monitor and enhance RR, as illustrated in Figures 6 and 7.

**Response to Weaknesses 2**

The first referenced paper studies how repeated nesting of multiple kernel functions can lead to degradation in nonlinear modeling capabilities, ultimately causing the kernel to lose its nonlinear modeling ability. Introducing residual connections within the nested kernel pathways can effectively alleviate this phenomenon, enhancing nonlinear modeling capabilities. The second referenced paper indicates that, in the absence of residual connections, gradients tend to diminish during backpropagation, such that the shallowest layers of the network receive almost no updates, resulting in poor performance in deep networks.

Figure 6 shows that as the number of layers increases, the RR of MLP decreases surprisingly rapidly, indicating that the effective information output by MLP diminishes (i.e., it can be linearly represented by other samples). Conversely, the RR of RES, which incorporates residual connections, remains robust, suggesting that the effective information output by RES hardly decreases with increasing depth. **This phenomenon provides an additional perspective on why residual connections are widely adopted.**

In our effort to include as much information as possible within the limited page constraints, we may have sacrificed some readability. We apologize for any inconvenience this may have caused.
Response #1 to Reviewer VW2o
**Response to Questions 1**

Due to the possibility of negative correlations between sequences, the value of $c$ may be negative. 

Ensuring that $c$ is positive through the sigmoid function has two advantages. First, it avoids numerical errors caused by $c$ being zero. Second, it assigns weights more accurately in cases of negative correlation. 

CEP enhances RR by suppressing homogeneous information and amplifying heterogeneous information. As shown in Equation 1, when performing the dictionary lookup, it is necessary to strengthen the weights of sequences that are dissimilar to the current sequence. The similarity between sequences is measured from two perspectives: the strength $c$ and the distance $ind$. The reason for considering the distance $ind$ is that, in industrial processes, many sensors measure the same data with different time delays, resulting in a large amount of redundant homogeneous information. Such copies of the same feature at different time delays should also be considered homogeneous information and thus be suppressed. Therefore, the expression $\frac{ind}{c}$ represents our requirement. When $ind$ is very small and $c$ is very large, it indicates that there is almost no time delay between the two sequences and high correlation; such homogeneous information needs to be suppressed. When $ind$ is large and $c$ is small, it indicates a significant time delay between the two sequences and low correlation, which can be considered independent information sources; such heterogeneous information needs to be extracted. Generally, $c$ is not negative because we calculate $c$ and $ind$ through argmax. Therefore, when $c$ is negative, it means a very rare situation has occurred where there are no positively correlated segments between the two sequences. This special case needs to be noted and given higher attention. Therefore, it is necessary to make $c$ positive while preserving the magnitude relationships; otherwise, softmax would assign extremely low weights. Hence, we choose to use the sigmoid function to impose this constraint.

**Response to Questions 2**


While CEP can be embedded in other models as well, only COPU can explicitly leverage the various advantages brought by high RR. Figure 8 shows that the improvement of CEP itself over other base models isn't very outstanding. That is because CEP is proposed to address the issue of AOPU's robustness issue to batch size, rather than as an independent end-to-end solution. We believe that once the time-series deep learning community becomes aware of RR, much more outstanding results will emerge.


**Response to Questions 3**

The code for CEP and COPU will be made publicly available after acceptance.


We sincerely believe that the research on COPU and RR is of significant value. We would greatly appreciate it if the reviewer could consider raising COPU's rating. If the reviewer has any further concerns, we are more than willing to address them.
Response #2 to Reviewer bobw
**Response to Reviewer mbUm**

We thank the reviewer for taking the time to assess our paper and giving the positive feedback. Your recognition is a great encouragement to us. We have addressed each concern raised by the reviewer. Thank you for your careful review and valuable suggestions.

**Response to Weaknesses 1**

The structured design of COPU necessitates our study of RR, which distinguishes it from other indicators. In the objective function presented in Figure 4 (i.e., Equation 9), the matrix must be invertible to correctly compute the loss, which naturally leads to an analysis of RR. 

Neural networks such as LSTM, GRU, and Attention are widely used as base modules in many complex network structures to extract abstract features, owing to their outstanding performance in end-to-end tasks under MSE or entropy indicators. Even though these neural networks perform exceptionally well in end-to-end tasks, **it is challenging to quantify and analyze whether they are qualified as feature extractors**, and whether the features they extract are conducive to and consistent with the overall model. This issue, where the optimization objectives of sub-modules are inconsistent with that of the overall model, is referred to as mesa-optimization. A classic example is in CNNs, where the overall model's optimization goal is classification, while the shallow kernels aim to capture local textures and directional vectors. Although we cannot quantify and analyze the texture capture of CNNs in the field of computer vision, we can quantify and analyze the impact of COPU on RR in the time-series domain. 

As RR approaches one, the precision loss in Equation 9 decreases, and the gradient computation of COPU becomes more accurate. When RR equals one, the network's output is designed as the minimum variance estimator, and the gradients of the trackable parameters are conditional natural gradients, which incorporate manifold information and ensure that the network converges faster and more stably. These characteristics are not possessed by other common indicators, such as MSE, T$^2$, MAE, and so forth. Therefore, COPU places emphasis on analyzing RR.

**Response to Weaknesses 2**

Figure 4 illustrates the schematic relationship between CEP and COPU. Section 3, titled COPU: Cross-Correlation Enhanced Approximated Orthogonal Projection Unit on line 237, describes how to achieve COPU from a textual perspective. Specifically, CEP is introduced as the augmentation block in Equation 8. Subsequently, through computations in the dual space and the objective function, we obtain the loss, update the parameters via truncated gradient propagation as shown in Figure 4, and obtain the predicted output through the trackable parameters. The construction methods of the dual parameters and the objective function are shown in Equations 7 and 9, respectively. 

This structured design has a solid mathematical foundation and theoretical assurance, ensuring that when RR equals one, the network possesses the aforementioned excellent characteristics (natural gradients and minimum variance estimation). However, this design is highly sensitive to batch size; when RR is too small, the model may even fail to converge. For this reason, COPU is proposed to enhance model robustness and improve performance.

**Response to Weaknesses 3**

We greatly appreciate the reviewer’s valuable suggestions. Since the authors' team focuses on AI in industries, we habitually chose experiments from the field. The authors' team will endeavor to incorporate more datasets to provide a more comprehensive experimental analysis for COPU.

**Response to Weaknesses 4**

Figures 5 to 14 provide a detailed presentation of the static and dynamic changes in RR under different hyperparameter configurations in neural networks. We particularly emphasize the impact of different network depths on RR, as well as the influences of batch size and sequence length. These experiments offer valuable references for us to determine the hyperparameters of COPU. Except for AOPU, which follows the settings in its original paper, the hyperparameter settings of other comparative models are consistent with those of COPU, as indicated in Section 4.3, line 404.
Response #1 to Reviewer bobw
**Response to Reviewer mbUm**

We are deeply grateful for the time and effort the reviewer has invested in reviewing our manuscript. Your recognition and support are crucial to us. We hold each of your concerns in the highest regard and will address them thoughtfully in the following.

**Response to Strengths**

Thank you for your gracious recognition and commendation. Through our practical deployment of deep learning methodologies, we have found that complex models do not invariably outperform simpler ones, particularly within industrial environments. This phenomenon starkly contrasts observations in other domains such as computer vision. We introduce the research of COPU to unravel the underlying reasons for this phenomenon. We believe that COPU will illuminate new directions for future endeavors in time series analysis within the deep learning community and make good contributions to the field.

**Response to Weaknesses**

The reviewer's observation is accurate. COPU focuses on analyzing how deep learning can properly adapt to time-series modal data through the examination of RR, making it challenging to generalize to other data types. Due to the involvement of time-frequency conversions in CEP, COPU incurs additional computational time overhead compared to traditional network architectures. 

Current research on neural networks in the time-series domain remains heuristic, relying on empirical conjecture and lacking in-depth domain knowledge. COPU proposes a universal monitoring metric in time-series domain that significantly enhances the interpretability of NNs, providing guidance for the design of deep network structures, which is of great significance. Therefore, we believe that some extra computational overhead is acceptable.

**Response to Questions**
1.   Essentially, RR is a metric that evaluates the degree to which a model approximates the minimum variance estimator. The minimum variance estimator is the optimal unbiased estimator and is considered the ceiling for regression tasks. Therefore, the analysis of RR does not involve considerations of discrete or continuous properties. For other data modalities, such as computer vision, the minimum variance estimator may not be their task of interest; hence, we assert that COPU is not suitable for other modalities.

2.   As RR approaches one, the gradient backpropagation of the objective function in Figure 4 becomes more precise, and the neural network acquires more favorable properties. When RR equals one, the gradient of the trackable parameters is equivalent to the conditional natural gradient, which incorporates manifold information and ensures that the network converges faster and more stable. Simultaneously, the network's output becomes equivalent to the minimum variance estimator, ensuring that the neural network can reach the performance ceiling. Conversely, as RR approaches zero, the error in gradient backpropagation increases (due to the matrix inversion in Equation 9), leading to less stable model performance.
Therefore, COPU's emphasis on suppressing homogeneous features and amplifying heterogeneous ones aims to increase RR. The series of experiments from Figures 5 to 8 were conducted to investigate how these elements impact RR, while Tables I and II illustrate how the RR influences performance. We hope this explanation resolves your confusion.

3.   We also find this topic exceptionally interesting and engaging. Through Figures 5 to 14, we meticulously present the inherent RR of the data under different batch sizes and sequence length settings, the RR of different layers during initialization, the dynamic changes of RR in various layers within networks of differing depths during training, and the performance monitoring results of networks with different depths throughout training. This approach enables us to deeply understand the internal changes occurring during the network training process and to determine how to choose the stack depth. We have discovered that depths capable of significantly enhancing RR during training tend to exhibit the best performance, rather than those depths and structures where RR remains constant over time. Our experimental results partially corroborate our findings: when COPU is deeper, RR is higher, and the model becomes more robust; when RR increases substantially, the model's performance also improves.



We believe that COPU's thorough and meticulous analysis of the neural network training process highlights a highly valuable research direction in the time-series domain. We would be grateful if the reviewer could consider raising COPU's rating. If the reviewer has any further concerns, we are more than happy to help address them.
Response #1 to Reviewer mbUm
Yes
N/A
No ethics review needed.
Generalizability of the RR Metric: The paper shows that RR is effective for assessing nonlinear modeling capabilities in time series data, but it is unclear if RR can be reliably applied to other data types (e.g., continuous vs. discrete time series, multimodal data).

Balancing Homogeneous and Heterogeneous Elements: While the approach of suppressing homogeneous features and amplifying heterogeneous ones is innovative, the paper does not detail how the balance between these elements impacts performance. More insight into how this balance can be adjusted and its effect on model performance would be valuable.

Effect of Stack Depth on COPU’s Performance: Although increasing the depth of COPU stacks improves performance, the model exhibits diminishing returns at higher depths. It would be helpful to have guidelines on optimizing stack depth for this model.
Timely and Novel Research Direction: In time series forecasting research, most studies have traditionally focused on increasing model capacity. This paper takes a novel approach by addressing time series analysis from a fresh perspective that emphasizes data-specific structural improvements. This unique angle offers high novelty by shifting from merely increasing the capacity of NNs to a more data-driven design that aligns closely with the characteristics of time series data.

Innovative Methodology: COPU stands out for emphasizing heterogeneous over homogeneous elements in the data, which is well-suited to the nature of time series data. Unlike previous studies that focus on extracting homogeneous features, this paper proposes the Cross-correlation Enhanced Perceptron (CEP) to align features based on their correlations, suppressing redundant homogeneous information while amplifying unique features. This approach enhances the NN's ability to capture nonlinear aspects effectively, making it highly suitable for time series analysis.

Performance and Stability: COPU demonstrates superior performance compared to traditional NN structures, especially in terms of capturing diverse features without overfitting. This model excels in real-world scenarios by effectively learning complex and varied representations in time series data, showcasing both stability and robustness.
This paper, titled COPU: Recognizing Time Series' Heterogeneity in Stacked Neural Network, addresses limitations in traditional neural network (NN) architectures when applied to time series data. The authors propose the COPU (Cross-correlation Enhanced Approximated Orthogonal Projection Unit), a new framework designed to enhance the NN's nonlinear modeling capabilities by emphasizing heterogeneous features. Traditional NN structures often focus on homogeneous feature extraction, which limits their performance on time series data where diverse feature extraction is essential. The COPU framework uses the Rank Ratio (RR) metric to measure the model's ability to capture unique information, enhancing heterogeneous components while compressing homogeneous ones. Experimental results demonstrate that COPU significantly outperforms existing NN structures in capturing nonlinear patterns, especially in time series datasets like SRU.
Limited Applicability: COPU is specifically designed for time series data, which could restrict its application across different data types, such as image or text data. Although the focus on time series data is intentional, this may limit its adaptability to more general purposes.

Model Complexity: The architecture of COPU, particularly with CEP integration, is relatively complex, potentially requiring more computational resources than conventional NN models. This complexity could present challenges for implementation, especially for large-scale datasets or real-time applications.
Yes
No ethics review needed.
1. In the design process of CEP, why may the c value be negative? Why is it feasible to use sigmoid to force a negative number to be positive?
2. Does the CEP module have to be fixed with your network structure or can it be embedded in other models as well?
3. Will the paper provide code? I want to know about the implementation of CEP and COPU.
1. The author clearly demonstrates the problem that this paper aims to solve: that complex networks can effectively solve problems related to computer vision and natural language processing; But its ability to solve the problem of time series analysis is insufficient.
2. The author proposes a CEP scheme, which quantifies the similarity between features through Cross-correlation to ensure alignment without directly ignoring differences in time, which is quite innovative.
3. The author summarized relevant literature, explained the characteristics of time series analysis problems, and proposed that RR index has a good evaluation significance in measuring time series analysis problems, and this index can be relatively convincing in the subsequent research
4. The author cited sufficient literature to demonstrate the point of view, and designed ablation experiments to enhance feasibility.
This paper proposes a new neural network component, called the cross-correlation Enhanced Perceptron (CEP), to solve the difficult problem of nonlinear modeling in time series analysis. CEP facilitates the learning of unique features by performing alignment and cross-correlation calculations on input features in a single step to distinguish and amplify different features while suppressing the influence of similar features. To further improve the model performance, the authors integrate CEP into the approximate orthogonal projection unit (AOPU) to form the COPU framework. COPU uses CEP to enhance nonlinear modeling capabilities and solves the limitations of AOPU in computational accuracy and expressiveness. The experimental results show that COPU is significantly superior to the existing methods in several real regression tasks. The contribution of this paper is to develop a CEP component specially used in time series analysis, which provides a new idea for feature modeling in a specific domain. At the same time, the authors also focus on an evaluation metric Rank Ratio (RR), because it can be interpreted as the proportion of linear dependencies that are translated into independence by neural networks.
1. The advantages of indicator RR compared with other common indicators are not fully elaborated.
2. The relationship between CEP and COPU is not clearly stated, and it should be explained in detail how to achieve it.
3. The datasets used in the experimental design are insufficient, and more datasets are recommended to be added for verification.
4. In terms of experimental Settings, the author should provide setting parameters of different networks to compare the results more reasonably.
Yes
No ethics review needed.
The same to Weaknesses
They introduced RR as a metric to quantify the proportion of linear dependencies transformed into independence by the network. Performing a comprehensive exploration of NN’s nonlinear modeling capability.  This paper proposes a new NN component termed CEP that has strong nonlinear representation performance in time series analysis and serves as the foundation for developing the COPU framework.
This paper think that conventional NN architectures overemphasize homogeneous feature extraction, impeding the learning of diverse features and diminishing their nonlinear modeling capability. To address this gap, they propose the Cross-correlation Enhanced Approximated Orthogonal Projection Unit (COPU) to quantify and augment the NN’s nonlinear modeling capacity.
1.  In page 1, the author said :  For these methods to be effectively applied in this field, it is
essential to recognize that time series has more ambiguous discriminative patterns than other forms
like images and text (Alec et al., 2021). Such ambiguity hinders the model’s ability to extract diverse
features from the input, obstructing its capacity for nonlinear modeling.

Then you give simple explanation, but the problem is you explanation is wrong. You classify original text and image data, while you do convoltuion on timeseries dat, and then you said "it is more challenging to discern the effect of two input sequences on the output
of a system". That is not resonable.


2. Some sentence is hard to understand like:
Page 7: This offers a novel perspective that elucidates the underlying efficacy of residual connections in NNs beyond the kernel (Duvenaud et al., 2014) and gradient (Kaiming et al., 2016) explanations.

3. In Figure 7, except for layer 7, it is hard for me to find difference CEP and other method, and I think with the iteration, the CEP curve will  close to other curve in the end.

4. You need to add explanation to different lines in Figure 8.

5. Last and most important Problem:
You propose a new method to solve the problem in time series prediction, but you did not conduct experiments on popular dataset and compare with some current SOTA models, like you refered in your paper : Are transformers effective for time series forecasting?
Yes
No ethics review needed.
Overall, the paper is not yet ready for publication. Significant improvements are required in both the presentation and the experimental setup and evaluation in order to adequately support authors' claims. While the authors are welcome to respond to these shortcomings, I believe the issues are too fundamental to be addressed through a rebuttal alone and will likely require a major revision of the paper.
- The authors motivate the proposed approach using realistic examples.

- They propose using rank ratio as a metric for measuring nonlinear modeling capability.

- Experiments conducted on real world datasets.
In this paper, the authors present an approach called Cross-correlation Enhanced Approximated Orthogonal Projection Unit (COPU) to quantify and augment a neural network’s (NN’s) nonlinear modeling capacity. The authors motivate their approach based on the observed inability of NNs to learn diverse features and effectively perform nonlinear modeling. The proposed method amplifies heterogeneous components and reduces homogeneous ones, enabling the learning of unique and independent features, thereby enhancing the NN's nonlinear modeling capability.
- The presentation of the paper is quite confusing. There are several mistakes, such as Eq. (6) not being referenced anywhere, and in line 251, the notation "x \in \mathbb{R}^{d,b}" is unclear. Additionally, there are several sentences that are difficult to follow.
For example: 

- "This process enables the differentiation between redundant information and innovation, further suppressing homogeneous information among features while amplifying their differences"
what innovation means here?

- "while the augmentation modular focuses on diverse and informative feature extraction"
module?

- "We implement this approach by  employing dictionary lookup Ashish et al. (2017) to achieve feature mapping."
wrong reference type.

- Using scientific notation for MAPE (which is typically presented as percentage) is also confusing.	

- S4 acronym not introduced.

These are just a few examples, but there are numerous similar presentation issues throughout the paper, making it very difficult to follow. This is a significant shortcoming in itself, bringing the paper below ICLR  standards.

- Results on standardized and well-known datasets are expected to support the authors' claims and facilitate the reproduction of the presented results.

- In many cases, an MLP performs similarly to (or better than) the proposed approach, raising questions about the actual difficulty of the datasets used in the evaluation and whether the results truly support the authors' claims. The authors should also provide the results of a linear model to highlight the necessity of modeling non-linear components.