Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis
Weight initialization;Signal propagation;Physics informed neural networks
Yes
No ethics review needed.
See above in weakness section.
The strength of this paper lies in the development of a novel weight initialization technique that facilitates faster convergence and enhances performance in physics-informed neural networks (PINNs) utilizing the tanh activation function.
The paper introduces a novel weight initialization technique specifically designed for neural networks using the tanh activation function. This technique is evaluated against the well-known Xavier initialization method using benchmark datasets. The experimental results demonstrate that the proposed initialization method enhances the convergence speed of Physics-Informed Neural Networks (PINNs) utilizing the tanh function, showing greater robustness to variations in network size. The findings indicate that the new initialization technique outperforms Xavier initialization in solving various problems related to Partial Differential Equations (PDEs).
1. The comparison of the proposed weight initialization technique solely with Xavier is insufficient; it should also be experimentally evaluated against other state-of-the-art weight initialization methods.
2. Using tanh activation function in entire neural network is not good practice that it has the drawback of the vanishing gradients for the very high and very low values of x.
3. The formulation is more complex than standard methods, which could complicate implementation as shown in Equation (1). 
4. The optimal value of 𝛼 can be highly context-dependent, varying across different architectures, datasets, and tasks, which makes it less universally applicable. Additionally, the choice of 𝛼 can interact with other hyperparameters, such as learning rate and batch size, complicating the overall tuning process during backpropagation, as described in Equation (2).
5. In Section 4.1, the evaluation process utilizes three datasets—MNIST, FMNIST, and CIFAR-10—employing the tanh activation function in every layer. As shown in Table 2, as the number of hidden layers increases, loss gradually increases, which is indicative of overfitting. It would be more effective to use the proposed weight initialization in conjunction with state-of-the-art architectures for training deep neural networks.
Yes
No ethics review needed.
In sections 4.1 and 4.2, the experiment trains for 20 or 40 epochs. Do networks converge to their best accuracy? What is the difference in accuracy when training for more epochs?

The experiments in section 4 are not too computationally intensive, is it possible to include more
data sets or PDE can show the improvements are general?

Can the code used in the experiment can be provided to improve reproducibility?
Unlike optimization problems with theoretical guarantees on fixed points, weight initialization
is an important task in deep learning. An initialization scheme with theoretical backing can
have a long-lasting impact, even just for a sub-field of deep learning.

Experiments do show significant improvement when using PINNs to solve PDE.
This work first provides a theoretical analysis of weight initialization when exclusively using tanh
as an activation function. Providing reasons for the clustering behavior when initializing networks.
Based on the developed theory, an initialization scheme is proposed, and the hyperparameter σz
is empirically determined. There are two types of experiments mainly comparing the proposed
initialization with Xavier. The first type of experiment concerns the classification accuracy in the
early training phase. Showing there is an improvement in accuracy across different data sets and
configurations. The second type of experiment concerns solving PDE with PINNS, showing that the
proposed method has a good performance.
The impact depended on exclusively using tanh as an activation function is fundamentally
beneficial in PINNs. As the current state of the paper, there is not enough support for this.

Given that the experiments are not too computationally intensive and the experiment section
only considers a few data sets or PDEs, the demonstrated improvement may not be general.
Yes
No ethics review needed.
The novelty of this work is strong, and the topic sounds interesting. However, the writing and structure should be revised again. There are some questions that the authors should be concerned about. 

1. In Eq. 2, $\sigma_{z}$ is set to $\alpha/\sqrt{N^{l}-1}$ and $\alpha = 0.085$. From Figure 2, we can find that the optimal value of $\alpha$ is $0.085$. Is there any theoretical reason why $\alpha$ should be $0.085$. Or should we manually try the value accordingly?

2. Additionally, please scribe what is $\alpha$. There is no definition of $\alpha$.

3. In Figure 3 (c) and (d), the proposed method seems to decrease after 6 epochs. Although the accuracy curve can rapidly reach the peak (faster than Xavier), the robustness of this method also should be discussed. For example, the initialization method can first provide prior knowledge to neural networks, but if it can keep the stability of training or not. Or is it the reason for the high $\alpha$?

4. In Appendix A.1, the authors discussed different conditions. When $x = 0$, whether the vanishment problem will abscond. Please highlight the strategy on how this method can process it.
1. this article proposes one novel weight initialization method for the FFNN and PINN.
2. the authors prove that the activation values cannot vanish when increasing the depth of the neural network by using a fixed-point analysis.
Weight initialisation is an old topic, and most studies have verified that weight initialization methods can improve the performance of neural networks. However, because that neural network’s depth increasing rapidly,  most neural networks, especially Feedforward Neural Networks (FFNNs) should face the gradient vanishment problem.  In this article, authors proposed one weight initialisation method for FFNNs and Physics-Informed Neural Networks (PINNs). Based on an analysis of the fixed points of the function tanh(ax), this method determines values of $a$ that prevent the saturation of activations during the training progress. In terms of robustness, this method presents a stronger and more efficient performance. In the experiment, verified on MNIST, Fashion MNIST, and CIFAR10 datasets, this method also shows acceptable results compared to the Xavier.
1. The universality of this method should be improved. For example, more experiments on the feasibility of other neural networks except FNN and PINN.
2. The details of hyperparameters should be mentioned, such as what is the $Threshold$ of FFNN, which training strategy (supervised or unsupervised) of FFNN is used?   
3. FFNN is specifically designed to visualize the trained features, which also should be discussed.
4. The consistency of results should be guaranteed. For example, in Figure 3 (c) and (d), there is a different trend (Xavier tends to equal 
the proposed method), which also should be discussed. In case that after 20 epochs, the performance would be totally different to the presented result.
Yes
No ethics review needed.
1. Layer norm is added to handle the diminishing activation problem. Any reason why you did not compare the performance of the proposed approach with Xavier weight initialization with Layer norm?

2. In equation 2, the paper claims that a_i^(k+1) follows normal distribution with unit mean. But when number of neurons in layer l-1 (N_(l-1)) is greater than number of neurons in layer l (N_l), then the mean can be greater than 1. When N_(l-1) = 2 * N_l, the mean will be 2. This is not clearly discussed in the paper. If the mean is > 1, then that can lead to tanh saturation.
Originality
The paper presents a novel weight initialization method specifically designed for tanh-based neural networks, addressing an understudied area in neural network initialization. This approach is distinct in its use of fixed-point analysis to prevent activation saturation and improve training robustness across network sizes, particularly in the context of Physics-Informed Neural Networks (PINNs). By emphasizing robustness and performance consistency in both traditional classification tasks and PINNs, the paper makes a valuable contribution to the field of weight initialization. The originality is strong, given the lack of prior research focusing on tanh-based initialization methods.

Quality
The paper demonstrates high quality in both theoretical and experimental aspects. The method is grounded in rigorous mathematical analysis, leveraging fixed-point theory to derive conditions that ensure stable activation propagation. The provided lemmas, proofs, and propositions add credibility and depth to the approach.
Experiments are well-designed and span various network configurations, datasets (MNIST, Fashion MNIST, CIFAR-10), and applications (PINNs for solving differential equations). The results consistently show that the proposed method outperforms Xavier initialization, particularly in deeper networks and varying network sizes.

Clarity:
Overall the paper is very well written, with some exceptions mentioned in the weakness section. All the sections in the paper are laid out clearly. The notations are consistent across the paper.
The paper proposes a new method to initialize weights for FCNNs and PINNs with tanh activation. The paper claims that the proposed weight initialization method will not lead to diminishing activations for very deep networks unlike Xavier weight initialization. The paper claims that the proposed weight initialization is robust to network depth and number of units in hidden leayers.
Significance:
The paper compares their proposed weight initialization with Xavier weight initialization for FCNN with tanh activation. Xavier is known to show diminishing gradients and activations problem for deeper networks, but this is solved by using layer normalization. I am therefore considering this work not significant because the problem that the authors are trying to solve does not exist for Xavier + Layer norm and the authors did not do any comparative analysis with and without layer norm.

Other major issue:
1. In equation 2, the paper claims that a_i^(k+1) follows normal distribution with unit mean. But when number of neurons in layer l-1 (N_(l-1)) is greater than number of neurons in layer l (N_l), then the mean will be greater than 1. When N_(l-1) = 2 * N_l, the mean will be 2. This is not clearly discussed in the paper. If the mean is > 1, then that leads to tanh always saturated.
@inproceedings{
anonymous2024robust,
title={Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Es4RPNDtmq},
note={under review}
}
As a neural network's depth increases, it can achieve strong generalization performance. Training, however, becomes challenging due to gradient issues. Theoretical research and various methods have been introduced to address this issue. However, research on weight initialization methods that can be effectively applied to tanh neural networks of varying sizes still needs to be completed. This paper presents a novel weight initialization method for Feedforward Neural Networks with the tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, our proposed method aims to determine values of $a$ that prevent the saturation of activations. A series of experiments on various classification datasets demonstrate that the proposed method is more robust to network size variations than existing methods. Furthermore, when applied to Physics-Informed Neural Networks, the method exhibits faster convergence and robustness to variations of the network size compared to Xavier initialization in various problems of Partial Differential Equations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Weight initialization Signal propagation Physics informed neural networks
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9b444528195fc0b06bd65eef9a35f203a5ef087a.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission