Unbounded Activations for Constrained Monotonic Neural Networks
Monotonic;Neural Networks
Yes
No ethics review needed.
Why is the empirical improvement so marginal, despite the theoretical insights shown in the paper?
* The paper proposes a novel construction for monotonic MLPs with unbounded activation functions by introducing an activation function whose direction is dynamically determined depending on the sign of the weight in the linear layer. 
* The paper theoretically proves that the constructed MLPs are universal approximators for monotonic functions.
The paper is about constructing monotonic MLPs with unbounded activation functions, to ease the optimization and mitigate the issue of saturation when bounded activation functions are used in prior works. The proposed construction involves an activation function whose direction is dynamically determined depending on the sign of the weight in the linear layer.
*  The goal of the paper is on addressing optimization issues, but empirical gains are very limited. There is no gain on three out of the five experimented datasets. On the other two datasets, although the mean of the results is improved, the variance is also much larger. Overall, the empirical improvement is quite marginal, which makes the paper not convincing enough. 
* The experiment section is also very short and the authors have not investigated the results further. For example, there is no result showing whether the optimization has been eased (with loss curves, scaling behavior, etc.) 

Thus, on the empirical side, I think the paper is still quite preliminary and not yet ready for publication.
Yes
The paper uses the COMPAS dataset, which has also been utilized in related studies that serve as benchmarks in this field. Given that research on monotonicity often overlaps with fairness studies, the use of this dataset can be justified. However, it is essential to address the ethical concerns associated with the COMPAS dataset. This dataset has been widely criticized for potential biases and fairness issues, as it includes sensitive data and has been shown to sometimes produce biased outcomes against specific demographic groups. Including a discussion on these ethical issues, as well as the measures taken to mitigate potential bias in the analysis, would strengthen the paper’s ethical rigor and transparency.
Yes, Discrimination / bias / fairness concerns
1) If I understand correctly, since they use "point-symmetric activation functions" simultaneously, wouldn’t the authors' claim in "We prove that contrary ...  even convex ones like ReLU, is a universal approx"(at contribution section in Introduction) that it is "even convex once" be incorrect? 

2) In the text, does "Constrained MLP" actually refer to "Constrained MNN"?

3) Is there any other reason why the experimental results from paper [3] were excluded?

4) It would be better to cite the officially published versions of papers, including paper [3], rather than the arXiv versions where possible.

5) Based on my understanding of the paper, it seems that having both "convex" and "concave" types of activation functions within the network is a more important key point than simply having "left-saturating" and "right-saturating" activations. Could you explain the difference between these two concepts in more detail?

(minor)
1) page9 line 473 "The only requirement for this method to work, is to have the input features to be", Isn't it an incomplete sentence?
$\bullet$ The proof is constructed with solid mathematical rigor.

$\bullet$ The proposed method (activation switch) is explained clearly and in an easy-to-understand manner.

$\bullet$ The experiments on real-world datasets for the proposed method were conducted appropriately.
This paper proposes a new neural network structure to reflect monotonic relationships between a specific features and the output in neural networks.  Traditional monotonic MLPs rely on non-negative constrained weights and saturating activation functions (e.g. Sigmoid, Tanh) to maintain monotonic relationships, which can cause optimization difficulties and limit expressiveness. The paper proves that by using alternating left-saturating and right-saturating monotonic activation functions, an MLP with either non-negative or non-positive constrained weights can serve as a universal approximator for monotonic functions. Additionally, the authors propose a new parameterization method (activation switch) that eliminates the need for weight constraints, thereby enhancing optimization stability and performance. Experimental results demonstrate that the proposed method achieves better approximation accuracy than existing monotonic neural network structures while preserving monotonicity and universal approximation properties.
Weakness 1. The categorization of related literature is unclear, and explanations are insufficient. 
-  The explanations of related work in Section 1 (Introduction) and Section 2 (Related Work) are incomplete and would benefit from integration. (soft vs hard / CONSTRAINED MONOTONIC ARCHITECTURES vs HEURISTIC AND REGULARIZED APPROACHES)
-  There is an unclear expression on page 1, line 49-50. ("Such guarantees usually come at the cost of effectiveness.") Please explain the following sentence more clearly.

Weakness 2. Additionally, many relevant references are missing([1], [2]). 

-   Both papers [1] and [2] are recent works in the field and should be included as comparison targets as related works. These studies should be cited in the paper.


Weakness 3. There are doubts about the contributions claimed by the authors, and most aspects raise concerns regarding novelty.

-  Despite the authors' efforts in their proof, it seems that the essential point is that a monotonic neural network requires positive weights (or an even repetition(e.g. $2$-layers, $4$-layers ... $2n$-layers) of negative weights) and a structure with activation functions that support monotonic increasing properties (including both convex and concave functions). I believe the proposed structure $activation switch$ in this paper does not deviate from this approach in the broader sense. If I missed anything, please explain more clearly how the proposed method differs from existing methods.

Weakness 4. The practical issue of gradient vanishing problem.
-  Concerns about gradient vanishing should be critically considered, especially given that monotonic neural networks can generally serve as universal approximators for arbitrary monotonic functions even with a shallow depth of 4. In fact, papers [2] and [3] report better performance than the proposed method and appear to have much simpler architectures, using lower depth network. 

- As I understand it, using batch normalization along with existing methods seems to significantly alleviate the gradient vanishing problem. Is my understanding correct?



Weakness 4. The experimental results in this paper exclude several recent works in the field from comparative analysis. Although I do not believe that solely achieving SOTA performance defines the contribution of this study, there are concerns about cherry-picking in the reported results, or revisions may be necessary to better reflect the authors' claims in the experimental section.

- Paper [3] is mentioned in the main text but was excluded from the experiments. Paper [3] uses exactly the same datasets as the authors, and even shows better performance on some of them. To address concerns about cherry-picking, paper [3] should be added to the experiment.
- Paper [2] also constrains weights to be non-negative through re-parameterization (exponential transformation) and uses a saturated activation function. However, it achieves very good performance (SOTA) on several datasets. So, paper [2] also should be added to the experiment.

Weakness 5. (minor) There are multiple comma-separated phrases within single sentences, or sometimes incomplete sentences, making it difficult to understand.

- page1, line 71-74
- page3, line144-146
- page7, line 326-328
- page7, line 332-334
- page9, line 473 (incomplete sentence)

Weakness 6. It would be beneficial to include a discussion on the ethical considerations associated with using the COMPAS dataset. Given that this dataset involves sensitive information and has been widely discussed in terms of fairness and bias, it would strengthen the paper to address potential ethical concerns and how these were considered in your analysis.

References

[1] Yanagisawa, H., Miyaguchi, K., & Katsuki, T. (2022). Hierarchical lattice layer for partially monotone neural networks. Advances in Neural Information Processing Systems, 35, 11092-11103.

[2] Kim, H., & Lee, J. S. Scalable Monotonic Neural Networks. In The Twelfth International Conference on Learning Representations.

[3] Nolte, N., Kitouni, O., & Williams, M. (2022, September). Expressive monotonic neural networks. In The Eleventh International Conference on Learning Representations.
Yes
No ethics review needed.
1. Could the authors clarify why ReLU and similar unbounded activations were chosen over traditional bounded activations in monotonic settings? What specific advantages do these activations bring in practical applications, beyond the theoretical universal approximation guarantee?

2. How does the proposed parameterization handle sensitivity to weight initialization? Did the authors test different initialization schemes, and if so, which were most effective? Further details here would be valuable for readers aiming to replicate the setup.

3. The experiments focus on a limited set of datasets. How does the model perform in other domains requiring monotonicity, such as environmental modeling or medical risk assessment? Expanding the range of tested applications could reinforce the paper's claims of generalizability.

4. The paper mentions that the new parameterization requires double matrix multiplications for weight splitting. Did the authors measure the computational overhead introduced by this approach? Reporting training times or memory usage compared to other methods would give insights into the scalability of this model.
1. The paper introduces a new parametrization technique for monotonic MLPs, which removes the need for weight constraints and allows the network to adjust activations dynamically. This could improve flexibility and ease of optimization.

2. The authors provide a new theoretical result showing that MLPs with alternating unbounded activations are universal approximators for monotonic functions. This strengthens the theoretical foundation of monotonic neural networks.

3. By focusing on monotonic architectures, the paper addresses a key need in applications where interpretability is critical (e.g., fairness and transparency), making this work relevant for real-world deployment.
This paper introduces a new method for building monotonic neural networks using unbounded activations like ReLU, avoiding traditional weight constraints. The proposed architecture dynamically adjusts activations to ensure monotonicity, simplifying optimization. Theoretical results show it can approximate any monotonic function, and experiments demonstrate improved accuracy on several datasets. This approach enhances the usability of monotonic networks in interpretable AI applications.
1. The experiments cover only a small number of datasets. Extending the evaluation to more diverse datasets and tasks would provide stronger evidence of the model’s generalizability and practical utility.

2. Although the authors propose a solution to avoid weight constraints, they do not fully address potential optimization challenges, such as sensitivity to initialization or convergence rates, which may impact the method’s robustness.

3. The paper’s theoretical section relies heavily on specific activation behaviors without enough real-world validation, which may limit the practical applicability of the theoretical claims.
Yes
No ethics review needed.
I have an additional question regarding the algorithm. The authors convert the constraint on networks weights to constraints on the prediction, i.e., design different forward pass for positive and negative weights. They simply claim this would mitigate the optimization difficulty because seemingly this will not lead to gradient vanishing. However, I did not find sufficient theoretical/experimental results supporting this other than the main results presented in Table 1. It would be better if the authors could clarify this.
The theoretical results are sufficiently impressive, and the proof techniques are clear and well-motivated. I found the proof easily understood. The authors also support their theorems with adequate numerical experiments.
This paper studies learning monotone functions with networks. The authors show a powerful expressivity result: non-negative weight networks with left- and right-saturated activation functions can approximate all monotonically increasing functions (thus all monotone functions via sign flip). They further show that non-positive weights lead to universal approximation of monotone functions even with convex activations such as ReLU, while these activations combined with non-negative weights can only approximate convex monotone functions. Overall, the authors present novel and impressive theoretical results.

This work further proposes an algorithm to utilize their theorems, achieving sufficient improvements on the tasks they studied.
The paper is barely written and not sufficiently revised. For example, numerous typos exist: Line 267, I suppose a $\gamma$ is missing before $1_A(x)$; Fig 2, caption looks messy; Line 473, a sentence is incomplete. I suggest all authors, especially those senior, to carefully revise the paper end to end since the current writing is a bit in fragments.

Improvements achieved in Table 1 looks marginal except for the heart disease dataset. However, I would not indicate negative opinions due to this as this is acceptable for this mostly theoretical paper. In contrast, I find this quite interesting because they were able to design a trick other than constraining the weights to enforce the monotonicity constraint.
This paper proves that constrained MLPs are universal approximators for a broad class of activations,  This includes most modern activations, even convex ones like ReLU. A novel parametrization is introduced, reducing sensibility to initialization.
@inproceedings{
anonymous2024unbounded,
title={Unbounded Activations for Constrained Monotonic Neural Networks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=N1DKrLIKhT},
note={under review}
}
Monotonic multi-layer perceptrons (MLPs) are crucial in applications requiring interpretable and trustworthy machine learning models, particularly in domains where decisions must adhere to specific input-output relationships. Traditional approaches that build monotonic MLPs with universal approximation guarantees often rely on constrained weights and bounded activation functions, which suffer from optimization issues. 
In this work, we prove that non-negative constrained weights MLPs with activations that saturate on alternating sides are universal approximators for the class of monotonic functions. Thanks to this new result, we show that non-positive constrained weights MLPs with convex monotone activations, contrary to their non-negative constrained counterpart, are universal approximators. 
Despite such guarantees, we also show that such classes of MLPs are hard to optimize. Therefore, we propose a novel parametrization that eliminates the need for weight constraints, allowing the network to dynamically adjust activations based on weight signs, thus enhancing optimization stability and performance. 
Experiments demonstrate that our approach maintains theoretical guarantees and significantly outperforms existing monotonic architectures in approximation accuracy.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Monotonic Neural Networks
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ce064e472fe59cce33e462f4d9ba32460e1d4fa4.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/594bd60c3a57ba38fb1ae758a4329acac8c1273f.zip
Unbounded Activations for Constrained Monotonic Neural Networks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission