KAAN: Kolmogorov-Arnold Activation Network --- a Flexible Activation Enhanced KAN
Kolmogorov-Arnold representation Theorem;Kolmogorov-Arnold Network;Multi-Layer Perceptrons
Yes
No ethics review needed.
1. I would suggest authors to reevaluate the core contributions and rewrite the paper. If the main contribution is empirical in nature, I would suggest doing more experiments on transformer-like architectures or showing taks where MLPs or KANs fail to learn underlying functions correctly but the proposed method can.
2. What is the meaning of “KAN faces the challenges of being unintuitive and inflexible.” This is a highly subjective statement, giving concrete examples of what inflexible and unintuitive means would help readers. Does KAAN help give more flexibility or intuition? If so, how? What is the takeaway?
1. The paper is written clearly and concisely, and is easy to read. 
2. The proposed activation makes KANs more flexible and easy to deploy which would encourage the scientific community to experiment with these networks.
3. Experiments clearly demonstrate that the proposed activation function allows KANs to be trained while achieving comparable performance to MLPs and even ResNets.
Authors propose Kolmogorov-Arnold activations inspired from KANs (Kolmogorov-Arnold Networks) and replace B-splines in KANs to achieve similar or better performance than MLPs. Authors show that MLPs can be represented in a form conforming to Kolmogorov-Arnold representation Theorem (KAT). Using MLP-like equipped with Kolmogorov-Arnold activations, authors experiment and compare different basis functions. Experiments also demonstrate successful integration with Convolutional Neural Networks (CNNs) which achieving comparable performance.
1. Novelty is missing: KAN arxiv report (Liu et. al 2024) already gives a MLP-like interpretation of KANs which allows stacking of layers similar to MLPs which is similar to section 3 in the paper.
2. Authors have essentially replaced splines, which is a core contribution of the original KAN paper (provides higher degree of control to model univariate functions) with learnable activation functions. There is already literature covering learnable activation functions with different basis like Polynomial or sinusoidal basis (in context of MLPs). Therefore I feel the paper doesn’t bring new insights into Neural Networks or KANs.
Yes
No ethics review needed.
1. What are the parameter counts/VRAM consumption/running time for the tested KAANs vs. MLPs/CNNs? 
2. Is it possible to compare KAANs with standard networks that use the same number of parameters?
1. The proposed approach clearly works on the presented tasks, and in some cases provides a performance improvement.
2. The KAAN parametrization is compatible with standard ANN architectures.
3. Related to the previous point, this parametrization might be helpful for neural architecture search/meta-learning/similar approaches that adapt neural networks’ architectures, as the nonlinearity parameters are designed to be differentiable.
This paper presents Kolmogorov-Arnold Activation Networks, an extension of Kolmogorov-Arnold Networks, that uses MLP/CNN-like architecture with flexible activation functions defined for each edge between neurons.
2. Memory and computation time requirements

The computational requirements of KAANs appear to be much higher than for corresponding standard MLPs/CNNs. Eq. 6 uses several weights per connection (one for each activation type) and additionally parametrizes the activations. This should increase both memory consumption and running time of KAANs compared to standard networks. 

The increased number of parameters in KAANs also (unless I missed something) suggests the performance improvements (Tabs. 3-5) are very modest compared to standard networks that use several times fewer parameters. 

3. Lack of interpretability

Throughout the paper, KAANs are called intuitive. However, I do understand how KAANs are more intuitive than standard MLPs (if anything, they are more convoluted). The results in Tabs. 3-5 indirectly confirm my concern: there’s no clear winner across different combinations of activation functions.

Lines 300-311 discuss the potential uses cases for each activation function, but all of those apply to standard ANN architectures that don’t define edge-based nonlinearities. 

4. Poor writing

The paper needs some writing improvements. Here are some instances I’ve noticed, although text needs overall polish.
1. [Line 30] “There were not many breakthroughs until KANs” [rephrased] – I would disagree, and suggest Transformers as an obvious architectural breakthrough. But, the list can expand with for instance capsule networks (https://www.sciencedirect.com/science/article/pii/S1319157819309322) and gflownets (https://arxiv.org/abs/2111.09266). 
2. The introduction contains many terms, such as LANs and TANs, but they’re not cited until related work. 
3. “No many” instead of “not many” in line 30, extra bracket in line 81, typo in line 205, non-plural “Experiment” name for Sec. 5
Yes
No ethics review needed.
My opinion could shift towards acceptance if the authors could address one of the following points:

Develop a method to identify the most optimal combination of basis functions.

Find a specific combination of basis functions that consistently outperforms others.

Demonstrate that in certain specific tasks, KAANs offer a significant advantage.
The theoretical framework is elegantly and solidly constructed.

It points out that “MLP represents a specific instance of KAN”

It points out that“any continuous univariate basis functions can be used as activation function”

KAANs offer greater flexibility and fewer limitations than traditional KANs, making them more adaptable to various structures.

The paper conducts extensive experiments across a multitude of AI-related tasks.
This paper introduces a novel architecture named KAANs, which enhances the efficiency of MLP by incorporating a method inspired by KANs. Theoretically, the paper begins by establishing that MLPs are a subset of KANs and then deviates from traditional KANs by replacing B-spline activation functions with linear combinations of basis functions. Experimentally, the paper evaluates 7 different combinations of basis functions as activation functions across various AI-related tasks, demonstrating that KAANs achieve higher accuracy than both MLPs and KANs.

While the theoretical foundation is robust and compelling, the KAANs just replace activation functions in MLPs with more complex functions. when trying to search for the optimal combination of basis functions along with the most effective weights, the concept go back to the learnable activation functions.  Therefore, it appears that the paper has elegant theory but not enough contributions on practical level.
The paper experiments with various combinations of basis functions, where different combinations excel in different tasks. This variability raises questions about how to determine the most effective combination for a given task.

Although KAANs outperform MLPs and KANs in the experiments, the comparison may not be entirely fair. The more complex activation functions used in KAANs require greater computational power compared to MLPs, potentially skewing the results. Similarly, comparing KAANs to KANs without adjusting for KANs' longer training requirements may not provide a balanced view of their respective efficiencies.
Yes
No ethics review needed.
1. It would be interesting to elaborate more on the perspective in Sec 3.2 and gain more motivation on the comparison between KANs and MLPs.

2. How does (C)KAAN perform on more challenging tests?
1. The authors proposed a novel framework of viewing MLPs and a special case of KANs 

2. They conducted extensive experiments on challenging datasets including Tabular datasets and Cifar-10, and introduced a convolutional version as well.
The authors proposed a novel framework of viewing MLPs and a special case of KANs and proposed as a inspiration KAAN, where each nonlinear activation function is parametrized by a linear combination of basis functions. They conducted extensive experiments on challenging datasets including Tabular datasets and Cifar-10, and introduced a convolutional version as well. The article presented an interesting perspective and should be treated as a nice improvement on KANs, with the following limitations.

1. While KAAN seems interesting, it seems still such a way of parametrization of nonlinearity in KANs, with more complicated nonlinearity. This improvement is at best incremental and would need more support from numerical evidence.

2. The referee would envision that KAANs suffer from less interpretability than KANs; especially on symbolic regression. Could the authors comment on this restriction?

3. It would be interesting to elaborate more on the perspective in Sec 3.2 and gain more motivation on the comparison between KANs and MLPs.

4. How does (C)KAAN perform on more challenging tests?
1. While KAAN seems interesting, it seems still such a way of parametrization of nonlinearity in KANs, with more complicated nonlinearity. This improvement is at best incremental and would need more support from numerical evidence.

2. The referee would envision that KAANs suffer from less interpretability than KANs; especially on symbolic regression. Could the authors comment on this restriction?
@inproceedings{
anonymous2024kaan,
title={{KAAN}: Kolmogorov-Arnold Activation Network --- a Flexible Activation Enhanced {KAN}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=3VOKrLao5g},
note={under review}
}
Kolmogorov-Arnold Networks (KAN) have led to a significant breakthrough in the foundational structures of machine learning by applying the Kolmogorov-Arnold representation theorem. Through this approach, the target conditional distribution is expressed as the summation of multiple continuous univariate B-spline functions. However, KAN faces the challenges of being unintuitive and inflexible. To address this issue, we analyze the structural configurations of Multi-Layer Perceptrons (MLPs) and KANs, finding that MLP can be represented in a form conforming to Kolmogorov-Arnold representation Theorem (KAT). Therefore, we propose MLP style KAN framework Kolmogorov-Arnold Activation Network (KAAN), which is more intuitive, flexible and transferable. To verify the flexibility and transferability of our approach, we extend it to Convolutional Neural Network (CNN). Also, we demonstrate that parameter sharing is beneficial not only for efficiency but also for effectiveness. KAAN shows better representation capacity than MLP on several benchmarks. Furthermore, our experiment results lead us to conclude that this method is feasible for integrating modern network approaches such as CNNs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Kolmogorov-Arnold representation Theorem Kolmogorov-Arnold Network Multi-Layer Perceptrons
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7d6f9894896cebea42c320cce2d79f7174c06ec5.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/b403efd039b2bac6ed4967c70a81be67ce993dde.zip
KAAN: Kolmogorov-Arnold Activation Network --- a Flexible Activation Enhanced KAN
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission