BioNAS: Incorporating Bio-inspired Learning Rules to Neural Architecture Search
Feedback Alignment;Neural Architecture Search;Learning Rules;Biologically Plausible
Yes
No ethics review needed.
Whether it is applicable to larger datasets.
The integration of bio-inspired learning rules with NAS is a fresh perspective that could significantly contribute to the field of neural network optimization. The paper is well-organized and clearly written, making it accessible for readers with varying levels of expertise in NAS and bio-inspired techniques.
The paper explores a method that combines bio-inspired learning rules with Neural Architecture Search (NAS) to optimize the performance of neural networks. The authors propose BioNAS, a framework capable of automatically searching for the best architecture and learning rules. Through a series of experiments, particularly on the CIFAR-10 and CIFAR-100 datasets, the paper demonstrates the advantages of BioNAS in terms of accuracy and adversarial robustness, comparing it with existing backpropagation training methods.
Is there an analysis of energy consumption.
Yes
No ethics review needed.
Questions:
1. How might other forms of bio-inspired learning algorithms (e.g., Hebbian, predictive coding, target propagation) be integrated in the proposed framework?
2. Does the distribution of different learning algorithms in the generated networks show interesting patterns? And if so, what might be the (intuitive) explaination for generating such patterns, what might be the advantages for such patterning, and can such patterns be generalized to different network architectures to produce competitive performance?
Adversarial attacks on generated bio-inspired networks are introduced to illustrate the advantage of their framework.
This paper proposes a neural architecture search framework, BioNAS, that supports certain types of biologically plausible learning algorithms. The framework is built upon existing NAS framworks DARTS and EG-NAS, and incorporates different feedback alignment techniques from BioTorch. The authors compare the performance of their BioNAS generated networks against some previous works and claim they can get state-of-the-art bio-inspired network performance on several benchmarks.
Although the direction of this work is interesting, my major concern is that this work appears to be prelimilary, and some of the key conclusions are *overclaimed*.  

Major Weakness
1. The authors claim that their proposed BioNAS is designed for bio-inspired networks, but currently only several variants of FA algorithms are supported, making this work less significant, and their claim should be toned down.
2. The proposed BioNAS is implemented by adding operations backended by BioTorch to existing NAS frameworks, which does not show much novelty nor effort.
3. The authors claim that they can get state-of-the-art bio-inspired network performance on several benchmarks, but for CIFAR-10 they only compare against two previous Hebbian-based results, and for CIFAR-100 and ImageNet16-120 no previous results of bio-inspired algorithms are shown.
Yes
No ethics review needed.
One important concern is whether the authors have verified if there is a gradient obfuscation issue within their model. The observation that the multi-step PGD attack performs worse than the single-step FGSM attack may indicate potential gradient masking or unsuccessful attacks [1]. This issue can often lead to misleading conclusions about a model’s robustness.

[1] Athalye, Anish, Nicholas Carlini, and David Wagner. "Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples." ICML 2018.
The authors utilize a more bio-plausible learning rule instead of the commonly used BP algorithm, which is interesting.

The authors provide strong empirical support for their approach by conducting extensive experiments across multiple datasets and varied settings.
This paper presents BioNas, a neural architecture search algorithm tailored for bio-inspired neural networks. BioNas focuses on biologically plausible learning algorithms and simultaneously optimizes both the network architecture and the learning rule. The proposed framework achieves state-of-the-art performance on the CIFAR-10 and CIFAR-100 datasets, demonstrating its effectiveness in enhancing bio-inspired model design.
The motivation of combining FA approaches and NAS is not clear. While the concept of applying different learning rules to train SNNs is intriguing, the combination of NAS and FAs in this context raises questions. NAS traditionally involves exploring a vast space of candidate models to find an optimal architecture with significant computational cost. Its primary aim is to achieve high performance without regard to training efficiency, which contrasts with the efficient and light weight bio-inspired learning rules. 

The novelty is limited, as the BioNAS framework closely resembles DARTS and EG-NAS. The primary difference introduced here is the addition of various existing learning rules to the search space.
Yes
No ethics review needed.
1. Are the attack image gradients for the white-box attack method calculated by end-to-end backpropagation or the searched learning algorithm?

2. Methods such as FA use a new matrix to replace the weight matrix in the back propagation process, which also needs end-to-end learning. Why are these training methods related to bio-inspired?

3. After the search is completed, does the subnet undergo random weight retraining or inherit the supernet's weight for fine-tuning?
This paper introduces the different learning rules into the network search method for the first time. Experiment results show that mixed learning rules can effectively improve the performance of the network and its white-box adversarial robustness. I think the integration of NAS with various learning methods represents a promising approach.
This paper uses the NAS method to search for suitable edge and learning rules of the network cell, constructing a biologically inspired neural network that achieves the comparable performance of the network trained by end-to-end backpropagation algorithm. Simultaneously, this paper demonstrates that the mixed biologically inspired learning rules can reduce gradient variance and enhance the network’s white-box adversarial robustness.
1. This paper does not detail how to simultaneously search for cell architecture and the learning rules. And the search space is much larger than the original DARTS and EGNAS; the search time will increase about threefold. Therefore, if DARTS and EGNAS adopt the same search time as this paper, will their performance be improved?

2. Different rows (training methods) in Table 2 use different network structures, but they need to use the same network structure for comparison.

3. This paper lacks the theoretical analysis that adopting the mixed learning rule can enhance the white-box adversarial robustness.
neural architecture search with different feedback alignment techniques improves accuracy and robustness
@inproceedings{
anonymous2024bionas,
title={Bio{NAS}: Incorporating Bio-inspired Learning Rules to Neural Architecture Search},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tBB8hCG5I7},
note={under review}
}
Bio-inspired neural networks have gained traction due to their adversarial robustness, energy efficiency, and for being biologically plausible. While these bio-inspired networks have shown significant progress, they still fall short in terms of accuracy and are hard to scale to complex tasks. In this paper, we propose to use neural architecture search to further improve state-of-the-art bio-inspired neural networks. We
achieve this thanks to BioNAS, a framework for neural architecture search that explores different bio-inspired neural network architectures and learning rules. The novelty of BioNAS lies in exploring the use of different bio-inspired learning rules for the different layers of the model. The motivation for this choice comes from recent work in the field suggesting that different learning mechanisms might be used in different
regions of the human brain. Using BioNAS, we get state-of-the-art bio-inspired neural network performance achieving an accuracy of 94.86 on CIFAR10, 76.48 on CIFAR-100 and 43.42 on ImageNet16-120, surpassing state-of-the-art bio-inspired neural networks. We show that a part of this improvement comes from the use of different learning rules instead of using a single algorithm for all the layers. We release BioNAS to the community and make the code available via this link (https://anonymous.4open.science/r/LR-NAS-DFE1)
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Feedback Alignment Neural Architecture Search Learning Rules Biologically Plausible
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e33a2ece3d20c7b10aca99e25fc21b95dd8fffad.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
BioNAS: Incorporating Bio-inspired Learning Rules to Neural Architecture Search
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission