Multi-objective Differentiable Neural Architecture Search
hardware efficiency;neural architecture search;network compression
Yes
No ethics review needed.
__1-__ Using hypernetwork for the NAS and prate-frontier learning is well known. 

That limits the novelty of the proposed method.

[A]Brock, A., Lim, T., Ritchie, J., and Weston, N. (2018). SMASH: One-shot model architecture search through hypernetworks. In the International Conference on Learning Representations

[B[ Lorraine, Jonathan, and David Duvenaud. "Stochastic hyperparameter optimization through hypernetworks." arXiv preprint arXiv:1802.09419 (2018).

[C] Pan, Z., Liang, Y., Zhang, J., Yi, X., Yu, Y., and Zheng, Y. (2018). Hyperst-net: Hypernetworks for spatio-temporal forecasting. arXiv preprint arXiv:1809.10889

[D]  Zhang, C., Ren, M., and Urtasun, R. (2019). Graph hypernetworks for neural architecture search. In International Conference on Learning Representations.
 
However, there are major concerns about the performance, initialization, and scalability as stated here (Section 6 in the following paper):
Chauhan, Vinod Kumar, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A. Clifton. "A brief review of hypernetworks in deep learning." Artificial Intelligence Review 57, no. 9 (2024): 250.


__2-__ Hypernrtwork is not performing well on unseen networks.
https://arxiv.org/abs/2110.13100
With that, Could you elaborate on any specific constraints or limitations when transferring MODNAS to devices significantly different from the ones used in the training phase?


__3-__ MiLeNAS paper (CVPR’20) shows that the gradient errors caused by approximations of second-order methods in bi-level optimization results in suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution.  However, it seems that the authors still use a bi-level optimization technique.  It seems using advanced techniques like MiLeNAS might be more useful.
 
__4-__ Can the MODNSD be applied to object detection tasks as well?

__5-__ MobileNet search space is pretty small DNN. How does it work on more complex DNN search space?  

__6-__ The paper mentions scalability but doesn’t mention potential bottlenecks. For example, are there search space complexities that MODNAS struggles with?



__7-__ The paper doesn't provide the code. That limits reproducibility of the proposed method.



__8-__ I am not sure how accurate is the energy model that the paper is using considering even Eyeriss is a pretty old paper (2016).  The authors need to provide more details about energy modeling. 

__9-__ I think it is better to use power rather than energy. Considering energy is power x  latency, when you minimize the latency, if the power is fixed, energy is minimized automatically. Considering power, accuracy, and latency can be a better metric.


__10-__ The paper employs the Frank-Wolfe solver for optimizing scalarizations. Could you discuss any trade-offs in this choice, and if other optimizers were considered?

__11-__ Given that preference vectors affect the optimization landscape, an analysis of how different preference configurations influence the final architectures would clarify MODNAS’s versatility.


__12-__ MODNAS is tested for three objectives—accuracy, latency, and energy. Could this method scale effectively if additional objectives were introduced, or would this necessitate modifications?

__13-__ The approach appears tailored to devices with GPU-like architectures. It would be valuable to understand how MODNAS performs on other architectures, like mobile hardware.




__14-__ Minor: 

A)The paper is pretty dense. I hardly can read the legends in Figures 7, 8.

B)Minor Typo:
Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging" – A comma after "i.e." would improve readability: "i.e., finding a diverse set..."

"To search across devices, we frame the problem as a multi-task multi-objective optimization problem" – Add a comma, “multi-task, multi-objective optimization problem.”
1-The method is adaptable to a range of devices by conditioning the hypernetwork on device embeddings, making it highly versatile for deployment on diverse hardware.

2-MODNAS is tested on various hardware devices and tasks, including image classification, machine translation, and language modeling, showcasing its applicability across multiple domains.
The paper (MODNAS) presents an approach to Neural Architecture Search (NAS) that balances competing objectives—like performance, latency, and energy efficiency—across multiple hardware devices. By encoding user preferences as a scalarization vector, MODNAS efficiently searches for Pareto-optimal solutions across diverse devices in a single run. The method employs a hypernetwork to generate architectures for specific hardware configurations, leveraging multiple gradient descents for optimization.  The method has been evaluated on MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation, and a decoder-only space for language modeling.
1-Using Hypernetworks for NAS is well known but doesn’t seem a promising solution. It is like a heuristic solution.

2- I think energy and latency are not necessarily conflicting metrics.
Yes
No ethics review needed.
- How are user preference and hardware device embeddings designed?
- Please explain how $\textbf{Architect}$ makes architecture parameters differentiable and propagate them to the $\textbf{MetaHypernetwork}$.
- In Figure 4, solutions on the Pareto frontier can be achieved by modulating user preferences. However, in HDX [1], which is a one-shot NAS with hardware constraints, claims that modulating hyperparameters linearly doesn’t lead to linearly distributed results. Is $\textbf{MetaHypernetwork}$ free from this problem? Can real experimental results be plotted like Figure 4 to substantiate the integrity of $\textbf{MetaHypernetwork}$?
- To search network architectures with other works, the paper sets the search time budget up to 2.5 times compared to that of MODNAS. (Or fixed time budget, e.g., 192 hours) That is, MODNAS outperforms prior works in terms of search time, and the quality of solutions is better than that of others under given time budgets. What the review wonders is whether can other works find near-GT solutions with a larger time budget. If a target network is distributed extensively and used frequently, huge search costs can be tolerable if there are better solutions.

[1] Deokki Hong, Kanghyun Choi, Hye Yoon Lee, Joonsang Yu, Noseong Park, Youngsok Kim, and Jinho Lee. Enabling Hard Constraints in Differentiable Neural Network and Accelerator Co-Exploration. In Proceedings of the 59th ACM/IEEE Design Automation Conference, 2022.
- The paper proposes to execute one-shot NAS while search networks satisfy multiple objectives about accuracy and hardware efficiency with target hardware.
- The paper analyzes the proposed method with various aspects such as efficacy, and robustness of the training process.
- The paper provides extensive experiments and visualizations to support the proposal and its analysis.
This paper proposes a Neural Architecture Search (NAS) algorithm that optimizes several performance metrics on various hardware.
The paper adopts Multi Objective Optimization (MOO) to optimize multiple objectives simultaneously.
The key idea is to train $\textbf{MetaHypernetwork}$ which takes hardware features and user preferences about performance metrics as inputs and returns optimized network architectures as outputs.
To train $\textbf{MetaHypernetwork}$, the paper exploits $\textbf{Supernetwork}$ for NAS and $\textbf{MetaPredictor}$ for optimization in terms of efficiency.

$\textbf{MetaHypernetwork}$ takes user preferences and device features, and returns network architecture designs.
Then, $\textbf{Architect}$ makes those architecture designs into differentiable architecture parameters.
With these architecture parameters, $\textbf{Supernetwork}$ computes accuracy-related loss of the given network design.
Also, $\textbf{MetaPredictor}$ computes efficiency-related loss with the network architecture under the given hardware features.
Further, the paper proposes to update $\textbf{MetaHypernetwork}$ using Multiple Gradient Descent (MGD) to get optimized network architectures that satisfy multiple objectives concurrently.

For NAS, the proposed method is adaptable to diverse pretrained $\textbf{Supernetwork}$.
For efficiency, $\textbf{MetaHypernetwork}$ can be updated with a lot of hardware-related loss functions at once.
For optimization, the paper achieves Pareto frontier solutions that are hard to reach with sequential or averaged gradient updates.
- It may be hard to regulate the trade-off among user preferences with scalarization. Figure 4 can be a support, but it is just an abstract depiction, not experimental results.
- The proposed method can help search optimized network architectures quickly at low cost. However, network architectures the same as or near ground truth solutions may hard to be reach with $\textbf{MetaHypernetwork}$, where other works can reach with huge search costs.
Yes
No ethics review needed.
- What assumptions do the MetaPredictors make about the underlying devices when estimating latency and energy requirements? For example, do they assume a particular choice of operating system and process scheduler used, other parallel and system processes running, core utilisation of the DNN inference, runtime library, or HW design used to execute the DNN on the target device?

- Especially on smaller systems, memory requirements are often a major bottleneck of DNN inference: Could the approach proposed by the authors be extended to include memory? So far, the evaluation seems to focus only on accuracy, latency and energy.

- How exactly are "preference vector" and a "hardware device feature vector" defined? The authors cover many different types of HW in their evaluation: GPUs (e.g. 1080 ti), smarthpones (e.g. pixel 2, 3), edge devices (e.g. raspi 4) and dedicated HW accelerators (eyeriss, FPGA), which can be characterised by a number of different metrics, often exclusive to the type of HW considered. (e.g. #cores, processor speed, RAM, ... for the raspi4 vs. #streaming cores, VRAM, bus interface, ... for the 1080 ti vs. #LUTs, #BRAM, ... for the FPGA). So it would be really interesting to know how the authors put these metrics into relation and unified them into one vector.

- Which FPGA did the authors use? Since an FPGA is just freely programmable hardware, it would also be interesting to know which DNN accelerator design the authors implemented on the FPGA to perform their evaluation.

- Fig. 5: Why is MODNAS shown as an upper threashold line (+/- variance)? Based on the caption, I would have expected to see an HVI curve that stops after 24 evaluations, similar to the other curves in the plot.
- The extension of existing zero-shot NAS techniques with Hypernetworks for HW aware NAS is motivated well and contextualized nicely for alredy existing techniques. 
- The authors provide an extensive evaluation for different applications (language modeling, vision, translation), multiple well known NAS searchspaces, and different target spaces (2-3 dimensional).
The authors propose a multi-objective HW aware NAS algorith that can emit optimized DNN architectures across multiple different target devices using a single search run.
To overcome the challange of having to solve multiple different optimization runs for each considered HW platform, the authors propose using a hypernetwork to generate an architectural distribution across multiple devices based on a "preference vector" and a "hardware device feature vector" from which discrete architectures can then be sampled.
For objective function evaluators, the authors use a Supernetwork as a standin for a accuracy and "MetaPredictors", one for each target device, to estimate hardware metrics.
- The authors do not show how the proposed DNN architectures would actually perform on the different target systems. As far as I understand it, the HV results shown are calculated using the estimated results from the "MetaPredictors". While this still allows for a relative comparision with the other techniques and algorithms evaluated in the paper, it makes it hard to evaluate the actual usefulness and effectiveness of the apporach.
Yes
No ethics review needed.
- Could you provide the training and validation loss curves?
- In Figure 13, do you quantize the probability of $r_m$​ to obtain the vector in the embedding layer $e_m$​? As I understand it, embedding layers typically take indices as inputs.
- In Figure 13, if I understand correctly, $e_{\phi_0}$ is a linear layer that maps the device feature vector to a k-dimensional vector. If this is the case, referring to it as an 'embedding' layer may be somewhat confusing, given the use of $e_m$ for embeddings.
- Suppose an application requires a network with a latency of under 30 ms on a given platform. How does the user preference vector, where each element is a probability within [0, 1], map to specific latency (and energy consumption) in real-world scenarios?
The differentiable NAS framework proposed in the paper introduces a novel hypernetwork that enables zero-shot transferability to new devices, which, to the best of my knowledge, is an original contribution. I appreciate how this framework addresses key challenges faced by the NAS community. Previous approaches directly optimize network structures and hardware metrics using differentiable operators for one specific platform, while two-stage NAS frameworks separate supernet training from subnet searching. However, the search process in two-stage frameworks is often costly, as it relies on hardware-specific predictors and estimators for each objective on each platform. Although some works, such as  [1, 2], have attempted to reduce the search cost in two-stage frameworks, I believe that a unified solution that provides zero-shot transferability and multi-objective optimization across new platforms has been lacking. This paper fills that gap by proposing a unified framework that combines hardware and objective embeddings with differentiable NAS through the use of a hypernetwork. I believe this is an exciting progress for neural architecture search.

The paper is well-presented and well-organized. The extensive experiments that compare it to several state-of-the art NAS frameworks and the in-depth analysis of the framework fully support the claims made in the paper.

[1] HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning

[2] BRP-NAS: Prediction-based NAS using GCNs
The paper introduces a novel framework for differentiable neural architecture search (NAS) that integrates a generalizable hardware embedding with multi-objective embeddings during supernet training. This approach enables the NAS framework to generalize to unseen hardware platforms and efficiently sample Pareto-optimal subnets based on user preferences. To achieve this, the authors design a MetaHypernetwork that uses a sampled preference vector and representative hardware embeddings to guide architecture sampling within the supernet. During training, instead of randomly sampling subnets, the framework selects and trains subnets based on the hardware and objective embeddings, resulting in a better Pareto front. Extensive experiments demonstrate that this framework consistently produces superior Pareto-optimal subnets across various devices, outperforming several state-of-the-art NAS frameworks.
My only concern is that training differentiable NAS with multi-objective hardware metrics can often be unstable and may struggle to converge, leading to issues with reproducibility. While the authors have shared the hyperparameters and plan to release the implementation, it would be beneficial to include a discussion of the training techniques in the Appendix to address these challenges.
We introduce a novel hardware aware differentiable multi-objective architecture search method that efficiently profiles the whole Pareto-front.
@inproceedings{
anonymous2024multiobjective,
title={Multi-objective Differentiable Neural Architecture Search},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=9mjZ800m7Y},
note={under review}
}
Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives that require training a neural network. Typically, in MOO for neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences to trade-off performance and hardware metrics, yielding representative and diverse architectures across multiple devices in just a single search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments involving up to 19 hardware devices and 3 different objectives demonstrate the effectiveness and scalability of our method. Finally, we show that, without any additional costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only space for language modelling.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
hardware efficiency neural architecture search network compression
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/833bf46aff5eb428082a984c9ba23f80a34482de.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Multi-objective Differentiable Neural Architecture Search
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission