Pretrained Hybrids with MAD Skills
hybrid architectures;large language models;transformers;state space models;model merging;neural architecture search;mechanistic search
Yes
No ethics review needed.
1. Have you done experiments to validate whether the performance continues to hold at scale?
2. What is the overhead of the projectors?
1. The paper is well-written and the the motivation is clear and convincing.
2. The figures are well-designed and helpful.
3. Interesting results that prove the effectiveness of their framework on specific models.
The paper builds by proposing a new framework for automating the design of hybrid language models by re-using existing pre-trained models using ideas from Neural Architecture Search. The authors show that their approach allows for merged language models to be competitive with their component models and also outperform them on fine-tuning tasks.
1. The experiments done are mostly on smaller models and it's not clear if MAD remains effective with larger models. The authors could do additional experiments with other open-source models to validate this.
Yes
No ethics review needed.
1. How would Manticore perform relative to its component models if they were fine-tuned under the same computational budget? Would this comparison validate Manticore’s primary claim of outperforming its component models?
2. Could the authors elaborate on the memory and computational efficiency of Manticore hybrids, especially compared to traditional single-model architectures? Does the use of projectors introduce significant overhead?
+ Manticore’s approach to combining pretrained models from different architectures using projectors and mixture weights is innovative and extends beyond typical model merging methods.
+ Manticore’s design, which allows for fine-tuning and programming pretrained hybrids, offers a degree of flexibility, making it potentially beneficial for practitioners looking to leverage diverse model architectures.
+ Testing across LRA and MAD tasks provides an initial sense of the framework's potential, although the evaluation depth limits the conclusions drawn.
The paper introduces Manticore, a framework for creating hybrid architectures by combining pretrained components from different language models (LMs). Manticore aims to automate hybrid model design by reusing pretrained model components from distinct architectures, such as Transformers and state-space models, using projectors to align feature representations between these models. The framework's goal is to merge model architectures in a way that preserves their respective strengths, ideally achieving better performance than the individual components. Through experiments on the Long Range Arena (LRA) and MAD tasks, Manticore demonstrates comparable or improved performance over some individual models and existing hybrid models.
- The main claim, "Pretrained hybrids can outperform their component models on fine-tuning tasks," is not well-supported. A fair comparison would entail fine-tuning Manticore and its component models under the same budget to evaluate relative gains. Without this, it’s unclear whether the hybrid approach provides substantial benefits beyond those of individually optimized models.
- Manticore requires a dedicated training process for projector layers and mixture weights, potentially adding overhead and limiting applicability in constrained environments.
- The absence of publicly available code restricts reproducibility and verification of the results, limiting the community's ability to assess the framework’s impact fully.
Yes
No ethics review needed.
A fair comparison would either be a model of the same architecture with 2x size or the hybrid being discretized and having the same amount of FLOPs/parameters of the one of the individual models. This would address weakness 2.

 A stronger test would be to consider models that perform better in some tasks than others and then validate if the hybrid does achieve the best of both worlds. This would address the weakness 3.
This paper focuses on an important and relevant problem of using architectural components from different state-of-the-art model architectures to construct a hybrid model that provides the best of all worlds without incurring expensive pre-training and search space exploration overheads.

It introduces a novel idea of projectors that enable different architectures to interact in each other feature space by projecting an intermediate shared feature space that acts as a translator for them.
Manticore is a framework for automating the creation of hybrid language model architectures by reusing pre-trained models from different architectures. By leveraging NAS and projectors, Manticore enables pre trained hybrid models that combine the strengths of multiple architectures, allowing for flexible and high-performing LLMs without extensive manual design or retraining.
Although the idea of projectors is novel but using gating to combine the contributions of different architectures has been explored in Mixture of experts [1], weighted ensemble averaging and finds a direct use in this paper.

The evaluation compares the combined hybrid that has 2x the number of parameters and >2x FLOPs due to projectors and gating against individual models of half the size.

In Table 1, Mamba is already better than Pythia in all of the tasks, in Table 2, Mambaformer is also better in all tasks but one, In Table 4, GPT-Neo is better than all tasks, So creating a hybrid is achieving almost the same score as the better model.

[1] https://arxiv.org/abs/1701.06538
We develop a framework for creating pretrained hybrid models from existing pretrained models.
@inproceedings{
anonymous2024pretrained,
title={Pretrained Hybrids with {MAD} Skills},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=rXrYdOtBfs},
note={under review}
}
While Transformers underpin modern large language models (LMs), a growing list of alternative architectures with new capabilities, promises, and tradeoffs is emerging. This makes choosing the right LM architecture challenging. Recently proposed *hybrid architectures* seek a best-of-all-worlds approach that reaps the benefits of all architectures. Hybrid design is difficult for two reasons: it requires manual expert-driven search, and new hybrids must be trained from scratch. We propose **Manticore**, a framework that addresses these challenges by *automating the design of hybrid architectures* while reusing pretrained models to create *pretrained* hybrids. Our approach augments ideas from differentiable Neural Architecture Search (NAS) by incorporating simple projectors that translate features between pretrained blocks from different architectures. We then fine-tune hybrids that combine pretrained models from different architecture families---such as the GPT series and Mamba---end-to-end. With Manticore, we enable LM selection without training multiple models, the construction of pretrained hybrids from existing pretrained models, and the ability to *program* pretrained hybrids to have certain capabilities. Manticore hybrids match existing manually-designed hybrids, achieve strong performance on the Long Range Arena benchmark, and improve on pretrained transformers and state space models on various natural language tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
hybrid architectures large language models transformers state space models model merging neural architecture search mechanistic search
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d37db6ca8e7792671dfe339f66dac9a0a6887b16.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Pretrained Hybrids with MAD Skills
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission