Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Safety Self-Alignment
mixture of experts;lora;chain of thoughts;LLM safety
Yes
No ethics review needed.
I am unclear about the relationship between \(x\), \(x_a\), \(x_g\), \(y_{\text{cot}}\), and MoTE. From my understanding, this paper uses a CoT prompt to produce the question analysis first, followed by answer guidance, and finally the safe answer. Additionally, the paper uses this generated data as fine-tuning data for SFT on this model. Is the data generated by the model used to retrain the same model?

- Line 128: It is not clear whether the question analysis, answer guidance, and final safe answer are produced by the same model, or if question analysis and answer guidance can be generated by other models.

- Line 128: Where does the training data (\(x_a\), \(x_g\), \(y_{\text{cot}}\)) for SFT come from?

- Figure 2: Some subfigure titles have a period while others do not. Additionally, the x-axis is not clearly labeled.

- Lines 182-210: Should this part be located in the Experiments section? It seems to be an analysis of the method.

- Line 185: An introduction to the metrics would be helpful. In Table 1, there is no explanation for "Help," "Harm," and "Harm-Help."

- Line 367: What fine-tuning datasets are used for the baselines? \(D_a\), \(D_g\), \(D_{\text{ans}}\), etc.? If it is \(D_{\text{ans}}\), since MoTE is based on the AlignCoT dataset, this could lead to an unfair comparison.
Strengths:

1. The innovative idea of using multiple LoRA experts.

2. Leveraging CoT and introducing AlignCoT.
This paper proposes a new AlignCoT dataset, which includes the original question, question analysis, answer guidance, and final safe answer by prompting the model with Chain of Thought (CoT). A new SFT method is also introduced, based on the AlignCoT dataset. This method incorporates multiple LoRA matrices corresponding to different parts: question, question analysis, answer guidance, and answer. Through comparative analysis, MoTE demonstrates better alignment efficacy compared to benchmark alignment techniques.
Weaknesses:

1. The writing could be improved.

2. The MoTE method is heavily reliant on the AlignCoT dataset, making it difficult to extend to other datasets.
Yes
N/A
No ethics review needed.
See Weakness.
- The paper provides detailed ablation studies, clearly analyzing the contributions of each component, including AlignCoT and MoTE, to the overall performance.
This paper presents a method to improve the safety alignment of LLMs using AlignCoT, a Chain-of-Thought (CoT) based prompting technique that structures datasets with intermediate reasoning steps, and MoTE, a multi-LoRA approach where different parts of the sequence embedding are processed by specialized LoRA modules. The authors train and evaluate their approach on the Safe-RLHF and HH-RLHF datasets, assessing both helpfulness and harmlessness in single-step and multi-step inference settings.
- The formulations in Section 3, including Equation 1, are incomplete and should be revised with more standard notations. For instance, rather than using 𝑥 for context-response pairs, (𝑥,𝑦) should be used, and the source (policy/distribution) of 𝑥 should be clarified. Additionally, the justification for modeling CoT as a joint probability distribution between different steps is not clear. If this formulation is kept, parts of Appendix A should be integrated into the main text for clarity.

- The related work section lacks coverage of self-alignment methods and other fine-tuning approaches based on CoT. A dedicated subsection comparing AlignCoT with previous work is needed to clarify its novelty.

- Despite the point above, Section 3 provides minimal technical discussion. The formulation could be more effectively replaced with a figure illustrating the structural prompting process.

- The "Efficient training of MoTE" section offers little novelty, as its function appears similar to existing masked attention techniques.

- The paper's clarity needs improvement. For example:
  - It is unclear which 7B LLM is referred to in Line 177. Is it Alpaca-7B?
  - The metrics used in Figure 2 are not specified or explained in the text.

- Important details about the methods used in training and evaluation should be moved from Appendix B to the main text.

- The paper's reliance on Alpaca-7B as a baseline is concerning, especially given the rapid advancements in LLM alignment and training. Using an older model might mean addressing issues that have already been mitigated in more recent models. This raises the question of whether the improvements demonstrated are still relevant in the context of modern LLMs. It would be more convincing to evaluate the method on current models like Llama-3-8B, Llama-2-7B, or Mistral-7B, which better reflect the state-of-the-art in language model performance and alignment challenges.

- The evaluation presented in Table 1 is somewhat narrow, relying heavily on PKU-SafeRLHF and HH-RLHF, which are closely related datasets, which limits the evaluation's breadth. Cross-testing results should be included, and other benchmarks like XSTest, WildChat, or win rate comparisons against Llama-Guard or Perspective API (less recommended) should be considered.

- Unless I overlooked it, the code for the proposed methods has not been provided. This makes it difficult to verify the results or apply the techniques described in the paper.
Yes
No ethics review needed.
1. Do author expect MoTE to be extended to other self-alignment methods that involves multiple experts? Could the authors provide more insight on why shared expert can improve the performance?
2. I am a bit confused by single-step inference (line 161) and figure 2. Why the paper aims for improving the single-step inference? Single_Sequence dataset is crafted for Single-step Inference setting, but MoTE can even achieve better performance. Could the author elaborate more on this result?
1. The paper demonstrates comprehensive experiments, draws comparison with enough baselines, and carefully examines multiple design choices in both AlignCoT and MoTE. 
2. The paper is very-well organized and the presentation is clear.
The paper introduces AlignCoT a self-alignment method for generating dataset that facilitating the safe responses of LLMs. Furthermore, the paper also proposes Mixture of Insightful Expert (MoTE) that extends a mixture of experts (MoE) framework with a share expert to facilitate the exchange of knowledge across different stages of AlignCoT. The two contributions are relatively orthogonal, but the sufficient experiments are conducted to demonstrate the necessity of both methods on HH-RLHF benchmark.
1. The benefit of the shared expert in MoTE is a bit unclear. It sounds like a mixture of MoE with single expert LoRA. The authors mentioned information exchange. This is very generic, does MoTE extends to other self-alignment methods?
2. The novelty from each method is relatively limited. For example, AlignCoT is just one of CoT frameworks. A large amount of CoT frameworks have been explored these days. Also MoTE sounds like a mixture of MoE with single expert LoRA. Although, tricks like step skipping and the process to streamline the training of MoTE have been explored. The overall novelty of MoTE compared to MoE is relatively limited.
Yes
N/A
No ethics review needed.
Check the weaknesses section above.
This paper shows the promising performance of AlignCoT and MoTE based on Alpaca-7B and Wizard-Vicuna-Uncensored 7B.
This paper proposes a self-alignment method that utilizes a Chain of Thought approach (AlignCoT) to address two challenges: 1. Recent studies rely on the assumed emergent capabilities of LLMs.  2. Recent studies discard all intermediate reasoning steps, when aligning the model with updated answers.
1. In Table 1, it is unclear if the results are based on Alpaca-7B or Wizard-Vicuna-Uncensored 7B.
2. For evaluation, it would make this paper stronger if the authors could provide more details about how human annotators provide a final verification for precise results. For example, what are the annotation guidelines provided to annotators, how disagreements between annotators were resolved, how human annotators are picked, and how many human annotators are in the verification process?
3. It would be better if the authors could show if their method has the same effects in bigger LLMs (such as picking some LLMs with 13B or even larger).
4. The metrics, help, harm, and harm-help are too vague and may be sensitive to human annotators. It would make this paper stronger if the authors could provide better metrics. For example, how the authors deal with the annotator bias or subjectivity in current metrics.
@inproceedings{
anonymous2024mixture,
title={Mixture of insighTful Experts (Mo{TE}): The Synergy of Thought Chains and Expert Mixtures in Safety Self-Alignment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nTAC2NCQUO},
note={under review}
}
As the capabilities of large language models (LLMs) have expanded dramatically, aligning these models with human values presents a significant challenge. Recent studies demonstrate that powerful LLMs can achieve self-alignment by either correcting their initial unsafe responses or autonomously ranking answers without human intervention. In this work, we identify two key limitations: first, they rely on the assumed emergent capabilities of LLMs, and second, they discard all intermediate reasoning steps when aligning the model with updated answers. To address these challenges, we propose a novel self-alignment method that utilizes a Chain of Thought (CoT) approach, termed AlignCoT. This method encompasses stages of Question Analysis, Answer Guidance, and Safe Answer production. It is designed to enable LLMs, even smaller and weaker models like 7B LLMs, to produce high-quality, safe responses. Furthermore, we introduce the Mixture of insighTful Experts (MoTE) architecture, which applies mixture of experts to enhance each component of the AlignCoT process, markedly increasing alignment efficiency. The MoTE approach not only outperforms existing methods in aligning LLMs with human values but also highlights the benefits of using self-generated data, revealing the dual benefits of improved alignment and training efficiency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
mixture of experts lora chain of thoughts LLM safety
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bb2086fd15578eda363907fada138a6081e08836.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Safety Self-Alignment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission