Conditional Enzyme Generation Using Protein Language Models with Adapters
protein;enzyme;protein language model;generative model;fine-tuning;conditional generation;adapter;machine learning
*Our Response to (1): Thank you for your support of our study and for the insightful feedback. We would first like to clarify our dataset splitting details. In Table 2, "Swissprot Heldout 70%" and "Swissprot Heldout 90%" are held-out protein sequences used only for the evaluation of perplexity. By contrast, we used the "Swissprot Heldout ECs" split to evaluate generation quality from OOD EC numbers. This split was constructed by holding out all protein sequences associated with certain EC numbers. These sequences are rarely homologous to the training sequences (as they belong to different enzyme families entirely), and nearly all have <40% identity to the training set. Therefore, when we ask the model to generate EC numbers that are OOD (not present in the training set), this is a very difficult task, and we see underwhelming performance in Figure 4D. We must admit that there is a lot of room for improvement here; by being one of the first to explore this question, we encourage others to tackle it with different approaches.* 

*Our Response to (2): As for the heldout ECs, we note that they are sampled across diverse EC numbers at level 4 (X.X.X.X), based on the "medium" split for Task 2 in the CARE benchmarks (https://arxiv.org/abs/2406.15669v1, details in Table 3 and Figure A.3B). In fact, these EC numbers are intentionally spread across functional space (details in CARE Benchmarks Appendix A.2). This essentially means that if 4.2.1.20 is held-out, there will still be sequences with other EC numbers falling under 4.2.1.- (similar functions) in the training set. You may also note that the "hard" split in the CARE benchmarks is more difficult because all EC numbers under a certain EC level 3 (X.X.X.-) are held out. Thus, the most similar EC numbers still present in the training set in that case would be X.X.-.- . However, this presents a more difficult task, so we only explored the "medium" split in this study. We hope this clarifies why there is no data leakage and that the held-out EC numbers were thoughtfully chosen to be spread across function and sequence space. Happy to explain further if there are remaining questions and/or concerns!*

*The reviewer also raises an interesting point about more detailed analysis of the generated sequences. It is correct that certain enzymes have important domains, such as active sites, which may be more highly conserved. However, because most enzymes have complex and poorly understood sequence-function relationships, we have not had a chance to examine these subdomains in detail and primarily focused evaluation on local sequence alignment with BLAST. That being said, we are currently working on structure evaluation of generated sequences (using pLDDT), which we be included in this revision as another metric to evaluate generation quality. We are also performing additional experiments to strengthen the overall study, as suggested by the other reviewers.*

*Please let us know if you have any further questions or concerns that we can address.*
Yes
No ethics review needed.
see weaknesses.
The use of adapters allows for parameter-efficient training, and easy accommodates various types of conditioning information, such as enzyme function and taxonomy for different applications.

ProCALM shows capabilities to generalize to rare and unseen enzyme families and taxonomies, which is an unique advantage over existing methods.
The paper introduces ProCALM (Protein Conditionally Adapted Language Model), an approach for the conditional generation of proteins, specifically enzyme in this paper, using adapter tuning with protein language models.
In silico experiments show that matches existing methods (ie, ZymCTRL) in generating sequences from target enzyme families while offering several advantages, such as being parameter-efficient to train, allowing multiple conditionings (by adding more adapter and re-train accordingly(, and can generalize to rare and unseen enzyme families and taxonomies. 
ProCALM demonstrates potential in generating sequences for unseen functions, although there is room for improvement and future research.
Despite the promise of ProCALM in generating functional protein sequences, the current form of the manuscript can be significantly improved if the following concerns are addressed in the future.

**Lack of technical novelty in the machine learning side:**
This paper appears to be a simple adaptation of adapter tuning for PLMs for one specific application scenario. Neither a new ML method nor new insights into how to properly tailor existing ML methods to protein problems are presented. Protein LMs (ProGen) are existing models, and adapter tuning is widely used in LMs, including protein LMs [1, 2, 3, 4]. The integration method seems fairly straightforward, with the only difference being the task (enzyme design).

**The performance is not strong and the evaluation is not sufficient**.
Compared to existing methods, ProCALM's performance is comparable and not impressive. 
More efforts need to be put in providing clear evidence of ProCALM's superiority over existing methods.
Despite the promise in generating sequences for unseen functions, this paper only studies enzyme generation. Given the mild contributions from the ML aspect, the authors should provide more evaluations on various conditional protein sequence design applications using PLMs and conditional adapters to demonstrate ProCALM's generality and versatility for functional design.


---

[1] Structure-informed language models are protein designers. ICML 2023

[2] Adapting protein language models for structure-conditioned design. BioRxiv 2024

[3] ShapeProt: Top-down Protein Design with 3D Protein Shape Generative Model. BioRxiv 2023

[4] InstructPLM: Aligning Protein Language Models to Follow Protein Structure Instructions.  BioRxiv 2024
Yes
No ethics review needed.
1. Previous work [1] has validated the effectiveness of scaling. Considering that ProGen2 also has 2.7B and 6.4B versions, can scaling to bigger protein language model further improve the generation performance of the method proposed in this paper?
1. The proposed method ProCALM is parameter-efficient and computationally inexpensive to finetune protein language models for conditional generation tasks.
2. ProCALM can handle multiple types of conditioning information, such as enzyme function and taxonomy, and different representations of the same condition.
3. The paper shows that ProCALM can successfully generalize to out-of-distribution conditions.
This paper proposes a novel approach, ProCALM (Protein Conditionally Adapted Language Model), for the conditional generation of proteins using adapters in protein language models. They leverage a seperate encoder to capture information of given condition into latent representation. The conditional adapter layer, presented within each transformer layer, integrates this latent representation with the language model's embeddings to ensure conditional generation. Experimental results demonstrate that ProCALM can successfully generate protein sequences conditioning on enzyme function and taxonomy, and it shows significant generalization capabilities to unseen enzyme families and taxonomies.
1. This paper does not cite LM-Design [1], which introduces a lightweight adapter into protein language models to perform structure-conditioned sequence generation and is very related to this work.
2. The method proposed in this paper is similar to LM-Design, and the experimental results, such as generalization to OOD distribution, have already been demonstrated by LM-Design. Therefore, I believe this paper lacks sufficient novelty.
3. The current results still lag behind existing methods (such as ZymCTRL) in terms of generation quality, diversity and perplexity, and only shows advantage in training costs, which may not be sufficiently strong to prove the effectiveness of this method.
4. The evaluation metrics are not comprehensive enough. For sequence quality, the pLDDT metric has widely been used in sequence generation evaluation [2,3]. And for sequence diversity, comparing generated sequences among themselves, rather than against a reference database, can more directly reflect diversity. For example, this can be evaluated from both sequence and structure dimensions: by calculating pair-wise sequence identity and by using a structure prediction model (such as ESMFold) to first obtain the structure of the sequence and then calculating the pair-wise structural similarity.

[1] Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei YE, and Quanquan Gu. Structure-informed language models are protein designers.

[2] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex Xijie Lu, Nicolo Fusi, Ava Pardis Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need.

[3] Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners.
Yes
No ethics review needed.
I have listed my concerns and questions in 'Weaknesses'.
1. Adapter is indeed a parameter-efficient method that enables controllable generation at a relatively low cost and effectively mitigates overfitting.
2. The authors conducted extensive experiments to demonstrate the effectiveness of the proposed method, such as generating unseen families, and some necessary ablation studies like the performance in different function representations.
The paper proposes ProCALM, a method for the conditional generation of proteins using protein language models enhanced with conditional adapters. This approach aims to generate protein sequences tailored to meet specific types or molecular functions.
The authors conduct several experiments and demonstrate that, for seen enzyme families, ProCALM achieves comparable to previous methods like ZymCTRL. Additionally, it shows the ability to generalize to rare and novel enzyme families and taxonomies.
1. Compared to previous fine-tuning models designed to generate protein sequences with specific functions, the approach tries to learn the mapping of functional distribution, and indeed holds some potential to generate sequence with functions not encountered during training. However, it seems to rely on the quality of the functional representation, and there seems to be a risk of “data leakage” during the stage of function representation learning. 
2. Although ProCALM has a lower training cost than previous methods, its ability to generate sequences with both seen and unseen protein families is limited. For seen functions, its performance is slightly inferior to existing methods, and for unseen families, its capabilities remain quite limited.
Yes
No ethics review needed.
I think that the authors make a convincing case that ProCALM is broadly applicable and useful given that it is 1) parameter efficient and inexpensive to train while matching ZymCTRL and 2) flexible to allow multiple types of conditioning. I also think that the paper would benefit from addressing the concerns about the OOD generalization claims.

I may have misunderstood the data-splitting, evaluations, or something else entirely, and I am willing to increase my score if this is the case and/or the authors address my concerns listed in the weaknesses section.

I'm also curious if the authors have examined the properties of the subsequences in the ProCALM-generated enzymes. For example, a histidine kinase will contain a histidine kinase domain (which may be as short as 70 amino acids), and this is probably the region of the protein to which it is most important to generate something similar in order to yield a working histidine kinase. I think it would be valuable to see if the generated sequences for a specific EC class contain the domain(s) that the target sequences contain and in what order (for multi-domain enzymes)
1. The paper is written very clearly and does a good job of contextualizing the contributions of ProCALM against prior work.
2. While this paper does not represent the first use of conditional adapters for protein generation, the authors explore conditioning on information other than structure (like taxonomy and enzyme class)
3. The evaluation metrics are well-chosen and reasonable for sequence-level similarity and diversity, though it may be worthwhile to perform more fine-grained evaluation on generated sequences (i.e. domain-level)
4. Based off the evaluations provided, ProCALM is clearly performant and compares well to the existing method tested, ZymCTRL
5. The authors perform an ablation study to test the benefits of parallel adapters vs. a shared adapter in the case of conditioning both taxonomy and enzyme class,
The authors propose ProCALM, an adapter-based method to allow for conditional generation of proteins from pre-trained protein language models, and demonstrate the performance of ProCALM on conditional enzyme generation. Their main contributions are 1) demonstrating that conditional adapters, while being parameter efficient and inexpensive to train in both time and memory, yield comparable performance to existing methods and 2) creating a flexible framework for conditional generation -- ProCALM is not limited to any one particular class of condition, and the authors demonstrate individual and joint conditioning across enzyme class and taxonomy. Lastly, the authors assess the ability of ProCALM to generalize to rare and completely held-out/out-of-distribution enzymes (this includes unseen combinations of taxonomy and enzyme class). The authors also evaluate across a broad range of metrics and perform several ablation experiments.
My main concern is about the out-of-distribution (OOD) generalization claims, which I believe are currently unsupported by the evaluations/data-splitting:
1. The 70% and 90% sequence identity thresholds used for clustering to get held-out clusters for evaluation (Table 2) are really high thresholds. 
- It seems like it would be totally feasible for a sequence in train and a sequence in one of these evaluation sets to be very closely related homologs (~40% shared sequence similarity and higher). 
- In such a case, it seems like ProCALM could simply generate, for a given enzyme commission (EC) class, a sequence very similar to one seen with the same EC class in training. 
- I believe that adding evaluations for held-out clusters at other thresholds will strengthen the overall results of this work.

2. The Heldout ECs for generation evaluation are randomly sampled ECs, which doesn't account for the hierarchy of EC numbers. 
- While it would not be reasonable to hold out say all sequences where the first digit of the EC number starts with a 7, there should be some standardizing of what level of EC number gets held out. For example, maybe if all sub-subclasses are held out for a specific subclass, that would be an informative evaluation to see. 
- In light of the random sampling of ECs for the heldout group, the mean accuracy level in Figure 4.D seems a little underwhelming. If the mean accuracy level is below 1, then the accuracy level is more about the ability of ProCALM to generate examples of the same overall class, which probably relies more on overall sequence similarity between examples of the target class and the training data, which comes back to the point about high percent identity clustering thresholds.
@inproceedings{
anonymous2024conditional,
title={Conditional Enzyme Generation Using Protein Language Models with Adapters},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=dWReNWEj5b},
note={under review}
}
The conditional generation of proteins with desired functions and/or properties is a key goal for generative models. Existing methods based on prompting of language models can generate proteins conditioned on a target functionality, such as a desired enzyme family. However, these methods are limited to simple, tokenized conditioning and have not been shown to generalize to unseen functions. In this study, we propose ProCALM (Protein Conditionally Adapted Language Model), an approach for the conditional generation of proteins using adapters to protein language models. Our specific implementation of ProCALM involves finetuning ProGen2 to incorporate conditioning representations of enzyme function and taxonomy. ProCALM matches existing methods at conditionally generating sequences from target enzyme families. Impressively, it can also generate within the joint distribution of enzymatic function and taxonomy, and it can generalize to rare and unseen enzyme families and taxonomies. Overall, ProCALM is a flexible and computationally efficient approach, and we expect that it can be extended to a wide range of generative language models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
protein enzyme protein language model generative model fine-tuning conditional generation adapter machine learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/bb74041bdf293f9a3988d931501b795761659be5.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Conditional Enzyme Generation Using Protein Language Models with Adapters
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission