SurfDesign: Effective Protein Design on Molecular Surfaces
Molecular Surfaces;Protein Design;Geometric Deep Learning
Thank you for your thorough and constructive review. We’re glad that you found SurfDesign’s approach and benchmarks promising and that our methodology demonstrated strong performance across tasks. It is a pleasure and honor to gain your 8-score! 

(1) **Biological Significance in Protein-Protein Interaction Design:** We appreciate your emphasis on protein-protein interaction applications and agree this is a crucial direction for SurfDesign. To this end, we followed SurfPro’s setup [A] and applied SurfDesign to design binders with high affinity to specific target receptors. Binding affinity was evaluated using AlphaFold2's pAE_interaction metric, which has proven effective in distinguishing experimentally validated binders from non-binders. We used <binder, target protein> pairs across six categories from Bennett et al. (2023) as benchmarks. Among the 10 available categories, only six showed significant separation between positive and negative binders based on AF2 pAE_interaction. All baseline models were fine-tuned on a binder design dataset following pretraining on CATH 4.2 to ensure a fair comparison. The results, shown below, demonstrate that SurfDesign achieves the lowest average AF2 pAE_interaction across the six target proteins, underscoring the strength of our surface-based approach for functional protein design! 

| **Models** | **InsulinR** | **PDGFR** | **TGFb** | **H3** | **IL7Ra** | **TrkA** | **Average** |
|---|---|---|---|---|---|---|---|
| Positive Binder | **5.9996** | **14.1366** | **15.4884** | **21.2631** | **20.9102** | **10.2791** | **14.7061** |
| Negative Binder | 19.7167 | 18.0937 | 23.2664 | 22.4556 | 26.0540 | 24.7567 | 21.1335 |
| Random Baselines | 19.9880 | 21.2690 | 21.4971 | 24.4997 | 24.1541 | 23.1147 | 22.2020 |
| ProteinMPNN  | 18.3393 | 25.2919 | 25.8559 | 24.5968 | 25.5278 | 27.0980 | 23.4462 |
| PiFold  | 12.9809 | 21.8230 | 24.4737 | 23.3924 | 26.6738 | 19.7172 | 20.5785 |
| LM-DESIGN  | 13.6440 | 22.0749 | 23.3725 | 23.8332 | 24.3937 | 22.3987 | 20.7728 |
| SurfPro | 10.2608 | 17.9862 | 17.7364 | 21.2916 | 20.8594 | 10.6535 | 16.9485 |
| **SurfDesign** | **9.9827** | **17.5460** | **17.4381** | **21.0558** | **20.8213** | **10.5094** | **16.22555** |

[A] SurfPro: Functional Protein Design Based on Continuous Surface. ICML 2024.

(2) **Clarification on “O optionally” in Backbone Representation:** “O optionally” indicates that the backbone representation may or may not include the oxygen atom of the peptide bond, based on context-specific requirements. We’ve clarified this in the revised manuscript.

(3) **Calculation of Surface Properties:** Surface properties like hydrophobicity, hydrogen bonding potential, and charge are computed using established descriptors derived from biophysical studies. We follow MaSIF[B] and calculate three key invariant point inputs, including the Poisson Boltzmann electrostatics using APBS: https://github.com/LPDI-EPFL/masif/blob/master/source/triangulation/computeAPBS.py, the hydrophobicity:https://github.com/LPDI-EPFL/masif/blob/master/source/triangulation/computeHydrophobicity.py, and the free electrons/protons: https://github.com/LPDI-EPFL/masif/blob/master/source/triangulation/compuSurfDesignteCharges.py.

[B] Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods.

(4) **Clarifications on Table 5:** You’re correct—there was a mislabeling in Table 5, and it should indeed refer to SurfDesign rather than MoE-DSR. We have corrected this in the text for accuracy.

(5) **Figure 4 Labeling:** Thank you for pointing this out. We’ve corrected the erroneous reference in Line 423 from Figure 3.3 to Figure 4.

(6) **Surface Isomer Experiments:** Regarding the surface isomer section, we realize further detail is needed. We conducted these tests with two well-documented lysozyme surface isomers to explore SurfDesign's robustness to minor structural variations. The isomers we used share the same amino acid sequence but differ in the crystal structure (thus resulting in different surface geometry) due to crystallization conditions, with slight variations in backbone RMSD and ligand/ion interactions. By designing sequences robust to these surface isomers, SurfDesign demonstrates the potential for applications where protein surfaces might slightly differ due to environmental or crystallization conditions, which is relevant in drug discovery and functional protein design. We have expanded these findings in the appendix and provided a visualization of this pair of selected surface isomers (**1lyz** and **193l**) to provide the comprehensive data requested. We expect more future efforts to find *surface isomers* with more significant distinction in surface geometries and investigate SurfDesign's robustness to these variants. 

-----------------------
Thank you again for your valuable insights, which will help us further strengthen the presentation and impact of our work.
A Kind Reply to Dear Reviewer u1Ve
Thank you for the thoughtful and comprehensive review, as well as for acknowledging the contributions of SurfDesign to surface-based protein sequence generation. It is a great honor to receive a 8-score, which confirms our efforts in this work. Let me address and respond to your questions point by point:

(1) **Potential Data Leakage via PLM Training Dataset:** We appreciate your suggestion to address possible data leakage. Even though no-PLM ablation experiments have already demonstrated better performance than SurfPro, we further examined the sequences generated by SurfDesign with the training corpus of ESM-2 (UniRef90). The results show that **the sampled sequences of SurfDesign have no overlap with any sequence from ESM's training set**, namely, the sequence identity is less than 100%. Additionally, the distinct recovery rates for surface and core residues (as shown in Figure 4) support that the model does not merely memorize sequences, as surface recovery is generally lower than core recovery, indicating the model's nuanced understanding of structural features rather than memorization.

(2) **Surface Description Differentiability:** We acknowledge that the current surface extraction and encoding methods are not differentiable with respect to atomic coordinates, which may limit certain downstream applications. However, as you noted, this work focuses on introducing a robust, invariant representation of protein surfaces, and incorporating differentiability is an exciting direction for future exploration.

(3) **Choice of Surface Extraction Method (PyMOL vs. Biopython):** We opted for PyMOL over Biopython for surface extraction because PyMOL can provide normal vectors, whereas Biopython only computes surface coordinates (see https://biopython.org/docs/1.75/api/Bio.PDB.ResidueDepth.html) without generating normals. Normal vectors are essential for our surface-based equivariant message passing (SEMP), as they enable us to capture finer geometric features like dihedral angles within the surface point cloud. The "minor difference among toolkits" refers to processing efficiency—both PyMOL and other tools are comparably fast. However, PyMOL’s reliability and precise control over surface generation, even in edge cases, make it the preferred choice. We have clarified this in the paper for transparency.

Minor Errors: Thank you for pointing out these text errors. We have updated the manuscript to correct the unclear wording on line 134, replace "studies" with "studied" on line 457, and ensure consistency in the formatting of numerical data across lines 315, 317, and 322.

------------------
Once again, we appreciate your detailed feedback and suggestions, which have strengthened the presentation and robustness of our work.
Response to Reviewer H1WD
(4) **Statistics on Generated Protein Surfaces:** This table (also the same table in Appendix Table 7) documents the vertex count statistics for the CATH datasets. We observe an equal distribution over vertex in different splits. 

| | CATH 4.2 |  |  | CATH 4.3 |  |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  | Train | Validation | Test | Train | Validation | Test |
| Average Vertex Count Per Residue | 53.47 | 53.56 | 53.31 | 53.36 | 55.27 | 53.11 |
| Maximum Vertex Count | 27,817 | 25,614 | 25,433 | 27,110 | 27,817 | 25,968 |
| Minimum Vertex Count | 1,923 | 2,315 | 2,022 | 1,923 | 2,011 | 2,000 |

------------------
Thank you once again for your valuable feedback, which has helped us improve both the clarity and scope of our work. If you have any further questions or comments, please do not hesitate to ask. We are looking forward to your reply!
A Kind Rebuttal to Reviewer WG2F (Part II)
Thank you for your constructive feedback and insightful questions. We appreciate the opportunity to address your concerns, as they touch on several core aspects of our work.

(1) **Clarification on Practical Applicability and Source of Protein Surfaces:**
You raise an important point regarding SurfDesign’s reliance on all-atom protein structures for generating surfaces and whether the observed performance improvements may stem from incorporating this additional structural information. While it is true that SurfDesign currently relies on all-atom structures to compute surfaces, we envision future applications where surfaces could be inferred from **low-resolution experimental data or predicted structures**, expanding SurfDesign’s practical utility. Here, we conducted an additional experiment to investigate the robustness of SurfDesign over predicted surfaces. Specifically, we leverage ESM-Fold to forecast the structures of 1,120 test samples in CATH 4.2 and generate corresponding surfaces. Then we require SurfDesign to design sequences based on these pseudo-surfaces. We discovered that the **AAR decreases from 74.13% to 52.66%, which demonstrates SurfDesign's generalization to designing proteins without experimental structures.** We hope this ablation study can help you understand SurfDesign's practical application!

Additionally, we note that the inclusion of surface features was carefully designed to enhance functional site representation and complement the protein language model rather than simply encode side-chain details. The improved performance thus reflects SurfDesign’s ability to leverage surface-derived functional features that capture biologically relevant information beyond traditional backbone-centric methods.

(2) **Efficiency Analysis:** We recognize the importance of assessing SurfDesign’s computational efficiency, especially given its use of surface encoding, which involves more points compared to backbone-only models. After preprocessing the surface data, the primary computational component is our equivariant message-passing neural network (SEMP) for the surface point cloud. To maintain efficiency, we typically stack fewer than five layers of SEMP, and we construct a KNN graph with $𝑘 =4$, so each node only gathers information from a limited set of neighbors. By controlling the number of layers and the node’s receptive field, we ensure that SEMP remains lightweight and computationally efficient while still effectively capturing the geometric and chemical characteristics of protein surfaces.

(3) **Additional Experiments on Functional Protein Design:** We appreciate your suggestion to explore SurfDesign’s utility beyond inverse folding for functional design tasks such as protein optimization. This aligns well with our goal to apply SurfDesign to tasks that emphasize functional and stability constraints in design. As you recommended, we follow the setting of SurfPro [A] and employ SurfDesign to design proteins that bind to a target receptor with high affinity. Specifically, AF2 pAE_interaction is used to evaluate the binding affinities between the designed binders and target proteins, which is found very effective in distinguishing experimentally validated binders from non-binders. Experimentally confirmed positive complexes of <binder, target protein> pairs across six categories from Bennett et al. (2023) are collected for experiments. Among the 10 categories available, 4 exhibit indistinguishable AF2 pAE_interaction between negative and positive binders. As all baselines were finetuned using the binder design dataset after pretrained on the CATH 4.2 dataset, we keep the same setup and report the result below. The results illustrate that our SurfDesign achieves the lowest average AF2 pAE interaction across six target proteins, which strongly supports our motivation to incorporate surface for functional protein design! 

| **Models** | **InsulinR** | **PDGFR** | **TGFb** | **H3** | **IL7Ra** | **TrkA** | **Average** |
|---|---|---|---|---|---|---|---|
| Positive Binder | **5.9996** | **14.1366** | **15.4884** | **21.2631** | **20.9102** | **10.2791** | **14.7061** |
| Negative Binder | 19.7167 | 18.0937 | 23.2664 | 22.4556 | 26.0540 | 24.7567 | 21.1335 |
| Random Baselines | 19.9880 | 21.2690 | 21.4971 | 24.4997 | 24.1541 | 23.1147 | 22.2020 |
| ProteinMPNN  | 18.3393 | 25.2919 | 25.8559 | 24.5968 | 25.5278 | 27.0980 | 23.4462 |
| PiFold  | 12.9809 | 21.8230 | 24.4737 | 23.3924 | 26.6738 | 19.7172 | 20.5785 |
| LM-DESIGN  | 13.6440 | 22.0749 | 23.3725 | 23.8332 | 24.3937 | 22.3987 | 20.7728 |
| SurfPro | 10.2608 | 17.9862 | 17.7364 | 21.2916 | 20.8594 | 10.6535 | 16.9485 |
| **SurfDesign** | **9.9827** | **17.5460** | **17.4381** | **21.0558** | **20.8213** | **10.5094** | **16.22555** |

[A] SurfPro: Functional Protein Design Based on Continuous Surface. ICML 2024.
A Kind Rebuttal to Reviewer WG2F
(2) **Limited Analysis of Multi-chain Complex Dataset:**
Thank you for noting this aspect. To provide a more in-depth evaluation of multi-chain complexes, we conducted additional experiments to assess SurfDesign’s performance on complex interface design tasks. Specifically, we evaluated the generated sequences for their ability to preserve interaction-site functionality while introducing sequence diversity. Three additional metrics are included: **Binding site ratio (BSR)** compares the overlap between the binding sites of the generated and native complexes after alignment.  **Affinity** is defined as the percentage of generated complexes with higher binding affinities (lower binding energies) than the native complexes, calculated by Rosetta. **Foldable diversity (FD)** only considers the diversity between pairs of sequences that are both structurally consistent with the input protein backbone, computed in [A]. We compared our model with LM-design, focusing on metrics that capture interaction-specific properties to highlight SurfDesign's capacity to handle interface designs effectively. This result strongly demonstrates that our surface-based inverse-folding model outperforms other models on the interface design of protein complexes.

|  | **BSR** | **Affinity** | **FD** |
|---|---|---|---|
| LM-Design | 47.3% | 6.5% | 11% |
| SurfDesign | **55.9%** | **8.0%** | **18%** |

[A] Reinforcement learning on structure-conditioned categorical diffusion for protein inverse folding. arXiv 2024.10.

(3) **Lack of Computational Efficiency Comparison:** 
The primary distinction between SurfDesign and conventional inverse folding methods is the computation of surface geometry and chemical properties, which we have designed to be highly efficient. Specifically, we generate the protein surface using PyMol, a process that takes less than 0.1 second per protein. For physicochemical properties, we utilize the same approach as in MaSIF, which is similarly efficient.

| | CATH 4.2 |  |  | CATH 4.3 |  |  |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  | Train | Validation | Test | Train | Validation | Test |
| Average Vertex Count Per Residue | 53.47 | 53.56 | 53.31 | 53.36 | 55.27 | 53.11 |
| Maximum Vertex Count | 27,817 | 25,614 | 25,433 | 27,110 | 27,817 | 25,968 |
| Minimum Vertex Count | 1,923 | 2,315 | 2,022 | 1,923 | 2,011 | 2,000 |
| Preprocess Time Per Protein |  | 0.38 s |  |  | 0.36 s |  |

Additionally, we compared PyMol with MSMS, the latter used by SurfPro (the first surface-based baseline), and observed that the time cost is nearly equivalent between the two. Consequently, SurfDesign introduces minimal additional computational expense, maintaining competitive runtime with other methods in this domain.

We hope this reassures you that SurfDesign’s efficiency aligns closely with conventional inverse folding approaches, adding only negligible overhead due to surface-based feature computations.

----------------
Thank you again for your constructive feedback. We believe these additions will strengthen our model's presentation and provide more comprehensive evidence of its efficacy in surface-based protein design. We hope to address all your concerns and improve our paper with your advice!
A Kind Rebuttal to Reviewer Gipq (Continued)
Thank you for your thoughtful review and detailed feedback. We appreciate your valuable insights, which highlight areas for further improvement and help us enhance our model's utility and rigor. We address each of your points below.

(1) **Insufficient Metrics for Functional Preservation:** Thank you for your insightful suggestion on introducing metrics to evaluate SurfDesign’s effectiveness in designing sequences for specific functional regions. Following your guidance, we have integrated metrics from [A] and [B] to provide a more comprehensive assessment of SurfDesign’s performance.

**Structural Compatibility and Stability (Ref-TM and Ref-pLDDT):** Following [A] and PDB-Struct [C], we have incorporated two structure-aware metrics to evaluate how well SurfDesign preserves structural integrity and stability. Ref-TM measures how closely the generated sequences align with the ground truth structure through the TM-score, assessing SurfDesign’s ability to respect structural conditions. Ref-pLDDT provides a per-residue folding stability score for generated sequences, with values based on AF2’s pLDDT confidence. On the same CATH test samples used in [C], SurfDesign achieves outstanding results: a Ref-TM of 0.89 and a Ref-pLDDT of 89.42, demonstrating structural robustness on par with ProteinMPNN (which achieves a Ref-TM of 0.87 and Ref-pLDDT of 87.89). These results substantiate SurfDesign's effective, generalized design capabilities beyond mere memorization, supporting robust performance across diverse structures.

| Design method | ESMFold |  | OmegaFold |  | AlphaFold2 |  | Recovery% |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  | TM | pLDDT | TM | pLDDT | TM | pLDDT |  |
| Wildtype | 0.80 | 74.91 | 0.75 | 78.39 | 0.90 | 89.87 | 100 |
| Uniform | 0.05 | 27.68 | 0.05 | 31.53 | 0.06 | 33.68 | 5.00 |
| Natural frequencies | 0.07 | 30.53 | 0.07 | 35.59 | 0.06 | 35.02 | 5.84 |
| AF-Design | 0.53 | 61.37 | 0.53 | 72.04 | 0.52 | 75.29 | 15.95 |
| ESM-Design | 0.38 | 59.65 | 0.38 | 62.66 | 0.37 | 60.02 | 17.33 |
| StructTrans | 0.72 | 68.85 | 0.64 | 70.35 | 0.79 | 80.66 | 35.89 |
| GVP | 0.73 | 69.67 | 0.67 | 74.33 | 0.83 | 84.29 | 39.46 |
| ProteinMPNN | $\underline{0.80}$ | $\underline{76.53}$ | $\underline{0.76}$ | $\mathbf{80.75}$ | $\underline{0.87}$ | $\underline{87.89}$ | 41.44 |
| PiFold | 0.71 | 67.55 | 0.64 | 70.21 | 0.82 | 82.54 | 44.86 |
| LM-Design | 0.73 | 72.12 | 0.70 | 77.58 | 0.85 | 87.26 | $\underline{51.23}$ |
| SurfDesign | $\mathbf{0.81}$ | $\mathbf{79.35}$ | $\mathbf{0.76}$ | $\underline{80.11}$ | $\mathbf{0.89}$ | $\mathbf{89.42}$ | $\mathbf{70.19}$ |

**Foldable Diversity and Self-consistency (Foldable Diversity and sc-TM):** In addition to the amino acid recovery rate, we have incorporated Foldable Diversity and Self-consistency TM score (sc-TM) as recommended in [B] to further verify the diversity and self-consistency of the generated sequences. Foldable Diversity evaluates only those sequence pairs that are structurally consistent with the input protein backbone, providing a more targeted diversity metric that avoids penalizing high-quality, diverse designs. Self-consistency TM score (sc-TM), following [D], gauges the consistency of structural predictions for generated sequences, leveraging a fixed threshold of $TM_{min}=0.7$ as implemented by [B]. We refer to https://github.com/flagshippioneering/pi-rldif for computation, and the results are shown below (numbers of all baselines are copied from [B] and we thank authors for their contributions). The analysis shows that SurfDesign maintains high structural consistency with competitive diversity, outperforming other methods on foldable diversity metrics and providing substantial evidence of the model’s capability to generate high-quality, diverse sequences that remain faithful to the structural constraints of input proteins.

| Dataset | Model | Foldable Diversity $\uparrow$ | sc-TM $\uparrow$ |
| :--- | :--- | :---: | :---: |
|  | ProteinMPNN (T=0, RD) | 20% | 0.80 |
|  | ProteinMPNN (T=0.1) | 23% | 0.67 |
|  | ProteinMPNN (T=0.2) | 3% | 0.30 |
|  | ProteinMPNN (T=0.3) | 0.1% | 0.14 |
| CATH-all | PiFold (T=0.1) | 23%| 0.72 |
|  | PiFold (T=0.2) | 8% | 0.38 |
|  | KWDesign (T=0.1) | 18% | 0.79 |
|  | KWDesign (T=0.2) | 23% | 0.58 |
| | SurfDesign (Ours) |  **23%** | **0.84** |

[A] ShapeProt: Top-down Protein Design with 3D Protein Shape Generative Model. arXiv 2023.12.

[B] Reinforcement learning on structure-conditioned categorical diffusion for protein inverse folding. arXiv 2024.10. 

[C] PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design. arXiv 2023

[D] Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. ICLR 2022. 

Thank you for prompting these enhancements, which have allowed us to highlight SurfDesign’s advantages in both structural stability and diversity, underscoring its suitability for functional and structurally informed sequence design.
A Kind Rebuttal to Reviewer Gipq
(4) **k-NN Parameter in Surface Graph Construction:** We apologize for the omission. For our surface graph construction, we used a k of 4, chosen empirically based on performance in preliminary tests balancing computational efficiency and graph completeness. We will include this information in the manuscript and provide more context on the selection rationale.

(5) **Curvature Feature Proof or Reference:** We appreciate your request for more rigor here. The curvature feature ψ was designed to be roto-translation invariant, based on theoretical work on differential geometry. We have added a brief proof and references in the Appendix to support this claim (please see the revised version).

(6) **Adaptability to Other Tasks:** Thank you for your interest in exploring the adaptability of SurfDesign. In principle, the surface-based framework could be extended to tasks such as backbone or all-atom generation. In a concurrent study, we investigated **incorporating surface-based features into full-atom design using ProtPardelle** [C]. The results indicate that integrating surface geometric and chemical properties enhances performance. We will include a discussion on these potential adaptations in the future work section of the paper.

| Model | Length 100 |  |  |  |
| :---: | :---: | :---: | :---: | :---: |
|  | Quality |  | Diversity | Novelty |
|  | scTM $\uparrow$ | scRMSD $\downarrow$ | Max Clust. $\uparrow$ | Max TM $\downarrow$ |
| Native PDBs | 0.91 | 2.98 | 0.75 | N/A |
| ProtPardelle | 0.56 | 12.9 | 0.57 | 0.66 |
| ProtPardelle  + Surface | 0.61 | 9.84 | 0.61 | 0.60 |

[C] An all-atom protein generative model. PNAS 2024. 

-------------------------
We thank you again for your constructive feedback and the opportunity to clarify and strengthen our work. We look forward to discussing this with you shortly.
A Kind Rebuttal to Reviewer FM5g (Part II)
Thank you very much for your detailed review and thoughtful feedback. We appreciate your comments and suggestions, which will help us enhance our model and presentation. We address each of your points below.

(1) **Metrics for Protein Design:** We agree that sequence recovery and perplexity alone may not fully capture SurfDesign’s effectiveness for practical protein design tasks, and we appreciate your insight into this potential limitation. While wet-lab experiments are beyond the scope of our current study, we have included additional metrics as suggested by PDB-Struct [A] to provide a more robust assessment of SurfDesign. Specifically, **Ref-TM** assesses whether the generated sequences can respect the structure condition, which evaluates the agreement of the ground truth structure with the predicted structures using theTM-score. **Ref-pLDD** evaluates the folding stability of the generated sequences, which is computed as the mean value of the per-residue confidence estimate pLDDT predicted by the structure prediction models. 

We examine SurfDesign on the same 82 test samples of the CATH dataset and results are reported below (numbers of all relevant baselines are copied from PDB-Struct [A] and we thank authors for their contributions). We observe that SurfDesign stands out as the leading design method across the refoldability metrics, competitive with ProteinMPNN. It achieves 0.89 Ref-TM and 89.42 Ref-pLDDT with AlphaFold2 prediction. ProteinMPNN is slightly behind with a 0.87 Ref-TM and 87.89 Ref-pLDDT, followed by LM-Design. This additional analysis, including functional enrichment, diversity, and secondary structure compatibility, provides stronger evidence that **SurfDesign's 60-70% sequence recovery rate is not indicative of overfitting but instead suggests effective design, supporting its relevance beyond the training distribution.** 

| Design method | ESMFold |  | OmegaFold |  | AlphaFold2 |  | Recovery% |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  | TM | pLDDT | TM | pLDDT | TM | pLDDT |  |
| Wildtype | 0.80 | 74.91 | 0.75 | 78.39 | 0.90 | 89.87 | 100 |
| Uniform | 0.05 | 27.68 | 0.05 | 31.53 | 0.06 | 33.68 | 5.00 |
| Natural frequencies | 0.07 | 30.53 | 0.07 | 35.59 | 0.06 | 35.02 | 5.84 |
| AF-Design | 0.53 | 61.37 | 0.53 | 72.04 | 0.52 | 75.29 | 15.95 |
| ESM-Design | 0.38 | 59.65 | 0.38 | 62.66 | 0.37 | 60.02 | 17.33 |
| StructTrans | 0.72 | 68.85 | 0.64 | 70.35 | 0.79 | 80.66 | 35.89 |
| GVP | 0.73 | 69.67 | 0.67 | 74.33 | 0.83 | 84.29 | 39.46 |
| ProteinMPNN | $\underline{0.80}$ | $\underline{76.53}$ | $\underline{0.76}$ | $\mathbf{8 0 . 7 5}$ | $\underline{0.87}$ | $\underline{87.89}$ | 41.44 |
| PiFold | 0.71 | 67.55 | 0.64 | 70.21 | 0.82 | 82.54 | 44.86 |
| LM-Design | 0.73 | 72.12 | 0.70 | 77.58 | 0.85 | 87.26 | $\underline{51.23}$ |
| SurfDesign | $\mathbf{0.81}$ | $\mathbf{79.35}$ | $\mathbf{0.76}$ | $\underline{80.11}$ | $\mathbf{0.89}$ | $\mathbf{89.42}$ | $\mathbf{70.19}$ |

[A] PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design. arXiv 2023

(2) **Features and Potential for Data Leakage:** We appreciate your concern regarding potential data leakage from features like hydrophobicity and charge. However, we want to restate and clarify that the task we want to solve in this paper is **surface-based protein design** instead of conventional backbone-based inverse folding. This idea of surface-based design was originally proposed by SurfPro [B], where models design proteins conditioned on both geometric shape and biochemical properties of molecular surfaces. In other words, **surface-conditioned protein design targets functional proteins that fold into specified surfaces, denoted as $\mathcal{Q}$, with associated biochemical properties—without relying on explicit backbone information**.

Importantly, the goal of protein design extends beyond predicting a sequence that simply folds into a given backbone. For functional design, we aim to create proteins with targeted activities, such as substrate binding or inhibition. Traditional inverse folding is limited in specifying function-related constraints, as it uses only the backbone structure. By conditioning on surface properties like charge and hydrophobicity, our approach imposes additional biochemical constraints crucial for functionality. For example, complementary shapes alone may not ensure effective binding if charges, polarity, or hydrophobicity are poorly positioned.

[B] SurfPro: Functional Protein Design Based on Continuous Surface. ICML 2024.

(3) **Typos and Clarity Issues:** Thank you for pointing out these minor errors, which we will correct in the revised manuscript. Specifically, we’ll revise “Priliminary” to “Preliminary” (L 104) and replace “LM-DESIGN” with “SurfDesign” in L 365 to eliminate confusion.
A Kind Rebuttal to Reviewer FM5g
Yes
No ethics review needed.
- Line 89 What does O optionally, mean? Is it included in the backbone representation or not? 
- How is $h_i$ computed, the description reads “such as hydrophobicity, hbond, and charge”. The "such as" description is not specific. 
- It would be interesting to further explore the models performance on heterogeneous and homogeneous multi-chain complexes, examples of both are in the PDB database. 
- Line 423 I believe Figure3.3 should say Figure 4 here. 
- It would be nice to expand Figure 4 to include exposed vs buried, helical vs sheet vs loop/turn, for completeness and possibly some metrics of hydrophobicity and electrostatics. 
- Table 5 – should the row MoE-DSR be SurfDesign, based on the accompanying text? Otherwise MoE-DSR is not defined in the text anywhere 
- The section on surface isomers feels incomplete, and I couldn’t find accompanying information in the appendix. 
    - What experiment was done? 
    - Were only 2 lysozyme isomers compared?
    - Do the isomers have different amino acid sequences? if so how different are they. 
    - How structurally different are the isoforms (backbone RMSD), and what are the conditions under which they were crystalized (are ligands or ions present in either crystal structure?). 
    - What is the biological significance of designing sequences that are robust to structural isomers
The inverse design method proposed in this work seems to be powerful and promising. The authors perform an exhaustive set of benchmarks where they rank best against strong baselines, including another surface-based method. The formulation is intuitive and straightforward and appears to significantly improve sequence design.
In this work the authors seek to address a longstanding challenge in functional protein design by augmenting inverse protein modeling with continuous surface representations. To this end, they present SurfDesign, a model trained on surface manifolds, augmented with their surface-based message passing scheme and knowledge from published PLMs. The generated sequences are evaluated on CATH 4.2 and 4.3 test sets, including short (less than 100AAs), and  single-chain ablations. Further, proteins are evaluated in a multi-chain setting using PDB structures, and on zero-shot generalization to unseen proteins on a held out test set. SurfDesign is superior on all benchmarks. The authors also evaluate the model on structure recovery, and providing scaling data, which align with prior PLMs.
The main biological significance of this method would to be in using a surface-based method to design stronger protein-protein interacting systems, however demonstration of the model in this application (even via in-silico metrics, and an exemplary binder problem) seems to be missing.
Yes
No ethics review needed.
1. Minor errors in text:
- line 134: "discover ignorable differences". The line is unclear; are the differences insignificant? Could you state why you chose cumbersome method of using pymol for surface extraction instead of convenient biopython package is the differences are minor?
- line 457: studies -> studied
- lines 315, 317, 322: please use different delimiters between decimals and numbers in the dataset splits
The main strength of the work is that authors archive state of the art performance on a variety of benchmarks with extensive validation of their claims. The contribution to the surface description of a protein is novel and relevant to the field. Their design of equivariant surface message passing algorithm is also novel because it integrates curvature information due to use of surface-continuous Darboux frames instead of point clouds. These two contributions coupled with extensive validation already warrant the publication of this work.
The paper proposes a new method to generate sequence of a protein conditioned on the surface descriptors. This paper proposes improvement to the surface description by incorporating two continuous tangent vectors in addition to the normal vector at each point on the surface (Darboux frame). Moreover, the constructed frame is continuous w.r.t the point on the surface (Eq. 3-4). Additionally authors propose equivariant message passing network based on 3D Spherical Fourier-Bessel decomposition of the previously constructed Darboux frames relative positions and orientations; the rest of the message passing graph NN follows standard design. Finally, authors use fine-tuning of pretrained protein language model as a sequence decoder.
The authors extensively test their approach on CATH-4.2, CATH-4.3 benchmarks. Additionally they tested their approach to multi-chain sequence recovery using Dauparas et al dataset. SurfDesing archieves state of the art performance on CATH-4.2 and CATH-4.3 datasets, compared to the previous methods in terms of perplexity and sequence recovery. 
Authors provide evidence that each contribution (model equivariance and inclusion of PLM) contribute to the performance of their approach (Table 1, shaded rows). Additionally they show state of the art metrics in structure and surface recovery. Finally, they demonstrate that scaling PLM improves performance and that their approach is tolerant to changes in surface depending on different conditions used to crystallize the protein.
The main weakness of this work is the possibility of data leakage between training and test sets. The authors use PLM as the sequence decoder, which training dataset contains sequences from CATH database.  Although the no-PLM ablation experiment shows performance on par with SurfPro (similar sized model), we suggest showing that generated sequences do not have 100% identity to any sequence from the PLM training datasets. The second possible data leakage pathway is that exact curvatures of the surface may perfectly encode the identities of surface residues, therefore is may be useful to show sequence recovery, however the Figure 4 seems to disprove this possibility, due to core residues having higher recovery rates than the surface residues. Altogether we think that this possible weakness of this work is inconsequential. 
Another weakness is that this approach to surface description is not differentiable w.r.t atomic coordinates, but it falls outside of the scope of this work.