Pretrained pLM and fine-tuning As well as the amino acid descriptors and molecular fingerprints, the pretrained pLMs were treated as feature extractor. The per-residue averaged embeddings from the final layer of the pLMs were used as input for the GPR surrogate model. We studied the out-of-box performance of these pretrained pLMs without any fine-tuning as well as after fine-tuning them on peptide sequences.