Pre-trained language model and fine tuning. We pre-trained the Gemma-based model with 3 billions parameters on the whole SMILES strings of ChemBERTa's PubChem-10M. It is worth noting that we used adapters with the aim of training on all the 10M data while saving computational resources. We trained our model without fixing parameters for a small number of epoch, then we fixed the model parameters and the adapters were added to pre-trained model. We used Low-Rank Adaptation (LoRA) , greatly reducing the number of trainable parameters for downstream tasks. In the subsequent pre-training, we only trained the parameters within the adapters. We utilized the Adam optimizer along with cosine annealing for learning rate scheduling, with the initial and final learning rates set to 5e-4 and 5e-8, respectively. We set the batch size 64 and the maximum sequence length 64. Our SMILES tokenizer uses character-level byte-pair encoding instead of bytelevel encoding during tokenization. We initially reserved 72 characters from the SMILES alphabet as our base vocabulary and expanded it with the 1000 most frequent merges. The model incorporates parameterized token and position embeddings, with 8 attention heads and 4 attention blocks. Featuring an embedding and hidden dimension of 512.