For pre-training, we randomly split the 706,126 molecular crystal dataset with a train-validation ration of 90%:10%. The model was trained for 50 epochs with a batch size of 512. The AdamW optimizer with a learning rate of 10 -4 and weight decay of 10 -2 was used . The learning rate was warmed up during the first 5% of the total epoch and was then linearly decayed to zero for the remaining epochs. For the SEP task we assigned higher weights to the elements with fewer occurrences due to the great variance in the frequency of the occurrence of different symmetry elements. The individual weights are calculated as follows: