To account for the lack of previous information on optimal hyper-parameters for molecule generation with S4, we implemented a two-step procedure for hyper-parameter tuning. First, 242 models were trained to prioritize hyper-parameters (see supporting information, Table ). High-performing hyperparameter values in terms of validation accuracy were advanced to the second phase, where 108 experiments were conducted. Hyper-parameter search was conducted for 10 days on multiple NVIDIA A100 40GB GPUs to maximize the validity during pre-training.