In recent years, the application of deep learning methods have had a dramatic impact in the field of chemistry and drug discovery . Advances in machine learning, including transformers , which have already shown remarkable success in natural language processing and computer vision , are readily adapted to solve domain-specific problems including molecular design. Many different deep learning architectures have been proposed to explore the chemical space. proposed recurrent neural networks, investigated generative adversarial networks, and used variational autoencoders to sample new molecules. All of these techniques allow sampling compounds from the chemical space, but they do not naturally encode the localized search characteristic of molecular optimization. The generated molecules need to be refined with e.g. reinforcement learning approaches or via coupling the latent representation to a predictive model.