The tokenizers are central for working with text-based formats such as SMILES strings. The tokenizer in PySMILESUtils consists of an abstract class with methods that are considered common to tokenizers. A provided subclass, the SMILESAtomTokenizer, uses regular expression token patterns for tokenization. Regular token strings can be provided as well. Given token patterns and potential tokens, the dataset is analyzed and the found tokens are used to build the vocabulary that will be used to encode and decode the SMILES strings into tensors of indices ( ). Optionally one-hot encoded tensors can be created. Code Box 1 shows an example of how to work with the SMILESAtomTokenizer using the default regular expression patterns to create a vocabulary from the SMILES in the dataset. The tokens created are chemical symbols, where e.g. Chlorine is not split into "C" and "l", but rather kept as "Cl". After fitting the vocabulary it's easy to convert a given SMILES string into a PyTorch tensor of indices that could go into an embedding layer of a deep learning model. Page 4 of 8