Accurately) pretraining (Fig. ) . The ELECTRA approach is based on training a model to distinguish between "real" input characters and "corrupt" ones, which was previously shown to be useful for contextual representation of natural language . We adapted ELECTRA for the CLM training with an LSTM model and SMILES strings as input . The training data contained corrupted input SMILES strings generated by randomly substituting multiple characters with other characters of the SMILES language. The CLM was trained to spot the corrupted characters. We hypothesized that, compared to autoregressive pretraining, ELECTRA pretraining has a more appropriate inductive bias (i.e., the set of algorithmic assumptions to solve a given task) to extract useful features for ordinal classification. The inductive bias of autoregressive pretraining is particularly suited for generating SMILES strings because the training and generative tasks are the same, namely, adding characters iteratively. However, ligands of the same macromolecular target tend to have similar chemical substructures, and, therefore, the ability of a model to distinguish small structural changes was deemed relevant. At the same time, small structural changes might lead to drastic variation of the biological activity (the so-called activity cliffs) . Hereinafter, the model that was pretrained with the ELECTRA method is referred to as "E-CLM." Fig. | Bioactivity prediction. a, A CLM for molecule generation iteratively predicts the next character in a SMILES string given the preceding characters ("autoregressive" approach). b, An E-CLM (a CLM pretrained with the ELECTRA method) is trained on corrupted SMILES strings aiming to predict, for each string character, whether it is the original (correct) or a corrupted (substituted) character. c, Activity distribution of the PI3Kγ ligands. Compounds with annotated pIC50 ≤ 4.0 were considered "inactive", and a pIC50 value of 6.5 was used to separate the "moderately active" from the "highly active" compounds. d, The molecular structures (in the form of a SMILES string) of the fine-tuning set were used to focus the CLM (pretrained on the US patent database) on the chemical space of the target of interest (PI3Kγ). To account for the uncertainty in the predictions, we employed an ensemble of 100 models to rank the generated molecules by the number of "votes".