Joint model architecture We leverage an encoder-encoder architecture, which allows us to reuse existing models from the literature. To encode SMILES strings, we use the MolFormer model from Ross et al. , which utilized a transformer architecture with rotary positional encoding . For 13 C NMR we use a simple multi-layer perceptron (MLP), while for 1 H NMR and IR we opted for a CNN architecture . We describe the encoder architectures in more detail in Appendix D. Due to varying embedding sizes, we also attach a linear embedding projection layer to each modality encoder, similar to the implementation of Girdhar et al. No encoder weights are frozen as this led to a drop in retrieval accuracy in our preliminary experiment. We attribute this to the limited expressivity of a linear layer.