We hypothesise that the questions about the intuitive attributions of random models posed by Zafar et al. can be answered by assessing the influence of the initial tokenization or architecture artifacts. This is further supported by our findings that a native encoder sometimes even outperformed pre-trained encoders during transfer learning, as well as research from Ucak et al. who showed that different tokenizations can have significant effects on final performance in NLP-based molecular representation models.