Additionally, we also studied whether there was a bias towards choosing a compound depending on its position (left/right) during annotation. After the first initial preliminary round was completed, we had received qualitative feedback from the chemists on some of the presented pairs. Specifically some criticism was expressed in regards to some pairs being inherently hard, as both compounds contained clearly problematic motifs (e.g., "plague vs. cholera" The probability of one winning over the other is provided by feedback as supplied by chemists. For this, the chemists are asked to select one or the other depending on a pre-specified question prompt on a web application. An implicit score model is then learned based on this feedback, which can later be used for downstream cheminformatics tasks. pairs where both compounds featured known toxicophores). These were then removed according to the procedure detailed in Section 2.3. A second round with identical number of pairs was subsequently carried out. Note that in the first and second preliminary rounds, all chemists were handed out the same pairs (i.e., we performed inter-rater repetitions), so as to adequately evaluate the points presented above. After both preliminary rounds yielded satisfactory results (see Section 4.1), we set out for a production run where we obtained over 5000 responses over the course of several months. Furthermore, since a reasonable degree of agreement between the chemists in the preliminary rounds was observed, we forwent the pair repetition requirement in the production runs and considered all participating chemists as a single labeling oracle.