There is a difference, however, when the labels were permuted. During training, the decline in F1-score after label shuffling was more pronounced, across all data splits (Figure ). If labels were shuffled during testing, on the other hand, this was not necessarily the case (Supp. Figure ). In the kinase and fam-ily splits, test-phase label permutation did not produce substantially different F1-scores except when SNA was performed, though a drop is still seen with the random split. AUROC, on the other hand, did not show a difference between label shuffling during training versus testing, regardless of data splitting strategy, as it declined relative to the non-shuffled baseline in all splits and rebalancing conditions (Figure , S6 and Supp. Table ).