To demonstrate this effect, we carried out a simulation experiment to illustrate how the measured performance of a de novo sequencing algorithm can strongly depend on the selection of validation data. We evaluated a pre-trained Casanovo model on a series of datasets of varying quality, each containing 20,000 spectra. Each dataset was randomly sampled from the disjoint test set derived from MassIVE-KB prior to Casanovo training in two steps: first, by selecting the subset of all PSMs with the desired proportion of matched b-and y-ions; second, by randomly downsampling these PSMs to obtain sets of 20,000 spectra each. The results of this experiment (Figure ) clearly demonstrate how a model's apparent performance is contingent on the quality of the data used for evaluation: the average precision changes from 0.99 on spectra with over 50% of the total spectrum intensity explained by b-and y-ions, to 0.84 when the percentage ranges from 30-50%, and to 0.37 when it is below 30%. This effect would likely be even more pronounced with models trained on datasets of varying quality.