All benchmarks and datasets experimented with in this work, as well as the code used to generate them, are freely available online, and we hope the benchmarks will make reaction prediction tasks more accessible to ML practitioners with limited domain knowledge. ORDerly presents a fully open-source pipeline to go from raw ORD data to a fully trained condition prediction model, allowing for an avenue to leverage the growing contributions to open source chemistry.