In this work, we presented ORDerly, an open-source framework for preparing chemical reaction data stored in the Open Reaction Database (ORD) for machine learning applications. ORDerly was used to generate benchmark datasets for forward prediction (ORDerly-forward), retrosynthesis (ORDerly-retro), and condition prediction (ORDerly-condition) based on US patent data. Transformer models were trained on the forward prediction and retrosynthesis datasets, and they were found to only generate invalid SMILES strings very infrequently, while also achieving similar test accuracy to that found in the literature on a held-out set of US patents. ORDerly was also used to generate test sets from all non-patent data from ORD, which could serve as a better indication of model generalisation. Accuracy for the forward prediction task was comparable on the non-USPTO test set, while accuracy on the retrosynthesis task was somewhat lower. The condition prediction task was used to investigate different strategies for assigning reaction roles and frequency filtering of the spectator molecules. When building datasets for condition prediction using the labeling in ORD we found contamination of the inputs (reactants) with the outputs (agents), resulting in a problem that was unrealistically easy. We therefore chose to use chemically informed logic to better assign reaction roles for the ORDerly-condition benchmark.