Importantly, ORCA inputs lack the hierarchical complexity and linguistic properties that typical code and NL translation metrics assess (e.g., word order), making these metrics unsuitable here. Instead, the presence of specific terms necessary for correct execution is key for ORCA inputs. Hence, we framed this task as a classification. Accordingly, we evaluated all models via the F1 total and F1avg scores on the ORCAExtracted test set, and determined the fraction of executable input files. In contrast to F1 total , F1avg weighs all three ORCA sections equally rather than assigning more weight to the section containing more terms.