Second, Figure shows the validation performance of all our model configurations utilizing RAG as a function of the number of retrieved documentation pages. It shows that non-finetuned models tend to slightly increase in validation performance as K increases. However, for the finetuned models, we generally observe a decline in performance as K grows until the model context window is filled (K =Limit). The only exception is the bruteforce model with CoT, where retrieving five documentation pages shows best validation performance. It seems that the finetuned models handle external context poorly. We hypothesize that they either 'forget' how to handle the additional context or get 'confused' by it. This could be explained by the models not being trained to use RAG. Alternatively, RAG might only be useful when the model has little knowledge about ORCA input files.