Explanation of random forest performance difference to when using categories and when using dummy variables
It should work: the variable is ordinal so using numerical values makes sense.
So there's a bug somewhere, here are a few suggestions of things to look at:

Possibly a type conversion error somewhere: make sure the variable is interpreted as numerical.
Check whether the model actually uses the variable: if not then it's likely some type error; if yes then I would investigate what goes wrong: for example it might help to plot this variable vs. target in the two cases where the variable is categorical or numerical.
Maybe some difference between the preprocessing of the training and test set: apply the model on the training set, if the performance is good then it's likely that there's something wrong in the preprocessing of the test set.