Best way to evaluate performance for my case
It's not possible to obtain a single f1-score when there are several classes, probably you are using micro or macro f1-score?

First don't use the micro F1-score, since it gives the same weight to every instance and therefore gives a lot of weight to the "X" class.

Macro F1-score should be closer to what you want to evaluate, since it gives every class equal weight. If you want you could even calculate the macro F1-score over only classes A,B,C (i.e. excluding X), but that seems unnecessary to me.