Is it worthless to cite a paper as X et al. when X is not first author but performed the work?
References have one major purpose: to be able to trace the information provided in that source. This is why the first (named) author is always given in "et al." references. The problem you encounter is that references are also used to cast merit in the academic system but that is actually a secondary task. So if you decide to use another authors name, you make it more difficult for others to locate the paper n question. So I would strongly advise against the action your propose since it violates the purpose of the referencing system. So even though one can trace papers with for example, doi-numbers breaking the systematic representation of cited sources is not good.

There are other ways to state the source of the work in the text by stating that "the work by X (A et al. yyyy)..." or something similar. But, I would not recommend doing this for just reasons that are beyond anyone's means to control and the risk is that you will not be perceived positively or at least a bit odd. I am sure that the circumstances, if based on unethical behaviour or the like, will become common knowledge in the field.