How is the quality of a paper evaluated for application purposes?
The quality of a mathematical paper isn't an absolute. It may not even be apparent for years. Quality is judged by individuals and two judges may come to opposite conclusions on the value. 

Number of citations is no more than an indication. A person working in an especially arcane sub-field with few other researchers will naturally have fewer citations than someone working in a popular field. Who cites your work may mean more, actually. 

The quality of the journal in which you find something is an indication, but partly that is a matter of coincidence. Perhaps the paper in the "lesser" journal is, again, in too arcane a field for the editors to have included it. It isn't that this means nothing, it is just that it isn't an absolute indication. 

If the purpose of the question is anything beyond a hypothetical, related to who is more likely to advance, then that really depends on other factors. Certainly one paper isn't a great indicator. As a comment by Prof. Santa Claus notes, a future advisor is more likely to want to work with the student who has more to contribute to the actual work of the subfield of that advisor. Something in either paper may help that person to make a decision, but other factors will also come in to play. 



This doesn't, of course, address the issue that some problems in mathematics are considered important. You mention famous unsolved questions of course. Giving a first correct proof of a problem already considered important is, of course, quality work.