Should I be a visiting student rather than a non-degree seeking graduate student?
Enrolling as a non-degree-seeking graduate student in a U.S. mathematics graduate program is almost never a good idea, and this is in no way a mainstream option.  (I'm talking about not seeking a degree anywhere.  For comparison, it's not uncommon to be officially seeking a degree at University X but visiting University Y while your advisor visits there.)

It's easy to get the wrong impression from course catalogs, since they give short descriptions of options that may not reflect how they are used in practice.  For example, I doubt the Princeton math department ever admits anyone as a qualifying student; if it happens at all, it is exceptionally rare.  I can imagine it might happen for a clearly brilliant student from a deprived background, but not for the vast majority of applicants.  There's just too much competition for admission.

If you are accepted as a non-degree-seeking graduate student:


You won't be treated in any way like an ordinary graduate student.  To the extent anyone in the department is aware of you, you'll be in a special category of "person who wasn't admitted to the graduate program but is paying a lot of money to take courses anyway", which is not a flattering description.  In particular, you should not expect faculty to supervise your research or interact with you any more or differently than they treat the undergraduates in these classes.  (It could happen, but I'd guess it probably won't.)
It won't help with admission, compared with doing equally well in similar courses elsewhere.  Specifically, any admissions committee will have members who want to make sure this isn't used as an easier back door to admission, and they will be sure to enforce strict standards.



  I can graduate this year (as a junior) and become a "qualifying student", or become a "visiting student" and I can graduate a year later.


Are you sure you're interpreting these programs right?  In this listing of the categories, visiting students are enrolled in graduate programs elsewhere, while qualifying students are non-degree-seeking students who are trying to make up for weak backgrounds in the hope of future admission.  I'm not aware of any option for undergraduates to spend a year at Princeton, except for some international exchange programs.  However, I might well be missing some possibility.