Acceptance into graduate math programs
I don't see this question as being definitively answerable without a lot more information, both about your background and qualifications and about where you are applying.  Grad schools differ enormously in their approach to admissions in unusual cases.  Some lower-tier schools are eager to take a chance on smart but nontraditional applicants who will be rejected by more prestigious universities.  Others have established rigid rules after having had bad experiences with admitted students who had inadequate backrounds or didn't understand what they were getting themselves into.

Regarding grades, keep in mind that the world has seen a lot of grade inflation, and there are startlingly large numbers of applicants with nearly perfect grades.  You generally cannot impress an admissions committee by having excellent grades, although poor grades can hurt your chances.  Math departments typically care a lot about grades in mathematics and heavily mathematical fields (e.g., theoretical physics or CS), a little about grades in other scientific or engineering fields, and not at all about other grades, except that exceptionally poor grades in any field suggest the applicant is not good at getting things they don't care about done (which is not a good omen for their career).  I.e., math departments don't care if you get a B- in history, but getting an F in history could be a problem.

Recommendation letters are absolutely crucial.  In my experience they are by far the most important factor: other aspects of the application could hurt your chances even if you get good letters, but those other aspects cannot get you admitted by themselves.  If you can't arrange for enthusiastic letters that the admissions committee will trust, then you have no chance at all of being admitted at a higher-ranked school.  It's possible that a lower-ranked school might take a chance on you, but as I understand it every graduate program cares a lot about letters.

As for math papers, it's unfortunately not hard to write a near-vacuous or unoriginal paper and get it published somewhere obscure, and carefully evaluating a paper can take a lot of time and effort (refereeing is hard).  If you simply list a paper in your application, with no evidence of its quality, then it's unlikely anyone will have the time to investigate carefully.  The important thing for graduate admissions is not the publication per se, but rather the research experience that led to it, and you need a letter of recommendation that talks about this research and vouches for it.  (In fact, having such a letter can be very valuable even if the research did not lead to a published paper.)  This means you need a letter from your supervisor or mentor, or at the very least from someone who has read the paper and knows something about how it came about (for example, someone who can single out your own contribution if you had coauthors).  In principle, if you publish in an especially prestigious journal and without senior coauthors, then the journal's high standards might serve as enough of signal of quality by themselves.  However, this very rarely happens with undergraduate research.

So in short:


Bad grades could hurt you but good grades won't serve as a strong argument for admission.
Recommendation letters are crucial.
Research experience could help you get great letters but is very difficult to judge except through the letters.


However, as I said in the beginning, there are a lot of graduate programs out there, with different approaches to admissions.  If all goes well, you'll get strong letters from math faculty and will be admitted in more or less the usual way.  Otherwise, you may have to find a school that is willing to take a chance (usually because they have trouble attracting strong students with conventional backgrounds).