I need a reference letter but I've been out of school for 3 years
In your case it sounds like the strong letters you want from your former instructors are just not going to happen.  None of your professors from 2011 or before can speak to your recent good work in a mathematically relevant branch of industry.  They can only try to remember what kind of student you were back then, and it doesn't sound great.

In your case I would go with letters from people in industry who have skills and qualifications that would be respected by those in Financial Mathematics.  If possible, all but at most one of your letters should come from someone with an advanced degree, and you should make sure that they know you well enough to convey the positive arc of your trajectory in recent years.

You should also take any relevant standardized tests especially seriously: you certainly want a great score on the MATH part of the GRE.  I'm not sure whether the GRE math subject exam is required or applicable for master's programs in financial mathematics: I guess it would be at some programs but not others.  I think that either way it would be good to take some practice versions of this exam, and if you can see that you'll do well (over the 50th percentile is now a good performance from an American student) then you should consider taking it even if it's not required.

You may also have to face the reality that you'll be admitted to a lesser quality program because of your "checkered academic past".  If you're really serious about graduate school, you can spend one or two years in a lesser program, really light up the place, and then transfer into a better program.

Good luck.