If you want to do a PhD in mathematics, how important is it to start immediately after finishing undergraduate studies?
But I've been told by a few people that if you want to do a PhD in mathematics, you have to go pretty much right after undergrad, mainly because recent letters of recommendation are so important, and professors forget you after a time. 


You do not have to start a PhD program "pretty much right after undergrad".  It is most common to do so, but there is a substantial minority of students who are older and/or spent several years out of school.  In (American, at least) academia, your age counts for nothing; the ticking clock in the sky keeps track of the number of years since your PhD.  I know several people who spent years off from undergrad in the sense of leaving school but clearly kept up with their mathematical reading and learning -- while in the Israeli army, culinary school, creative writing programs... -- and started grad school with skills at least at good as those around them and a maturity that most 22 year-olds lack.  I've looked through hundreds verging on thousands of job applicants' CVs, and I am struck by how often the stronger candidates were in their 30's rather than their 20's when they got their PhD.  

Of course the biggest risk in taking time off between undergrad and grad is that you will get distracted by the rest of the world and not come back for graduate study.  But that's only a risk in the context of your original plan: if you found something else that you like better than being a graduate student, good for you.  It is also relatively common that after a fairly small time away -- one or two years -- people realize that they really do prefer an academic career.  (For some reason this seems to be most common among high school teachers.  Isn't that a bit sad?)  If you're not totally committed to a PhD program, taking time off and seeing whether your desire waxes or wanes is a pretty smart idea.

Of all things in your decision, I don't think that going straight to a PhD program because you're worried that your professors will forget about you is a good strategy.  Professors don't forget about students that quickly, but after a few years, they may.  To combat this, I would say: if you are not sure whether you want to do a PhD right away, why not apply right away to PhD programs?  Certainly knowing where you can get in and seeing the programs that admitted you are all factors in your decision.  If you apply right away, professors will write letters for you, and if you go away even for a long time, those letters should still be equally usable afterwards: your past undergraduate performance is not a function of time.