Masters as a stepping stone to PHD
I'm in chemistry, and I also went to a small, top-20 liberal arts school that did not offer much additional, higher-level coursework. 

I experienced two main challenges: 


Transitioning into graduate-level courses. This was difficult, because of not having experienced similar class-work before. However, the fundamentals you learn at liberal arts colleges are solid, and you've learned how to learn. 
Transitioning to the difference in teaching quality. At most graduate programs, the course instructors are research faculty and their priority is research. It can take some time to adjust to how to learn in a class that was not well-constructed for learning. 


That said, unless the masters programs are very different than the phd programs in mathematics,I think you'll encounter the same challenges in either the masters or the doctoral program. In chemistry and physics, masters students take the same coursework. Additionally, usually you have to write a thesis for a masters, so during your coursework there is more pressure to do successful research, since you only have ~2 years (instead of ~5-6, as in a PhD program). Also in my experience graduate schools only rarely grant course exemptions for masters work. 

You have several options:


Apply directly to PhD program. If you do this, I highly recommend studying some of the advanced coursework over the summer before you start. MIT Open Courseware has some great materials for this. 
Apply to a masters program at a less selective school, then apply to more selective PhD programs. In a less-selective school, the classes might start at a lower level, there might be more support for students, and you might be able to take advanced undergraduate coursework not offered at your undergraduate institution, if you need it.
Get a job or internship and take a few courses. You could take a gap-year in which you get a job or internship in your field. You'll grow your skills and content knowledge via that position, and you may also have time to take ~1 class at a time at a local university, online (through a university or a free service like coursera), or via independent study (like MIT OCW). This way you'll also earn money, if that's a concern. (Masters students may or may not be stipended.)