How bad is it to take another year for my undergraduate degree?
If you have a really good relationship with one (or more) of your math profs then you could make  a fifth year work to your advantage. Ask them what research opportunities you have with them, perhaps as an independent study. If you could put together a nice, but do-able, project to fill up the rest of the time for the additional year, then it could work out to be a plus.
But this assumes, for purposes of graduate study, that your GPA is good enough and you have some insight into one or more math subfields. You don't need (nor is it possible to have) universal insight, but something with which you can be clearly predicted to be a success by letter writers.
You are in a difficult place, I think, but is may still be possible to recover through hard work and faculty contacts.