To what extent do grades matter when applying to graduate school in mathematics?
A single bad grade is unlikely to matter much. A history of bad grades is worrisome. Your application to a doctoral program needs to show that you have high likelihood of success in (math) research. This implies that you can demonstrate insight into areas of math that you want to specialize in.
But insight in mathematics isn't uniform over sub-fields and mathematicians understand that (I hope, anyway).
In the US, there are many things taken into account besides GPA and even individual grades. So, you are right to be &quot;a little worried&quot;, but not to panic.
But, for some people, even a poor academic history (not like yours, I think) can be overcome by other factors.
But you might want to be ready to answer a question or two about why you got lower grades in a few courses. Lack of insight is only one possibility. Taking too many courses in a term is sometimes a factor.