I feel like my mathematical competence is fading
First there’s no magical “being good at mathematics,” so it’s not something you can suddenly lose.  If you want to get better you can practice more and get better, and if you practice less you’ll get rustier, but neither state is permanent.  Your anxiety around your math skills may be due to this kind of black-and-white thinking about math skill.
In terms of practical advice, one thing that really helps keep you sharp on the basics is teaching.  Maybe try signing up to do a little tutoring, or do something with the undergrad math club.