Doubting my PhD ability
I was also enrolled in a Math PhD (Numerical Optimization), with difficulties due to lack of time (I was working as a Math teacher simultaneously); when I finished my PhD courses my advisor proposed a research topic... One year at a half later, I also was at a dead end, unable to find any good result and doubting myself.

When I told my advisor at last that I'd decided to move to another program in another university, he propose another topic, assuring me success. He was sure I would get it (as we know, advisors spend time with their doctoral students, and a student failure is also a fiasco for them) because he knew the subject well (the first topic was more unknown to him, and therefore more challenging, but it was also possible that there was no solution!).

He was right: In the new topic, I quickly got results, tutored for my advisor, I was able to present partial results in congresses, got some articles accepted in journals, and at last, two years later, I finally got my PhD. I lost almost two years, but seen from afar (ten years have passed since then) everything is OK.

By the way, recently I saw (in www.researchgate.net) that somebody has the answer to the first problem that I wasn't able to solve ... so it was solvable, and my advisor was right in proposing it to me!

Well, that was my experience, I think similar to yours. Hope this is helpful.