Phantom podcasts after restore that won't stay deleted
I am having a similar problem: I had 18 podcast eposodes that were stuck attempting to download. I was able to get rid of 17 of them by manually downloading each one by searching the name in the podcast app. After manually downloading them, a restart of the phone would remove the stuck downloads. That might fix your issue, if you haven't solved it.
My further problem is that the 18th stuck episode is no longer available.

I am going to attempt a restore but I expect to have the same issue afterward.