Church-Turing and physical PDEs
The branch of mathematics and computer science that studies these questions is computable mathematics. The general answer is that things tend to be computable. I would add to that the observation that it often takes some work to establish computability. For instance, you mention finite element methods and the problems with their convergence. This proves absolutely nothing about computability of PDEs because there are, or might be, other methods for solving PDEs.

Some references that migth interest you, in order of relevance:


H. Zenil (editor): A computable universe, Word Scientific, 2012.
The nLab has a page on computable physics.
K. Weihrauch: Computable analysis - An introduction, Springer, 2000.