Is there a programming language that is not a formal language
No, at least not a reasonable programming language by what we commonly understand under that term.

As soon as you have a compiler (or an interpreter), you have an algorithm that decides whether a given string is a valid program. This compiler then defines the language. (Not a good way to define a programming language, but a workable one.)