How to hide non-openable in the finder while using an application to browse/open
No, there is no way to hide these files. It's a usability decision by Apple.

The dialog always shows the same list of files but greys out the ones you currently cannot open. Technically you could go to the pop up at the bottom and change it to All files and open any of the files listed. Of course they may not make sense in the application, but you could open them.