Authentication by hardcoding public key in client software
Yes, the basic idea of hardcoding a public key is secure. It is sometimes recommended as an alternative to the complexity TLS and PKI bring – otherwise it can be easy to skip a crucial step and end up with little or no security.

However, the "encrypt a secret for server" scheme has some weaknesses compared to TLS. The clearest is lack of forward secrecy (which admittedly some TLS suites also lack). If an attacker breaks the public key encryption or gets the private server key, they can decrypt all sessions past and future.

A slightly more complex protocol would fix that issue. You could e.g. hardcode a signing key and authenticate an ephemeral Diffie–Hellman key exchange.

(You also miss out on using the available implementations of TLS, needing to write your own code risking mistakes. That is harder to quantify. And like you said, key management may be more of an issue.)