What's the downside of using X.509 in SSH instead of PKI ssh uses public keys for authentication and not signed X.509 certificates. Specify a drawback which would be the use of X.509 certificates instead of public keys.

I've tried to look up a lot of different things, but the certificate seems superior in every way.
The fundamental difference is in how (and wether) the clients and servers identity is validated.

In X.509, keys are authenticated by virtue of being issued by a trusted certificate authority. That makes it easy to use when the client starts out knowing nothing about the server's identity, and the server does not need to know anything about the client's identity. But can be a particular nightmare to manage when the client also needs to authenticate against the server.

In SSH, keys are self-generated (similar, although not the same, to self-signed X.509 certificates), and are authenticated by relying on a fingerprint, which has to be transmitted out-of-band. That makes it easier to use in scenarios where both ends are reasonably trusted. SSH also has built-in mutual authentication. The client verifies the server's identity by its host key (what a server X.509 does in TLS), and the server verifies the client's authority through the entry in the authorized_keys file (that's the equivalent of X.509 client certificates).

SSH is mostly used for internal communications within an organization, where both ends are, at least somewhat, trusted. The overhead of generating and distributing X.509 certificates would put a serious burden on many use cases (just think of spinning up test servers on a daily basis in a development environment!).

It is also designed to not rely on the destination name (so you can use the DNS name most of the time, and IP addresses if your DNS system fails for some reason).

That said, you can actually configure SSH to use X.509 certificates. As far as I understand it, though, it only uses the public and private keys, and ignores the rest of the certificate.

And you can also get most of the benefits of an X.509 certificate with SSH by adding an SSHFP record to your DNS (this is non-trivial as you first have to enable DNSSEC). With an SSHFP record, one could argue you are actually in better shape than with an X.509, because you get the benefit of an authenticated host key, without relying on the somewhat broken certificate-authority trust model.

Bottom line: The SSH mechanism and X.509 are designed for different scenarios. Both are reasonably secure in their respective niches.