Authentication impossible without pre-shared public key Correct me if I am wrong, digital certificates require the user to have the public key for the certificate in their web browser, right?

Which means it has to be pre-shared.

(Say I am designing a cryptosystem which is an application for encrypted chat that uses TLS. However, I want the cryptosystem to be end-to-end, meaning, the client must directly connect to a server of their choice.

How can I make this cryptosystem's authentication work if I allow my clients to create their own servers for direct end-to-end encryption?
In other words, the application does NOT contain a pre-shared public key for the digital certificate.

In this scenario, is my application doomed without a pre-shared public key?
You dont need the public key/certificate to be present in the browser. You just need to devise a method to ensure the public key/certificate which is presented by the peer is valid. In a Web Browser scenario this is typically done by a number of known CAs which sign those certificates. But you can also ask the user to authenticate the certificate by a fingerprint or by registering it at a central directory you trust.

You can also what is called "Trust-on-First-Use" which is basically "install this cert". This has the one-time risk of a person in the middle attack. It is how SSH is used most of the time today.

You do not even need a known/verified certificate but can use instead anonymous key agreement. This has of course the big problem that each time this key exchange is done a person in the middle can spoof the server. TLS has aDH ciphers for this.

And finally you can use TLS based on a shared password as well. It depends if both parties know each other and can actually verify each other out of band or not.

If you do not have a central directory authority the most common method is to use a trust on first use setup with the additional option to manually verify the peers (this is for example also used in Whats App in addition to the weak identity proof with phone number).