Why is it not possible to tap into the HTTPS traffic specifically going from the server to the client?
I'm assuming you're talking about RSA. The more common DH algorithm uses ephemeral public and private keys generated by both the server and the client, although that's not relevant to this answer. The public and private keys are not used to exchange traffic. Instead, they're used to exchange a symmetric key which is used to encrypt data going both directions. This is what's called a hybrid cryptosystem.

Now, TLS in particular is a little more complicated than that. It uses two symmetric keys, one for each direction, in order to protect from certain kinds of attacks. however, the public key encryption is used to exchange a single master key which is used to generate the two symmetric keys, as well as some other keys used in other parts of the TLS protocol (such as MAC keys used for integrity).

For more details, see How does SSL/TLS work?