How is RSA able to prevent brute forcing using the public key?
If Jeff has the public key and an encrypted message, why can't he guess the message, encrypt it with the public key, and see if he gets the encrypted message.


Well, that particular attack is foiled by the RSA padding method.  All RSA encryption padding methods include randomness in the padding (specifically to foil this attack).  Whenever John encrypts a message, he (actually, his crypto library) selects a number of random bits, and pads the message (that is, prepares the message for the core RSA operation) using those random bits. Hence the encrypted message is a function of both the plaintext message and those random bits.

Jeff can certainly take the public key, and encrypt a guess of the message with it; however unless he happens to pick the same random bits that John used, his encrypted message will look different, and so comparing the actual ciphertext to his test one (with high probability) tells him nothing.  And, we use enough random bits so that Jeff can't brute force those.