Is it possible to get an RSA encryption key by comparing the unencrypted and encrypted file? Exactly what the title says. If I have a RSA encrypted file, and the exact same file but decrypted, can I tell the key that was used from that information?
No, if the RSA cryptosystem is secure, i.e. when it uses random padding such as PKCS#1 v1.5 padding or OAEP, then you cannot. As Stephen already mused, it is pretty likely that you can find out the key simply because it is included or can be derived; generally public keys are not meant to be secure.

If textbook (raw modular exponentiation) RSA is used then you can encrypt with the public key. If you get the same value then it is pretty likely that you've found the public key. But usually textbook RSA isn't used because it is insecure in several other ways as well. Ciphertext distinguishablity is just one of the problems.

Now some information can of course still be extracted from the ciphertext. First of all, the length of the RSA ciphertext is generally identical to the key size. So you can distinguish between keys of several lengths. Furthermore, if you find an RSA ciphertext that is higher than the modulus of the public key then that key is certainly not used to generate the ciphertext.



You explicitly asked for the encryption key which is the public key. If you meant the private decryption key then, no, the key cannot be derived except for the little amount of data also available for the public key.



Note that RSA is never directly used to encrypt files. Instead a hybrid cryptosystem needs to be used. The usual RSA schemes are not secure to encrypt data larger than the modulus minus the padding overhead. Furthermore, encrypting large amounts of data with RSA would be very CPU intensive and decrypting it would be worse.