How to add passphrase to an RSA key? If I want to protect a private RSA key, do I just XOR the password into the key, or do I do something else? Is it secure?

Edit:
I got a good answer from a comment - Hash the password/passphrase with PBKDF and use that as a key for AES to encrypt the RSA private key. Also the passphrase is never stored on the disk. The user is prompted to enter it when starting the programm and it is stored in RAM for a limited amount of time(It will be overwritten later and RAM is cleared anyway on shutdown.)
First things first: Don't roll your own crypto.

As for your current approach: This is basically a vigenere cipher which is inherently broken, provides not integrity protection and wouldn't even encrypt known / predictable bit positions (where the ASCII code is constant zero or one).

As for an improved version:
Use a well-known encryption algorithm (e.g. AES-GCM) to encrypt and authenticate the key. Don't forget that you have to use a nonce (a number used only once) with it. Do not feed the user password directly into the cipher, but rather feed it into a proper password based key derivation function (PBKDF) (e.g. Argon2, scrypt, bcrypt, PBKDF2). Make sure you use a salt and that it is random and that the derivation parameters are so large that key access takes about 100ms of full CPU load on your system.