How to add passphrase to an RSA key?
First things first: Don't roll your own crypto.

As for your current approach: This is basically a vigenere cipher which is inherently broken, provides not integrity protection and wouldn't even encrypt known / predictable bit positions (where the ASCII code is constant zero or one).

As for an improved version:
Use a well-known encryption algorithm (e.g. AES-GCM) to encrypt and authenticate the key. Don't forget that you have to use a nonce (a number used only once) with it. Do not feed the user password directly into the cipher, but rather feed it into a proper password based key derivation function (PBKDF) (e.g. Argon2, scrypt, bcrypt, PBKDF2). Make sure you use a salt and that it is random and that the derivation parameters are so large that key access takes about 100ms of full CPU load on your system.