Could a fixed RSA encryption be used instead of a salt to prevent rainbow table attacks? It's common to use a salt before hashing a password in order to prevent an attack by rainbow tables.
Would it also work to encrypt the password for instance by RSA encryption - by a permanent RSA private key stored on the server - before hashing?
In other words: $hash = SHA(RSA(password))$
For me this looks secure because rainbow tables wouldn't work with the encrypted cipher $RSA(password)$, would they?
(Side note: Since that RSA private key is not used for encryption at all, it would never have to been replaced...it would just work as a totally deterministic pseudo-random function in this case!)
I don't see a point of using RSA for password hashing. Using SHA and RSA will not make the bruteforce attack slower. The massive GPU/ASIC attacks will still work if we assume the public key $(e,n)$ is known. That is why we need memory hard functions to make the attacks slower. Sticking the standard is still better like using Argon2id ( Argon2 was the winner of the Password Hashing Competition in 2015). The unique salt also helps to eliminate the rainbow tables. Rainbow tables are dead for passwords system that deploys unique salts!.
A minor point is that one doesn't need to store the RSA private key $(d,n)$ since one cannot reverse the SHAx. So it is useless.
Back to Rainbow
in case of protection against the rainbow tables, one needs to make sure that every password needs a domain separations. This is achieved by unique salt for all. If you want to use the RSA then you need to use OAEP padding or PKCS#1 v1.5. padding. Both are probabilistic encryption scheme that as long as you have a good random number source like /dev/urandom then if you encrypt the same message again and again you will get different results, up to a huge limit of course ( the size of $r$ in the OAEP). One can think of the salt as this randomization.
A side note: the pepper, which is unique salt for each application server, is used to separate the domains of the applications in the case of hitting the same salt for the same user. Also, If an attacker downloads the users' table with only an SQL injection, then they cannot apply even brute-force without the pepper of the server.
Note 2: According to Hashcat list, only OpenSSH uses RSA in a combined mode RSA/DSA/EC/OpenSSH

Addendum
This part is based on the comments of @fgrieu in the case that @marcus considered these;

$(\text{salt},hash = \text{PasswordHash}(\text{salt},\text{DeterministicPadding}(\text{password})^d\bmod n))$

$(\text{salt}, hash = \text{Hash}(\text{DeterministicPadding}(\text{salt}\mathbin\|\text{password})^d\bmod n))$


Here the deterministic padding stands for padding the message for RSA encryption, but deterministically, like RSASSA-PKCS1-v1_5.
It is obvious that if the salt is unique for each user then it is already secure against the rainbow tables. The password crackers if access to the passwords' hashes, cannot test it without the knowledge of the private key.
The biggest problem is the protection of the RSA private key. The usual approach is using HSM to handle those encryptions where the RSA key is stored, too, however, for heavy systems, it may be a bottleneck for the speed. This is no a real comparison, and the usual advice for the password hashing algorithms is adjusting the iteration so that it takes around 1 second per user. This is for user-friendliness. i.e. the general user may not want to wait too much for the login process.
Knowing the public key $(n,e)$ won't help the attackers since they, as of public knowledge, cannot break RSA &gt; 829-bit. See current records on How big an RSA key is considered secure today?
We can consider this RSA operation as a pepper of the application server, too. Also, instead of RSA, one can use HMAC-SHA256 for the same usage, which has a lower key size.
In short, if the key can be protected, It has more protection against the usual approach.