Encrypt large data with RSA through blocks in java
The system is probably meant to effectively use an "RSA signature" on top of RSA encryption. If you used raw RSA it could "work", but only if the client's modulus is larger than the server's – or you get lucky and the encrypted key is small enough. Since you are using a library that handles padding for you, it expects a small enough input that it can pad it to size for encryption.

If you can, you could instead use an actual RSA signature algorithm in the second step. This would let the server to verify who the encrypted key is from.

However, if the protocol is something you must use, you should double check that you have understood it correctly. As you have stated it, it seems broken.