Is it possible to use the RSA algorithm, or a variant, for software licensing? I'm almost a newbie about RSA encryption, so my question could it be banal.
I'm beginning to create a licensing system for my software.
I need to distribute a crypted license file to many customers.
I was thinking to use RSA public key to encrypt license file (with OAEP padding or PKCS#1 v1.5).
The software that will be licensed to customers will have inside the private key in order to be able to decrypt the license content.
Now the question is if can be possible to know the public key giving the encrypted file, the private key and the final plain text.
This because an hacker could know these data because are all inside the distributed software.

In other words I need a crypto algorithm that use two different keys.
One pw for encrypt the data, and another to decrypt the data.
And where will not be possible to obtain the encrypt password giving the decrypt password and crypted text.
The answer to the question in the title is: Yes, it is possible to use RSA signature for software licensing (e.g. in the context either RSASSA-PKCS1-V1_5 or RSASSA-PSS in PKCS#1 as defined here, or any of the three schemes in ISO/IEC 9796-2 free but partial preview). One minor issue is that, for a decent security level, what protects the license file will be much too big to be keyed in.

There are a lot of terminology errors in the body of the question:


What's wanted is distribute a crypted encrypted signed license file to many customers
What's needed is use the RSA public private key to encrypt sign that license file; in signature schemes with message recovery (e.g. ISO/IEC 9796-2) this will have the (perhaps desirable) side effect of making at least some of the signed content not directly intelligible, though recoverable without any secret information;
The software that will be licensed to customers will have inside the private public key in order to be able to decrypt verify the license content (and recover what portion of the signed information was not directly intelligible, if applicable)
The question is if it can be possible to know obtain the public private key giving given the encrypted signed file, the private public key and the final plain text signed data.


The short answer to that other question is: No, there is not a reason to worry that the private key (or anything allowing to generate new signed license files) could be obtained, provided the legitimate private key is kept and used securely, and was adequately drawn.

You'll get one private key to sign (and perhaps, with message recovery, make not directly intelligible some or all of) the data forming the license file, and an associated public key to verify the integrity (and perhaps, with message recovery, recover all) of that data; and where it will not be possible to obtain the private key (or some equivalent mean) given the public key and the data in any of its form.

On the other hand, no amount of cryptography will really prevent that someone skilled modify the software to bypass the license check, or allow the use of a license duplicated from an authorized one even if the software attempts to lock a license to a particular computer. Without temper-resistant hardware, the best you could do is that some portion of the software is unusable without access, at least temporarily, to a valid license to use that portion of the software.