CKM_RSA_PKCS vs CKM_RSA_X_509 mechanisms in PKCS#11 What is the exact difference between the CKM_RSA_PKCS and CKM_RSA_X_509 mechanisms in the PKCS#11 v2.20 specification (Please explain with an example)?
As very clearly indicated by the specification, CKM_RSA_X_509 performs &quot;raw&quot; RSA. Raw RSA is simply modular exponentiation. So it performs just the RSASP1 function in the PKCS#1 standards. This means that a user should - at the minimum - also provide a secure padding mechanism. Otherwise the conditions to perform a secure RSA signing operation are not met.
CKM_RSA_PKCS on the other hand also performs the padding as defined in the PKCS#1 standards. This padding is defined within EMSA-PKCS1-v1_5, steps 3, 4 and 5. This means that this mechanism should only accept messages that are 11 bytes shorter than the size of the modulus. To create a valid RSASSA-PKCS1-v1_5 signature, you need to perform steps 1 and 2 of EMSA-PKCS1-v1_5 yourself.
Both of these mechanisms are only building blocks for creating a signing algorithm. You won't get a valid PKCS#1 signature by supplying random data to either mechanism. These methods have been defined to give more control to the user of the mechanism.
Sometimes protocols such as older SSL specifications do use their own signature schemes (concatenating an MD5 and SHA-1 hash, leaving out the hash indicator). Those protocols cannot be supported without a direct implementation or lower level primitive being available.
Another reason that CKM_RSA_PKCS exist is that it may not be very efficient to perform the full PKCS#1 v1.5 signature generation on the security token. It is very inefficient to send an entire text document to a smart card. Just sending a hash structure calculated on the host computer is much faster.

A final note: you should never let an attacker be able to use CKM_RSA_X_509 for the keys in your token, nor should you give him alter the padding you supply to that mechanism. In general it is better to use up to date mechanisms that represent complete signature schemes.
EDIT: apparently access to the private key operation in itself is not enough to break the RSA key. Of course letting an attacker have access to the private key operations is insecure in itself, but it won't allow copying / breaking the private key if the operation is well implemented. I presume it does open some attack vectors for the simple reason that an attacker has more control over the input to the modular exponentiation.