question about RSA sign/verify scheme
The RSA operation always produces an answer modulo the modulus.

Normally, we are careful to have a 'plaintext' which is smaller than the modulus, and hence the end result is the same.  After all, the fundamental identity for RSA is:

$$(M^d \bmod N)^ e \bmod N = M \bmod N$$

If $M &lt; N$, then the right hand side simplifies to $M$

However, in this case, you have an initial input FEDCBA9876543210 which is actually larger than the modulus BE1EE24B5ACD9DE5.  When you do this, you end up with an end result which is FEDCBA9876543210 mod BE1EE24B5ACD9DE5 = 40BDD84D1B86942B

That answers the immediate question you asked; however as you dig further into RSA, you should be aware of the following:


When we use RSA, we (almost) never apply the RSA operation directly to the value given by the application (hash result, plaintext); instead, we apply a 'padding' operation first, to avoid potential attacks that exploit the homomorphic properties (and determinism in the case of encryption) that the raw RSA operation has
During the signature operation, we really can't call the RSA operation 'decryption' (as we're not actually decrypting something that has been previously encrypted).  Normally, it is called a 'signing' or 'signature generation'.