Decrypt a public encrypted message and Sign a signature, how the math is different? As I understand, when you want to send a confidential message to someone, you encrypt the message with his public key. And he use his private key to decrypt the message.

At the same time, one can use his private key to sign a message, and other people use his public key to verify that it's him who sign it.

How the math is different between decrypt the message and sign a message?
The simplistic view that signing uses the same math as decryption is descriptive with textbook RSA; has some relation to practice in safe use of RSA; and is just wrong with most other common asymmetric cryptosystems, beside the commonality that a private key is involved.



In textbook RSA, the math to decrypt an encrypted message and compute a signature is the same: it is applied the textbook RSA private key function $x\to y=x^d\bmod N$, with $(N,d)$ the private key, and $x$ the encrypted message in decryption, or the message to sign in signature. Problem is, textbook RSA is unsafe for both encryption and signature.

When we move to secure use of RSA, the math becomes different:


To decrypt we check the bound of the cryptogram $x$, apply the above textbook RSA private key function, then


Extract the encrypted message from the result $y$, in a certain way that removes random padding that was added at encryption (and perhaps checking certain fields). There are at least two common classes of RSA encryption padding and corresponding decryption removal, defined in PKCS#1, and several implementation variants (not all are safe). That can be enough for very small messages (like few hundred bytes).
And/or what we really obtain is a secret key, which is then used to decipher the bulk of the message using symmetric-key cryptography (that's hybrid encryption).

To generate a signature we typically hash the message to sign then add redundancy in a certain way, then apply the above textbook RSA private key function to obtain the signature $y$ (in some variants we further keep $\min(y,N-y)$ ). There are at least three common classes of RSA signature padding: two in PKCS#1, and ISO/IEC 9796-2, which comes with several variations.




When we move to asymmetric cryptosystems other than RSA, one thing remains: the private key is involved in decryption, and/or computing a signature. Other than that, there is typically profound disjunction between decryption and computing a signature (one notable exception is the Rabin cryptosystem, which can be seen as a variant of RSA with even public exponent); and very often, one of the two features is just not offered by a particular cryptosystem.

For example, we have DSA, ECDSA, and Lamport signatures, but these cryptosystems do not offer encryption.

The converse can be true: we have Pailler encryption, but no useful signature analog. We have NTRUEncrypt, which latest versions are unbroken; but it is quite different from the related NTRUSign, and last time I checked work on the later stopped in the broken phase of multiple break/repair cycles.