Difference between Sign Message and Sign Transaction Using "Sign Message" one can prove ownership of an address.
This happens by signing a human-readable string using the private key belonging to the address.
The owner then provides the address, message and signature, and anyone can verify that they indeed hold the private key of the address.
The ECDSA signature is normally provided in 65-byte format (Recovery byte + 32-byte R + 32-byte S) and base64 encoded.

However, there is also another type of signature, which is used for "Sign Transaction".
For each input of a transaction, there is some data to be signed. This data is binary data, normally provided in hex format.
The corresponding signature is provided in 70-, 71-, or 72-byte DER format.

In both cases, the message (message string in the first case, and binary per-input data in the second case) is transformed into a byte array, and the private key is used to generate an ECDSA signature for that message byte array.
The output format is different, but the 65-byte format can easily be converted into DER.

Consequently, I thought that I could just use "Sign Message" (with a minor modification, so that it takes a hex string instead of an ASCII/UTF-8 string) to sign a transaction's hex-encoded binary per-input data.
I did that, converted the signature to DER format and tried to send the transaction, but it didn't work, because the signature was wrong.

So my question is, what is the difference between how "Sign Message" and "Sign Transaction" generate signatures, and why aren't they compatible?

Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?

Thank you!
Is the difference that for "Sign Message", first a digest is created, which is then signed, while for "Sign Transaction" (in the meaning written above), the input is already a digest, so it shouldn't be hashed again?


No. It has nothing to do with the hashing. Furthermore, the input is not already a digest, it must be hashed before signing. Both the message and the input data are hashed twice with SHA256.

The difference between message signing and transaction input signing is that message signing prepends the string Bitcoin Signed Message:\n (with \n being a newline character, not literally \n) to the message before it is hashed.

Additionally, the 65 byte signature produced by message signing has a byte at the beginning that is not part of the cryptographic signature itself. Instead it is a recovery id so that the correct public key can be recovered from the signature during signed message verification. In transaction signatures, there is a different byte appended to the end of the signature which is the sighash type, usually it is the byte 0x01.