Signing with RSA-SHA256 algorithm using EC keys Is it practical to sign a data using RSA-SHA256 algorithm with EC keys?
I used a small NodeJS script and crypto module (uses openSSL internally) to test this and I could successfully sign the data and verify the signature.
Pseudo-code:

Sign( data, EC-Key-p256, 'RSA-SHA256')
Verify( data, signature,public-key, 'RSA-SHA256')

Have used openSSL to create EC keys (p256v1) and used it for my testing.
Could you please help me understand whether this is a standard practice?
tl;dr  - RSA keys and ECDSA keys are not compatible and cannot be used in one anothers signature algorithms



In RSA you have the following:


A public key $(e, N)$ where $e$ and $N$ are integers.
A private key $d$ where $d$ is an integer and $ed \equiv 1 \bmod \phi(N)$, $\phi$ being Euler's totient function.


In ECDSA you have:


A public key $Q$ which is a point on an elliptic curve e.g. $Q = (q_x, q_y)$.
A private key $d$ which is an integer such that $dG = Q$, where $G$ is the base point of the elliptic curve.


Given those facts the question you pose doesn't make sense. Indeed an RSA private key and an ECDSA private key seem similar in that they are both integers, but in RSA $d$ needs to also have a corresponding $N$, which is usually left out since it is a public value (if you want to be pedantic the private key is really $(d, N)$ in RSA).

So say I generate an ECDSA private key $d$ and want to sign a message $m$ with it via RSA-SHA256. RSA-SHA256 would look something like $signature = \mathrm{sha256}(m)^d \bmod{N}$ (actually you would want to apply padding after the hash and before the exponentiation but that's not really important here).

But if only $d$ is generated, as one would do for an ECDSA private key, how does one perform the $\bmod{N}$ operation? You can't because $d$ by itself is an incomplete RSA private key, you also need the $N$ corresponding to the public key.

Let's say you do generate some $N$ after the fact so that you can perform signing. Verification would run into the same problem of keys not being compatible. RSA-256 signature verification for message $m$ and signature $s$ looks something like $\mathrm{sha256}(m) =^? s^e \bmod N$ where $(e, N)$ is the RSA public key. But if one does not have $(e, N)$ and is provided only a point $Q = (q_x, q_y)$ that lies on an elliptic curve, how does one then verify the signature? One of course can't.



SEJPM is probably correct in his comments that the library you're using sees that the signature algorithm doesn't match the key and chooses the correct signature algorithm for signing. If the library isn't raising a warning and quietly acting as though all parameters are valid I would consider switching libraries.