Are ECDSA keys and RSA keys interchangeable?
From the source code:

function createKeySigner(bits) {
 return function sign(thing, privateKey) {
    if (!bufferOrString(privateKey) &amp;&amp; !(typeof privateKey === 'object'))
      throw typeError(MSG_INVALID_SIGNER_KEY);
    thing = normalizeInput(thing);
    // Even though we are specifying "RSA" here, this works with ECDSA
    // keys as well.
    var signer = crypto.createSign('RSA-SHA' + bits);
    var sig = (signer.update(thing), signer.sign(privateKey, 'base64'));
    return base64url.fromBase64(sig);
  }
}


See in particular the comment. What happens here is that even though you specified "RSA with SHA-512" as signature algorithm (the 'RS512' parameter, in that library's weird terminology), the code perfectly knows that the private key is for ECDSA, not RSA (look at the contents of the private key file, it starts with "BEGIN EC PRIVATE KEY"), and so it "helpfully" uses ECDSA instead of RSA, transparently.

No mystery here then, just some Node.js code doing Node.js stuff (i.e. doing what it can instead of what you asked for, and hoping for the best -- a rather dubious methodology for all things related to security, in particular cryptography).