Is webkey a reasonably secure authentication system? I just created Webkey in the last couple days. This should allow any website to do single-password auth with a much higher level of auth security and convenience for users (single password with potentially hours or days expiry, text-entry-less single-click authentication if the password hasn't expired yet).

It uses RSA keys generated in-browser, stored in encrypted form using aes. The user's password is never sent over the internet (even in encrypted form). This utility can be hosted entirely statically (and is currently being hosted at http://webkey-auth.github.io/guest.html ). 

Is what I'm doing here theoretically secure? I think this could be a huge improvement over auth based on sending usernames and password - which is kind of the weakest link in internet security at the moment. What are potential problems with how this works?
First of all, you should make a more formal definition of the protocol. Security cannot be assessed without a proper definition.

Second you don't specify an key sizes. RSA-512 is such a low key size that it may be considered broken. On the other hand, you may run into performance issues if you choose a higher key size (Elliptic Curve crypto would make more sense). This is not theoretical security, I know, but it kind-of makes sure that this will never run for any practical key sizes.

The same for the challenge. 3 characters is way to low and almost begs for replay attacks if the same challenge is ever send. A 16 byte challenge should be the minimum.

You seem to be using a Node.js include for your RSA key pair code. That's strange if the key generation is offered within the browser as you suggest.

The biggest usability issue is the fact that you may loose your private key if the browser storage is destroyed. The RSA key seems to be local to the browser, so sharing different devices isn't specified.

It doesn't seem to be specified what happens if you loose your key.

I haven't seen any requirements for TLS (with server authentication). You'd need that otherwise an attacker may steal your key by simply injecting Java Script code.



All in all, there's too much missing from the description of your protocol to make a full security review. But it seems there is a lot that may need improvement before it can be used for any serious purposes.