Simplified encryption API wrapper for RSA and AES
I don't think you are over-thinking it, casting is always a code-smell. The fundamental problem is that you separate logic from data, the key (data) and the actual usage of that key (data). This is a problem in object-orientation.

Here is an alternative possible design, which does not do that:

public interface SecretKey {
    PublicKey generatePublicKey();

    CipherText encrypt(byte[] plaintext); // Symmetric encrypt
}

public interface PublicKey {
    CipherText encrypt(byte[] plaintext); // Asymmetric encrypt
} 

public interface CipherText {
    byte[] decrypt();
}


It all depends of course on your requirements, this may not exactly match what you are looking for, but the key idea here is that the data and function are not separated.