Can nesting different ciphers with different keys ever decrease security? Very related to this question but not addressed there.

Does the following construction risk introducing any vulnerabilities that render it less secure than using either of its individual components in isolation?:

EncryptAB(K1||K2, PT) = EncryptA(K1, EncryptB(K2, PT))

EncryptA and EncryptB may be different algorithms. K1 and K2 my be of different lengths. The result need not be more secure than the better of it's components, just not less secure.

A related question is: are there any potential problems with interleaving the rounds of two or more independent algorithms?

(BTW: I'm not planning on doing either, it's more a matter of curiosity.)



Edit: I'm only thinking of the bare block cipher primitives rather then chaining modes and such.
In your construct:

EncryptAB(K1||K2, PT) = EncryptA(K1, EncryptB(K2, PT))

it is easy to show that, as long as the keys K1 and K2 are independent, that this cannot be any less secure than the stronger of EncryptA and EncryptB.

Here is a sketch of how this is shown; suppose that there is a chosen plaintext attack against EncryptAB; that is, the attacker submits known values to EncryptAB, examines its results, and somehow rederive the value K1||K2.  Then, this method can be used to attack EncryptA; the attacker would choose a random value for K2; he would take his chosen plaintexts, EncryptB them with his K2 value, and then submit them as chosen plaintexts to the EncryptA with the known key.  Then, he can use those encrypted values to rederive the K1||K2 values, which includes the value K1 which he was interested in.  The attack against EncryptB works in exactly the same manner.

Now, this method doesn't prove that EncryptAB is any stronger than the stronger of EncryptA and EncryptB (and, indeed, it might not be); however, it does show that using the two of them doesn't make things any worse.  Also, we need to assume that that keys K1 and K2 are independent, because this simulation has the attacker picking one of those values; if they are interrelated, he wouldn't be able to do that.

Now, for your related question: how about interleaving the rounds of two different ciphers.  That doesn't have any such security reduction, and in practice, it would give me the willies.  Encryption round functions don't have that much strength in themselves (that's why we use multiple rounds); what a good round function is designed to do is play nicely with the round functions that'll be next to it in the designed cipher.  For example, if there is a good differential through one round, it's going to be the case that it doesn't have any good paths through the next one (or the one after that).  This is not by accident; the cipher designer (at least, if they knew what they were doing) went through some serious analysis to make sure that was the case.

In contrast, if you interleave the round functions from two different ciphers, well, no one has done any such analysis.  Yes, it might be secure, but without someone spending a lot of time looking at it, you couldn't be certain.  It'd be far better to use your original suggestion, and do a full set of rounds from your first cipher, and then do a second set of rounds from your second.