How vulnerable is one-time pad (OTP) encryption, if the OTP is used twice, with a random substitution scheme After reading up on the one-time pad (OTP) encryption method, I could see how it would offer unbreakable encryption if used properly.

Moreover, I looked at how the OTP could be broken if the OTP-key is used on more than one message.

What I now wonder is this: if classically you use a Vigenère Square to substitute letters according to your OTP-key, you can break the encryption by finding cribs and then applying the substitution of the letter in the second message. This is possible since I know the substitution scheme.

Now the interesting point: if I randomize my substitution table, I would think that a decryption using the same OTP-key would no longer be possible, unless I have enough messages to successfully map (almost) each letter to this unknown table.

Correct?
Bad idea (as far as I know &amp; understood).

First, let's make sure you do not confuse Vernam cipher (XOR) with OTP.


OTP (One Time Pad) requires you – as described here – to use (1) a truly random one-time pad value, (2) generated and exchanged in a secure way, (3) at least as long as the message, and (4) only to be used once. So, what you are describing is not a One-Time-Pad.
Vernam cipher applies this concept with a stream cipher (simple XOR)


The best example of “miss-using” a OTP is usually given as:

$c_1 = m_1 \oplus k\\
c_2 = m_2 \oplus k\\
c_1 \oplus  c_2 = m_1 \oplus k \oplus m_2 \oplus k = m_1 \oplus m_2$.

where using the same key twice defeats its purpose and provides interesting information on the plain text (statistical attack).

Why your idea is bad ?

Given Kerckhoffs' principle: 


The system must be practically, if not mathematically, indecipherable;
It should not require secrecy, and it should not be a problem if it falls into enemy hands;
It must be possible to communicate and remember the key without using written notes, and correspondents must be able to change or modify it at will;
It must be applicable to telegraph communications;
It must be portable, and should not require several persons to handle or operate;
Lastly, given the circumstances in which it is to be used, the system must be easy to use and should not be stressful to use or require its users to know and comply with a long list of rules.


Therefore you must consider that your Vigenere substitution table has to be publicly known : The secret should only rely on the OTP value. Hence the point of randomizing your substitution table is not that interesting any more.

Therefore, given the known substitution, statistical properties can be found that would defeat the randomization idea.

By the way, this idea is close to the design of Enigma, where the substitution table changes every round defined by an initial key, which is kept over the course of a day. Have a guess what happened during WW2... What would have happened if the Germans had changed their keys between every transmission?

PS: Rolling your own crypto is always a bad idea.