How to determine required number of rounds for RC6 cipher given custom W and B parameters? The RC6 symmetric block cipher is parameterized with three parameters :


W : The number of bits in a machine word (8 , 16 , 32 , 64)
B : The number of bytes specifying key length (0 to 255)
R : The number of rounds to execute (1 to 255)


A data block is specified as 'uint_t data [4]' where 'uint_t' can be 8, 16, 32, or 64 bits resulting in a 32, 64, 128, or 256 bit data block size respectively.

These parameters allow the end user to select block size, key size, and to trade off performance for security.

The question I have is : Given parameters W and B, along with a desired security strength (not sure how to specify this exactly, but it depends on R), how does one determine the minimum value of R to ensure this minimum security bound?

ie: I desire to parameterize an instance of RC6 for a 256 bit data block (W = 64) and a 256 bit key (B = 32).  I would like the cipher to have 256 bits of security (that of the key B) which clearly depends on the value of R. Assuming a full 256-bit key is used, what is the minimum number of rounds (R) required to ensure full diffusion and security equivalent to a brute-force search of the 256 key?

I have researched the original RC6 specification, alternative research papers that I could find which discuss various attacks on RC6, etc. but I have not yet seen any summary, equation, or guidelines on how to select values of R for tailored instances of RC6.

It seems to me that without an equation or clear instructions on how to select R, the specification is not 100% complete and layman implementing the cipher for a practical application will be left guessing, which will cause bad things to happen.

Note : I am not an expert cryptographer, but I do know enough not to "roll my own" and follow a standard to-the-letter.  This standard, while awesome, seems to be missing a letter... :)
In the case of AES, they had a lower bound on the highest probability differential (and the analogous linear hull), and they were able to use that information to determine at how many rounds would be required for a differential (and analogous linear) attack of a specific cost. 

Ideally, that is how you would proceed. 

However, Rijndael (later AES) has some specific properties that enable such a calculation to be made. All of the constituent operations are very cleanly and precisely defined, and they operate on sizes that are small enough to enable such analysis to take place.

With ciphers built from bitwise and arithmetic operations ad-hoc*, especially when those operations function on large words, finding the worst-case differential probability is more challenging, if/when it is possible. 

You can find the sort of average worst-case probability (which is not really the worst-case) and make the calculation for rounds using that, but that does not prove that no attack costs less then the desired amount, because there could be some higher probability differentials you don't know about yet.

Still, that is probably the best strategy you have.

Note

You can only perform this type of process for attacks that you know of already. You could prove that no basic differential attack costs less than $\frac{1}{2^n}$, but that does not mean that no attack costs less than $\frac{1}{2^n}$.

* Technically the operations in Rijndael are built from bitwise and a certain type of arithmetic operations - but all circuits are powered by these, at the circuit level.