How to determine required number of rounds for RC6 cipher given custom W and B parameters?
In the case of AES, they had a lower bound on the highest probability differential (and the analogous linear hull), and they were able to use that information to determine at how many rounds would be required for a differential (and analogous linear) attack of a specific cost. 

Ideally, that is how you would proceed. 

However, Rijndael (later AES) has some specific properties that enable such a calculation to be made. All of the constituent operations are very cleanly and precisely defined, and they operate on sizes that are small enough to enable such analysis to take place.

With ciphers built from bitwise and arithmetic operations ad-hoc*, especially when those operations function on large words, finding the worst-case differential probability is more challenging, if/when it is possible. 

You can find the sort of average worst-case probability (which is not really the worst-case) and make the calculation for rounds using that, but that does not prove that no attack costs less then the desired amount, because there could be some higher probability differentials you don't know about yet.

Still, that is probably the best strategy you have.

Note

You can only perform this type of process for attacks that you know of already. You could prove that no basic differential attack costs less than $\frac{1}{2^n}$, but that does not mean that no attack costs less than $\frac{1}{2^n}$.

* Technically the operations in Rijndael are built from bitwise and a certain type of arithmetic operations - but all circuits are powered by these, at the circuit level.