Under what circumstances does Batesian mimicry evolve?
While the attempt to consider this in a mathematical framework perhaps isn't so useful, it seems that the biological essence of your question is along the lines of: 


  Does a Batesian mimic initially need to be initially quite similar in morphology to the different species which it evolves to mimic?


The answer is no, morphology does not need to be similar, but it does help make mimicry likely. The only requisite to evolution is the presence of heritable genetic information with the potential for change, and this prerequisite is the same for any trait. Selection is not a prerequisite but mimicry is more likely to evolve by adaptive evolution than random evolution. Given that phenotype is often strongly affected by the DNA and that DNA has a potential to mutate, then any distance in phenotypic space can be covered so long as sufficient mutations occur. Sufficient mutations are more likely to occur when 

a) the distance in phenotypic space between the two initial species is small (the ancestral form of the mimic species is similar to the species it will mimic)

b) a large amount of time is allowed to elapse (more mutations can occur) 

The first of these is simply stating that there are fewer (or less extreme) mutations required to get the species to be perfect mimics. The latter simply states that time allows for a greater number of mutations to occurs, making it more likely that large phenotypic spaces can be bridged.

Consequently, allowing identical time, mutation rates, mutation effect sizes, and selection coefficients, two species that initially look similar are more likely to evolve mimicry than two species that initially look very different, because fewer steps need to be taken for convergence of phenotypes.

This is the same for any trait. One could ask can humans evolve to be ten feet tall, the answer is yes, as long as the genetic variance arises. One could ask can humans evolve to be 20 feet tall, and again the answer is yes. However, the latter is less likely because it requires a greater movement in phenotypic space. More time also makes both results more likely.