Genetic Algorithms: Trade-off between time and variance with regards to fitness function
You can probably get away with a relatively low X for two reasons:


The Central Limit Theorem. This tells us that the accuracy in the estimate of an agent's fitness will improve as the square root of the number of games played. 
In a GA, you don't need an absolute ranking of individuals, because your selection mechanism (see "related articles" here) typically isn't completely elitist. For example, if individuals in the top half of the population are allowed to breed, then your fitness function really just needs to separate the good from the bad reasonably well. It need not be perfect to work.


The correct value of X will still depend on the variance in the scores an agent might receive from playing the game, but this is easy to track. A good approach might be to incorporate this directly into your estimate. Compute the variance, and then prefer agents that not only score highly, but do so with low variance.