The Assumption of Weak Selection?
This is my take in this, without experience of using the Taylor series to analyse evolutionary game theory problems.

As you know the Taylor series expansion of $f(x)$ at point $a$ can be written as:

$f(x)= f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \frac{f^{(3)}(a)}{3!}(x-a)^3 + ... + \frac{f^{(n)}(a)}{n!}(x-a)^n + ...$

Often factors above second order are dropped as a simplification. However, to use the Taylor expansion, the function must be differentiable at point $a$, and to be differentiable it must be continuous.

As @Artem Kaznatcheev writes in his question, "...with weak selection meaning the game modifies overall fitness only slightly..." and "... it is typical to model organisms as having a base fitness that is modified slightly by the game interaction...". These statements imply that the overall fitness function/surface can be assumed to be relatively smooth and continuous (i.e. the payoff is negligible), which means that the Taylor series expansion can be used. If the game payoffs for a single game would determine a large proportion of the overall fitness, the fitness surface would be discontinuous.

An example where a Taylor series expansion is used to simplify the fitness function under weak selection is given in Andre &amp; Godelle (2006) (see equation 20 ff).

You might also find this supplement to Nowak et al (2010) interesting (a rather controversial paper), especially page 8.