Why a TS must have only one negative frequency
Well, let's just look at the simple case you mention. If we are at a transition state, then we are at a saddle point on the potential energy surface (PES). As you say, there are two directions where we go down in energy and two where we go up in energy which are along or perpendicular to the gradient vector.

Note that to confirm we are at a saddle point, in mathematics one could do the second-derivative test. Or, in chemistry (and in math) in order to do this, we construct the Hessian matrix and find its determinant. So, let our PES be given by a function $f(x,y)$. Then, the Hessian takes the form,
$$
\begin{bmatrix}
f_{xx} &amp; f_{xy} \\
f_{xy} &amp; f_{yy} \\
\end{bmatrix}
$$
where $f_{xx}$ denotes the second partial derivative of $f$, both with respect to $x$ and likewise for the rest of the terms. Notice that $f_{xy}=f_{yx}$, so this matrix is symmetric and hence all its eigenvalues are real. Also note that sometimes people will write this in terms of the force constants instead of the second-derivatives because a harmonic force-constant is a second derivative.

So, if we take the determinant of this matrix, its sign tells us about whether the function is concave up or concave down at the point we are interested in. This determinant is given by,
$$
f_{xx}f_{yy}-(f_{xy})^2.
$$
The sign of this actually tells us information about the eigenvalues of the matrix because the determinant of a matrix is equal to the product of its eigenvalues.
Notice that in order to find the eigenvalues of this matrix we would solve the following equation:
$$
(f_{xx}-\lambda)(f_{yy}-\lambda)-(f_{xy})^2=0
$$
Or, after rearranging,
$$
\lambda^2-\lambda(f_{xx}+f_{yy})+f_{xx}f_{yy}-f_{xy}^2=0
$$

These eigenvalues can be related to the frequencies of the vibrations if we first mass-weighted each second-derivative. All I want to show here is that for this case there are only two eigenvalues, and these eigenvalues do not have to do with the number of directions to move towards a minimum. One of the vibrations you can imagine as being the mode which in two directions have an increase in energy. So, along this mode, the molecule stays at the transition state and hence there is a physical frequency associated with this motion. If the second derivative test is worth anything, then the overall sign of this determinant will be negative at a transition state and hence there will be one and only one negative eigenvalue. Note that this only works at critical points, so that's why at the transition state there must be only one negative imaginary frequency (or negative eigenvalue).

Because for most every molecule we have more than two modes to worry about, you could imagine splitting a much larger matrix into many 2-d matrices and looking at two modes at a time. From this idea it is clear that a transition state need not be limited to having only one negative frequency. Rather, at second-order saddle points, (i.e. a saddle point where following the eigenvector corresponding to a negative eigenvalue leads you to another saddle point) there will be two negative imaginary eigenvalues.



To summarize:


The number of imaginary frequencies is the number of negative eigenvalues of the Hessian matrix (the matrix of all possible mixed-second-partial derivatives)
There can be more than one negative frequency at a higher-order saddle point, but physically speaking these are much less common than transition states with only one imaginary frequency
One cannot determine the number of imaginary frequencies by thinking about the number of paths to a minimum. You could think about the number of modes which do not kick you off of a saddle point if you prefer this.




Edit:

I mistakenly referred to negative frequencies which is not correct (as alluded to by a commenter). The eigenvalues of this matrix correspond to the force constants (second derivatives with respect to displacement coordinates), so one gets negative force constants, not negative frequencies at a saddle point. Because the force constant for one mode is negative at a transition state, the "frequency" of this mode is evaluated as $\nu=\sqrt{\frac{k}{\mu}}$ as usual, and the frequency is imaginary. So, one is looking for either one negative force constant or one imaginary frequency, not a negative frequency. Sorry I missed that.