Why can a neural network use more than one activation function?
First of all, it looks like you are under impression that a neural network is structured like this (example for 4 inputs and outputs):
$$ \begin{array}{rcl} 
y_1 &amp; = &amp; \text{sigmoid}(w_{11}x_1 + w_{12}x_2+w_{13}x_3+w_{14}x_4+b_1)\\
y_2 &amp; = &amp; \text{sigmoid}(w_{21}x_1 + w_{22}x_2+w_{23}x_3+w_{24}x_4+b_2)\\
y_3 &amp; = &amp; \text{softmax}(w_{31}x_1 + w_{32}x_2+w_{33}x_3+w_{34}x_4+b_3)\\
y_4 &amp; = &amp; \text{softmax}(w_{41}x_1 + w_{42}x_2+w_{43}x_3+w_{44}x_4+b_4)\\
\end{array}
$$
If that is how you understand it, then you've missed the point of &quot;deep&quot; neural networks. (Sequential) deep neural network are structured in several layers:
$$
 l_i^1 = \sigma^1(\sum_jw_{ij}^1x_j+b^1_i) \\
 l_i^2 = \sigma^2(\sum_jw_{ij}^2l^1_j+b^2_i) \\
 \ddots\\
 l_i^\alpha = \sigma^\alpha(\sum_jw_{ij}^\alpha l^{\alpha-1}_j+b^\alpha_i) \\
 y_i = \sigma^{out}(\sum_jw_{ij}^{out} l^\alpha_j+b^{out}_i) \\
$$
The activation function $\sigma^l$ of a single layer stays the same.

To me, it's like saying a function is approximated using sine functions with N different k values and then also randomly a few Hermite polynomials are thrown in as well.

To follow this analogy - you are mistaken by thinking that we are simultaneously trying to decompose over $\sin$ functions and hermitian polynomials. What actually happens is that we first decompose over $\sin$ functions and then decompose over hermitian polynomials. This actually can makes sense from practical point of view.

Hermite polynomials and the sine function aren't even orthogonal (to be honest I haven't checked but I'd assume they're not).

They are not, but sigmoids (or ReLUs) are not orthogonal to each other either. Orthogonality has noting to do with any of it.