Neural Network - exercise
But what happens if the input has 0.99 and 0.01?


indicates the point: if the third layer is perfect, then we're already done; but we need to account for small variations in the third layer.  We need to understand how far off we can get after multiplying by $W$.

Now, it appears that the answer gets this a little wrong.  If $\vec{x}$ consists of one entry at least 0.99 and nine entries of at most 0.01, then $W\vec{x}$ is off by some number depending on the ten columns of $W$, not the four rows.  And we need lower bounds for the correct slots, and upper bounds for the incorrect slots (rather than upper bounds for both).

For example, if $\vec{x}=\langle 0.05, 0.997, 0.1, 0.01, \dotsc\rangle$, we'd get

$$W\vec{x} =  \begin{pmatrix}
 0.997 +0.01 + \dotsb \\
 0.1 +0.01 + \dotsb \\
\vdots
\end{pmatrix}.$$

We need to use bias/activation/cutoffs to "push" this to the desired $\langle 1,0,0,0\rangle$ (and similarly for other $\vec{x}$).  Well, the entries where we desire a 1 are at least 0.99 in the relevant position and at least 0 elsewhere (seemingly the activations are supposed to make everything positive?), so we get at least 0.99.  The entries where we desire a 0 are at most 0.01 in every non-zero position, so we get at most $5*0.01$ (the first row of $W$ has the most nonzero entries, 5).  So setting any cutoff (with relevant a bias and/or activation function) between 0.05 and 0.99 will suffice.

[Another example, since that might have been too wordy.  Say the output in the third row is 
$$\vec{x}=\langle 0, 0.01, 0.99, 0.01, 0, 0.01, 0.01, 0.01, 0, 0.01\rangle.$$
Then 
$$W\vec{x} = \langle 0.05, 1.02, 0.03, 0.01\rangle.$$
You can see how I've cooked up $\vec{x}$ to get the largest possible first entry in $W\vec{x}$ given the constraints on $\vec{x}$.]