Calculation of VC dimension of simple neural network
I do not believe this is correct.  The entire network will represent a piecewise-constant function with at most $k+1$ pieces, and has VC dimension $k+1$.
Each hidden neuron is a step function, and together there are at most $k$ jump points among them.  Taking a linear combination of those, we still cannot create any new jump points, so at the output neuron before activation, we have a piecewise-constant function with at most $k$ jump points, and arbitrary constant values on the $k+1$ intervals between them.  After the activation, it's just piecewise constant with values 0 and 1 on at most $k+1$ intervals.