why is SVM cost function the norm of the hyperplane parameters, and not the mean square error?
Given a linearly separable data set, the intuition behind the SVM-algorithm is to find a hyperplane that seperates both classes and maximizes the margin (which is twice the distance of the hyperplane to the closest data point).

So from all hyperplanes that seperate both classes, we want to find the one with the biggest margin (to achieve the best generalization).

Let's imagine we have two parallel seperating hyperplanes defined by the vector $\theta$ and with no data points in between, their distance is then given by: $$m= \frac{2}{\|\theta\|}$$
To maximize this distance, the two hyperplanes must pass through data points of the opposing classes. 
In this case we can define another hyperplane in the middle (with same distance to both hyperplanes and the same vector $\theta$) that now has the margin $m$.

As you can see from the formula: the smaller $\|\theta\|$ the greater the margin $m$.

So maximizing the margin $m$ is equivalent to minimizing $\|\theta\|$. Thus, a separating hyperplane is optimal if it minimizes $\|\theta\|$.

This is just the rough intuition - if you want to go deeper into the maths behind it, I could recommend: http://www.svm-tutorial.com/2015/06/svm-understanding-math-part-3/