SVM regularization - minimizing margin? I'm currently studying from Andrew Ng's Stanford handouts here (I'm at part 8). Now from what I gathered from before, all the time our goal was to minimize ||w||^2 so that we can maximize the margin. However, now as he's writing about the regularization, he's saying:


  The parameter C controls the relative weighting between the twin goals of making the ||w||^2 large (which we saw earlier makes the margin small) and of ensuring that most examples have functional margin at least 1.


Why would making the margin small be currently a "goal"?
You are minimizing the entire loss equation. If it contains regularization, you force the weights to be small too. Having small weights is favorable characteristic because the algorithm is not focusing strongly on one feature, all happen to be important, so the risk of overfitting to some feature is smaller. 

So it is really some tradeoff as always, between the big margin but small focus on dataset distinct examples.