Support Vector Machine Errors
A SVM has 3 very important components: the support vectors, the separating hyperplane and the margin.




When a missclassification occurs, it is because a given point is on the
wrong side of the separating hyperplane, and that's called a
classification error. 
Whenever a point is inside the margin, that counts as a margin error.
The total error of a SVM, is the sum of the classification error and 
the margin error.


Now, to understand the C parameter, you've got to know that there are 2 types of SVMs:


Hard margin: these try to maximize the margin without introducing any
kind of errors.
Soft margin: which also share the same objective, but they allow for some classification and margin errors to occur. The number of errors allowed is controlled by the C parameter (which is often called the penalty parameter): if C is small, the SVM allows for some errors and therefore reaches better generalization; but as C increases, the SVM penalizes these errors more and more, eventually reaching the point that it allows no errors at all.


As you can see bellow on the left, a very large C restricts the model by not allowing any errors, showing very bad results when outliers are present. On the right, by using a small penalty parameter, we let that negative outlier to be in the positive area, therefore maximizing the margin value and reaching a more respectful separating hyperplane.



Although, observe that if you have a soft margin SVM with a very large C value, it will behave just like a hard margin SVM.