Are Support Vector Machines still considered "state of the art" in their niche? This question is in response to a comment I saw on another question.
The comment was regarding the Machine Learning course syllabus on Coursera, and along the lines of &quot;SVMs are not used so much nowadays&quot;.
I have only just finished the relevant lectures myself, and my understanding of SVMs is that they are a robust and efficient learning algorithm for classification, and that when using a kernel, they have a &quot;niche&quot; covering number of features perhaps 10 to 1000 and number of training samples perhaps 100 to 10,000. The limit on training samples is because the core algorithm revolves around optimising results generated from a square matrix with dimensions based on number of training samples, not number of original features.
So does the comment I saw refer some real change since the course was made, and if so, what is that change: A new algorithm that covers SVM's &quot;sweet spot&quot; just as well, better CPUs meaning SVM's computational advantages are not worth as much? Or is it perhaps opinion or personal experience of the commenter?
I tried a search for e.g. &quot;are support vector machines out of fashion&quot; and found nothing to imply they were being dropped in favour of anything else.
And Wikipedia has this: http://en.wikipedia.org/wiki/Support_vector_machine#Issues . . . the main sticking point appears to be difficulty of interpreting the model. Which makes SVM fine for a black-box predicting engine, but not so good for generating insights. I don't see that as a major issue, just another minor thing to take into account when picking the right tool for the job (along with nature of the training data and learning task etc).
SVM is a powerful classifier. It has some nice advantages (which I guess were responsible for its popularity)... These are:


Efficiency: Only the support vectors play a role in determining the classification boundary. All other points from the training set needn't be stored in memory.
The so-called power of kernels: With appropriate kernels you can transform feature space into a higher dimension so that it becomes linearly separable. The notion of kernels work with arbitrary objects on which you can define some notion of similarity with the help of inner products... and hence SVMs can classify arbitrary objects such as trees, graphs etc.


There are some significant disadvantages as well.


Parameter sensitivity: The performance is highly sensitive to the choice of the regularization parameter C, which allows some variance in the model.
Extra parameter for the Gaussian kernel: The radius of the Gaussian kernel can have a significant impact on classifier accuracy. Typically a grid search has to be conducted to find optimal parameters. LibSVM has a support for grid search.


SVMs generally belong to the class of "Sparse Kernel Machines". The sparse vectors in the case of SVM are the support vectors which are chosen from the maximum margin criterion. Other sparse vector machines such as the Relevance Vector Machine (RVM) perform better than SVM. The following figure shows a comparative performance of the two. In the figure, the x-axis shows one dimensional data from two classes y={0,1}. The mixture model is defined as P(x|y=0)=Unif(0,1) and P(x|y=1)=Unif(.5,1.5) (Unif denotes uniform distribution). 1000 points were sampled from this mixture and an SVM and an RVM were used to estimate the posterior. The problem of SVM is that the predicted values are far off from the true log odds.  

 

A very effective classifier, which is very popular nowadays, is the Random Forest. The main advantages are:


Only one parameter to tune (i.e. the number of trees in the forest)
Not utterly parameter sensitive
Can easily be extended to multiple classes
Is based on probabilistic principles (maximizing mutual information gain with the help of decision trees)