Why Decision Tree boundary forms a square shape and SVM a circular/oval one? I was going through a Udacity tutorial wherein a few data points were given and the exercise was to test which of the following models best fit the data: linear regression, decision tree, or SVM. Using sklearn, I was able to determine that that SVM is the best fit followed by decision tree. I got a very distinct decision boundary when these two algorithms were applied:



Is there any specific reason for the said shapes or does it just depend on the data sets?

The code was quite straightforward; just reading the CSV, separating the features and then applying the algorithms as shown below:

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

import pandas
import numpy

# Read the data
data = pandas.read_csv('data.csv')

# Split the data into X and y
X = numpy.array(data[['x1', 'x2']])
y = numpy.array(data['y'])

# import statements for the classification algorithms
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC

    # Logistic Regression Classifier
    classifier = LogisticRegression()
    classifier.fit(X,y)

    # Decision Tree Classifier
    classifier = GradientBoostingClassifier()
    classifier.fit(X,y)

    # Support Vector Machine Classifier
    classifier = SVC()
    classifier.fit(X,y)
Shape of the SVM decision boundary depends on the kernel (similarity function) used. The "standard" version of SVM has linear decision boundary. The one displayed could be using Gaussian kernel.

Decision boundary of a decision tree is determined by overlapping orthogonal half-planes (representing the result of each subsequent decision) and can end up as displayed on the pictures.

See more here:

https://shapeofdata.wordpress.com/2013/07/02/decision-trees/

https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM