# Plotting the decision boundary of a logistic regression model

In the notation of this previous post, a logistic regression binary classification model takes an input feature vector, $\boldsymbol{x}$, and returns a probability, $\hat{y}$, that $\boldsymbol{x}$ belongs to a particular class: $\hat{y} = P(y=1|\boldsymbol{x})$. The model is trained on a set of provided example feature vectors, $\boldsymbol{x}^{(i)}$, and their classifications, $y^{(i)} = 0$ or $1$, by finding the set of parameters that minimize the difference between $\hat{y}^{(i)}$ and $y^{(i)}$ in some sense.

These model parameters are the components of a vector, $\boldsymbol{w}$ and a constant, $b$, which relate a given input feature vector to the predicted logit or log-odds, $z$, associated with $\boldsymbol{x}$ belonging to the class $y=1$ through $$z = \boldsymbol{w}^T\boldsymbol{x} + b.$$ In this formulation, $$z = \ln \frac{\hat{y}}{1-\hat{y}} \quad \Rightarrow \hat{y} = \sigma(z) = \frac{1}{1+\mathrm{e}^{-z}}.$$ Note that the relation between $z$ and the components of the feature vector, $x_j$, is linear. In particular, for a two-dimensional problem, $$z = w_1x_1 + w_2x_2 + b.$$ It is sometimes useful to be able to visualize the boundary line dividing the input space in which points are classified as belonging to the class of interest, $y=1$, from that space in which points do not. This could be achieved by calculating the prediction associated with $\hat{y}$ for a mesh of $(x_1, x_2)$ points and plotting a contour plot (see e.g. this scikit-learn example).

Alternatively, one can think of the decision boundary as the line $x_2 = mx_1 + c$, being defined by points for which $\hat{y}=0.5$ and hence $z=0$. For $x_1 = 0$ we have $x_2=c$ (the intercept) and $$0 = 0 + w_2x_2 + b \quad \Rightarrow c = -\frac{b}{w_2}.$$ For the gradient, $m$, consider two distinct points on the decision boundary, $(x_1^a,x_2^a)$ and $(x_1^b,x_2^b)$, so that $m = (x_2^b-x_2^a)/(x_1^b-x_1^a)$. Along the boundary line, \begin{align*} & 0 = w_1x_1^b + w_2x_2^b + b - (w_1x_1^a + w_2x_2^a + b)\\ \Rightarrow & -w_2(x_2^b - x_2^a) = w_1(x_1^b - x_1^a)\\ \Rightarrow & m = -\frac{w_1}{w_2}. \end{align*} To see this in action, consider the data in linpts.txt, which maybe classified using scikit-learn's LogisticRegression classifier. The following script retrieves the decision boundary as above to generate the following visualization. import numpy as np
import matplotlib.pyplot as plt
import sklearn.linear_model
plt.rc('text', usetex=True)

X = pts[:,:2]
Y = pts[:,2].astype('int')

# Fit the data to a logistic regression model.
clf = sklearn.linear_model.LogisticRegression()
clf.fit(X, Y)

# Retrieve the model parameters.
b = clf.intercept_
w1, w2 = clf.coef_.T
# Calculate the intercept and gradient of the decision boundary.
c = -b/w2
m = -w1/w2

# Plot the data and the classification with the decision boundary.
xmin, xmax = -1, 2
ymin, ymax = -1, 2.5
xd = np.array([xmin, xmax])
yd = m*xd + c
plt.plot(xd, yd, 'k', lw=1, ls='--')
plt.fill_between(xd, yd, ymin, color='tab:blue', alpha=0.2)
plt.fill_between(xd, yd, ymax, color='tab:orange', alpha=0.2)

plt.scatter(*X[Y==0].T, s=8, alpha=0.5)
plt.scatter(*X[Y==1].T, s=8, alpha=0.5)
plt.xlim(xmin, xmax)
plt.ylim(ymin, ymax)
plt.ylabel(r'$x_2$')
plt.xlabel(r'$x_1$')

plt.show()

Current rating: 4
{"ft_lang_label":"__label__en","ft_lang_prob":0.71461904,"math_prob":0.99994636,"size":3431,"snap":"2020-45-2020-50","text_gpt3_token_len":1090,"char_repetition_ratio":0.10008754,"word_repetition_ratio":0.0,"special_character_ratio":0.33372194,"punctuation_ratio":0.16875872,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":1.0000031,"pos_list":[0,1,2],"im_url_duplicate_count":[null,4,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-10-26T01:15:27Z\",\"WARC-Record-ID\":\"<urn:uuid:cd66dc03-01ab-4712-ba63-4aeb04c6aca2>\",\"Content-Length\":\"34786\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:13b9cba0-6ba6-449f-9b4a-7fd69e90e28b>\",\"WARC-Concurrent-To\":\"<urn:uuid:43130ec5-3d3c-480a-bce7-0e925981b432>\",\"WARC-IP-Address\":\"31.170.123.78\",\"WARC-Target-URI\":\"https://scipython.com/blog/plotting-the-decision-boundary-of-a-logistic-regression-model/\",\"WARC-Payload-Digest\":\"sha1:MF5MGRJCZLHDRC2E5RUJUU36PGYFIN4U\",\"WARC-Block-Digest\":\"sha1:W2F72IW5BRLR34TLT2CQMQZ25LTO4I24\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-45/CC-MAIN-2020-45_segments_1603107890108.60_warc_CC-MAIN-20201026002022-20201026032022-00481.warc.gz\"}"}