# Machine Learning Notes—Linear Regression

August 14, 2013

This is my first post on machine learning, and hopefully not the last one. The main goal of these posts is to serve as a quick reference for simple machine learning problems and their solutions, meanwhile allowing me to get a better understanding of the field itself. That said, don’t take anything for granted.

## Linear Regression

Linear regression is one of the algorithms used to predict scalar values given some inputs. Linear regression can be modelled like this:

It can also be written in a vector form as:

We first need to find the model parameters $\beta$ before any predictions, $y$, can be made for our inputs $x$.

### Finding Betas

Given a set of inputs and outputs we would have to solve a set equations for $\beta$. The vector form can be solved using linear algebra; the solution takes the form of:

This particular solution will find betas using the method of ordinary least squares. If linear algebra is not an option, another way to find betas is to minimize the difference between actual values and predicted values by using some function optimization method, e.g. gradient descent.

## Data Set

We are going to use a minimal version of the Iris data set in the code examples.

## Implementing The Model

In the examples below I am going to implement a simple linear regression model that takes only a single input. This way we can much more easily visualize our data and model; however, the code can be easily extended to support more than one input.

In this example we are going to model the sepal length using sepal width only for setosa flower class.

One thing to note here is that we are adding intercept to our inputs (ones). It’s not very important in this particular case since our inputs will never evaluate to 0. In other cases it may be useful.

### Solving For Betas With NumPy And Linear Algebra

The fit() function expects a list of tuples as inputs, and a list of output values. The model parameters will be returned as a list. If we are going to run this function on our data set we will get a these beta coefficients [2.63900125 0.69048972]. Thus, our regression function looks as follows:

### Solving For Betas With scikit-learn

scikit-learn package comes with a linear model which can be used to solve linear regression problems.

The model parameters are stored in model.coef_ and for our dataset they are array([ 2.63900125, 0.69048972]). Predictions can be done by calling model.predict method.

By looking at the graphs it’s diffucult to tell which model is a better one. We can take a look at the coefficient of determination ($R^2$) for each of them. scikit-learn model provides score method which can be used to obtain it. For the first model we have $R^2$ of 0.551375580392 and for the second one—0.547248091457. The closer $R^2$ to 1 the better our model is.
{"extraction_info": {"found_math": true, "script_math_tex": 7, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.5095404386520386, "perplexity": 419.28051425231376}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2016-18/segments/1461860116878.73/warc/CC-MAIN-20160428161516-00123-ip-10-239-7-51.ec2.internal.warc.gz"}