2.4 Linear Regression with Multiple Variables|Gradient descent for multiple linear regression ---ML
 So, you've learned about gradient descent, about multiple linear regression, and also vectorization. Let's put it all together to implement gradient descent for multiple linear regression with vectorization. This would be cool. Let's quickly review what multiple linear regression looks like. Using our previous notation, let's see how you can write it more succinctly using vector notation. We have parameters w1 to wn, as well as b, but instead of thinking of w1 to wn as separate numbers, that is, separate parameters, let's instead collect all of the w's into a vector w, so that now w is a vector of length n. So, we're just going to think of the parameters of this model as a vector w, as well as b, where b is still a number, same as before. For us before, we had defined multiple linear regression like this. Now using vector notation, we can write the model as f sub wb of x equals the vector w dot product with the vector x plus b. And remember that this dot here means dot product. Our cost function can be defined as j of w1 through wn comma b, but instead of just thinking of j as a function of these n different parameters wj as well as b, we're going to write j as a function of parameter vector w and the number b. So this w1 through wn is replaced by this vector w, and j now takes this input, a vector w and a number b, and returns a number. Here's what gradient descent looks like. We're going to repeatedly update each parameter wj to be wj minus alpha times the derivative of the cost j, where j has parameters w1 through wn and b, and once again we just write this as j of vector w and number b. Let's see what this will look like when you implement gradient descent, and in particular, let's take a look at the derivative term. You'll see that gradient descent becomes just a little bit different with multiple features compared to just one feature. Here's what we had when we had gradient descent with one feature. We had an update rule for w and a separate update rule for b. So hopefully these look familiar to you. And this term here is the derivative of the cost function j with respect to the parameter w. And similarly, we have an update rule for parameter b with univariate regression, we had only one feature. We call that feature xi without any subscript. Now here's the new notation for when we have n features where n is two or more. We get this update rule for gradient descent. We have a w1 to be w1 minus alpha times this expression here. And this formula is actually the derivative of the cost j with respect to w1. The formula for the derivative of j with respect to w1 on the right looks very similar to the case of one feature on the left. The error term still takes a prediction f of x minus the target y. One difference is that w and x are now vectors. And just as w on the left has now become w1 here on the right, xi here on the left is now instead xi subscript 1 here on the right. And this is just for j equals 1. For multiple linear regression, we have j ranging from 1 through n, and so we'll update the parameters w1, w2 all the way up to wn. And then as before, we'll update b. And if you implement this, you get gradient descent for multiple regression. So that's it for gradient descent for multiple regression. Before moving on from this video, I want to make a quick aside or a quick side note on an alternative way for finding w and b for linear regression. And this method is called the normal equation. Whereas it turns out gradient descent is a great method for minimizing the cost function j to find w and b. There is one other algorithm that works only for linear regression and pretty much none of the other algorithms you see in this specialization for solving for w and b. And this other method does not need an iterative gradient descent algorithm. Called the normal equation method, it turns out to be possible to use an advanced linear algebra library to just solve for w and b all in one go without iterations. Some disadvantages of the normal equation method are, first, unlike gradient descent, this does not generalize to other learning algorithms, such as the logistic regression algorithm that you learn about next week, or the neural networks or other algorithms you see later in this specialization. The normal equation method is also quite slow if the number of features n is large. Almost no machine learning practitioners should implement the normal equation method themselves. But if you're using a mature machine learning library and call linear regression, there's a chance that on the backend, it will be using this to solve for w and b. So if you're ever in a job interview and hear the term normal equation, that's what this refers to. Don't worry about the details of how the normal equation works. Just be aware that some machine learning libraries may use this complicated method in the backend to solve for w and b. But for most learning algorithms, including how you implement linear regression yourself, gradient descent is often a better way to get the job done. In the optional lab that follows this video, you see how to define a multiple regression model in code, and also how to calculate the prediction f of x. You also see how to calculate the cost and implement gradient descent for multiple linear regression model. This will be using Python's NumPy library, so if any of the code looks very new, that's okay. But you should feel free also to take a look at the previous optional lab that introduces NumPy and vectorization for a refresher of NumPy functions and how to implement those in code. So that's it. You now know multiple linear regression. This is probably the single most widely used learning algorithm in the world today. But there's more. With just a few tricks such as picking and scaling features appropriately, and also choosing the learning rate alpha appropriately, you will be able to make this work much better. So just a few more videos to go for this week. Let's go on to the next video to see those little tricks that will help you make multiple linear regression work much better.
[{"start": 0.0, "end": 7.2, "text": " So, you've learned about gradient descent, about multiple linear regression, and also"}, {"start": 7.2, "end": 8.2, "text": " vectorization."}, {"start": 8.2, "end": 13.16, "text": " Let's put it all together to implement gradient descent for multiple linear regression with"}, {"start": 13.16, "end": 14.44, "text": " vectorization."}, {"start": 14.44, "end": 15.44, "text": " This would be cool."}, {"start": 15.44, "end": 19.52, "text": " Let's quickly review what multiple linear regression looks like."}, {"start": 19.52, "end": 24.04, "text": " Using our previous notation, let's see how you can write it more succinctly using vector"}, {"start": 24.04, "end": 25.18, "text": " notation."}, {"start": 25.18, "end": 33.24, "text": " We have parameters w1 to wn, as well as b, but instead of thinking of w1 to wn as separate"}, {"start": 33.24, "end": 39.04, "text": " numbers, that is, separate parameters, let's instead collect all of the w's into a vector"}, {"start": 39.04, "end": 43.760000000000005, "text": " w, so that now w is a vector of length n."}, {"start": 43.760000000000005, "end": 51.24, "text": " So, we're just going to think of the parameters of this model as a vector w, as well as b,"}, {"start": 51.24, "end": 54.56, "text": " where b is still a number, same as before."}, {"start": 54.56, "end": 59.080000000000005, "text": " For us before, we had defined multiple linear regression like this."}, {"start": 59.080000000000005, "end": 67.44, "text": " Now using vector notation, we can write the model as f sub wb of x equals the vector w"}, {"start": 67.44, "end": 70.92, "text": " dot product with the vector x plus b."}, {"start": 70.92, "end": 75.4, "text": " And remember that this dot here means dot product."}, {"start": 75.4, "end": 82.74000000000001, "text": " Our cost function can be defined as j of w1 through wn comma b, but instead of just thinking"}, {"start": 82.74, "end": 90.91999999999999, "text": " of j as a function of these n different parameters wj as well as b, we're going to write j as"}, {"start": 90.91999999999999, "end": 96.64, "text": " a function of parameter vector w and the number b."}, {"start": 96.64, "end": 105.91999999999999, "text": " So this w1 through wn is replaced by this vector w, and j now takes this input, a vector"}, {"start": 105.91999999999999, "end": 110.52, "text": " w and a number b, and returns a number."}, {"start": 110.52, "end": 112.72, "text": " Here's what gradient descent looks like."}, {"start": 112.72, "end": 119.67999999999999, "text": " We're going to repeatedly update each parameter wj to be wj minus alpha times the derivative"}, {"start": 119.67999999999999, "end": 128.32, "text": " of the cost j, where j has parameters w1 through wn and b, and once again we just write this"}, {"start": 128.32, "end": 134.0, "text": " as j of vector w and number b."}, {"start": 134.0, "end": 138.44, "text": " Let's see what this will look like when you implement gradient descent, and in particular,"}, {"start": 138.44, "end": 141.72, "text": " let's take a look at the derivative term."}, {"start": 141.72, "end": 146.64, "text": " You'll see that gradient descent becomes just a little bit different with multiple features"}, {"start": 146.64, "end": 149.44, "text": " compared to just one feature."}, {"start": 149.44, "end": 153.72, "text": " Here's what we had when we had gradient descent with one feature."}, {"start": 153.72, "end": 158.72, "text": " We had an update rule for w and a separate update rule for b."}, {"start": 158.72, "end": 162.24, "text": " So hopefully these look familiar to you."}, {"start": 162.24, "end": 167.88, "text": " And this term here is the derivative of the cost function j with respect to the parameter"}, {"start": 167.88, "end": 168.88, "text": " w."}, {"start": 168.88, "end": 175.79999999999998, "text": " And similarly, we have an update rule for parameter b with univariate regression, we"}, {"start": 175.79999999999998, "end": 178.24, "text": " had only one feature."}, {"start": 178.24, "end": 182.84, "text": " We call that feature xi without any subscript."}, {"start": 182.84, "end": 189.32, "text": " Now here's the new notation for when we have n features where n is two or more."}, {"start": 189.32, "end": 192.88, "text": " We get this update rule for gradient descent."}, {"start": 192.88, "end": 199.48, "text": " We have a w1 to be w1 minus alpha times this expression here."}, {"start": 199.48, "end": 207.35999999999999, "text": " And this formula is actually the derivative of the cost j with respect to w1."}, {"start": 207.35999999999999, "end": 213.12, "text": " The formula for the derivative of j with respect to w1 on the right looks very similar to the"}, {"start": 213.12, "end": 216.35999999999999, "text": " case of one feature on the left."}, {"start": 216.35999999999999, "end": 222.04, "text": " The error term still takes a prediction f of x minus the target y."}, {"start": 222.04, "end": 226.88, "text": " One difference is that w and x are now vectors."}, {"start": 226.88, "end": 234.68, "text": " And just as w on the left has now become w1 here on the right, xi here on the left is"}, {"start": 234.68, "end": 239.56, "text": " now instead xi subscript 1 here on the right."}, {"start": 239.56, "end": 245.07999999999998, "text": " And this is just for j equals 1."}, {"start": 245.08, "end": 252.52, "text": " For multiple linear regression, we have j ranging from 1 through n, and so we'll update"}, {"start": 252.52, "end": 260.40000000000003, "text": " the parameters w1, w2 all the way up to wn."}, {"start": 260.40000000000003, "end": 264.36, "text": " And then as before, we'll update b."}, {"start": 264.36, "end": 269.68, "text": " And if you implement this, you get gradient descent for multiple regression."}, {"start": 269.68, "end": 274.66, "text": " So that's it for gradient descent for multiple regression."}, {"start": 274.66, "end": 281.28000000000003, "text": " Before moving on from this video, I want to make a quick aside or a quick side note on"}, {"start": 281.28000000000003, "end": 287.36, "text": " an alternative way for finding w and b for linear regression."}, {"start": 287.36, "end": 291.44000000000005, "text": " And this method is called the normal equation."}, {"start": 291.44000000000005, "end": 296.20000000000005, "text": " Whereas it turns out gradient descent is a great method for minimizing the cost function"}, {"start": 296.20000000000005, "end": 299.32000000000005, "text": " j to find w and b."}, {"start": 299.32000000000005, "end": 304.38, "text": " There is one other algorithm that works only for linear regression and pretty much none"}, {"start": 304.38, "end": 310.08, "text": " of the other algorithms you see in this specialization for solving for w and b."}, {"start": 310.08, "end": 316.0, "text": " And this other method does not need an iterative gradient descent algorithm."}, {"start": 316.0, "end": 320.84, "text": " Called the normal equation method, it turns out to be possible to use an advanced linear"}, {"start": 320.84, "end": 327.32, "text": " algebra library to just solve for w and b all in one go without iterations."}, {"start": 327.32, "end": 332.68, "text": " Some disadvantages of the normal equation method are, first, unlike gradient descent,"}, {"start": 332.68, "end": 337.28000000000003, "text": " this does not generalize to other learning algorithms, such as the logistic regression"}, {"start": 337.28000000000003, "end": 341.96, "text": " algorithm that you learn about next week, or the neural networks or other algorithms"}, {"start": 341.96, "end": 344.56, "text": " you see later in this specialization."}, {"start": 344.56, "end": 351.0, "text": " The normal equation method is also quite slow if the number of features n is large."}, {"start": 351.0, "end": 356.84000000000003, "text": " Almost no machine learning practitioners should implement the normal equation method themselves."}, {"start": 356.84, "end": 363.44, "text": " But if you're using a mature machine learning library and call linear regression, there's"}, {"start": 363.44, "end": 368.64, "text": " a chance that on the backend, it will be using this to solve for w and b."}, {"start": 368.64, "end": 373.17999999999995, "text": " So if you're ever in a job interview and hear the term normal equation, that's what this"}, {"start": 373.17999999999995, "end": 375.35999999999996, "text": " refers to."}, {"start": 375.35999999999996, "end": 378.67999999999995, "text": " Don't worry about the details of how the normal equation works."}, {"start": 378.67999999999995, "end": 384.91999999999996, "text": " Just be aware that some machine learning libraries may use this complicated method in the backend"}, {"start": 384.92, "end": 386.76, "text": " to solve for w and b."}, {"start": 386.76, "end": 392.40000000000003, "text": " But for most learning algorithms, including how you implement linear regression yourself,"}, {"start": 392.40000000000003, "end": 396.36, "text": " gradient descent is often a better way to get the job done."}, {"start": 396.36, "end": 401.6, "text": " In the optional lab that follows this video, you see how to define a multiple regression"}, {"start": 401.6, "end": 408.20000000000005, "text": " model in code, and also how to calculate the prediction f of x."}, {"start": 408.20000000000005, "end": 414.04, "text": " You also see how to calculate the cost and implement gradient descent for multiple linear"}, {"start": 414.04, "end": 415.68, "text": " regression model."}, {"start": 415.68, "end": 421.40000000000003, "text": " This will be using Python's NumPy library, so if any of the code looks very new, that's"}, {"start": 421.40000000000003, "end": 422.56, "text": " okay."}, {"start": 422.56, "end": 428.36, "text": " But you should feel free also to take a look at the previous optional lab that introduces"}, {"start": 428.36, "end": 434.24, "text": " NumPy and vectorization for a refresher of NumPy functions and how to implement those"}, {"start": 434.24, "end": 436.0, "text": " in code."}, {"start": 436.0, "end": 437.16, "text": " So that's it."}, {"start": 437.16, "end": 439.82000000000005, "text": " You now know multiple linear regression."}, {"start": 439.82, "end": 444.71999999999997, "text": " This is probably the single most widely used learning algorithm in the world today."}, {"start": 444.71999999999997, "end": 446.08, "text": " But there's more."}, {"start": 446.08, "end": 450.92, "text": " With just a few tricks such as picking and scaling features appropriately, and also choosing"}, {"start": 450.92, "end": 455.88, "text": " the learning rate alpha appropriately, you will be able to make this work much better."}, {"start": 455.88, "end": 458.44, "text": " So just a few more videos to go for this week."}, {"start": 458.44, "end": 462.92, "text": " Let's go on to the next video to see those little tricks that will help you make multiple"}, {"start": 462.92, "end": 469.92, "text": " linear regression work much better."}]