2.2 Linear Regression with Multiple Variables | Vectorization part 1 --[Machine Learning |Andrew Ng]
 In this video, you see a very useful idea called vectorization. When you're implementing a learning algorithm, using vectorization will both make your code shorter and also make it run much more efficiently. Learning how to write vectorized code will allow you to also take advantage of modern numerical linear algebra libraries, as well as maybe even GPU hardware that stands for graphics processor unit. This is hardware originally designed to speed up computer graphics on your computer, but turns out can be used when you write vectorized code to also help you execute your code much more quickly. Let's look at a concrete example of what vectorization means. Here's an example with parameters w and b, where w is a vector with three numbers, and you also have a vector of features x with also three numbers. So here, n is equal to three. So notice that in linear algebra, the index or the counting starts from one, and so the first value is subscripted w1 and x1. In Python code, you can define these variables w, b, and x using arrays like this. Here I'm actually using a numerical linear algebra library in Python called NumPy, which is by far the most widely used numerical linear algebra library in Python and in machine learning. Because in Python, the indexing of arrays or counting in arrays starts from zero, you would access the first value of w using w square bracket zero, the second value using w square bracket one, and the third using w square bracket two. So the indexing here goes from zero, one, to two, rather than one, two, to three. Similarly, to access individual features of x, you would use x0, x1, and x2. Many programming languages, including Python, start counting from zero rather than one. Now, let's look at an implementation without vectorization for computing the model's prediction. In code, it would look like this. You take each parameter w and multiply it by its associated feature. Now, you could write your code like this, but what if n is in three, but instead n is 100 or 100,000? It's both inefficient for you to code and inefficient for your computer to compute. So here's another way, still without using vectorization, but using a for loop. In math, you can use a summation operator to add all the products of wj and xj for j equals one through n. Then outside the summation, you add b at the end. So the summation goes from j equals one up to and including n for n equals three, j therefore goes from one, two, to three. In code, you can initialize f to zero, then for j in range from zero to n, this actually makes j go from zero to n minus one. So from zero, one to two, you can then add to f the product of wj times xj. Finally, outside the for loop, you add b. Notice that in Python, the range zero to n means that j goes from zero all the way to n minus one and does not include n itself. And more commonly, this is written range n in Python. But in this video, I added a zero here just to emphasize that it starts from zero. While this implementation is a bit better than the first one, it still doesn't use vectorization and isn't that efficient. Now let's look at how you can do this using vectorization. This is the math expression that the function f, which is the dot product of w and x plus b. And now you can implement this with a single line of code. By computing fp equals np dot dot, I said dot dot because the first dot is the period and the second dot is the function or the method called dot. But it's fp equals np dot dot w comma x and this implements the mathematical dot products between the vectors w and x. And then finally, you can add b to it at the end. This NumPy dot function is a vectorized implementation of the dot product operation between two vectors. And especially when n is large, this will run much faster than the two previous code examples. I want to emphasize that vectorization actually has two distinct benefits. First, it makes the code shorter. It's now just one line of code. Isn't that cool? And second, it also results in your code running much faster than either of the two previous implementations that did not use vectorization. And the reason that the vectorized implementation is much faster is behind the scenes, the NumPy dot function is able to use parallel hardware in your computer. And this is true whether you're running this on a normal computer that is on a normal computer CPU, or if you are using a GPU, a graphics processor unit that's often used to accelerate machine learning jobs. And the ability of the NumPy dot function to use parallel hardware makes it much more efficient than the for loop or the sequential calculation that we saw previously. Now this version is much more practical when n is large because you are not typing w0 times x0 plus w1 times x1 plus lots of additional terms like you would have had for the previous version. But while this saves a lot on the typing, it's still not that computationally efficient because it still doesn't use vectorization. So to recap, vectorization makes your code shorter, so hopefully easier to write and easier for you or others to read. And it also makes it run much faster. But what is this magic behind vectorization that makes this run so much faster? Let's take a look at what your computer is actually doing behind the scenes to make vectorized code run so much faster.
[{"start": 0.0, "end": 7.28, "text": " In this video, you see a very useful idea called vectorization."}, {"start": 7.28, "end": 11.96, "text": " When you're implementing a learning algorithm, using vectorization will both make your code"}, {"start": 11.96, "end": 16.740000000000002, "text": " shorter and also make it run much more efficiently."}, {"start": 16.740000000000002, "end": 21.16, "text": " Learning how to write vectorized code will allow you to also take advantage of modern"}, {"start": 21.16, "end": 26.82, "text": " numerical linear algebra libraries, as well as maybe even GPU hardware that stands for"}, {"start": 26.82, "end": 28.68, "text": " graphics processor unit."}, {"start": 28.68, "end": 33.4, "text": " This is hardware originally designed to speed up computer graphics on your computer, but"}, {"start": 33.4, "end": 38.04, "text": " turns out can be used when you write vectorized code to also help you execute your code much"}, {"start": 38.04, "end": 39.04, "text": " more quickly."}, {"start": 39.04, "end": 42.8, "text": " Let's look at a concrete example of what vectorization means."}, {"start": 42.8, "end": 51.32, "text": " Here's an example with parameters w and b, where w is a vector with three numbers, and"}, {"start": 51.32, "end": 56.36, "text": " you also have a vector of features x with also three numbers."}, {"start": 56.36, "end": 60.019999999999996, "text": " So here, n is equal to three."}, {"start": 60.019999999999996, "end": 66.12, "text": " So notice that in linear algebra, the index or the counting starts from one, and so the"}, {"start": 66.12, "end": 71.03999999999999, "text": " first value is subscripted w1 and x1."}, {"start": 71.03999999999999, "end": 78.96000000000001, "text": " In Python code, you can define these variables w, b, and x using arrays like this."}, {"start": 78.96000000000001, "end": 85.0, "text": " Here I'm actually using a numerical linear algebra library in Python called NumPy, which"}, {"start": 85.0, "end": 92.12, "text": " is by far the most widely used numerical linear algebra library in Python and in machine learning."}, {"start": 92.12, "end": 99.4, "text": " Because in Python, the indexing of arrays or counting in arrays starts from zero, you"}, {"start": 99.4, "end": 106.92, "text": " would access the first value of w using w square bracket zero, the second value using"}, {"start": 106.92, "end": 112.76, "text": " w square bracket one, and the third using w square bracket two."}, {"start": 112.76, "end": 120.48, "text": " So the indexing here goes from zero, one, to two, rather than one, two, to three."}, {"start": 120.48, "end": 129.26, "text": " Similarly, to access individual features of x, you would use x0, x1, and x2."}, {"start": 129.26, "end": 134.88, "text": " Many programming languages, including Python, start counting from zero rather than one."}, {"start": 134.88, "end": 141.92000000000002, "text": " Now, let's look at an implementation without vectorization for computing the model's prediction."}, {"start": 141.92, "end": 144.35999999999999, "text": " In code, it would look like this."}, {"start": 144.35999999999999, "end": 150.35999999999999, "text": " You take each parameter w and multiply it by its associated feature."}, {"start": 150.35999999999999, "end": 156.67999999999998, "text": " Now, you could write your code like this, but what if n is in three, but instead n is"}, {"start": 156.67999999999998, "end": 158.83999999999997, "text": " 100 or 100,000?"}, {"start": 158.83999999999997, "end": 164.11999999999998, "text": " It's both inefficient for you to code and inefficient for your computer to compute."}, {"start": 164.11999999999998, "end": 170.07999999999998, "text": " So here's another way, still without using vectorization, but using a for loop."}, {"start": 170.08, "end": 178.10000000000002, "text": " In math, you can use a summation operator to add all the products of wj and xj for j"}, {"start": 178.10000000000002, "end": 180.36, "text": " equals one through n."}, {"start": 180.36, "end": 185.34, "text": " Then outside the summation, you add b at the end."}, {"start": 185.34, "end": 193.20000000000002, "text": " So the summation goes from j equals one up to and including n for n equals three, j therefore"}, {"start": 193.20000000000002, "end": 196.34, "text": " goes from one, two, to three."}, {"start": 196.34, "end": 204.4, "text": " In code, you can initialize f to zero, then for j in range from zero to n, this actually"}, {"start": 204.4, "end": 208.28, "text": " makes j go from zero to n minus one."}, {"start": 208.28, "end": 216.32, "text": " So from zero, one to two, you can then add to f the product of wj times xj."}, {"start": 216.32, "end": 220.32, "text": " Finally, outside the for loop, you add b."}, {"start": 220.32, "end": 225.92000000000002, "text": " Notice that in Python, the range zero to n means that j goes from zero all the way to"}, {"start": 225.92, "end": 229.88, "text": " n minus one and does not include n itself."}, {"start": 229.88, "end": 234.72, "text": " And more commonly, this is written range n in Python."}, {"start": 234.72, "end": 240.61999999999998, "text": " But in this video, I added a zero here just to emphasize that it starts from zero."}, {"start": 240.61999999999998, "end": 246.56, "text": " While this implementation is a bit better than the first one, it still doesn't use vectorization"}, {"start": 246.56, "end": 249.16, "text": " and isn't that efficient."}, {"start": 249.16, "end": 255.39999999999998, "text": " Now let's look at how you can do this using vectorization."}, {"start": 255.4, "end": 261.38, "text": " This is the math expression that the function f, which is the dot product of w and x plus"}, {"start": 261.38, "end": 262.72, "text": " b."}, {"start": 262.72, "end": 266.34000000000003, "text": " And now you can implement this with a single line of code."}, {"start": 266.34000000000003, "end": 273.56, "text": " By computing fp equals np dot dot, I said dot dot because the first dot is the period"}, {"start": 273.56, "end": 278.82, "text": " and the second dot is the function or the method called dot."}, {"start": 278.82, "end": 288.2, "text": " But it's fp equals np dot dot w comma x and this implements the mathematical dot products"}, {"start": 288.2, "end": 291.6, "text": " between the vectors w and x."}, {"start": 291.6, "end": 295.76, "text": " And then finally, you can add b to it at the end."}, {"start": 295.76, "end": 302.6, "text": " This NumPy dot function is a vectorized implementation of the dot product operation between two vectors."}, {"start": 302.6, "end": 307.8, "text": " And especially when n is large, this will run much faster than the two previous code"}, {"start": 307.8, "end": 309.32, "text": " examples."}, {"start": 309.32, "end": 313.8, "text": " I want to emphasize that vectorization actually has two distinct benefits."}, {"start": 313.8, "end": 316.2, "text": " First, it makes the code shorter."}, {"start": 316.2, "end": 318.12, "text": " It's now just one line of code."}, {"start": 318.12, "end": 319.36, "text": " Isn't that cool?"}, {"start": 319.36, "end": 324.88, "text": " And second, it also results in your code running much faster than either of the two previous"}, {"start": 324.88, "end": 329.2, "text": " implementations that did not use vectorization."}, {"start": 329.2, "end": 336.2, "text": " And the reason that the vectorized implementation is much faster is behind the scenes, the NumPy"}, {"start": 336.2, "end": 340.92, "text": " dot function is able to use parallel hardware in your computer."}, {"start": 340.92, "end": 345.84, "text": " And this is true whether you're running this on a normal computer that is on a normal computer"}, {"start": 345.84, "end": 352.58, "text": " CPU, or if you are using a GPU, a graphics processor unit that's often used to accelerate"}, {"start": 352.58, "end": 354.44, "text": " machine learning jobs."}, {"start": 354.44, "end": 359.52, "text": " And the ability of the NumPy dot function to use parallel hardware makes it much more"}, {"start": 359.52, "end": 367.52, "text": " efficient than the for loop or the sequential calculation that we saw previously."}, {"start": 367.52, "end": 374.91999999999996, "text": " Now this version is much more practical when n is large because you are not typing w0 times"}, {"start": 374.91999999999996, "end": 381.52, "text": " x0 plus w1 times x1 plus lots of additional terms like you would have had for the previous"}, {"start": 381.52, "end": 382.52, "text": " version."}, {"start": 382.52, "end": 388.2, "text": " But while this saves a lot on the typing, it's still not that computationally efficient"}, {"start": 388.2, "end": 392.08, "text": " because it still doesn't use vectorization."}, {"start": 392.08, "end": 397.52, "text": " So to recap, vectorization makes your code shorter, so hopefully easier to write and"}, {"start": 397.52, "end": 400.08, "text": " easier for you or others to read."}, {"start": 400.08, "end": 402.4, "text": " And it also makes it run much faster."}, {"start": 402.4, "end": 408.2, "text": " But what is this magic behind vectorization that makes this run so much faster?"}, {"start": 408.2, "end": 412.24, "text": " Let's take a look at what your computer is actually doing behind the scenes to make vectorized"}, {"start": 412.24, "end": 419.24, "text": " code run so much faster."}]