4.9 TensorFlow implementation | Data in TensorFlow --[Machine Learning | Andrew Ng]
 In this video, I want to step through with you how data is represented in NumPy and IntensiveLow so that as you're implementing new neural networks, you can have a consistent framework to think about how to represent your data. One of the unfortunate things about the way things are done in code today is that many, many years ago, NumPy was first created and became a standard library for linear algebra in Python. And then much later, the Google Brain team, the team that I had started and once led, created TensorFlow. And so unfortunately, there are some inconsistencies between how data is represented in NumPy and IntensiveLow. So it's good to be aware of these conventions so that you can implement correct code and hopefully get things running in your neural networks. Let's start by taking a look at how TensorFlow represents data. Let's say you have a data set like this from the coffee example. I mentioned that you would write x as follows. So why do you have this double square bracket here? Let's take a look at how NumPy stores vectors and matrices. In case you think matrices and vectors are complicated mathematical concepts, don't worry about it. We'll go through a few concrete examples and you'll be able to do everything you need to do with matrices and vectors in order to implement neural networks. Let's start with an example of a matrix. Here is a matrix with two rows and three columns. Notice that there are one, two rows and one, two, three columns. So we call this a two by three matrix. And so the convention is the dimension of the matrix is written as the number of rows by the number of columns. So in code to store this matrix, this two by three matrix, you just write x equals NP dot array of these numbers like these, where you notice that the square brackets tells you that one, two, three is the first row of this matrix and four, five, six is the second row of this matrix. And then this alphas square bracket groups the first and the second row together. So this says x to be this 2D array of numbers. So a matrix is just a 2D array of numbers. Let's look at one more example. Here I've written out another matrix. How many rows and how many columns does this have? Well we count this as one, two, three, four rows and it has one, two columns. So this is a number of rows by number of columns matrix. So it's a four by two matrix. And so to store this in code, you would write x equals NP dot array and then this syntax over here to store these four rows of a matrix in the variable x. So this creates a 2D array of these eight numbers. Matrices can have different dimensions. We saw an example of a two by three matrix and a four by two matrix. A matrix can also be other dimensions like one by two or two by one. And we'll see examples of these on the next slide. So what we did previously when setting x to be input feature vectors was set x to be equal to NP array with two square brackets 200 comma 17. And what that does is this creates a one by two matrix that is just one row and two columns. Let's look at a different example. If you were to define x to be NP array, but now written like this, this creates a two by one matrix that has two rows and one column. Because the first row is just a number 200 and the second row is just the number 17. And so this has the same numbers, but in a two by one instead of a one by two matrix. In nav, this example on top is also called a row vector. It's a vector that is just a single row. And this example is also called a column vector because it's a vector that just has a single column. And the difference between using double square brackets like this versus a single square bracket like this is that whereas the two examples on top are two D arrays where one of the dimensions happens to be one. This example results in a one D vector. So this is just a one D array that has no rows or columns. Although by convention, we may write x as a column like this. So on a contrast this with what we had previously done in the first course, which was to write x like this with a single square bracket. And that resulted in what's called in Python a one D vector instead of a two D matrix. And this technically is not one by two or two by one. It's just a linear array with no rows or no columns. And it's just a list of numbers. So whereas in course one, when we're working with linear regression, logistic regression, we use these one D vectors to represent the input features x. With TensorFlow, the convention is to use matrices to represent the data. And why is there this switch in conventions? Well, it turns out that TensorFlow was designed to handle very large datasets. And by representing the data in matrices instead of one D arrays, it lets TensorFlow be a bit more computationally efficient internally. So going back to our original example, for the first training example in this dataset with features 200 degrees Celsius in 17 minutes, we would represent it like this. And so this is actually a one by two matrix that happens to have one row and two columns to store the numbers 217. And in case this seems like a lot of details and really complicated conventions, don't worry about it. All this will become clearer. And you get to see the concrete implementations of the code yourself in the optional labs and in the practice labs. Going back to the code for carrying out for propagation or inference in the neural network, when you compute a one equals layer one applied to X, what is a one? Well, a one is actually going to be because there's three numbers is actually going to be a one by three matrix. And if you print out a one, you will get something like this is TF dot tensor point 2.7.3 is a shape of one by three, one three refers to that this is a one by three matrix. And this is TensorFlow's way of saying that this is a floating point number, meaning that it's a number that can have a decimal point represented using 32 bits of memory in your computer. That's what a float 32 is. And what is a tensor? A tensor here is a data type that the TensorFlow team had created in order to store and carry out computations on matrices efficiently. So whenever you see tensor, just think of it as matrix on these few slides. Technically a tensor is a little bit more general than the matrix, but for the purposes of this course, think of tensor as just a way of representing matrices. So remember I said at the start of this video that there's the TensorFlow way of representing a matrix and the NumPy way of representing matrix. This is an artifact of the history of how NumPy and TensorFlow were created. And unfortunately, there are two ways of representing a matrix that have been baked into these systems. And in fact, if you want to take a one, which is a tensor and want to convert it back to NumPy array, you can do so with this function a one dot NumPy and it will take the same data and return it in the form of a NumPy array rather than in the form of a TensorFlow array or TensorFlow matrix. Now let's take a look at what the activations output by the second layer would look like. Here's the code that we had from before. Layer two is a dense layer with one unit and a six point activation. And a two is computed by taking layer two and applying it to a one. So what is a two? A two may be a number like 0.8 and technically this is a one by one matrix. It's a 2D array with one row and one column. And so it's equal to this number 0.8. And if you print out a two, you see that it is a TensorFlow tensor with just one element, one number 0.8. And it is a one by one matrix. Again it is a flow 32 decimal point number taking up 32 bits in computer memory. Once again you can convert from a TensorFlow tensor to a NumPy matrix using a two dot NumPy and that will turn this back into a NumPy array that looks like this. So that hopefully gives you a sense of how data is represented in TensorFlow and in NumPy. I'm used to loading data and manipulating data in NumPy. But when you pass a NumPy array into TensorFlow, TensorFlow likes to convert it to its own internal format, the tensor, and then operate efficiently using tensors. And when you read the data back out, you can keep it as a tensor or convert it back to a NumPy array. I think it's a bit unfortunate that the history of how these libraries evolve has led us to do this extra conversion work when actually the two libraries can work quite well together. But when you convert back and forth, whether you're using a NumPy array or a tensor, it's just something to be aware of when you're writing code. Next, let's take what we've learned and put it together to actually build a neural network. Let's go see that in the next video.
[{"start": 0.0, "end": 9.16, "text": " In this video, I want to step through with you how data is represented in NumPy and IntensiveLow"}, {"start": 9.16, "end": 15.0, "text": " so that as you're implementing new neural networks, you can have a consistent framework"}, {"start": 15.0, "end": 18.6, "text": " to think about how to represent your data."}, {"start": 18.6, "end": 23.32, "text": " One of the unfortunate things about the way things are done in code today is that many,"}, {"start": 23.32, "end": 28.76, "text": " many years ago, NumPy was first created and became a standard library for linear algebra"}, {"start": 28.76, "end": 30.080000000000002, "text": " in Python."}, {"start": 30.080000000000002, "end": 35.120000000000005, "text": " And then much later, the Google Brain team, the team that I had started and once led,"}, {"start": 35.120000000000005, "end": 36.6, "text": " created TensorFlow."}, {"start": 36.6, "end": 42.56, "text": " And so unfortunately, there are some inconsistencies between how data is represented in NumPy and"}, {"start": 42.56, "end": 44.160000000000004, "text": " IntensiveLow."}, {"start": 44.160000000000004, "end": 49.2, "text": " So it's good to be aware of these conventions so that you can implement correct code and"}, {"start": 49.2, "end": 52.480000000000004, "text": " hopefully get things running in your neural networks."}, {"start": 52.480000000000004, "end": 56.84, "text": " Let's start by taking a look at how TensorFlow represents data."}, {"start": 56.84, "end": 61.88, "text": " Let's say you have a data set like this from the coffee example."}, {"start": 61.88, "end": 66.08, "text": " I mentioned that you would write x as follows."}, {"start": 66.08, "end": 71.16, "text": " So why do you have this double square bracket here?"}, {"start": 71.16, "end": 77.08000000000001, "text": " Let's take a look at how NumPy stores vectors and matrices."}, {"start": 77.08000000000001, "end": 83.12, "text": " In case you think matrices and vectors are complicated mathematical concepts, don't worry"}, {"start": 83.12, "end": 84.12, "text": " about it."}, {"start": 84.12, "end": 88.88000000000001, "text": " We'll go through a few concrete examples and you'll be able to do everything you need to"}, {"start": 88.88000000000001, "end": 93.80000000000001, "text": " do with matrices and vectors in order to implement neural networks."}, {"start": 93.80000000000001, "end": 97.12, "text": " Let's start with an example of a matrix."}, {"start": 97.12, "end": 103.56, "text": " Here is a matrix with two rows and three columns."}, {"start": 103.56, "end": 111.96000000000001, "text": " Notice that there are one, two rows and one, two, three columns."}, {"start": 111.96, "end": 117.36, "text": " So we call this a two by three matrix."}, {"start": 117.36, "end": 123.96, "text": " And so the convention is the dimension of the matrix is written as the number of rows"}, {"start": 123.96, "end": 126.16, "text": " by the number of columns."}, {"start": 126.16, "end": 133.88, "text": " So in code to store this matrix, this two by three matrix, you just write x equals NP"}, {"start": 133.88, "end": 141.56, "text": " dot array of these numbers like these, where you notice that the square brackets tells"}, {"start": 141.56, "end": 148.88, "text": " you that one, two, three is the first row of this matrix and four, five, six is the"}, {"start": 148.88, "end": 152.0, "text": " second row of this matrix."}, {"start": 152.0, "end": 158.35999999999999, "text": " And then this alphas square bracket groups the first and the second row together."}, {"start": 158.36, "end": 163.92000000000002, "text": " So this says x to be this 2D array of numbers."}, {"start": 163.92000000000002, "end": 168.84, "text": " So a matrix is just a 2D array of numbers."}, {"start": 168.84, "end": 171.4, "text": " Let's look at one more example."}, {"start": 171.4, "end": 174.28, "text": " Here I've written out another matrix."}, {"start": 174.28, "end": 177.08, "text": " How many rows and how many columns does this have?"}, {"start": 177.08, "end": 185.72000000000003, "text": " Well we count this as one, two, three, four rows and it has one, two columns."}, {"start": 185.72, "end": 190.06, "text": " So this is a number of rows by number of columns matrix."}, {"start": 190.06, "end": 192.6, "text": " So it's a four by two matrix."}, {"start": 192.6, "end": 199.76, "text": " And so to store this in code, you would write x equals NP dot array and then this syntax"}, {"start": 199.76, "end": 205.5, "text": " over here to store these four rows of a matrix in the variable x."}, {"start": 205.5, "end": 211.0, "text": " So this creates a 2D array of these eight numbers."}, {"start": 211.0, "end": 213.18, "text": " Matrices can have different dimensions."}, {"start": 213.18, "end": 219.08, "text": " We saw an example of a two by three matrix and a four by two matrix."}, {"start": 219.08, "end": 225.84, "text": " A matrix can also be other dimensions like one by two or two by one."}, {"start": 225.84, "end": 229.36, "text": " And we'll see examples of these on the next slide."}, {"start": 229.36, "end": 236.12, "text": " So what we did previously when setting x to be input feature vectors was set x to be equal"}, {"start": 236.12, "end": 242.88, "text": " to NP array with two square brackets 200 comma 17."}, {"start": 242.88, "end": 253.24, "text": " And what that does is this creates a one by two matrix that is just one row and two columns."}, {"start": 253.24, "end": 255.84, "text": " Let's look at a different example."}, {"start": 255.84, "end": 263.04, "text": " If you were to define x to be NP array, but now written like this, this creates a two"}, {"start": 263.04, "end": 271.15999999999997, "text": " by one matrix that has two rows and one column."}, {"start": 271.16, "end": 279.08000000000004, "text": " Because the first row is just a number 200 and the second row is just the number 17."}, {"start": 279.08000000000004, "end": 285.86, "text": " And so this has the same numbers, but in a two by one instead of a one by two matrix."}, {"start": 285.86, "end": 289.28000000000003, "text": " In nav, this example on top is also called a row vector."}, {"start": 289.28000000000003, "end": 292.64000000000004, "text": " It's a vector that is just a single row."}, {"start": 292.64000000000004, "end": 297.6, "text": " And this example is also called a column vector because it's a vector that just has a single"}, {"start": 297.6, "end": 300.12, "text": " column."}, {"start": 300.12, "end": 306.08, "text": " And the difference between using double square brackets like this versus a single square"}, {"start": 306.08, "end": 314.36, "text": " bracket like this is that whereas the two examples on top are two D arrays where one"}, {"start": 314.36, "end": 317.92, "text": " of the dimensions happens to be one."}, {"start": 317.92, "end": 322.32, "text": " This example results in a one D vector."}, {"start": 322.32, "end": 328.48, "text": " So this is just a one D array that has no rows or columns."}, {"start": 328.48, "end": 333.62, "text": " Although by convention, we may write x as a column like this."}, {"start": 333.62, "end": 339.88, "text": " So on a contrast this with what we had previously done in the first course, which was to write"}, {"start": 339.88, "end": 343.88, "text": " x like this with a single square bracket."}, {"start": 343.88, "end": 350.44, "text": " And that resulted in what's called in Python a one D vector instead of a two D matrix."}, {"start": 350.44, "end": 354.04, "text": " And this technically is not one by two or two by one."}, {"start": 354.04, "end": 358.44, "text": " It's just a linear array with no rows or no columns."}, {"start": 358.44, "end": 360.96, "text": " And it's just a list of numbers."}, {"start": 360.96, "end": 365.68, "text": " So whereas in course one, when we're working with linear regression, logistic regression,"}, {"start": 365.68, "end": 370.72, "text": " we use these one D vectors to represent the input features x."}, {"start": 370.72, "end": 376.72, "text": " With TensorFlow, the convention is to use matrices to represent the data."}, {"start": 376.72, "end": 378.68, "text": " And why is there this switch in conventions?"}, {"start": 378.68, "end": 384.42, "text": " Well, it turns out that TensorFlow was designed to handle very large datasets."}, {"start": 384.42, "end": 390.48, "text": " And by representing the data in matrices instead of one D arrays, it lets TensorFlow be a bit"}, {"start": 390.48, "end": 394.36, "text": " more computationally efficient internally."}, {"start": 394.36, "end": 399.40000000000003, "text": " So going back to our original example, for the first training example in this dataset"}, {"start": 399.40000000000003, "end": 405.90000000000003, "text": " with features 200 degrees Celsius in 17 minutes, we would represent it like this."}, {"start": 405.90000000000003, "end": 413.68, "text": " And so this is actually a one by two matrix that happens to have one row and two columns"}, {"start": 413.68, "end": 417.68, "text": " to store the numbers 217."}, {"start": 417.68, "end": 422.52, "text": " And in case this seems like a lot of details and really complicated conventions, don't"}, {"start": 422.52, "end": 423.52, "text": " worry about it."}, {"start": 423.52, "end": 425.32, "text": " All this will become clearer."}, {"start": 425.32, "end": 430.44, "text": " And you get to see the concrete implementations of the code yourself in the optional labs"}, {"start": 430.44, "end": 432.0, "text": " and in the practice labs."}, {"start": 432.0, "end": 438.68, "text": " Going back to the code for carrying out for propagation or inference in the neural network,"}, {"start": 438.68, "end": 446.08, "text": " when you compute a one equals layer one applied to X, what is a one?"}, {"start": 446.08, "end": 451.6, "text": " Well, a one is actually going to be because there's three numbers is actually going to"}, {"start": 451.6, "end": 455.36, "text": " be a one by three matrix."}, {"start": 455.36, "end": 464.52, "text": " And if you print out a one, you will get something like this is TF dot tensor point 2.7.3 is"}, {"start": 464.52, "end": 471.03999999999996, "text": " a shape of one by three, one three refers to that this is a one by three matrix."}, {"start": 471.03999999999996, "end": 476.14, "text": " And this is TensorFlow's way of saying that this is a floating point number, meaning that"}, {"start": 476.14, "end": 482.15999999999997, "text": " it's a number that can have a decimal point represented using 32 bits of memory in your"}, {"start": 482.15999999999997, "end": 483.15999999999997, "text": " computer."}, {"start": 483.15999999999997, "end": 484.84, "text": " That's what a float 32 is."}, {"start": 484.84, "end": 485.91999999999996, "text": " And what is a tensor?"}, {"start": 485.91999999999996, "end": 492.32, "text": " A tensor here is a data type that the TensorFlow team had created in order to store and carry"}, {"start": 492.32, "end": 495.52, "text": " out computations on matrices efficiently."}, {"start": 495.52, "end": 502.08, "text": " So whenever you see tensor, just think of it as matrix on these few slides."}, {"start": 502.08, "end": 505.71999999999997, "text": " Technically a tensor is a little bit more general than the matrix, but for the purposes"}, {"start": 505.71999999999997, "end": 510.98, "text": " of this course, think of tensor as just a way of representing matrices."}, {"start": 510.98, "end": 516.72, "text": " So remember I said at the start of this video that there's the TensorFlow way of representing"}, {"start": 516.72, "end": 520.12, "text": " a matrix and the NumPy way of representing matrix."}, {"start": 520.12, "end": 525.92, "text": " This is an artifact of the history of how NumPy and TensorFlow were created."}, {"start": 525.92, "end": 534.24, "text": " And unfortunately, there are two ways of representing a matrix that have been baked into these systems."}, {"start": 534.24, "end": 540.6, "text": " And in fact, if you want to take a one, which is a tensor and want to convert it back to"}, {"start": 540.6, "end": 546.6800000000001, "text": " NumPy array, you can do so with this function a one dot NumPy and it will take the same"}, {"start": 546.68, "end": 553.0799999999999, "text": " data and return it in the form of a NumPy array rather than in the form of a TensorFlow"}, {"start": 553.0799999999999, "end": 554.92, "text": " array or TensorFlow matrix."}, {"start": 554.92, "end": 559.9599999999999, "text": " Now let's take a look at what the activations output by the second layer would look like."}, {"start": 559.9599999999999, "end": 562.0799999999999, "text": " Here's the code that we had from before."}, {"start": 562.0799999999999, "end": 565.7199999999999, "text": " Layer two is a dense layer with one unit and a six point activation."}, {"start": 565.7199999999999, "end": 569.66, "text": " And a two is computed by taking layer two and applying it to a one."}, {"start": 569.66, "end": 571.16, "text": " So what is a two?"}, {"start": 571.16, "end": 578.88, "text": " A two may be a number like 0.8 and technically this is a one by one matrix."}, {"start": 578.88, "end": 582.8399999999999, "text": " It's a 2D array with one row and one column."}, {"start": 582.8399999999999, "end": 588.7199999999999, "text": " And so it's equal to this number 0.8."}, {"start": 588.7199999999999, "end": 595.48, "text": " And if you print out a two, you see that it is a TensorFlow tensor with just one element,"}, {"start": 595.48, "end": 597.4, "text": " one number 0.8."}, {"start": 597.4, "end": 600.64, "text": " And it is a one by one matrix."}, {"start": 600.64, "end": 608.76, "text": " Again it is a flow 32 decimal point number taking up 32 bits in computer memory."}, {"start": 608.76, "end": 617.88, "text": " Once again you can convert from a TensorFlow tensor to a NumPy matrix using a two dot NumPy"}, {"start": 617.88, "end": 622.92, "text": " and that will turn this back into a NumPy array that looks like this."}, {"start": 622.92, "end": 629.36, "text": " So that hopefully gives you a sense of how data is represented in TensorFlow and in NumPy."}, {"start": 629.36, "end": 633.72, "text": " I'm used to loading data and manipulating data in NumPy."}, {"start": 633.72, "end": 638.44, "text": " But when you pass a NumPy array into TensorFlow, TensorFlow likes to convert it to its own"}, {"start": 638.44, "end": 644.16, "text": " internal format, the tensor, and then operate efficiently using tensors."}, {"start": 644.16, "end": 648.88, "text": " And when you read the data back out, you can keep it as a tensor or convert it back to"}, {"start": 648.88, "end": 650.8000000000001, "text": " a NumPy array."}, {"start": 650.8000000000001, "end": 656.12, "text": " I think it's a bit unfortunate that the history of how these libraries evolve has led us to"}, {"start": 656.12, "end": 662.72, "text": " do this extra conversion work when actually the two libraries can work quite well together."}, {"start": 662.72, "end": 668.0, "text": " But when you convert back and forth, whether you're using a NumPy array or a tensor, it's"}, {"start": 668.0, "end": 671.28, "text": " just something to be aware of when you're writing code."}, {"start": 671.28, "end": 676.72, "text": " Next, let's take what we've learned and put it together to actually build a neural network."}, {"start": 676.72, "end": 686.76, "text": " Let's go see that in the next video."}]