6.13 Machine Learning development process |Transfer learning using data from a different task -ML Ng
 For an application where you don't have that much data, transfer learning is a wonderful technique that lets you use data from a different task to help on your application. This is one of those techniques that I use very frequently. Let's take a look at how transfer learning works. Here's how transfer learning works. Let's say you want to recognize the handwritten digits from zero through nine, but you don't have that much labeled data of these handwritten digits. Here's what you can do. Say you find a very large data set of one million images, of pictures of cats, dogs, cars, people, and so on, thousand classes. You can then start by training a neural network on this large data set of a million images with a thousand different classes, and train the algorithm to take as input an image X, and learn to recognize any of these 1,000 different classes. In this process, you end up learning parameters for the first layer of the neural network, W1, B1, for the second layer, W2, B2, and so on. W3, B3, W4, B4, and W5, B5 for the upper layer. To apply transfer learning, what you do is then make a copy of this neural network, where you would keep the parameters W1, B1, W2, B2, W3, B3, and W4, B4. But for the last layer, you would eliminate the upper layer and replace it with a much smaller upper layer, with just 10 rather than 1,000 upper units. These 10 upper units will correspond to the classes zero, one, through nine that you want your neural network to recognize. Notice that the parameters W5, B5, they can't be copied over because the dimension of this layer has changed. So you need to come up with new parameters, W5, B5, that you need to train from scratch rather than just copy it from the previous neural network. In transfer learning, what you can do is use the parameters from the first four layers, really all the layers except the final upper layer as a starting point for the parameters and then run an optimization algorithm such as gradient descent, or the Adam optimization algorithm with the parameters initialized using the values from this neural network up on top. So in detail, there are two options for how you can train this neural network's parameters. Option one is you only train the upper layers parameters. So you would take the parameters W1, B1, W2, B2, through W4, P4 as the values from on top and just hold them fixed and don't even bother to change them and use an algorithm like stochastic gradient descent or the Adam optimization algorithm to only update W5, B5 to lower the cost function, the usual cost function that you use for learning to recognize these digits zero to nine from a small training set of these digits zero to nine. So there's option one. Option two would be to train all the parameters in the network, including W1, B1, W2, B2, all the way through W5, B5. But the first four layers parameters would be initialized using the values that you had trained on top. If you have a very, very small training set, then option one might work a little bit better. But if you have a training set that's a little bit larger, then option two might work a little bit better. This algorithm is called transfer learning because the intuition is by learning to recognize cats, dogs, cars, people, and so on, it will hopefully have learned some plausible sets of parameters for the earlier layers for processing image inputs. Then by transferring these parameters to the new neural network, the new neural network starts off with the parameters in a much better place. So that with just a little bit of further learning, hopefully it can end up at a pretty good model. These two steps are first training on a large data set and then tuning the parameters further on a smaller data set. Go by the name of supervised pre-training for the step on top. That's when you train the neural network on a very large data set of say a million images, not quite a related task. And then the second step is called fine tuning, where you take the parameters that you had initialized or gotten from supervised pre-training and then run gradient descent further to fine tune the ways to suit the specific application of handwritten digit recognition that you may have. And so if you have a small data set, even tens or hundreds or thousands or just tens of thousands of images of the handwritten digits, being able to learn from this million images of a not quite related task can actually help your learning algorithms performance a lot. One nice thing about transfer learning as well is maybe you don't need to be the one to carry out supervised pre-training. For a lot of neural networks, there will already be researchers that have already trained a neural network on a large image and will have posted a trained neural networks on the internet, freely licensed for anyone to download and use. And what that means is rather than carrying out the first step yourself, you can just download a neural network that someone else may have spent weeks training and then replace the upper layer with your own upper layer and carry out either option one or option two to fine tune a neural network that someone else has already carried out supervised pre-training on and just do a little bit of fine tuning to quickly be able to get to neural network that performs well on your task. Downloading a pre-trained model that someone else has trained and provided for free is one of those techniques where by building on each other's work in the machine learning community, we can all get much better results by the generosity of other researchers that have pre-trained and posted their neural networks online. But why does transfer learning even work? How could you possibly take parameters obtained by recognizing cats, dogs, cars, and people and use that to help you recognize something as different as handwritten digits? Here's some intuition behind it. If you are training a neural network to detect, say, different objects from images, then the first layer of a neural network may learn to detect edges in the image. We think of these as somewhat low-level features in the image, which is to detect edges. Each of these squares is a visualization of what a single neuron has learned to detect, has learned to group together pixels to find edges in an image. The next layer of the neural network then learns to group together edges to detect corners. And so each of these is a visualization of what one neuron may have learned to detect and has learned to detect little simple shapes like corner-like shapes like this. And the next layer of the neural network may have learned to detect somewhat more complex but still generic shapes like basic curves or small little shapes like these. And that's why by learning on detecting lots of different images, you're teaching the neural network to detect edges, corners, and basic shapes. And that's why by training a neural network to detect things as diverse as cats and dogs and cars and people, you are helping it to learn to detect these pretty generic features of images. And finding edges, corners, curves, basic shapes, this is useful for many other computer vision tasks such as recognizing handwritten digits. One restriction of pre-training though is that the image type X has to be the same for the pre-training and the fine-tuning steps. So if the final task you want to solve is a computer vision task, then the pre-training step also has to be a neural network trained on the same type of input, namely an image of the desired dimensions. Conversely, if your goal is to build a speech recognition system to process audio, then a neural network pre-trained on images probably won't do much good on audio. Instead, you want a neural network pre-trained on audio data that you then fine-tune on your own audio data set. And the same for other types of applications. You can pre-train a neural network on text data, and if your application has the same feature input X of text data, then you can fine-tune that neural network on your own data. To summarize, these are the two steps for transfer learning. Step one is download a neural network with parameters that have been pre-trained on a large data set with the same input type as your application. And that input type could be images, audio, text, or something else. Or if you don't want to download a neural network, maybe you can train your own. But in practice, if you're using images, say, it's much more common to download someone else's pre-trained neural network. Then further train or fine-tune the network on your own data. And I found that if you can get a neural network pre-trained on a large data set, say a million images, then sometimes you can use a much smaller data set, maybe a thousand images, maybe even smaller, to fine-tune the neural network on your own data and get pretty good results. And I've sometimes trained neural networks on as few as 50 images that work quite well using this technique when it has already been pre-trained on a much larger data set. This technique isn't panacea. You can't get every application to work just on 50 images, but it does help a lot when the data set you have for your application isn't that large. And by the way, if you've heard of advanced techniques in the news like GPT-3 or BERT or neural networks pre-trained on ImageNet, those are actually examples of neural networks that someone else has pre-trained on a very large image data set or a text data set that can then be fine-tuned on other applications. If you haven't heard of GPT-3 or BERT or ImageNet, don't worry about it, but you have. Those have been successful applications of pre-training in the machine learning literature. If you haven't heard of GPT-3 or BERT or ImageNet, don't worry about it, but you have. Those have been successful applications of transfer learning in the machine learning literature. One of the things I like about transfer learning is it's been one of the ways that the machine learning community has shared ideas and code and even parameters with each other, because thanks to the researchers that have pre-trained large neural networks and posted the parameters on the internet freely for anyone else to download and use, this empowers anyone to take models they're pre-trained to fine-tune on potentially a much smaller data set. In machine learning, all of us end up often building on the work of each other, and that open sharing of ideas, of code, of trained parameters is one of the ways that the machine learning community, all of us collectively, manage to do much better work than any single person by themselves can. And so I hope that you, joining the machine learning community, will someday maybe find a way to contribute back to this community as well. So that's it for pre-training. I hope you find this technique useful. And in the next video, I'd like to share with you some thoughts on the full cycle of a machine learning project. So when building a machine learning system, what are all the steps that are worth thinking about? Let's take a look at that in the next video.
[{"start": 0.0, "end": 5.68, "text": " For an application where you don't have that much data,"}, {"start": 5.68, "end": 8.92, "text": " transfer learning is a wonderful technique that lets you use"}, {"start": 8.92, "end": 13.02, "text": " data from a different task to help on your application."}, {"start": 13.02, "end": 16.14, "text": " This is one of those techniques that I use very frequently."}, {"start": 16.14, "end": 19.32, "text": " Let's take a look at how transfer learning works."}, {"start": 19.32, "end": 22.54, "text": " Here's how transfer learning works."}, {"start": 22.54, "end": 24.76, "text": " Let's say you want to recognize"}, {"start": 24.76, "end": 28.72, "text": " the handwritten digits from zero through nine,"}, {"start": 28.72, "end": 32.82, "text": " but you don't have that much labeled data of these handwritten digits."}, {"start": 32.82, "end": 34.48, "text": " Here's what you can do."}, {"start": 34.48, "end": 39.68, "text": " Say you find a very large data set of one million images,"}, {"start": 39.68, "end": 42.16, "text": " of pictures of cats, dogs,"}, {"start": 42.16, "end": 45.76, "text": " cars, people, and so on, thousand classes."}, {"start": 45.76, "end": 49.2, "text": " You can then start by training a neural network on"}, {"start": 49.2, "end": 53.56, "text": " this large data set of a million images with a thousand different classes,"}, {"start": 53.56, "end": 57.0, "text": " and train the algorithm to take as input an image X,"}, {"start": 57.0, "end": 61.4, "text": " and learn to recognize any of these 1,000 different classes."}, {"start": 61.4, "end": 66.72, "text": " In this process, you end up learning parameters for the first layer of the neural network,"}, {"start": 66.72, "end": 71.0, "text": " W1, B1, for the second layer, W2, B2, and so on."}, {"start": 71.0, "end": 73.92, "text": " W3, B3, W4, B4, and W5,"}, {"start": 73.92, "end": 76.44, "text": " B5 for the upper layer."}, {"start": 76.44, "end": 79.2, "text": " To apply transfer learning,"}, {"start": 79.2, "end": 83.06, "text": " what you do is then make a copy of this neural network,"}, {"start": 83.06, "end": 87.32000000000001, "text": " where you would keep the parameters W1, B1,"}, {"start": 87.32000000000001, "end": 92.36, "text": " W2, B2, W3, B3, and W4, B4."}, {"start": 92.36, "end": 94.68, "text": " But for the last layer,"}, {"start": 94.68, "end": 101.04, "text": " you would eliminate the upper layer and replace it with a much smaller upper layer,"}, {"start": 101.04, "end": 107.24000000000001, "text": " with just 10 rather than 1,000 upper units."}, {"start": 107.24000000000001, "end": 110.76, "text": " These 10 upper units will correspond to the classes zero,"}, {"start": 110.76, "end": 114.68, "text": " one, through nine that you want your neural network to recognize."}, {"start": 114.68, "end": 116.96000000000001, "text": " Notice that the parameters W5,"}, {"start": 116.96000000000001, "end": 122.24000000000001, "text": " B5, they can't be copied over because the dimension of this layer has changed."}, {"start": 122.24000000000001, "end": 125.68, "text": " So you need to come up with new parameters,"}, {"start": 125.68, "end": 130.36, "text": " W5, B5, that you need to train from scratch rather than just"}, {"start": 130.36, "end": 134.6, "text": " copy it from the previous neural network."}, {"start": 134.6, "end": 138.64000000000001, "text": " In transfer learning, what you can do is use"}, {"start": 138.64, "end": 142.64, "text": " the parameters from the first four layers,"}, {"start": 142.64, "end": 146.6, "text": " really all the layers except the final upper layer as a starting point for"}, {"start": 146.6, "end": 151.04, "text": " the parameters and then run an optimization algorithm such as gradient descent,"}, {"start": 151.04, "end": 153.44, "text": " or the Adam optimization algorithm with"}, {"start": 153.44, "end": 158.23999999999998, "text": " the parameters initialized using the values from this neural network up on top."}, {"start": 158.23999999999998, "end": 164.48, "text": " So in detail, there are two options for how you can train this neural network's parameters."}, {"start": 164.48, "end": 168.04, "text": " Option one is you only train the upper layers parameters."}, {"start": 168.04, "end": 171.2, "text": " So you would take the parameters W1,"}, {"start": 171.2, "end": 173.23999999999998, "text": " B1, W2, B2, through W4,"}, {"start": 173.23999999999998, "end": 176.44, "text": " P4 as the values from on top and just hold them"}, {"start": 176.44, "end": 180.28, "text": " fixed and don't even bother to change them and use an algorithm like"}, {"start": 180.28, "end": 184.48, "text": " stochastic gradient descent or the Adam optimization algorithm to only update"}, {"start": 184.48, "end": 188.76, "text": " W5, B5 to lower the cost function,"}, {"start": 188.76, "end": 192.23999999999998, "text": " the usual cost function that you use for learning to recognize"}, {"start": 192.23999999999998, "end": 196.68, "text": " these digits zero to nine from a small training set of these digits zero to nine."}, {"start": 196.68, "end": 198.48000000000002, "text": " So there's option one."}, {"start": 198.48000000000002, "end": 201.92000000000002, "text": " Option two would be to train all the parameters in the network,"}, {"start": 201.92000000000002, "end": 204.28, "text": " including W1, B1, W2, B2,"}, {"start": 204.28, "end": 206.68, "text": " all the way through W5, B5."}, {"start": 206.68, "end": 209.52, "text": " But the first four layers parameters would be"}, {"start": 209.52, "end": 213.32, "text": " initialized using the values that you had trained on top."}, {"start": 213.32, "end": 216.96, "text": " If you have a very, very small training set,"}, {"start": 216.96, "end": 219.8, "text": " then option one might work a little bit better."}, {"start": 219.8, "end": 223.28, "text": " But if you have a training set that's a little bit larger,"}, {"start": 223.28, "end": 226.20000000000002, "text": " then option two might work a little bit better."}, {"start": 226.2, "end": 232.04, "text": " This algorithm is called transfer learning because the intuition is by"}, {"start": 232.04, "end": 235.6, "text": " learning to recognize cats, dogs, cars, people, and so on,"}, {"start": 235.6, "end": 239.72, "text": " it will hopefully have learned some plausible sets of parameters for"}, {"start": 239.72, "end": 243.35999999999999, "text": " the earlier layers for processing image inputs."}, {"start": 243.35999999999999, "end": 247.35999999999999, "text": " Then by transferring these parameters to the new neural network,"}, {"start": 247.35999999999999, "end": 252.2, "text": " the new neural network starts off with the parameters in a much better place."}, {"start": 252.2, "end": 255.04, "text": " So that with just a little bit of further learning,"}, {"start": 255.04, "end": 258.08, "text": " hopefully it can end up at a pretty good model."}, {"start": 258.08, "end": 262.03999999999996, "text": " These two steps are first training on a large data set and"}, {"start": 262.03999999999996, "end": 266.4, "text": " then tuning the parameters further on a smaller data set."}, {"start": 266.4, "end": 270.84, "text": " Go by the name of supervised pre-training for the step on top."}, {"start": 270.84, "end": 275.15999999999997, "text": " That's when you train the neural network on a very large data set of say a million"}, {"start": 275.15999999999997, "end": 278.44, "text": " images, not quite a related task."}, {"start": 278.44, "end": 282.24, "text": " And then the second step is called fine tuning,"}, {"start": 282.24, "end": 285.2, "text": " where you take the parameters that you had initialized or"}, {"start": 285.2, "end": 289.88, "text": " gotten from supervised pre-training and then run gradient descent further to"}, {"start": 289.88, "end": 294.08, "text": " fine tune the ways to suit the specific application of"}, {"start": 294.08, "end": 296.72, "text": " handwritten digit recognition that you may have."}, {"start": 296.72, "end": 301.6, "text": " And so if you have a small data set, even tens or hundreds or thousands or"}, {"start": 301.6, "end": 304.96000000000004, "text": " just tens of thousands of images of the handwritten digits,"}, {"start": 304.96000000000004, "end": 309.52, "text": " being able to learn from this million images of a not quite related task can"}, {"start": 309.52, "end": 314.03999999999996, "text": " actually help your learning algorithms performance a lot."}, {"start": 314.03999999999996, "end": 319.59999999999997, "text": " One nice thing about transfer learning as well is maybe you don't need to be"}, {"start": 319.59999999999997, "end": 322.88, "text": " the one to carry out supervised pre-training."}, {"start": 322.88, "end": 326.79999999999995, "text": " For a lot of neural networks, there will already be researchers that have already"}, {"start": 326.79999999999995, "end": 332.91999999999996, "text": " trained a neural network on a large image and will have posted a trained"}, {"start": 332.91999999999996, "end": 338.4, "text": " neural networks on the internet, freely licensed for anyone to download and use."}, {"start": 338.4, "end": 342.47999999999996, "text": " And what that means is rather than carrying out the first step yourself,"}, {"start": 342.47999999999996, "end": 346.15999999999997, "text": " you can just download a neural network that someone else may have spent weeks"}, {"start": 346.15999999999997, "end": 351.0, "text": " training and then replace the upper layer with your own upper layer and"}, {"start": 351.0, "end": 356.15999999999997, "text": " carry out either option one or option two to fine tune a neural network that"}, {"start": 356.15999999999997, "end": 360.47999999999996, "text": " someone else has already carried out supervised pre-training on and just do a"}, {"start": 360.47999999999996, "end": 364.12, "text": " little bit of fine tuning to quickly be able to get to neural network that"}, {"start": 364.12, "end": 366.03999999999996, "text": " performs well on your task."}, {"start": 366.04, "end": 370.84000000000003, "text": " Downloading a pre-trained model that someone else has trained and provided for"}, {"start": 370.84000000000003, "end": 375.48, "text": " free is one of those techniques where by building on each other's work in the"}, {"start": 375.48, "end": 380.08000000000004, "text": " machine learning community, we can all get much better results by the generosity"}, {"start": 380.08000000000004, "end": 384.88, "text": " of other researchers that have pre-trained and posted their neural networks online."}, {"start": 384.88, "end": 387.88, "text": " But why does transfer learning even work?"}, {"start": 387.88, "end": 391.84000000000003, "text": " How could you possibly take parameters obtained by recognizing cats, dogs,"}, {"start": 391.84, "end": 396.79999999999995, "text": " cars, and people and use that to help you recognize something as different as"}, {"start": 396.79999999999995, "end": 398.88, "text": " handwritten digits?"}, {"start": 398.88, "end": 401.71999999999997, "text": " Here's some intuition behind it."}, {"start": 401.71999999999997, "end": 409.15999999999997, "text": " If you are training a neural network to detect, say, different objects from images,"}, {"start": 409.15999999999997, "end": 415.32, "text": " then the first layer of a neural network may learn to detect edges in the image."}, {"start": 415.32, "end": 418.84, "text": " We think of these as somewhat low-level features in the image,"}, {"start": 418.84, "end": 420.96, "text": " which is to detect edges."}, {"start": 420.96, "end": 424.84, "text": " Each of these squares is a visualization of what a single neuron has learned to"}, {"start": 424.84, "end": 430.71999999999997, "text": " detect, has learned to group together pixels to find edges in an image."}, {"start": 430.71999999999997, "end": 435.08, "text": " The next layer of the neural network then learns to group together edges to detect"}, {"start": 435.08, "end": 436.84, "text": " corners."}, {"start": 436.84, "end": 441.88, "text": " And so each of these is a visualization of what one neuron may have learned to"}, {"start": 441.88, "end": 446.08, "text": " detect and has learned to detect little simple shapes like corner-like shapes"}, {"start": 446.08, "end": 447.28, "text": " like this."}, {"start": 447.28, "end": 451.15999999999997, "text": " And the next layer of the neural network may have learned to detect somewhat more"}, {"start": 451.15999999999997, "end": 456.28, "text": " complex but still generic shapes like basic curves or small little shapes like"}, {"start": 456.28, "end": 457.55999999999995, "text": " these."}, {"start": 457.55999999999995, "end": 463.32, "text": " And that's why by learning on detecting lots of different images, you're"}, {"start": 463.32, "end": 467.44, "text": " teaching the neural network to detect edges, corners, and basic shapes."}, {"start": 467.44, "end": 471.28, "text": " And that's why by training a neural network to detect things as diverse as"}, {"start": 471.28, "end": 475.88, "text": " cats and dogs and cars and people, you are helping it to learn to detect these"}, {"start": 475.88, "end": 479.76, "text": " pretty generic features of images."}, {"start": 479.76, "end": 485.96, "text": " And finding edges, corners, curves, basic shapes, this is useful for many other"}, {"start": 485.96, "end": 490.88, "text": " computer vision tasks such as recognizing handwritten digits."}, {"start": 490.88, "end": 496.36, "text": " One restriction of pre-training though is that the image type X has to be the"}, {"start": 496.36, "end": 499.71999999999997, "text": " same for the pre-training and the fine-tuning steps."}, {"start": 499.71999999999997, "end": 504.71999999999997, "text": " So if the final task you want to solve is a computer vision task, then the"}, {"start": 504.72, "end": 508.84000000000003, "text": " pre-training step also has to be a neural network trained on the same type of"}, {"start": 508.84000000000003, "end": 513.12, "text": " input, namely an image of the desired dimensions."}, {"start": 513.12, "end": 518.0, "text": " Conversely, if your goal is to build a speech recognition system to process"}, {"start": 518.0, "end": 522.8000000000001, "text": " audio, then a neural network pre-trained on images probably won't do much good on"}, {"start": 522.8000000000001, "end": 523.8000000000001, "text": " audio."}, {"start": 523.8000000000001, "end": 527.52, "text": " Instead, you want a neural network pre-trained on audio data that you then"}, {"start": 527.52, "end": 529.88, "text": " fine-tune on your own audio data set."}, {"start": 529.88, "end": 532.08, "text": " And the same for other types of applications."}, {"start": 532.08, "end": 537.1600000000001, "text": " You can pre-train a neural network on text data, and if your application has the"}, {"start": 537.1600000000001, "end": 542.1600000000001, "text": " same feature input X of text data, then you can fine-tune that neural network on"}, {"start": 542.1600000000001, "end": 543.5600000000001, "text": " your own data."}, {"start": 543.5600000000001, "end": 547.36, "text": " To summarize, these are the two steps for transfer learning."}, {"start": 547.36, "end": 553.24, "text": " Step one is download a neural network with parameters that have been pre-trained"}, {"start": 553.24, "end": 557.48, "text": " on a large data set with the same input type as your application."}, {"start": 557.48, "end": 561.2800000000001, "text": " And that input type could be images, audio, text, or something else."}, {"start": 561.28, "end": 565.4399999999999, "text": " Or if you don't want to download a neural network, maybe you can train your own."}, {"start": 565.4399999999999, "end": 570.28, "text": " But in practice, if you're using images, say, it's much more common to download"}, {"start": 570.28, "end": 572.6, "text": " someone else's pre-trained neural network."}, {"start": 572.6, "end": 579.1999999999999, "text": " Then further train or fine-tune the network on your own data."}, {"start": 579.1999999999999, "end": 583.3199999999999, "text": " And I found that if you can get a neural network pre-trained on a large data set,"}, {"start": 583.3199999999999, "end": 588.8, "text": " say a million images, then sometimes you can use a much smaller data set, maybe a"}, {"start": 588.8, "end": 593.4799999999999, "text": " thousand images, maybe even smaller, to fine-tune the neural network on your own"}, {"start": 593.4799999999999, "end": 596.4, "text": " data and get pretty good results."}, {"start": 596.4, "end": 601.3199999999999, "text": " And I've sometimes trained neural networks on as few as 50 images that work"}, {"start": 601.3199999999999, "end": 605.9599999999999, "text": " quite well using this technique when it has already been pre-trained on a much"}, {"start": 605.9599999999999, "end": 607.4799999999999, "text": " larger data set."}, {"start": 607.4799999999999, "end": 609.04, "text": " This technique isn't panacea."}, {"start": 609.04, "end": 613.92, "text": " You can't get every application to work just on 50 images, but it does help a lot"}, {"start": 613.92, "end": 618.8, "text": " when the data set you have for your application isn't that large."}, {"start": 618.8, "end": 623.8399999999999, "text": " And by the way, if you've heard of advanced techniques in the news like GPT-3"}, {"start": 623.8399999999999, "end": 630.3199999999999, "text": " or BERT or neural networks pre-trained on ImageNet, those are actually examples of"}, {"start": 630.3199999999999, "end": 634.8, "text": " neural networks that someone else has pre-trained on a very large image data"}, {"start": 634.8, "end": 639.36, "text": " set or a text data set that can then be fine-tuned on other applications."}, {"start": 639.36, "end": 642.92, "text": " If you haven't heard of GPT-3 or BERT or ImageNet, don't worry about it, but you"}, {"start": 642.92, "end": 644.04, "text": " have."}, {"start": 644.04, "end": 647.8399999999999, "text": " Those have been successful applications of pre-training in the machine learning"}, {"start": 647.8399999999999, "end": 648.56, "text": " literature."}, {"start": 648.56, "end": 652.12, "text": " If you haven't heard of GPT-3 or BERT or ImageNet, don't worry about it, but you"}, {"start": 652.12, "end": 652.8399999999999, "text": " have."}, {"start": 652.8399999999999, "end": 656.1999999999999, "text": " Those have been successful applications of transfer learning in the machine"}, {"start": 656.1999999999999, "end": 657.56, "text": " learning literature."}, {"start": 657.56, "end": 661.0799999999999, "text": " One of the things I like about transfer learning is it's been one of the ways"}, {"start": 661.0799999999999, "end": 666.0, "text": " that the machine learning community has shared ideas and code and even parameters"}, {"start": 666.0, "end": 670.5999999999999, "text": " with each other, because thanks to the researchers that have pre-trained large"}, {"start": 670.6, "end": 675.08, "text": " neural networks and posted the parameters on the internet freely for anyone else"}, {"start": 675.08, "end": 680.12, "text": " to download and use, this empowers anyone to take models they're pre-trained to"}, {"start": 680.12, "end": 683.6800000000001, "text": " fine-tune on potentially a much smaller data set."}, {"start": 683.6800000000001, "end": 688.5600000000001, "text": " In machine learning, all of us end up often building on the work of each other,"}, {"start": 688.5600000000001, "end": 694.76, "text": " and that open sharing of ideas, of code, of trained parameters is one of the ways"}, {"start": 694.76, "end": 699.36, "text": " that the machine learning community, all of us collectively, manage to do much"}, {"start": 699.36, "end": 702.6, "text": " better work than any single person by themselves can."}, {"start": 702.6, "end": 706.84, "text": " And so I hope that you, joining the machine learning community, will someday"}, {"start": 706.84, "end": 710.2, "text": " maybe find a way to contribute back to this community as well."}, {"start": 710.2, "end": 713.36, "text": " So that's it for pre-training."}, {"start": 713.36, "end": 715.8000000000001, "text": " I hope you find this technique useful."}, {"start": 715.8000000000001, "end": 721.12, "text": " And in the next video, I'd like to share with you some thoughts on the full cycle"}, {"start": 721.12, "end": 723.52, "text": " of a machine learning project."}, {"start": 723.52, "end": 727.2, "text": " So when building a machine learning system, what are all the steps that are"}, {"start": 727.2, "end": 728.44, "text": " worth thinking about?"}, {"start": 728.44, "end": 730.44, "text": " Let's take a look at that in the next video."}]