台大資訊 人工智慧導論 | FAI 3.5: Deep Learning Optimization 訓練深度學習模型需要很多小撇步
陳縕儂 Vivian NTU MiuLab
[{'start': '0.000', 'end': '2.400', 'text': 'Now that we have this loss function,'}, {'start': '2.400', 'end': '5.780', 'text': 'do we have a direction to optimize?'}, {'start': '5.780', 'end': '9.700', 'text': 'Our goal is to keep reducing our loss function,'}, {'start': '9.700', 'end': '11.940', 'text': 'right? We'}, {'start': '11.940', 'end': '14.720', 'text': 'can find a minimum fusion'}, {'start': '14.720', 'end': '20.420', 'text': 'so that the e_in theta can be as low as possible. So'}, {'start': '20.420', 'end': '24.260', 'text': "how is the optimization process done? Let's"}, {'start': '24.260', 'end': '28.260', 'text': 'review the e_in just now.'}, {'start': '28.260', 'end': '30.060', 'text': 'This is our loss function. That'}, {'start': '30.060', 'end': '34.000', 'text': 'is, the'}, {'start': '34.000', 'end': '39.940', 'text': 'difference between the current prediction and the observed y with this set of parameters.'}, {'start': '39.940', 'end': '42.840', 'text': 'Theta is this thing just now. So'}, {'start': '42.840', 'end': '46.980', 'text': 'how do we solve this optimization problem? The'}, {'start': '46.980', 'end': '51.580', 'text': 'reason is that it is a very multi-layer model, right?'}, {'start': '51.580', 'end': '56.420', 'text': 'And it has a lot of nonlinear tranforms in the middle.'}, {'start': '56.420', 'end': '59.960', 'text': 'So in fact, it has...'}, {'start': '59.960', 'end': '63.060', 'text': 'You see, there are so many kinds of'}, {'start': '63.060', 'end': '66.600', 'text': 'variable w and variable b here.'}, {'start': '66.600', 'end': '69.060', 'text': 'So the number of parameters is actually very amazing.'}, {'start': '69.060', 'end': '71.140', 'text': 'Even if you are like the example just now,'}, {'start': '71.140', 'end': '75.180', 'text': 'I only have five dimensions in the front and three dimensions in the middle,'}, {'start': '75.180', 'end': '78.280', 'text': 'which is actually five times three. And then add the ones at the back. In fact, it is'}, {'start': '78.280', 'end': '84.740', 'text': 'often hundreds, thousands, tens of thousands or even more parameters.'}, {'start': '84.740', 'end': '89.020', 'text': 'If we brew force all possible thetas and'}, {'start': '89.020', 'end': '93.020', 'text': 'then find the smallest one, this is actually even more impossible, right?'}, {'start': '93.020', 'end': '95.960', 'text': 'Because this number is unlimited.'}, {'start': '95.960', 'end': '99.720', 'text': 'So a more reasonable approach is to'}, {'start': '99.720', 'end': '103.840', 'text': 'start from a point and then'}, {'start': '103.840', 'end': '106.500', 'text': 'move this point. Going to a point means'}, {'start': '106.500', 'end': '110.540', 'text': 'a set of theta.'}, {'start': '110.540', 'end': '112.780', 'text': 'We change this set of theta'}, {'start': '112.780', 'end': '116.820', 'text': 'and gradually make our arrow smaller and smaller. So the concept is actually to use the'}, {'start': '116.820', 'end': '119.620', 'text': 'concept of calculus.'}, {'start': '119.620', 'end': '121.960', 'text': 'We start with a random set of theta,'}, {'start': '121.960', 'end': '123.460', 'text': 'so we will get an e hard, which is the'}, {'start': '123.460', 'end': '126.040', 'text': 'value of the cos function. Then'}, {'start': '126.040', 'end': '127.700', 'text': 'we have to see in'}, {'start': '127.700', 'end': '130.200', 'text': 'which direction the theta should move. For example, a certain w value'}, {'start': '130.200', 'end': '133.440', 'text': 'may increase, and then our'}, {'start': '133.440', 'end': '137.140', 'text': 'c will decrease. Then we know that if we move in this direction, our target loss function can become smaller. So we have a direction and then slowly adjust it. So in fact, it is to use the concept of calculus to do the partial differentiation of the cos function with respect to my theta. If its partial differential is 0, it means that it has reached an optimal value. It looks like it is'}, {'start': '137.140', 'end': '140.180', 'text': 'moving to the right or left, increasing or'}, {'start': '144.720', 'end': '148.320', 'text': 'decreasing. It will not increase the cos. So we'}, {'start': '148.320', 'end': '151.220', 'text': 'think that'}, {'start': '151.220', 'end': '157.700', 'text': 'this position is actually a position with the smallest cos. In fact, we want to find a place where'}, {'start': '157.700', 'end': '162.200', 'text': 'the partial differential'}, {'start': '178.580', 'end': '181.080', 'text': 'is 0. Of course, in the most ideal case Of course, I have a very magical math, and it can tell me that this can be substituted into a formula, and in this case the partial differential will be 0. But'}, {'start': '181.080', 'end': '189.460', 'text': 'in fact, this model is very complicated, so there is actually no way to use the method just mentioned, which is a close'}, {'start': '193.000', 'end': '195.940', 'text': 'form to solve it'}, {'start': '195.940', 'end': '199.140', 'text': 'directly, because it is actually a math problem that does not have a global optimal solution. The space of'}, {'start': '202.280', 'end': '205.420', 'text': 'this error function is'}, {'start': '205.420', 'end': '207.520', 'text': 'very complicated,'}, {'start': '207.520', 'end': '211.480', 'text': "so a more pragmatic approach is as I just said. Let's illustrate it. Let's simply think of this θ as if there is only"}, {'start': '211.480', 'end': '213.620', 'text': 'one differential, that is, if we only'}, {'start': '213.620', 'end': '219.200', 'text': 'have one w to adjust,'}, {'start': '219.200', 'end': '224.440', 'text': 'then its y side is its cosine. So we actually hope to find a point and then find a θ that can minimize this cθ. How do we'}, {'start': '224.440', 'end': '228.040', 'text': 'do it? As I just said,'}, {'start': '230.980', 'end': '233.180', 'text': 'we start from a random point, for example,'}, {'start': '234.440', 'end': '236.720', 'text': 'we start from θ0, and then'}, {'start': '239.120', 'end': '242.320', 'text': 'we have to see whether θ0 should move to the right or to the left to make'}, {'start': '244.220', 'end': '247.600', 'text': 'cθ smaller. How do we judge whether it should move to the right or to the left? I'}, {'start': '254.800', 'end': '257.040', 'text': 'want to see if this slope is currently this way'}, {'start': '259.800', 'end': '261.240', 'text': 'or this way, right?'}, {'start': '265.300', 'end': '266.840', 'text': 'How do we judge which way it is sloping? Do we just use partial differentials? If the partial differential is negative, For example, if the partial differential is negative here, it means that'}, {'start': '266.840', 'end': '268.580', 'text': 'if I move to the right, it will become smaller.'}, {'start': '268.580', 'end': '271.380', 'text': 'So I can'}, {'start': '277.560', 'end': '279.020', 'text': 'directly use its partial differential to know where'}, {'start': '285.400', 'end': '287.000', 'text': 'I want to move, and then move to, that is, move a little bit to the right, and then at the next time point, my cθ'}, {'start': '287.000', 'end': '290.200', 'text': 'will become lower,'}, {'start': '290.200', 'end': '291.900', 'text': 'and then I can use the partial differential here to'}, {'start': '291.900', 'end': '293.400', 'text': 'judge which direction I should continue to move. By analogy, when it reaches this point, it is actually when the partial differential is equal to 0, right? Because at this position, the'}, {'start': '300.780', 'end': '304.020', 'text': 'partial differential is actually like this, right? So the partial differential is equal to 0. So imagine that you throw a ball on this curve, and it follows'}, {'start': '307.220', 'end': '309.420', 'text': 'this slope, and then rolls and rolls and rolls to the lowest place. The lowest place'}, {'start': '309.420', 'end': '313.020', 'text': "is a minima point. It's just that in deep learning, that is, in the neural network,"}, {'start': '315.000', 'end': '317.700', 'text': 'it is not actually a convex. It may look'}, {'start': '317.700', 'end': '319.300', 'text': 'ugly like this, so it may not be a global minima, but a local minima. It will'}, {'start': '319.300', 'end': '323.260', 'text': 'stop at'}, {'start': '323.260', 'end': '325.540', 'text': 'this position and will not'}, {'start': '326.600', 'end': '328.880', 'text': 'roll anymore. So the actual'}, {'start': '333.520', 'end': '336.140', 'text': 'practice is like this. After we have θ0, we compute the partial differential at this point. For low θ, Then we find that the'}, {'start': '336.140', 'end': '338.780', 'text': 'slope is'}, {'start': '338.780', 'end': '340.420', 'text': 'negative,'}, {'start': '340.420', 'end': '342.720', 'text': 'so we can, the movement of θ1 is θ0, minus this thing,'}, {'start': '342.720', 'end': '344.320', 'text': 'this is the partial differential, which is negative, so if you subtract this, it'}, {'start': '344.320', 'end': '345.420', 'text': 'will increase a little value, that is,'}, {'start': '350.500', 'end': '351.860', 'text': 'θ will move to the right, because'}, {'start': '351.860', 'end': '356.000', 'text': 'we hope it can minimize,'}, {'start': '358.500', 'end': '361.040', 'text': 'right, so θ1 will move here, and then we will do partial differentiation for iterative, and we will find that it is still negative, so the negative value now means the slope is smaller, that is, it is not that steep, so it will'}, {'start': '372.820', 'end': '374.220', 'text': 'move a little bit to the right. So'}, {'start': '374.220', 'end': '377.280', 'text': 'we write it as an equation and it will grow like this. Each new'}, {'start': '377.280', 'end': '379.420', 'text': 'θ'}, {'start': '379.420', 'end': '381.920', 'text': 'will be the θ of the previous iteration minus the'}, {'start': '383.760', 'end': '386.260', 'text': 'value of the partial differential. This partial differential'}, {'start': '394.400', 'end': '398.100', 'text': 'is the partial differential of the current cθ with respect to θ. Then there is another thing here, what is this, this eTA? This'}, {'start': '400.280', 'end': '403.540', 'text': 'eTA is called learning rate, which means how big the step should I take and how aggressive should'}, {'start': '412.860', 'end': '415.280', 'text': 'I move. That is, you can move a small step to the right or a large step to the right. Then'}, {'start': '420.300', 'end': '423.300', 'text': 'you can'}, {'start': '423.300', 'end': '427.700', 'text': 'control the speed of its training by assigning this eTA, but this is actually not so naive. Because of course you would like it to move faster, so you might train it faster, right? But if you are in this place, for example, and you know that people are going to move to the right,'}, {'start': '427.700', 'end': '430.140', 'text': 'but if you'}, {'start': '430.140', 'end': '432.240', 'text': 'move too much, you'}, {'start': '432.240', 'end': '433.940', 'text': 'might end up here, right?'}, {'start': '439.980', 'end': '442.880', 'text': 'You completely miss the minimum point,'}, {'start': '442.880', 'end': '445.920', 'text': 'so the pace setting is actually very important. A better strategy should be that when'}, {'start': '450.560', 'end': '454.560', 'text': 'you are far away from the mark at the beginning, you can be more aggressive, and when the error is large,'}, {'start': '454.560', 'end': '455.860', 'text': 'you can take bigger steps, but'}, {'start': '455.860', 'end': '457.860', 'text': 'when your error is getting smaller, you have to start walking slower, that is, take'}, {'start': '457.860', 'end': '459.660', 'text': 'smaller steps. So'}, {'start': '463.540', 'end': '467.380', 'text': "the eTA can actually adjust its learning rate according to the training process. It doesn't have to be fixed, so"}, {'start': '474.020', 'end': '475.220', 'text': 'there are some methods that can automatically adjust the learning rate, so that the optimization process can be better. I remember that there is a very good optimization method'}, {'start': '475.220', 'end': '478.360', 'text': 'that automatically adjusts the learning rate. It was also presented at a conference this year, or last year, on the task of time paper, which was published ten'}, {'start': '491.500', 'end': '494.700', 'text': 'years ago. Ten years later, it has made a great contribution to the community. Just now, we only'}, {'start': '497.280', 'end': '499.140', 'text': 'looked at the situation of one circle, and the meaning of two circles is actually the same, except that we now have a'}, {'start': '499.140', 'end': '502.620', 'text': '2D dimension. Then your Z axis is its cost,'}, {'start': '502.620', 'end': '505.080', 'text': 'so it is a surface. So you are just dropping the ball and letting it roll to the local'}, {'start': '505.080', 'end': '511.420', 'text': 'minima.'}, {'start': '517.660', 'end': '522.100', 'text': 'Now there are two things you can adjust, which are the two dimensions. You just need to decide whether to adjust theta 1 and theta 2 to be larger or smaller. So after partial differentiation,'}, {'start': '528.900', 'end': '530.900', 'text': 'you can definitely get the values \u200b\u200bof the two dimensions, which are the slopes of'}, {'start': '536.980', 'end': '538.220', 'text': 'the two dimensions. Then'}, {'start': '538.220', 'end': '541.480', 'text': 'decide whether to'}, {'start': '541.480', 'end': '546.320', 'text': 'increase or decrease them. So you can generalize to multiple dimensions. Because we just saw that the theta of deep learning actually has very, very, very many dimensions.'}, {'start': '546.320', 'end': '547.760', 'text': 'So this is a 2-dimensional situation. The main injection method is'}, {'start': '547.760', 'end': '551.000', 'text': 'exactly the same. These two dimensions will each perform partial differentiation on the'}, {'start': '551.000', 'end': '553.560', 'text': 'two dimensions. So we will know in which direction'}, {'start': '553.560', 'end': '558.540', 'text': 'this dimension should move.'}, {'start': '558.540', 'end': '562.600', 'text': 'So it is actually the same as the'}, {'start': '562.600', 'end': '565.200', 'text': 'formula just now. When the two dimensions are clearly written,'}, {'start': '565.200', 'end': '568.980', 'text': 'it looks like this. So of course, you can think of it as a very high-dimensional vector. And'}, {'start': '575.920', 'end': '580.480', 'text': 'this side is also the same. Then the direction of adjustment for each dimension here is the partial differential of the theta of the previous iteration with respect to ctheta with respect to theta. The partial differential of theta corresponding'}, {'start': '586.620', 'end': '590.460', 'text': 'to that dimension is like this,'}, {'start': '592.800', 'end': '594.800', 'text': 'so the algorithm is'}, {'start': '594.800', 'end': '597.740', 'text': 'very simple. We initialize at a random point, and then when the next iteration will change, we will keep repeating this process. Compute gradient. Gradient'}, {'start': '598.700', 'end': '603.880', 'text': 'is its partial differential, and then we update the'}, {'start': '603.880', 'end': '606.120', 'text': 'theta of the next iteration. So like'}, {'start': '615.820', 'end': '619.460', 'text': 'this,'}, {'start': '622.260', 'end': '624.660', 'text': "we subtract the gradient because we want to minimize its value. So if its gradient is in this direction, then we have to go this way. Let's revisit this formulation. Now we have a lot of w and b in our theta,"}, {'start': '624.660', 'end': '628.800', 'text': 'and then we substitute the previous formula. This is our'}, {'start': '636.240', 'end': '637.840', 'text': 'theta,'}, {'start': '637.840', 'end': '645.520', 'text': 'and the update of our theta is that each ctheta will perform partial differentiation on all the parameters inside. So here, it will perform partial differentiation on w1112, a'}, {'start': '668.540', 'end': '671.940', 'text': 'lot of w, and then on all b, so here will be a very, very, very differential value. So why is its calculation very complicated? Because it has a lot of partial differentiations here. All w and b inside must be'}, {'start': '671.940', 'end': '675.580', 'text': 'partially differentiated, which is very troublesome. So this is actually the New Network One reason why it requires a lot of compute is that in the process of development, many people have begun to propose some methods to make the calculation more efficient. Next week, we will introduce'}, {'start': '694.460', 'end': '699.940', 'text': 'a very famous technique called bad propagation. It is equivalent to using the concept of DP and dynamic programming to make'}, {'start': '699.940', 'end': '702.040', 'text': 'the calculation faster.'}, {'start': '706.280', 'end': '710.580', 'text': 'Many things do not need to be calculated repeatedly. But even with DP,'}, {'start': '714.580', 'end': '718.480', 'text': 'it still requires a lot of calculations. When doing gradient descent, gradient descent means that we have to do partial differentiation of this thing. Usually,'}, {'start': '726.960', 'end': '729.400', 'text': 'our trending data here is because our this is the error between the y and the target y generated after we bring in a set of parameters. This is our cost. This is the error between our target y and the y we currently generate.'}, {'start': '737.740', 'end': '741.280', 'text': 'Then our loss function is here. So you see, suppose we now'}, {'start': '741.280', 'end': '743.940', 'text': 'have n data points.'}, {'start': '743.940', 'end': '745.720', 'text': 'In fact, it actually has n costs. Then the information combined together is this c theta. If'}, {'start': '754.280', 'end': '759.360', 'text': 'we do partial differentiation of this thing, it is'}, {'start': '759.360', 'end': '762.260', 'text': 'actually equivalent to the concept of partial differentiation for each point, right? Do you remember that in deep learning, in addition to compute There is another thing that is very big, that is, our data is very big, right? So the number of this trending data here, this large n, is'}, {'start': '795.460', 'end': '799.640', 'text': 'actually very large, right? But if we now, do we have to calculate this thing for one iteration? We only have one iteration, we have to calculate this thing. What is this thing? This thing is all these'}, {'start': '799.640', 'end': '802.900', 'text': 'data, all of which need to be partially differentiated and then averaged. Because we have a lot of data, so when you have looked at all the trending data, then you can calculate the partial'}, {'start': '813.280', 'end': '816.520', 'text': 'differential, and then you can update your model, that is, update the parameters, update a small step,'}, {'start': '816.520', 'end': '817.380', 'text': 'then you can imagine that'}, {'start': '824.860', 'end': '826.960', 'text': 'this process is already very slow. You have calculated so many things,'}, {'start': '830.840', 'end': '832.840', 'text': 'but you'}, {'start': '833.940', 'end': '835.940', 'text': 'can'}, {'start': '835.940', 'end': '837.680', 'text': 'only update a small step, and'}, {'start': '837.680', 'end': '838.680', 'text': 'you may have to update many steps. So when your trending'}, {'start': '839.480', 'end': '842.280', 'text': 'data is'}, {'start': '846.120', 'end': '848.120', 'text': 'very large, you'}, {'start': '848.120', 'end': '850.120', 'text': "use this gradient descent approach, which actually has this problem, that is, it is extremely slow, so slow that it can't be continued. So there are some other methods, such as SGD, which is called Stochastic Gradient Descent. In fact,"}, {'start': '858.300', 'end': '861.860', 'text': 'its idea is also very simple. Imagine whether we need to update the parameters and the model, then the thing updated here is the result of averaging all the data together, so Can we look at the error of a piece of data and update it directly instead of looking at all the data? Originally, we'}, {'start': '864.960', 'end': '866.740', 'text': 'need to look at all the data and then average it. But because there is'}, {'start': '866.740', 'end': '869.900', 'text': 'too much data, after you look at all the data, calculate the error and average it, and then update it a little bit, which is'}, {'start': '873.040', 'end': '875.120', 'text': 'too slow. So'}, {'start': '888.780', 'end': '892.520', 'text': 'can we use this to look at a'}, {'start': '892.520', 'end': '894.700', 'text': 'trending data and get a cost, and then update it directly? Then, because'}, {'start': '897.860', 'end': '900.300', 'text': 'each piece of data has the same probability of being sampled, we randomly sample'}, {'start': '905.100', 'end': '908.240', 'text': 'one and measure the'}, {'start': '908.240', 'end': '909.140', 'text': 'error, and then we update it after getting the direction. Then,'}, {'start': '914.680', 'end': '915.680', 'text': 'because we have a lot of data, but each piece of data actually has the same probability of being sampled, then our expected value, that is, the expected value of the gradient, is actually the same as what we want to do? The advantage of'}, {'start': '921.380', 'end': '923.660', 'text': 'this is that when we'}, {'start': '923.660', 'end': '924.860', 'text': 'sample a data point, the gradient'}, {'start': '924.860', 'end': '927.620', 'text': 'we calculate can'}, {'start': '927.620', 'end': '931.160', 'text': 'immediately update our model. Originally, we'}, {'start': '936.460', 'end': '939.000', 'text': 'had to calculate all the data, average it, and then update it'}, {'start': '939.000', 'end': '940.300', 'text': 'one step at a time. Now, when we sample a piece of data'}, {'start': '942.300', 'end': '944.800', 'text': 'and perform the gradient update, if our'}, {'start': '946.580', 'end': '948.680', 'text': 'overall long term expected value is actually'}, {'start': '948.680', 'end': '950.340', 'text': 'the same as what we want to update, then can we become faster?'}, {'start': '950.340', 'end': '952.120', 'text': 'So the idea is so'}, {'start': '955.080', 'end': '957.780', 'text': 'simple. Originally, it just looked at all of them and then updated. Now, we just look at'}, {'start': '960.620', 'end': '963.280', 'text': 'one and apply the'}, {'start': '966.060', 'end': '967.920', 'text': 'gradient and'}, {'start': '967.920', 'end': '970.060', 'text': 'update. Because the expected value is the same as this is our gradient descent, but it can be faster. So here is a diagram. The left side'}, {'start': '970.060', 'end': '972.400', 'text': 'is the gradient descent. It updates once after it has looked at all the'}, {'start': '976.100', 'end': '978.560', 'text': 'data. So'}, {'start': '980.900', 'end': '982.980', 'text': 'if this is the target point, the darker the blue, the smaller the error. So it'}, {'start': '986.780', 'end': '990.480', 'text': 'has to move from the red side. So it looks at all the data points and then updates once and moves over like this. So the'}, {'start': '990.480', 'end': '993.280', 'text': 'direction of the movement is pretty good, and it also moves a lot. Now suppose we have a'}, {'start': '993.280', 'end': '995.120', 'text': 'total of 20 examples, so it is equivalent to looking at each piece of data once. In fact, I update'}, {'start': '996.920', 'end': '998.980', 'text': '20 times. It'}, {'start': '998.980', 'end': '1001.920', 'text': 'actually'}, {'start': '1004.260', 'end': '1006.960', 'text': 'moves like this. Look at the first'}, {'start': '1006.960', 'end': '1010.040', 'text': 'piece and then there is a'}, {'start': '1010.040', 'end': '1011.400', 'text': 'gradient, and then move here. So'}, {'start': '1014.160', 'end': '1016.740', 'text': 'it moves 20 times. This side only'}, {'start': '1022.480', 'end': '1024.580', 'text': 'moves once. So the one on the right looks at one piece of data'}, {'start': '1027.480', 'end': '1030.120', 'text': 'and moves once. Then you will find that if you compare the one on the right with the one on the left, it feels'}, {'start': '1036.280', 'end': '1038.360', 'text': 'more efficient. It looks at the same'}, {'start': '1038.360', 'end': '1045.200', 'text': '20 pieces of data, but it feels like it has moved closer to the target point than it. We call the fact that'}, {'start': '1048.140', 'end': '1050.140', 'text': 'all the data have been viewed once an'}, {'start': '1050.140', 'end': '1052.340', 'text': 'airpod, which means that all the data have been'}, {'start': '1063.520', 'end': '1065.560', 'text': 'viewed once. So the left side is also an airpod, and the right side is also an airpod, which means that it has been moved 20 times, which means 20 pieces of data. So'}, {'start': '1065.560', 'end': '1066.960', 'text': 'you see, the advantage of SGD is that it will get'}, {'start': '1076.400', 'end': '1079.900', 'text': 'closer to the target point faster than gradient descent because you can imagine that it is more dynamic and updates faster. But the left and right gradient descent and SGD are actually a bit too extreme. One is to'}, {'start': '1079.900', 'end': '1082.900', 'text': 'view all, and the other is to move once after viewing one.'}, {'start': '1087.940', 'end': '1090.340', 'text': 'So is there a middle point where'}, {'start': '1096.180', 'end': '1100.880', 'text': 'you can view several and then move again to be more balanced? So in the end, the most commonly used method is actually called mini-batch SGD, which is not SGD, but it is actually a'}, {'start': '1114.700', 'end': '1116.100', 'text': 'bit similar to the concept of SGD. The idea is also very simple. It is actually to find a balance between the two. Here, a trending sample is updated once, and here,'}, {'start': '1116.100', 'end': '1117.200', 'text': 'all are'}, {'start': '1117.200', 'end': '1119.700', 'text': 'updated once. Of course, you can naturally think of, can I choose'}, {'start': '1119.700', 'end': '1123.240', 'text': 'a small set? The small set is called B-big-B trending sample. We call this small set a batch, which is'}, {'start': '1123.240', 'end': '1128.480', 'text': 'several It may be 20 or 100, and this is considered a batch.'}, {'start': '1128.480', 'end': '1133.620', 'text': 'So usually when using'}, {'start': '1133.620', 'end': '1138.120', 'text': 'mini-batch trending, we will'}, {'start': '1143.400', 'end': '1146.760', 'text': 'set a batch size, that'}, {'start': '1146.760', 'end': '1150.600', 'text': 'is, how many samples we need to update after looking at them. So it is written as try like this, which'}, {'start': '1153.300', 'end': '1155.240', 'text': 'means that'}, {'start': '1155.240', 'end': '1158.680', 'text': 'we have looked at all the'}, {'start': '1158.680', 'end': '1162.940', 'text': 'samples in this batch, and then we will average them, and'}, {'start': '1172.420', 'end': '1176.160', 'text': 'then we will update once. So its update frequency is not as fast as SGD, but it is much faster than gradient descent. According to your batch size setting, this means that'}, {'start': '1178.860', 'end': '1181.000', 'text': 'we will'}, {'start': '1181.000', 'end': '1184.500', 'text': 'update once after looking at all the data once. This is actually updating once after looking at a piece of information. This'}, {'start': '1184.500', 'end': '1186.240', 'text': 'is the middle one. In an iteric, it'}, {'start': '1186.240', 'end': '1189.200', 'text': 'looks at'}, {'start': '1189.200', 'end': '1191.980', 'text': 'b samples'}, {'start': '1191.980', 'end': '1194.480', 'text': 'and then it can be updated once.'}, {'start': '1196.620', 'end': '1200.960', 'text': 'So an apart will only be updated once here. How many times will an apart be updated here? According to how many trending data are in this apart, this side is also in an apart, and then divided by b, it is actually the number of updates. There is nothing special'}, {'start': '1209.660', 'end': '1213.920', 'text': 'about Alberts of mini-batch, there is just a middle sample minibatch of m example, so its batch The size is set to small m, which means to'}, {'start': '1223.640', 'end': '1227.640', 'text': 'sample this small group. Then we will calculate the loss'}, {'start': '1227.640', 'end': '1230.380', 'text': "and then update after averaging. Let's"}, {'start': '1238.920', 'end': '1241.420', 'text': 'compare these three different methods. Red is trending, blue is trending time, which is the training time, and blue is its performance. Batch size is our exodome. If batch size is 1, is it sgd, which is the stochastic gradient exam'}, {'start': '1241.420', 'end': '1243.760', 'text': 'on the left? If the'}, {'start': '1247.720', 'end': '1250.760', 'text': 'batch size is the'}, {'start': '1252.260', 'end': '1256.500', 'text': 'largest, is it the same as the gradient exam, which is all the trending data?'}, {'start': '1257.400', 'end': '1259.700', 'text': 'So the left side is sgd and the right side is gradient exam. In the middle is your different batch size settings, mini-batch settings.'}, {'start': '1269.340', 'end': '1273.180', 'text': 'Then you look at performance wise. If we'}, {'start': '1273.180', 'end': '1274.080', 'text': 'look at this side, the'}, {'start': '1285.820', 'end': '1288.720', 'text': 'batch size is equal to 1, and this side is the batch size equal to the amount of your trending data, which is the'}, {'start': '1294.200', 'end': '1295.740', 'text': 'gradient exam. If we look at acuracy, acuracy wise, several of them are relatively high. So obviously you can find that'}, {'start': '1295.740', 'end': '1297.200', 'text': 'if the update frequency is faster, it will approach the target point faster in this way.'}, {'start': '1303.140', 'end': '1305.980', 'text': 'But this one and this'}, {'start': '1305.980', 'end': '1309.320', 'text': 'one have a big difference, which is its trending time.'}, {'start': '1314.760', 'end': '1317.060', 'text': 'If we can use less trending time and get similar performance So would you actually choose this one? Overall, it feels like the CV'}, {'start': '1317.060', 'end': '1319.120', 'text': 'value is the best, the acuracy is high, but the trending time is actually low.'}, {'start': '1324.460', 'end': '1328.600', 'text': 'So if we compare the trending'}, {'start': '1328.600', 'end': '1331.340', 'text': 'speed, mini-batch is more efficient, and SGD is second. The gradient exam is'}, {'start': '1331.340', 'end': '1334.000', 'text': 'terrible because it only updates once after reading all of them. At this time, you may feel a little puzzled why mini-batch is'}, {'start': '1336.740', 'end': '1342.080', 'text': "faster than SGD? Doesn't SGD update once after reading each one? So why is it that when"}, {'start': '1353.120', 'end': '1357.920', 'text': 'you trend the same number of airpods, that is, you'}, {'start': '1361.160', 'end': '1363.660', 'text': 'look at the same data and the same number of times, why is mini-batch more efficient? Why is mini-batch faster? The main reason is that it is actually related to the'}, {'start': '1363.660', 'end': '1364.700', 'text': 'calculation on the computer.'}, {'start': '1371.800', 'end': '1374.440', 'text': 'If we are using'}, {'start': '1379.120', 'end': '1381.520', 'text': "SGD, let's see if we multiply x by w to get z. Then you"}, {'start': '1383.380', 'end': '1386.380', 'text': 'calculate each x separately, that is, you update it once, and'}, {'start': '1386.380', 'end': '1389.320', 'text': 'then you sample the next trending data and calculate it'}, {'start': '1392.720', 'end': '1394.860', 'text': 'again. So your x is a trending data, right?'}, {'start': '1394.860', 'end': '1396.600', 'text': 'So it is a'}, {'start': '1396.600', 'end': '1398.500', 'text': 'vector, and then this vector is'}, {'start': '1398.500', 'end': '1401.340', 'text': 'multiplied by this metric to get the loss. Of course, there are many'}, {'start': '1405.540', 'end': '1407.180', 'text': 'layers here. Here is a simple illustration. But'}, {'start': '1420.960', 'end': '1422.660', 'text': 'if you think about it, if we set the batch size to 2, then this data and this data are actually combined and multiplied'}, {'start': '1422.660', 'end': '1426.760', 'text': 'together, right? Then it gets this. You see, SGD needs to be calculated twice. In fact, it is a metric here. Just'}, {'start': '1428.960', 'end': '1433.800', 'text': 'multiplying this metric, you can calculate the cost of both. In computers, evidence multiplication is actually very well optimized, especially on GPUs. NVIDIA has'}, {'start': '1435.500', 'end': '1438.400', 'text': 'already increased the computing speed of these evidence multiplications in many tools.'}, {'start': '1439.900', 'end': '1443.280', 'text': 'So the two metric vector multiplications'}, {'start': '1444.740', 'end': '1448.080', 'text': 'are actually not as good as the direct metric matrix'}, {'start': '1448.080', 'end': '1449.880', 'text': 'multiplication. This is because the mini-batch'}, {'start': '1453.580', 'end': '1457.960', 'text': 'calculation actually uses evidence multiplication. So it can actually calculate these errors in a more efficient way. And it is a compromise solution that can balance its performance and its computing efficiency. Metric vector multiplication is relatively inefficient. So the speed here is not related to the data selection situation just now, but to the'}, {'start': '1477.540', 'end': '1480.960', 'text': 'design of your computer. In addition, there is actually another issue. The g of the target we found actually often falls on the local minima. If its current curve and surface are so ugly, then if I start from'}, {'start': '1484.720', 'end': '1487.400', 'text': 'different places, I may fall on'}, {'start': '1489.560', 'end': '1494.240', 'text': 'different g’s. That’s right. So actually, there is no guarantee that the new network can find the global optimal. The global optimal may be here,'}, {'start': '1494.240', 'end': '1498.740', 'text': 'but you may find these three, none of which are the global optimal, but only the local optimal.'}, {'start': '1503.140', 'end': '1506.940', 'text': 'So this is why we need so many training tips. For example, if the initialization starts at different points, it'}, {'start': '1511.080', 'end': '1513.660', 'text': 'may roll to different places. If it starts from here, it may roll down. If it starts from here, it may roll to'}, {'start': '1528.060', 'end': '1531.240', 'text': 'this side. So you may need some'}, {'start': '1531.240', 'end': '1535.900', 'text': 'tips. For example, if I have a model, I may have to sprinkle different initialization points. In this way, for example, sprinkle three different starting points, one of which can'}, {'start': '1549.260', 'end': '1551.620', 'text': 'reach a lower place. Then, overall, that should be a solution that is closer to the global optimal. So remember that when you initialize your parameters, don’t start from the same place every time. Because if you start from the same place every time, you will always fall into the same local minima. The other thing is what I just said, the setting of the learning rate is'}, {'start': '1564.200', 'end': '1566.540', 'text': 'actually very non-naive. Because if we set the learning'}, {'start': '1579.200', 'end': '1582.860', 'text': 'rate too high, it means that we take too much in one step. So we clearly know that we want to go to the right here, but I want to'}, {'start': '1582.860', 'end': '1584.760', 'text': 'go too much and just walk over'}, {'start': '1586.460', 'end': '1588.440', 'text': "here. In fact, it is not good to set it too high, because your cost will increase. So let's set it a little higher, not too high to walk here, but this step is still quite large. If I want to go to the right from here, I"}, {'start': '1591.880', 'end': '1595.240', 'text': 'may'}, {'start': '1595.240', 'end': '1597.580', 'text': 'walk to this position and miss the'}, {'start': '1600.180', 'end': '1602.280', 'text': 'lower point. So this is the green situation. Of course, if I set it very low, I can walk slowly and slowly, and there is a chance that the performance can'}, {'start': '1622.300', 'end': '1626.560', 'text': 'get better later, that is, the error is lower, but it will be'}, {'start': '1628.480', 'end': '1632.880', 'text': 'very, very slow. You see, I set the green one to be relatively'}, {'start': '1632.880', 'end': '1636.560', 'text': "large, isn't it, I will"}, {'start': '1636.560', 'end': '1638.720', 'text': 'drop it'}, {'start': '1638.720', 'end': '1641.320', 'text': 'quickly, but you are still there slowly, and you will feel that the training is very inefficient. Of course, ideally, it can be like this in the front, and then you can still walk'}, {'start': '1641.320', 'end': '1644.320', 'text': 'more finely and then walk to'}, {'start': '1644.320', 'end': '1647.160', 'text': 'this'}, {'start': '1649.240', 'end': '1653.040', 'text': 'lower place. So how to find a good learning rate, how can I automatically adjust the learning rate according to the process'}, {'start': '1664.680', 'end': '1668.280', 'text': 'is actually a very important knowledge, so setting the learning rate is also'}, {'start': '1668.280', 'end': '1671.280', 'text': "very important. Of course, there are some tips for mini-batch training, such as each airpods One airpod is to see all the data once, and all my data will be seen many times. Every time our airpod is seen, we don't want it to see the same order"}, {'start': '1671.280', 'end': '1673.160', 'text': 'every time, because there may be some information in these orders. If it'}, {'start': '1680.300', 'end': '1683.800', 'text': 'sees the same order every time, it'}, {'start': '1687.600', 'end': '1690.440', 'text': "may accidentally learn the information we don't want it to learn. We want it to focus on these data every time it uses the airpods, rather than the relationship between data. So we will"}, {'start': '1694.580', 'end': '1698.140', 'text': "shuffle these training data, and then don't remember the"}, {'start': '1702.820', 'end': '1704.460', 'text': 'order because'}, {'start': '1706.220', 'end': '1709.720', 'text': 'each piece of information is actually independent. What it needs to learn is actually from X to Y, not the relationship between X and X. Of course, in the'}, {'start': '1709.720', 'end': '1713.900', 'text': 'settings on the computer, your batch size should not be changed, so that it can perform calculations efficiently. Because the multiplication of the metric just now, your batch size must be fixed,'}, {'start': '1722.240', 'end': '1724.480', 'text': 'and the size of'}, {'start': '1727.740', 'end': '1729.120', 'text': 'each multiplication is the same, so it can perform calculations more efficiently. I just mentioned that the learning rate will actually be adjusted according to your batch size. There is a'}, {'start': '1731.420', 'end': '1735.020', 'text': 'theory here.'}, {'start': '1735.020', 'end': '1736.460', 'text': 'If your batch size is k-based, then your learning rate'}, {'start': '1736.460', 'end': '1738.220', 'text': 'may be based on the square root of k. Here are the key techniques mentioned earlier. Then, corresponding to the challenge, in deep learning,'}, {'start': '1756.340', 'end': '1758.740', 'text': 'because there are actually many difficulties, there are various strategies, methods, and architectures designed to adjust such problems. So we will see that there are many, many things that our course will not introduce all of them,'}, {'start': '1761.280', 'end': '1763.380', 'text': 'and because many of them are constantly being developed, but you will find'}, {'start': '1763.380', 'end': '1768.960', 'text': 'that in the pure learning process of deep learning, there will be many architecture methods and many'}, {'start': '1771.060', 'end': '1773.660', 'text': 'regulation methods,'}, {'start': '1773.660', 'end': '1777.860', 'text': 'because neural network is a model that is very easy to overfit and very challenging. Then, how to find'}, {'start': '1790.740', 'end': '1794.820', 'text': 'a good algorithm to find the right G efficiently is actually a very important thing. Finally, let me summarize what we talked about today. At the beginning, we talked about how to do model selection, which is actually to build a validation set, and then we can validate our current model. Which model of different complexity is more suitable? Then we'}, {'start': '1803.120', 'end': '1811.840', 'text': 'also let everyone talk about the concept from neural network to deep learning,'}, {'start': '1811.840', 'end': '1813.600', 'text': 'and understand it a little bit from the perspective of biology, and then simulate the human brain, and then'}, {'start': '1817.700', 'end': '1819.480', 'text': 'use mathematical'}, {'start': '1819.480', 'end': '1821.480', 'text': 'methods to'}, {'start': '1822.820', 'end': '1825.020', 'text': 'simulate it. Now there is also a more general neural network setting, which is actually a layer with many layers, and can simulate more'}, {'start': '1831.620', 'end': '1834.620', 'text': 'complex A complex function is used in many current applications. In fact, it is like the previous model automatically helps'}, {'start': '1834.620', 'end': '1836.600', 'text': 'you find which important'}, {'start': '1838.300', 'end': '1840.100', 'text': 'patterns can help make the final application judgment. After extracting these features, a linear combination will be made to make subsequent judgments. The design of the loss function actually also'}, {'start': '1840.100', 'end': '1841.700', 'text': 'has the difference between the GX you'}, {'start': '1855.520', 'end': '1860.220', 'text': 'predicted and the target Y. We will call it the loss function cost function. Finally,'}, {'start': '1860.220', 'end': '1863.220', 'text': 'we also briefly talked about the deep learning optimization method. We started with the concept of gradient descent. Then because it was too slow, it became sgd. Can we find a middle and more moderate method? So we developed the mini-batch sgd and how to make random initialization'}, {'start': '1866.020', 'end': '1868.900', 'text': 'and ensure that the'}, {'start': '1872.560', 'end': '1874.560', 'text': 'points'}, {'start': '1874.560', 'end': '1877.640', 'text': 'we'}, {'start': '1881.980', 'end': '1884.840', 'text': 'learn'}, {'start': '1962.980', 'end': '1966.920', 'text': 'are probably better and can be'}, {'start': '1966.920', 'end': '1968.460', 'text': 'adjusted with the learning rate.'}, {'start': '1968.460', 'end': '1969.460', 'text': 'Thank you everyone.'}]