Post
Cancel

# 元学习文章阅读（MAML）

MAML 是2017年 Chelsea Finn 大佬提出的一种基于优化（Optimized-based）的小样本学习方法，核心在两个不同的数据集中分别计算梯度和更新参数。

# 1. MAML

2017.《Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks》

The key idea underlying our method is to train the model’s initial parameters such that the model has maximal performance on a new task after the parameters have been updated through one or more gradient steps computed with a small amount of data from that new task. ## 1.1. 算法

1. 利用某一批数据集训练元模型 $M_{meta}$
2. 在另外一批数据集上精调（fine-tune）得到最终的模型 $M_{fine-tune}$ 。

MAML在监督分类中的算法伪代码如下： 1：随机初始化模型参数 $\theta$；

2：循环，对于每个epoch，进行若干batch；

3：随机选取若干个（比如4个） $\mathcal T$ 形成一个batch；

4：对于每个batch中的第 $i$ 个 $\mathcal T$ ，进行第一次梯度更新「inner-loop 内层循环」

5：选取 $\mathcal T_i$ 中的 support set，共 $N\cdot K$个样本（5-way 5-shot=25个样本）

6：计算每个参数的梯度。原文写对每一个类下的 $K$ 个样本做计算。实际上参与计算的总计有 $N\cdot K$ 个样本。这里的loss计算方法，在回归问题中就是MSE；在分类问题中就是cross-entropy；

7：进行第一次梯度更新得到 $\theta’$，可以理解为对 $\mathcal T_i$ 复制一个原模型 $f(\theta)$ 来更新参数；

8：挑出训练集中的 query set 数据用于后续二次梯度更新；

9：完成第一次梯度更新。

10：进行第二次梯度更新，此时计算出的梯度直接通过GD作用于原模型上，用于更新其参数。「outer-loop 外层循环」

• 隐含了二重梯度，需要计算 $\mathcal L_{T_i}f(\theta’)$ 对 $\theta$ 的导数，而 $\mathcal L_{T_i}f(\theta’)$ 是 $\theta’$ 的函数， $\theta’$ 又是 $\theta$ 的函数（见步骤7）；

• 不再分别利用每个 $\mathcal T$ 的loss更新梯度，而是计算一个 batch 中模型 $L_{T_i}f(\theta’)$ 的 loss 总和进行梯度下降；

11：结束在该batch中的训练，回到步骤3，继续采样下一个batch。

## 1.2. 梯度下降数学分析

$\boldsymbol{\theta} = [\theta_1,\theta_2,...,\theta_n]^T$

$\boldsymbol M_{in} = \begin{bmatrix} ^1x_1& ^1x_2& \cdots& ^1x_6\\ ^2x_1& ^2x_2& \cdots& ^2x_6 \\ \vdots& \vdots & \ddots & \vdots\\ ^{10}x_1& ^{10}x_2& \cdots& ^{10}x_6 \end{bmatrix}$

$\boldsymbol M_{out} = \begin{bmatrix} ^1y_1& ^1y_2& \cdots& ^1y_4\\ ^2y_1& ^2y_2& \cdots& ^2y_4 \\ \vdots& \vdots & \ddots & \vdots\\ ^{10}y_1& ^{10}y_2& \cdots& ^{10}y_4 \end{bmatrix}$

$\boldsymbol M_{pred}(\boldsymbol \theta) = \begin{bmatrix} ^1\hat{y}_1(\boldsymbol \theta)& ^1\hat{y}_2(\boldsymbol \theta)& \cdots& ^1\hat{y}_4(\boldsymbol \theta)\\ ^2\hat{y}_1(\boldsymbol \theta)& ^2\hat{y}_2(\boldsymbol \theta)& \cdots& ^2\hat{y}_4(\boldsymbol \theta) \\ \vdots& \vdots & \ddots & \vdots\\ ^{10}\hat{y}_1(\boldsymbol \theta)& ^{10}\hat{y}_2(\boldsymbol \theta)& \cdots& ^{10}\hat{y}_4(\boldsymbol \theta) \end{bmatrix}$

$L_\tau(\boldsymbol \theta) = MSE_\tau = \frac{1}{10\cdot4} \cdot [\frac{1}{2} \sum_i^{10} \sum_j^{4}(^i\hat{y}_j(\boldsymbol \theta) - {}^iy_j)^2]$

\begin{aligned} \boldsymbol g = \nabla_{\boldsymbol \theta} L_\tau(\boldsymbol \theta) = \frac{\partial L_\tau(\boldsymbol \theta)}{\partial \boldsymbol \theta} = \begin{bmatrix} \partial L_\tau(\boldsymbol \theta) / \partial \theta_1\\ \partial L_\tau(\boldsymbol \theta) / \partial \theta_2\\ \vdots\\ \partial L_\tau(\boldsymbol \theta) / \partial \theta_n \end{bmatrix} \end{aligned}

\begin{aligned} \boldsymbol g_1 & = \nabla_{\boldsymbol \theta} L_\tau(\boldsymbol \theta_0)\\ \boldsymbol \theta_1 &= \boldsymbol \theta_0 - \alpha \boldsymbol g_1 \end{aligned}

\begin{aligned} \boldsymbol g_2 &= \nabla_{\boldsymbol \theta_1} L_\tau(\boldsymbol \theta_1)\\ \boldsymbol \theta_2 &= \boldsymbol \theta_1 - \alpha \boldsymbol g_2\\ & = \boldsymbol \theta_0 - \alpha \boldsymbol g_1 - \alpha \boldsymbol g_2 \end{aligned}

\begin{aligned} initialization:\quad&\boldsymbol \theta_0 = \boldsymbol \theta\\ 1^{st}\;gradient\;step:\quad&\boldsymbol \theta_1 = U^1_\tau(\boldsymbol \theta)=\boldsymbol \theta - \alpha \boldsymbol g_1\\ 2^{nd}\;gradient\;step:\quad&\boldsymbol \theta_2 = U^2_\tau(\boldsymbol \theta)=\boldsymbol \theta- \alpha \boldsymbol g_1- \alpha \boldsymbol g_2\\ ...&...\\ k^{th}\;gradient\;step:\quad&\boldsymbol \theta_k = U^k_\tau(\boldsymbol \theta)=\boldsymbol \theta- \alpha \boldsymbol g_1- \alpha \boldsymbol g_2-...- \alpha \boldsymbol g_k\\ &\boldsymbol g_k = \nabla_{\boldsymbol \theta_{k-1}} L_\tau(\boldsymbol \theta_{k-1}) \end{aligned}

## 1.3. 基于优化的元学习目标

MAML 的目标是：找寻一组模型初始参数 $\boldsymbol \theta$，使得模型在面对随机选取的新任务 $\tau$ 时，经过 $k$ 次梯度更新，在 $\tau$ 上的损失函数就能达到很小。

We consider the optimization problem of MAML: find an initial set of parameters, $\boldsymbol \theta$, such that for a randomly sampled task $\tau$ with corresponding loss $L_\tau$, the learner will have low loss after $k$ updates. ——–[Reptile]

\begin{aligned} \mathop{minimize}_{\theta} \; \mathbb E_{\tau}[L_{\tau}(^{k}_\tau\boldsymbol \theta)] = \mathop{minimize}_{\theta} \; \mathbb E_{\tau}[L_{\tau}(U^k_\tau(\boldsymbol \theta))] \end{aligned}

## 1.4. MAML数学分析

\begin{aligned} \mathop{minimize}_{\theta} \; \mathbb E_{\tau}[L_{B}(U_{A}(\boldsymbol \theta))] \end{aligned}

MAML 中只进行 $k=1$ 次梯度算子更新，作者号称有如下四个原因：

• Meta Learning会快很多；

• 如果能让模型只经过一次梯度下降就性能优秀，当然很好；

• Few-shot learning的数据有限，多次梯度下降很容易过拟合；

• 刚才说的可以在实际应用中多次梯度下降。

$\boldsymbol g_{MAML} = \nabla_{\boldsymbol \theta} L_{B}(U_{A}(\boldsymbol \theta))$

$U^{k=1}_\tau(\boldsymbol \theta)={}\tau \boldsymbol \theta_1$

\begin{aligned} \boldsymbol g_{MAML} &= \nabla_{\boldsymbol \theta} L_{B}(U_{A}(\boldsymbol \theta))= \frac{\partial}{\partial \boldsymbol \theta} L_{B}(U_{A}(\boldsymbol \theta))\\ &= L_{B}'({}_{A}\boldsymbol \theta_1) U_{A}'(\boldsymbol \theta)\quad where \quad {}_{A} \boldsymbol \theta_1 = U_{A}(\boldsymbol \theta) \end{aligned}

$U^1_\tau(\boldsymbol \theta)=\boldsymbol \theta - \alpha \boldsymbol g_1$

$U_{A}'(\boldsymbol \theta) = \frac{\partial U_{A}(\boldsymbol \theta)}{\partial \boldsymbol \theta}= \frac{\partial \boldsymbol \theta}{\partial \boldsymbol \theta}-\alpha \frac{\partial \boldsymbol g_1}{\partial \boldsymbol \theta}$

\begin{aligned} \frac{\partial \theta_i}{\partial \theta_j} = \left\{\begin{matrix} 1 \quad i=j\\ 0 \quad i \neq j \end{matrix}\right. \end{aligned}

\begin{aligned} \frac{\partial \boldsymbol \theta}{\partial \boldsymbol \theta} = \begin{bmatrix} 1\;0\;\cdots \; 0\\ 0\;1\;\cdots \; 0\\ \vdots\;\vdots\;\ddots \; \vdots\\ 0\;0\;\cdots \; 1 \end{bmatrix} = \boldsymbol I_{n \times n} \end{aligned}

\begin{aligned} \boldsymbol g_1 = \frac{\partial L_\tau(\boldsymbol \theta)}{\partial \boldsymbol \theta} = \begin{bmatrix} \partial L_\tau(\boldsymbol \theta) / \partial \theta_1\\ \partial L_\tau(\boldsymbol \theta) / \partial \theta_2\\ \vdots\\ \partial L_\tau(\boldsymbol \theta) / \partial \theta_n \end{bmatrix} \end{aligned}

\begin{aligned} \alpha \frac{\partial \boldsymbol g_1}{\partial \boldsymbol \theta} &= \alpha \frac{\partial L_\tau(\boldsymbol \theta)}{\partial \boldsymbol \theta} / \partial \boldsymbol \theta\\ & = \alpha \begin{bmatrix} \partial L_\tau(\boldsymbol \theta) / \partial \theta_1\\ \partial L_\tau(\boldsymbol \theta) / \partial \theta_2\\ \vdots\\ \partial L_\tau(\boldsymbol \theta) / \partial \theta_n \end{bmatrix} / \partial \boldsymbol \theta\\ & = \alpha \begin{bmatrix} \partial (\frac{\partial L_{A}}{\partial \theta_1}) / \partial \theta_1 & \partial (\frac{\partial L_{A}}{\partial \theta_1}) / \partial \theta_2& \cdots & \partial (\frac{\partial L_{A}}{\partial \theta_1}) / \partial \theta_n \\ \partial (\frac{\partial L_{A}}{\partial \theta_2}) / \partial \theta_1 & \partial (\frac{\partial L_{A}}{\partial \theta_2}) / \partial \theta_2& \cdots & \partial (\frac{\partial L_{A}}{\partial \theta_2}) / \partial \theta_n \\ \vdots & \vdots & \ddots & \vdots\\ \partial (\frac{\partial L_{A}}{\partial \theta_n}) / \partial \theta_1 & \partial (\frac{\partial L_{A}}{\partial \theta_n}) / \partial \theta_2& \cdots & \partial (\frac{\partial L_{A}}{\partial \theta_n}) / \partial \theta_n \\ \end{bmatrix}_{n \times n}\\ &= \alpha \begin{bmatrix} \partial^2 L_{A} / \partial \theta_1^2 & \partial^2 L_{A} /\partial \theta_1 \partial \theta_2 & \cdots & \partial^2 L_{A} /\partial \theta_1 \partial \theta_n \\ \partial^2 L_{A} /\partial \theta_2 \partial \theta_1 & \partial^2 L_{A} / \partial \theta_2^2 & \cdots & \partial^2 L_{A} /\partial \theta_2 \partial \theta_n \\ \vdots & \vdots & \ddots & \vdots\\ \partial^2 L_{A} /\partial \theta_n \partial \theta_1 & \partial^2 L_{A} /\partial \theta_n \partial \theta_2& \cdots & \partial^2 L_{A} / \partial \theta_n^2 \\ \end{bmatrix}_{n \times n}\\ &= \alpha \boldsymbol H_{A}(\boldsymbol \theta)_{n \times n} \end{aligned}

\begin{aligned} U_{A}'(\boldsymbol \theta) &= \frac{\partial U_{A}(\boldsymbol \theta)}{\partial \boldsymbol \theta}= \frac{\partial \boldsymbol \theta}{\partial \boldsymbol \theta}-\alpha \frac{\partial \boldsymbol g_1}{\partial \boldsymbol \theta}\\ &= \boldsymbol I_{n \times n} - \alpha \boldsymbol H_{A}(\boldsymbol \theta)_{n \times n} \end{aligned}

MAML 的作者表示，大部分计算量都在于计算 $H_{A}(\boldsymbol \theta)_{n \times n}$ 这个包含二重梯度的 Hessian 矩阵，导致 MAML 的计算量很大，使得 MAML 以难以训练成名。

\begin{aligned} initialization:\quad&\boldsymbol \theta_0 = \boldsymbol \theta\\ 1^{st}\;gradient\;step:\quad&\boldsymbol \theta_1 = U^1_A(\boldsymbol \theta)=\boldsymbol \theta_0 - \alpha \boldsymbol g_1\\ 2^{nd}\;gradient\;step:\quad&\boldsymbol \theta_2 = U^2_A(\boldsymbol \theta)=\boldsymbol \theta_1 - \alpha \boldsymbol g_2\\ ...&...\\ k^{th}\;gradient\;step:\quad&\boldsymbol \theta_k = U^k_A(\boldsymbol \theta)=\boldsymbol \theta_{k-1}- \alpha \boldsymbol g_k\\ &\boldsymbol g_k = \nabla_{\boldsymbol \theta_{k-1}} L_\tau(\boldsymbol \theta_{k-1}) \end{aligned}

$\boldsymbol \theta \leftarrow \boldsymbol \theta - \beta \cdot \boldsymbol g_{MAML}$

\begin{aligned} \boldsymbol g_{MAML} &= \nabla_{\boldsymbol \theta} L_{B}(U^k_{A}(\boldsymbol \theta))\\ &= \nabla_{\boldsymbol \theta} L_{B}(\boldsymbol \theta_k)\\ &= \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k)\cdot(\nabla_{\boldsymbol \theta}\boldsymbol \theta_k)\\ &= \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k)\cdot(\nabla_{\boldsymbol \theta_{k-1}}\boldsymbol \theta_k) \cdots (\nabla_{\boldsymbol \theta_{1}}\boldsymbol \theta_2)\cdot (\nabla_{\boldsymbol \theta_{0}}\boldsymbol \theta_1)\\ &= \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k)\cdot\prod_{i=1}^k (\nabla_{\boldsymbol \theta_{i-1}}\boldsymbol \theta_i)\\ &= \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k)\cdot\prod_{i=1}^k (\nabla_{\boldsymbol \theta_{i-1}}(\boldsymbol \theta_{i-1}- \alpha \boldsymbol g_i))\\ &= \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k)\cdot\prod_{i=1}^k (\nabla_{\boldsymbol \theta_{i-1}}(\boldsymbol \theta_{i-1}- \alpha (\nabla_{\boldsymbol \theta_{i-1}} L_\tau(\boldsymbol \theta_{i-1})))\\ &= \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k)\cdot\prod_{i=1}^k (\boldsymbol I - \alpha \nabla_{\boldsymbol \theta_{i-1}} (\nabla_{\boldsymbol \theta_{i-1}} L_\tau(\boldsymbol \theta_{i-1})))\\ \end{aligned}

\begin{aligned} \prod_{i=1}^{k=1} (\boldsymbol I - \alpha \nabla_{\boldsymbol \theta_{i-1}} (\nabla_{\boldsymbol \theta_{i-1}} L_\tau(\boldsymbol \theta_{i-1}))) &= \boldsymbol I - \alpha \nabla_{\boldsymbol \theta_0} \nabla_{\boldsymbol \theta_0} L_\tau\boldsymbol (\theta_0)\\ & = \boldsymbol I - \alpha \boldsymbol H_{A}(\boldsymbol \theta)\\ \end{aligned}

$\boldsymbol g_{MAML} = \nabla_{\boldsymbol \theta_k} L_{B}(\boldsymbol \theta_k) \cdot (\boldsymbol I - \alpha \boldsymbol H_{A}(\boldsymbol \theta))$

## 1.5. FOMAML

FOMAML 假设学习率 $\alpha \rightarrow 0^+$，则前面与 $\alpha$ 相乘的项（$k=1$ 时是 Hessian 矩阵，$k>1$ 时是连乘项）因为乘以 $0^+$ 被消去。那么整个 $U_{A}’(\boldsymbol \theta)$ 就等于单位阵了，

$\boldsymbol g_{FOMAML} = L'_{B}({}_A\boldsymbol \theta_k)=\nabla_{\boldsymbol \theta_k} L_{B}({}_A\boldsymbol \theta_k)$

• 采样任务 $\tau$；
• 计算在数据集 A 上的梯度因子 $U^{k=1}_{A}(\boldsymbol \theta)= _{A} \boldsymbol \theta_1 = \phi$;
• 计算在数据集 B 上的损失函数 $L_{B}$ 对 $\phi$ 的（偏）导数：$g_{FOMAML}=\nabla_{\phi} L_{B}(\phi)$；
• 将 $g_{FOMAML}$ 插入外循环更新参数。

\begin{aligned} \theta_i' \leftarrow \theta - \alpha \nabla_\theta L_{Ai}(\theta)\\ \theta \leftarrow \theta - \beta \nabla_{\theta'} \sum L_B(\theta')\\ \end{aligned}

This approximation removes the need for computing Hessian-vector products in an additional backward pass, which we found led to roughly 33% speed-up in network computation. ## 1.6. 缺点

MAML的缺点：

1. Hard to train：paper中给出的backbone是4层的conv加1层linear，试想，如果我们换成16层的VGG，每个task在算fast parameter的时候需要计算的Hessian矩阵将会变得非常大。那么你每一次迭代就需要很久，想要最后的model收敛就要更久。

2. Robustness一般：不是说MAML的robustness不好，因为也是由一阶online的优化方法SGD求解出来的，会相对找到一个flatten minima location。然而这和非gradient-based meta learning方法求解出来的model的robust肯定是没法比的。

# 2. 各类实现

dragen-1860 的 Pytorch 实现：https://github.com/dragen1860/MAML-Pytorch

Tensorflow实现：https://github.com/dragen1860/MAML-TensorFlow

# 3. 参考文献

 Tianhao Wei翻译自Lilian.元学习：学习如何学习【译】
{"ft_lang_label":"__label__zh","ft_lang_prob":0.51808023,"math_prob":0.99988174,"size":16340,"snap":"2022-40-2023-06","text_gpt3_token_len":8262,"char_repetition_ratio":0.34396425,"word_repetition_ratio":0.18983768,"special_character_ratio":0.32141984,"punctuation_ratio":0.042664345,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9999945,"pos_list":[0,1,2,3,4,5,6,7,8],"im_url_duplicate_count":[null,null,null,null,null,null,null,null,null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2022-09-28T19:50:08Z\",\"WARC-Record-ID\":\"<urn:uuid:913dc1fb-0910-4a5c-8d46-48333af894dc>\",\"Content-Length\":\"49626\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:77d54f46-44cf-4701-969d-c3b56b9a1db3>\",\"WARC-Concurrent-To\":\"<urn:uuid:c2b76079-6b72-40b0-8a72-f308c2e27e61>\",\"WARC-IP-Address\":\"185.199.111.153\",\"WARC-Target-URI\":\"https://sirlis.cn/meta-learning-MAML/\",\"WARC-Payload-Digest\":\"sha1:UD6MXFZGJ4JP6HQNUNMVP4PG32TUDORQ\",\"WARC-Block-Digest\":\"sha1:RMJQKTA6UNOY2A6E4QVYPB43JQDNRV4A\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2022/CC-MAIN-2022-40/CC-MAIN-2022-40_segments_1664030335276.85_warc_CC-MAIN-20220928180732-20220928210732-00501.warc.gz\"}"}