## 基本概念 #

\begin{equation}L_{\theta}=\mathbb{E}_{z\sim p_{\theta}(z)}[f(z)]\label{eq:base}\end{equation}

## 连续情形 #

\begin{equation}L_{\theta}=\int p_{\theta}(z) f(z)dz\label{eq:lianxu}\end{equation}

### 形式 #

\begin{equation}L_{\theta}=\mathbb{E}_{\varepsilon\sim q(\varepsilon)}[f(g_{\theta}(\varepsilon))]\label{eq:reparam}\end{equation}

### 例子 #

\begin{equation}\mathbb{E}_{z\sim \mathcal{N}\left(z;\mu_{\theta},\sigma_{\theta}^2\right)}\big[f(z)\big] = \mathbb{E}_{\varepsilon\sim \mathcal{N}\left(\varepsilon;0, 1\right)}\big[f(\varepsilon\times \sigma_{\theta} + \mu_{\theta})\big]\end{equation}

## 离散情形 #

\begin{equation}L_{\theta}=\mathbb{E}_{y\sim p_{\theta}(y)}[f(y)]=\sum_y p_{\theta}(y) f(y)\label{eq:lisan}\end{equation}

\begin{equation}p_{\theta}(y)=softmax\big(o_1,o_2,\dots,o_k\big)=\frac{1}{\sum\limits_{i=1}^k e^{o_i}}\left(e^{o_1}, e^{o_2}, \dots, e^{o_k}\right)\label{eq:softmax}\end{equation}

### 形式 #

\begin{equation}\mathop{\arg\max}_i \Big(\log p_i - \log(-\log \varepsilon_i)\Big)_{i=1}^k,\quad \varepsilon_i\sim U[0, 1]\end{equation}

\begin{equation}softmax \Big(\big(\log p_i - \log(-\log \varepsilon_i)\big)\big/\tau\Big)_{i=1}^k,\quad \varepsilon_i\sim U[0, 1]\end{equation}

\begin{equation}softmax \Big(\big(o_i - \log(-\log \varepsilon_i)\big)\big/\tau\Big)_{i=1}^k,\quad \varepsilon_i\sim U[0, 1]\end{equation}

Gumbel Max的证明：

Gumbel Max的形式看上去有点复杂，远没有正态分布的重参数简单，但事实上只要鼓起勇气去看它，它连证明都不困难。我们想要证明Gumbel Max最后输出$i$的概率是$p_i$，不失一般性，这里我们证明输出1的概率是$p_1$。

\begin{equation}\begin{aligned}
&\log p_1 - \log(-\log \varepsilon_1) > \log p_2 - \log(-\log \varepsilon_2) \\
&\log p_1 - \log(-\log \varepsilon_1) > \log p_3 - \log(-\log \varepsilon_3) \\
&\log p_1 - \log(-\log \varepsilon_1) > \log p_k - \log(-\log \varepsilon_k)
\end{aligned}
\end{equation}

\begin{equation}\varepsilon_2 < \varepsilon_1^{p_2 / p_1}\leq 1 \end{equation}

\begin{equation}\varepsilon_1^{p_2 / p_1}\varepsilon_1^{p_3 / p_1}\dots \varepsilon_1^{p_k / p_1}=\varepsilon_1^{(p_2 + p_3 + \dots + p_k) / p_1}=\varepsilon_1^{(1/p_1)-1}\end{equation}

\begin{equation}\int_0^1 \varepsilon_1^{(1/p_1)-1}d\varepsilon_1 = p_1\end{equation}

### 例子 #

https://github.com/bojone/vae/blob/master/vae_keras_cnn_gs.py

### 溯源 #

Gumbel Max由来已久，但首次提出并应用Gumbel Softmax的是论文《Categorical Reparameterization with Gumbel-Softmax》，这篇论文主要探讨了部分隐变量是离散型变量的变分推断问题，比如基于VAE的半监督学习（方法上有点类似《变分自编码器（四）：一步到位的聚类方案》）。其后，在文章《GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution》中，Gumbel Softmax首次被用在离散序列生成，但还不是文本生成，而是比较简单的人造字符序列。

## 背后的故事 #

### SF估计 #

\begin{equation}\begin{aligned}\frac{\partial}{\partial\theta}\int p_{\theta}(z) f(z)dz=&\int f(z) \frac{\partial}{\partial\theta} p_{\theta}(z) dz\\
=&\int p_{\theta}(z)\times\frac{f(z)}{p_{\theta}(z)}\frac{\partial}{\partial\theta} p_{\theta}(z) dz\\
=&\mathbb{E}_{z\sim p_{\theta}(z)}\left[\frac{f(z)}{p_{\theta}(z)}\frac{\partial}{\partial\theta} p_{\theta}(z)\right]\\
=&\mathbb{E}_{z\sim p_{\theta}(z)}\Big[f(z)\frac{\partial}{\partial\theta} \log p_{\theta}(z)\Big]
\end{aligned}\label{eq:sf}\end{equation}

### 梯度方差 #

\begin{equation}\mathbb{E}_{z\sim p_{\theta}(z)}\Big[f(z) \frac{\partial}{\partial\theta} \log p_{\theta}(z)\Big]\approx f(\tilde{z}) \frac{\partial}{\partial\theta} \log p_{\theta}(\tilde{z}),\quad \tilde{z}\sim p_{\theta}(z)\end{equation}

### 降方差 #

\begin{equation}\begin{aligned}\frac{\partial}{\partial\theta}\mathbb{E}_{\varepsilon\sim q(\varepsilon)}[f(g_{\theta}(\varepsilon))]=&\mathbb{E}_{\varepsilon\sim q(\varepsilon)}\left[\frac{\partial}{\partial\theta}f(g_{\theta}(\varepsilon))\right]\\
=&\mathbb{E}_{\varepsilon\sim q(\varepsilon)}\left[\frac{\partial f}{\partial g} \frac{\partial g_{\theta}(\varepsilon)}{\partial\theta}\right]
\end{aligned}\end{equation}

1、SF估计中包含了$\log p_{\theta}(z)$，我们知道，作为一个合理的概率分布，一般都在无穷远处（即$\Vert z\Vert \to \infty$）都会有$p_{\theta}(z)\to 0$，取了$\log$之后反而会趋于负无穷，换句话说，$\log p_{\theta}(z)$这一项实际上放大了无穷远处的波动，从而一定程度上增加了方差；

2、SF估计中包含的是$f$而重参数之后变成了$\frac{\partial f}{\partial g}$，$f$一般是神经网络，而通常我们定义的神经网络模型其实都是$\mathscr{O}(z)$级别的模型，从而我们可以预期它的梯度是$\mathscr{O}(1)$级别的（不严格成立，只能说在平均意义下基本成立），所以相对情况下更平稳一些，因此$f$的方差也比$\frac{\partial f}{\partial g}$的方差要大。

## 文章小结 #

@online{kexuefm-6705,
title={漫谈重参数：从正态分布到Gumbel Softmax},
author={苏剑林},
year={2019},
month={Jun},
url={\url{https://spaces.ac.cn/archives/6705}},
}
{"ft_lang_label":"__label__zh","ft_lang_prob":0.79552203,"math_prob":0.9999778,"size":15699,"snap":"2022-40-2023-06","text_gpt3_token_len":11184,"char_repetition_ratio":0.18770309,"word_repetition_ratio":0.022375215,"special_character_ratio":0.27154595,"punctuation_ratio":0.0676308,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":1.0000076,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-02-02T20:43:45Z\",\"WARC-Record-ID\":\"<urn:uuid:362ac52c-e152-47d6-b1e8-b0c2c6b952b2>\",\"Content-Length\":\"117731\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:3573d33f-404f-45d6-821f-666774130c19>\",\"WARC-Concurrent-To\":\"<urn:uuid:2686b837-2d10-4b87-9bc2-33d954909844>\",\"WARC-IP-Address\":\"120.77.64.127\",\"WARC-Target-URI\":\"https://spaces.ac.cn/archives/6705/comment-page-1\",\"WARC-Payload-Digest\":\"sha1:ZV4TWDT7TKH4G27WCUHZOGRCCMS3DOSQ\",\"WARC-Block-Digest\":\"sha1:F4B3AJEH6DXJPKJNZUGCC5ATU6C4ZV55\",\"WARC-Identified-Payload-Type\":\"application/xhtml+xml\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-06/CC-MAIN-2023-06_segments_1674764500041.18_warc_CC-MAIN-20230202200542-20230202230542-00469.warc.gz\"}"}