Soft-Q公式推导

带最大熵的最优策略可表示为：

$$\pi_{ME}^\ast =\underset \pi {argmax} \sum_{t} E_{(s,a)\sim \rho_\pi}[r(s, a)+\alpha H(\pi(\cdot | s))]$$

则优化目标为：

$$J(\pi) =\sum_{t} E_{(s,a)\sim \rho_\pi}[r(s, a)+\alpha H(\pi(\cdot | s))]$$

定义带熵的Soft Q-value为：

$$Q_{soft}^\pi (s, a) = r_0 + E_{\tau \sim \pi}\left [ \sum_{t=1}^\infin \gamma^t(r_t + H(\pi(\cdot | s_t))) \right]$$

其中$H$ 为熵，则：

$$\begin{aligned} H(\pi(\cdot | s_t)) &= - \int_{a_t}\pi(a_t|s_t)log(\pi(a_t|s_t))da_t \\ &= -\sum_{a_t} \pi(a_t|s_t)log(\pi(a_t|s_t)) \\ & = E_{a_t \sim \pi}[- log (\pi(a_t|s_t))] \end{aligned}$$

对于策略$\pi(a|s)$，本文使用能量模型来建模，相较于高斯的单峰分布，他可以学习到多模式分布：

$$\pi(a_t|s_t) \propto exp (-\varepsilon (s_t|a_t))$$

其中，$\varepsilon$是能量函数，可用神经网络拟合。

要想对策略$\pi$建模就必须确定能量函数，论文中使用了$Q_{soft}^\pi $来表示能量，结合玻尔兹曼分布来定义了能量函数 $\varepsilon $。然后证明了这样可以使用策略改进来学到最优策略。

根据定义，新的策略为：

$$\widetilde \pi (a_t|s_t) = \frac {exp (Q_{soft}^\pi (s_t, a_t))} {\int exp(Q_{soft}^\pi (s_t, a))da } \tag{1}$$

这个形式与$softmax$是一致的，在Q-learning中策略是$max(Q)$ ，而基于能量模型的策略似乎就是将策略替换为$softmax(Q)$ 。策略被表示为每个动作的$Q$值相较于其他动作的$Q$值有多大优势，而这种表示是可以存在多峰的。

下面证明该策略也能通过价值迭代的方式收敛：

这个证明依赖一个不等式：

$$H(\pi(\cdot|s)) +E_{a\sim \pi}[Q_{soft}^\pi(s, a)] \le H(\widetilde \pi(\cdot|s))+E_{a\sim \widetilde \pi }[Q_{soft}^\pi(s, a)] \tag 2$$

这个不等式可通过$(3)$直接得到，

$$H(\pi(\cdot|s)) + E_{a\sim \pi}[Q_{soft}^\pi (s,a)] = -D_{KL}(\pi(\cdot|s)|| \widetilde \pi(\cdot | s)) + log \int exp(Q_{soft}^\pi (s, a)) da \tag 3$$

  因为当左式的$\pi$替换为$\widetilde \pi$后，右侧只有KL散度式会变化，而替换后的KL散度值为0，因此替换后左式值增加，即得到$(2)$。对于$(3)$式证明如下：

$$\begin{aligned} D_{KL}(\pi(\cdot|s)||\widetilde \pi (\cdot|s)) &= \int \pi(\cdot|s)log \frac {\pi(\cdot|s)} {\widetilde \pi(\cdot|s)} \\ &= \int \pi(\cdot|s)\left [\; log\pi(\cdot|s) - Q_{soft}^\pi(s,a) + log \int exp(Q_{soft}^\pi(s, a)) da \right ] da \\ &= E_{a\sim \pi}[log(\pi(\cdot | s))] - E_{a\sim \pi}[Q_{soft}^\pi(s, a)] + \int \pi(\cdot|s)log\int exp(Q_{soft}^\pi(s, a))da da \\ &= - H(\pi(\cdot | s)) - E_{a\sim \pi}[Q_{soft}^\pi(s,a )] + log\int exp(Q_{soft}^\pi(s, a))da \\ \end{aligned}$$

注意第4个等号最后一个式子是因为中间已经将a积分掉变成了一个常数，外面积分为1。

先根据$Q_{soft}^\pi$推导其bellman递归形式：

$$\begin{aligned} Q_{soft}^\pi (s, a) &= r_0 + E_{\tau \sim \pi,s_0=s,a_0=a}[\sum_{t=1}^\infin \gamma^t(r_t + H(\pi(\cdot | s_t)))] \\ &= r_0 + E_{s',a'}\left[\gamma \left(r_1 + H(\pi(\cdot | s')) +E_{\tau \sim \pi,s_1=s',a_1=a'}\left [ \sum_{t=2}^\infin \gamma^t(r_t+ H(\pi(\cdot | s_t))) \right]\right)\right] \\ &= r_0 + \gamma E_{s',a'} \; [Q_{soft}^\pi (s', a') + H(\pi(\cdot | s'))] \\ &= r_0 + \gamma E_{s'} \left [ H(\pi(\cdot | s'))+ E_{a'\sim \pi}[Q_{soft}^\pi(s',a')] \right] \\ \end{aligned}$$

这里特意凑出了$(2)$式中的 $H(\pi(\cdot|s)) + E_{a\sim \pi}[Q_{soft}^\pi (s,a)]$ ，然后将该不等式带入得

$$\begin{aligned} Q_{soft}^\pi (s, a) &\le r_0 + \gamma E_{s'} \left [ H(\widetilde \pi(\cdot | s'))+ E_{a'\sim \widetilde \pi}[Q_{soft}^\pi(s',a')] \right] \\ & = r_0 + \gamma E_{s'} \left [ H(\widetilde \pi(\cdot | s'))+ E_{a'\sim \widetilde \pi}[ r_1 + \gamma[E_{s''}[H(\pi(\cdot | s'')) + E_{a''\sim \pi}[Q_{soft}^\pi (s'',a'')]]]]\right] \\ &=r_0 + \gamma E_{s'}\left [ H(\widetilde \pi(\cdot|s'))+ r_1 + \gamma E_{s''}[H(\pi(\cdot|s'') ] + E_{a''\sim \pi}[Q_{soft}^\pi(s'',a'')] \right] \\ &\le ... 不断带入不等式(2) \\ & \le r_0 + E_{\tau \sim \widetilde \pi}\left[\sum_{t=1}^\infin \gamma^tH(\widetilde \pi(\cdot|s_t)) + r_t \right] \\ & = Q_{soft}^{\widetilde \pi} (s,a) \end{aligned}$$

这就证明了根据定义的$\widetilde \pi$ 可以进行策略迭代不断改进。 即

$$\pi_{i+1}(\cdot |s) \propto exp(Q_{soft}^\pi(s, \cdot))$$

n-step-bootstrapping

这里用了蒙特卡洛中的Variance Reduction技术——控制变量法(control variates)。

$$\begin{aligned} (\rho_{t+1}G_{t+1:t+n})^{\ast} =& \;(\rho_{t+1}G_{t+1:t+n}) + c(\rho_{t+1}Q(S_{t+1},A_{t+1}) - E_{b}[\rho_{t+1}Q(S_{t+1}, \cdot )])\\ =& \;(\rho_{t+1}G_{t+1:t+n}) + c(\rho_{t+1}Q(S_{t+1},A_{t+1}) - \sum_a b(a|s)\frac {\pi(a|s)}{b(a|s)}\cdot Q(s,a) \\ =&\;(\rho_{t+1}G_{t+1:t+n}) + c(\rho_{t+1}Q(S_{t+1},A_{t+1}) - \sum_a {\pi(a|s)}\cdot Q(s,a) \\ =&\; (\rho_{t+1}G_{t+1:t+n}) + c(\rho_{t+1}Q(S_{t+1},A_{t+1}) - E_\pi[Q(S_{t+1},\cdot)] \\ =&\;(\rho_{t+1}G_{t+1:t+n}) + c(\rho_{t+1}Q(S_{t+1},A_{t+1}) - \overline V _ {t+n-1}(S_{t+1}) \end{aligned}$$