【机器学习基础】系列博客为参考周志华老师的《机器学习》一书，自己所做的读书笔记。
本文为原创文章，未经本人允许，禁止转载。转载请注明出处。
1.表示定理
无论SVM还是SVR，学得的模型总能表示成核函数$\kappa (\mathbf x,\mathbf x_i)$的线性组合。不仅如此，事实上我们有下面这个称为“表示定理”（representer theorem）的更一般的结论：
表示定理：令$\mathbb{H}$为核函数$\kappa$对应的再生核希尔伯特空间，$|| h||_{\mathbb{H}}$表示$\mathbb{H}$空间中关于$h$的范数，对于任意单调递增函数$\Omega:[0,\infty] \mapsto \mathbb{R}$和任意非负损失函数$\ell : \mathbb{R}^m \mapsto [0,\infty]$，优化问题
\[\min \limits_{h \in \mathbb{H}} F(h)=\Omega(|| h||_{\mathbb{H}})+\ell(h(\mathbf x_1),h(\mathbf x_2),...,h(\mathbf x_m)) \tag{1}\]的解总可写为
\[h^*(\mathbf x)=\sum^m_{i=1}\alpha_i \kappa (\mathbf x,\mathbf x_i) \tag{2}\]表示定理对损失函数没有限制，对正则化项$\Omega$仅要求单调递增，甚至不要求$\Omega$是凸函数，意味着对于一般的损失函数和正则化项，优化问题(1)的最优解$h^*(\mathbf x)$都可表示为核函数$\kappa (\mathbf x,\mathbf x_i)$的线性组合；这显示出核函数的巨大威力。
2.核方法
人们发展出一系列基于核函数的学习方法，统称为“核方法”（kernel methods）。最常见的，是通过“核化”（即引入核函数）来将线性学习器拓展为非线性学习器。下面以线性判别分析为例来演示如何通过核化来对其进行非线性拓展，从而得到“核线性判别分析”（Kernelized Linear Discriminant Analysis，简称KLDA）。
我们先假设可通过某种映射$\phi : \chi \mapsto \mathbb{F}$将样本映射到一个特征空间$\mathbb{F}$，然后在$\mathbb{F}$中执行线性判别分析，以求得：
\[h(\mathbf x)=\mathbf w^T \phi (\mathbf x) \tag{3}\]KLDA的学习目标是：
\[\max \limits_{\mathbf w} J(\mathbf w)=\frac{\mathbf w^T \mathbf S_b^{\phi} \mathbf w}{\mathbf w^T \mathbf S_w^{\phi} \mathbf w} \tag{4}\]其中$\mathbf S_b^{\phi}$和$\mathbf S_w^{\phi}$分别为训练样本在特征空间$\mathbb{F}$中的类间散度矩阵和类内散度矩阵。令$\mathbf{X}_i$表示第$i \in \{0,1\}$类样本的集合，其样本数为$m_i$；总样本数$m=m_0+m_1$。第$i$类样本在特征空间$\mathbb{F}$中的均值为：
\[\mathbf{\mu}_i^{\phi}=\frac{1}{m_i} \sum_{\mathbf x \in \mathbf X_i} \phi(\mathbf x) \tag{5}\]两个散度矩阵分别为：
\[\mathbf{S}_b^{\phi}=(\mathbf{\mu}_1^{\phi} - \mathbf{\mu}_0^{\phi})(\mathbf{\mu}_1^{\phi} - \mathbf{\mu}_0^{\phi})^T \tag{6}\] \[\mathbf{S}_w^{\phi}=\sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i}(\phi (\mathbf{x})-\mathbf{\mu}_i^{\phi})(\phi (\mathbf{x})-\mathbf{\mu}_i^{\phi})^T \tag{7}\]通常我们难以知道映射$\phi$的具体形式，因此使用核函数$\kappa(\mathbf x,\mathbf x_i)=\phi(\mathbf x_i)^T \phi(\mathbf x)$来隐式地表达这个映射和特征空间$\mathbb{F}$。把$J(\mathbf w)$作为式(1)中的损失函数$\ell$，再令$\Omega \equiv 0$，由表示定理，函数$h(\mathbf x)$可写为：
\[h(\mathbf x)=\sum^m_{i=1}\alpha_i \kappa (\mathbf x,\mathbf x_i) \tag{8}\]$\equiv$为恒等于，即无论条件如何变化，等式始终保持不变。
因为有式(3)等于式(8)：
\[\mathbf w^T \phi (\mathbf x) = \sum^m_{i=1}\alpha_i \kappa (\mathbf x,\mathbf x_i) \tag{9}\]将$\kappa(\mathbf x,\mathbf x_i)=\phi(\mathbf x)^T \phi(\mathbf x_i)$代入式(9)：
\[\mathbf w^T \phi (\mathbf x) = \sum^m_{i=1}\alpha_i \phi(\mathbf x)^T \phi(\mathbf x_i) \tag{10}\] \[\mathbf w^T \phi (\mathbf x) = \phi(\mathbf x)^T \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \tag{11}\]由于$\mathbf w^T \phi (\mathbf x)$的计算结果为标量，而标量的转置等于其本身，所以：
\[\mathbf w^T \phi (\mathbf x) = (\mathbf w^T \phi (\mathbf x))^T = \phi(\mathbf x)^T \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \tag{12}\] \[\mathbf w^T \phi (\mathbf x) = \phi(\mathbf x)^T \mathbf w = \phi(\mathbf x)^T \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \tag{13}\] \[\mathbf w = \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \tag{14}\]令$\mathbf K \in \mathbb{R}^{m\times m}$为核函数$\kappa$所对应的核矩阵，$(\mathbf K)_{ij}=\kappa(\mathbf x_i,\mathbf x_j)$。令$\mathbf l_i \in \{ 1,0 \}^{m\times 1}$为第$i$类样本的指示向量，即$\mathbf l_i$的第$j$个分量为1当且仅当$\mathbf x_j \in \mathbf X_i$，否则$\mathbf l_i$的第$j$个分量为0。再令：
\[\hat{\mathbf{\mu}}_0=\frac{1}{m_0} \mathbf{Kl}_0 \tag{15}\] \[\hat{\mathbf{\mu}}_1=\frac{1}{m_1} \mathbf{Kl}_1 \tag{16}\] \[\mathbf{M}=(\hat{\mathbf \mu}_0-\hat{\mathbf{\mu}}_1)(\hat{\mathbf \mu}_0-\hat{\mathbf{\mu}}_1)^T \tag{17}\] \[\mathbf{N}=\mathbf{KK}^T-\sum_{i=0}^1 m_i \hat{\mathbf{\mu}}_i \hat{\mathbf{\mu}}_i^T \tag{18}\]于是，式(4)等价为：
\[\max \limits_{\mathbf{\alpha}} J(\mathbf{\alpha}) \frac{\mathbf{\alpha}^T \mathbf{M} \mathbf{\alpha}}{\mathbf{\alpha}^T \mathbf{N} \mathbf{\alpha}} \tag{19}\]显然，使用线性判别分析求解方法即可得到$\mathbf{\alpha}$，进而可由式(8)得到投影函数$h(\mathbf x)$。
2.1.式(15)、式(16)的推导
为了详细地说明此公式的计算原理，下面首先先举例说明，然后再在例子的基础上延展出其一般形式。假设此时仅有4个样本，其中第1和第3个样本的标记为0，第2和第4个样本的标记为1，那么此时：
\[m=4 \tag{20}\] \[m_0=2,m_1=2 \tag{21}\] \[X_0=\{\mathbf x_1,\mathbf x_3 \},X_1=\{\mathbf x_2,\mathbf x_4 \} \tag{22}\] \[\mathbf{K}=\begin{bmatrix} \kappa(\mathbf{x}_1,\mathbf{x}_1) & \kappa(\mathbf{x}_1,\mathbf{x}_2) & \kappa(\mathbf{x}_1,\mathbf{x}_3) & \kappa(\mathbf{x}_1,\mathbf{x}_4) \\ \kappa(\mathbf{x}_2,\mathbf{x}_1) & \kappa(\mathbf{x}_2,\mathbf{x}_2) & \kappa(\mathbf{x}_2,\mathbf{x}_3) & \kappa(\mathbf{x}_2,\mathbf{x}_4) \\ \kappa(\mathbf{x}_3,\mathbf{x}_1) & \kappa(\mathbf{x}_3,\mathbf{x}_2) & \kappa(\mathbf{x}_3,\mathbf{x}_3) & \kappa(\mathbf{x}_3,\mathbf{x}_4) \\ \kappa(\mathbf{x}_4,\mathbf{x}_1) & \kappa(\mathbf{x}_4,\mathbf{x}_2) & \kappa(\mathbf{x}_4,\mathbf{x}_3) & \kappa(\mathbf{x}_4,\mathbf{x}_4) \\ \end{bmatrix} \in \mathbb{R}^{4\times 4} \tag{23}\] \[\mathbf{l}_0 = \begin{bmatrix} 1 \\ 0 \\ 1 \\ 0 \\ \end{bmatrix} \in \mathbb{R}^{4\times 1} \tag{24}\] \[\mathbf{l}_1 = \begin{bmatrix} 0 \\ 1 \\ 0 \\ 1 \\ \end{bmatrix} \in \mathbb{R}^{4\times 1} \tag{25}\]所以：
\[\hat{\mathbf{\mu}}_0=\frac{1}{m_0} \mathbf{Kl}_0=\frac{1}{2} \begin{bmatrix} \kappa(\mathbf{x}_1,\mathbf{x}_1)+\kappa(\mathbf{x}_1,\mathbf{x}_3) \\ \kappa(\mathbf{x}_2,\mathbf{x}_1)+\kappa(\mathbf{x}_2,\mathbf{x}_3) \\ \kappa(\mathbf{x}_3,\mathbf{x}_1)+\kappa(\mathbf{x}_3,\mathbf{x}_3) \\ \kappa(\mathbf{x}_4,\mathbf{x}_1)+\kappa(\mathbf{x}_4,\mathbf{x}_3) \\\end{bmatrix} \in \mathbb{R}^{4\times 1} \tag{26}\] \[\hat{\mathbf{\mu}}_1=\frac{1}{m_1} \mathbf{Kl}_1=\frac{1}{2} \begin{bmatrix} \kappa(\mathbf{x}_1,\mathbf{x}_2)+\kappa(\mathbf{x}_1,\mathbf{x}_4) \\ \kappa(\mathbf{x}_2,\mathbf{x}_2)+\kappa(\mathbf{x}_2,\mathbf{x}_4) \\ \kappa(\mathbf{x}_3,\mathbf{x}_2)+\kappa(\mathbf{x}_3,\mathbf{x}_4) \\ \kappa(\mathbf{x}_4,\mathbf{x}_2)+\kappa(\mathbf{x}_4,\mathbf{x}_4) \\\end{bmatrix} \in \mathbb{R}^{4\times 1} \tag{27}\]根据此结果易得$\hat{\mathbf{\mu}}_0,\hat{\mathbf{\mu}}_1$的一般形式为：
\[\hat{\mathbf{\mu}}_0=\frac{1}{m_0} \mathbf{Kl}_0=\frac{1}{m_0} \begin{bmatrix} \sum_{\mathbf{x}\in X_0} \kappa(\mathbf{x}_1,\mathbf{x}) \\ \sum_{\mathbf{x}\in X_0} \kappa(\mathbf{x}_2,\mathbf{x}) \\ \vdots \\ \sum_{\mathbf{x}\in X_0} \kappa(\mathbf{x}_m,\mathbf{x}) \end{bmatrix} \in \mathbb{R}^{m\times 1} \tag{28}\] \[\hat{\mathbf{\mu}}_1=\frac{1}{m_1} \mathbf{Kl}_1=\frac{1}{m_1} \begin{bmatrix} \sum_{\mathbf{x}\in X_1} \kappa(\mathbf{x}_1,\mathbf{x}) \\ \sum_{\mathbf{x}\in X_1} \kappa(\mathbf{x}_2,\mathbf{x}) \\ \vdots \\ \sum_{\mathbf{x}\in X_1} \kappa(\mathbf{x}_m,\mathbf{x}) \end{bmatrix} \in \mathbb{R}^{m\times 1} \tag{29}\]2.2.式(19)的推导
首先将式(14)代入式(4)的分子可得：
\[\begin{align} \mathbf w^T \mathbf S_b^{\phi} \mathbf w &= \left( \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \right) ^T \cdot \mathbf S_b^{\phi} \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \\&= \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) ^T \cdot \mathbf S_b^{\phi} \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i)\end{align} \tag{30}\]其中：
\[\begin{align} \mathbf{S}_b^{\phi}&=(\mathbf{\mu}_1^{\phi} - \mathbf{\mu}_0^{\phi})(\mathbf{\mu}_1^{\phi} - \mathbf{\mu}_0^{\phi})^T \\&= \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \phi (\mathbf{x}) -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \phi (\mathbf{x}) \right) \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \phi (\mathbf{x}) -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \phi (\mathbf{x}) \right)^T \\&= \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \phi (\mathbf{x}) -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \phi (\mathbf{x}) \right) \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \phi (\mathbf{x})^T -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \phi (\mathbf{x})^T \right) \end{align} \tag{31}\]将式(31)代入式(30)：
\[\begin{align} \mathbf w^T \mathbf S_b^{\phi} \mathbf w &= \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) ^T \cdot \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \phi (\mathbf{x}) -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \phi (\mathbf{x}) \right) \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \phi (\mathbf{x})^T -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \phi (\mathbf{x})^T \right) \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \\&= \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \sum^m_{i=1} \alpha_i \phi (\mathbf{x}_i)^T \phi (\mathbf{x}) -\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \sum^m_{i=1} \alpha_i \phi (\mathbf{x}_i)^T \phi (\mathbf{x}) \right) \left( \frac{1}{m_1}\sum_{\mathbf{x}\in X_1} \sum^m_{i=1} \alpha_i \phi (\mathbf{x})^T \phi (\mathbf{x}_i)-\frac{1}{m_0}\sum_{\mathbf{x}\in X_0} \sum^m_{i=1} \alpha_i \phi (\mathbf{x})^T \phi (\mathbf{x}_i) \right) \end{align} \tag{32}\]由于$\kappa(\mathbf x_i,\mathbf x)=\phi (\mathbf x_i)^T \phi (\mathbf x)$为标量，所以其转置等于本身，也即$\kappa(\mathbf x_i,\mathbf x)=\phi (\mathbf x_i)^T \phi (\mathbf x)=(\phi (\mathbf x_i)^T \phi (\mathbf x))^T=\phi (\mathbf x)^T \phi (\mathbf x_i)=\kappa (\mathbf x_i , \mathbf x)^T$，将其代入式(32)可得：
\[\mathbf w^T \mathbf S_b^{\phi} \mathbf w =\left( \frac{1}{m_1} \sum^m_{i=1} \sum_{\mathbf{x}\in X_1} \alpha_i \kappa(\mathbf x_i,\mathbf x) -\frac{1}{m_0} \sum^m_{i=1} \sum_{\mathbf{x}\in X_0} \alpha_i \kappa(\mathbf x_i,\mathbf x) \right) \left( \frac{1}{m_1} \sum^m_{i=1} \sum_{\mathbf{x}\in X_1} \alpha_i \kappa(\mathbf x_i,\mathbf x)-\frac{1}{m_0} \sum^m_{i=1} \sum_{\mathbf{x}\in X_0} \alpha_i \kappa(\mathbf x_i,\mathbf x) \right) \tag{33}\]令$\mathbf \alpha=(\alpha_1;\alpha_2;\cdots;\alpha_m)^T \in \mathbb{R}^{m\times 1}$，同时代入式(28)和式(29)，则式(33)可化简为：
\[\begin{align} \mathbf w^T \mathbf S_b^{\phi} \mathbf w &= (\mathbf \alpha^T \hat{\mathbf \mu}_1 - \mathbf \alpha^T \hat{\mathbf \mu}_0) \cdot (\hat{\mathbf \mu}_1^T \mathbf \alpha -\hat{\mathbf \mu}_0^T \mathbf \alpha) \\&= \mathbf \alpha^T \cdot (\hat{\mathbf \mu}_1 -\hat{\mathbf \mu}_0) \cdot (\hat{\mathbf \mu}_1^T - \hat{\mathbf \mu}_0^T) \cdot \mathbf \alpha \\&= \mathbf \alpha^T \cdot (\hat{\mathbf \mu}_1 -\hat{\mathbf \mu}_0) \cdot (\hat{\mathbf \mu}_1 - \hat{\mathbf \mu}_0)^T \cdot \mathbf \alpha \\&= \mathbf \alpha^T \mathbf M \mathbf \alpha \end{align} \tag{34}\]以上便是式(19)分子部分的推导，下面继续推导式(19)的分母部分。将式(14)代入式(4)的分母可得：
\[\begin{align} \mathbf w^T \mathbf S_w^{\phi} \mathbf w &= \left( \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \right) ^T \cdot \mathbf S_w^{\phi} \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \\&= \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) ^T \cdot \mathbf S_w^{\phi} \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i)\end{align} \tag{35}\]其中：
\[\begin{align} \mathbf S^{\phi}_w &= \sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i}\left( \phi (\mathbf{x})-\mathbf{\mu}_i^{\phi} \right) \left( \phi (\mathbf{x})-\mathbf{\mu}_i^{\phi} \right) ^T \\&= \sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i}\left( \phi (\mathbf{x})-\mathbf{\mu}_i^{\phi} \right) \left( \phi (\mathbf{x})^T-\left( \mathbf{\mu}_i^{\phi} \right)^T \right) \\&= \sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \left( \phi (\mathbf{x}) \phi (\mathbf{x})^T- \phi (\mathbf{x}) \left( \mathbf{\mu}_i^{\phi} \right)^T -\mathbf{\mu}_i^{\phi} \phi (\mathbf{x})^T +\mathbf{\mu}_i^{\phi} \left( \mathbf{\mu}_i^{\phi} \right)^T \right) \\&= \sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \phi (\mathbf{x}) \phi (\mathbf{x})^T-\sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \phi (\mathbf{x}) \left( \mathbf{\mu}_i^{\phi} \right)^T -\sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \mathbf{\mu}_i^{\phi} \phi (\mathbf{x})^T +\sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \mathbf{\mu}_i^{\phi} \left( \mathbf{\mu}_i^{\phi} \right)^T \end{align} \tag{36}\]由于：
\[\begin{align} \sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \phi (\mathbf{x}) \left( \mathbf{\mu}_i^{\phi} \right)^T &= \sum_{\mathbf{x}\in \mathbf{X}_0} \phi (\mathbf{x}) \left( \mathbf{\mu}_0^{\phi} \right)^T+\sum_{\mathbf{x}\in \mathbf{X}_1} \phi (\mathbf{x}) \left( \mathbf{\mu}_1^{\phi} \right)^T \\&= m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T +m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \end{align} \tag{37}\] \[\begin{align} \sum_{i=0}^1 \sum_{\mathbf{x}\in \mathbf{X}_i} \mathbf{\mu}_i^{\phi} \phi (\mathbf{x})^T &= \sum_{i=0}^1 \mathbf \mu_i^{\phi} \sum_{\mathbf{x}\in \mathbf{X}_i} \phi (\mathbf{x})^T \\&= \mathbf \mu_0^{\phi} \sum_{\mathbf{x}\in \mathbf{X}_0} \phi (\mathbf{x})^T+ \mathbf \mu_1^{\phi} \sum_{\mathbf{x}\in \mathbf{X}_1} \phi (\mathbf{x})^T \\&= m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T +m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \end{align} \tag{38}\]将式(37)和式(38)代入式(36)：
\[\begin{align} \mathbf S^{\phi}_w &= \sum_{\mathbf{x}\in D} \phi (\mathbf{x}) \phi (\mathbf{x})^T-2\left[ m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T +m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \right] +m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T +m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \\&= \sum_{\mathbf{x}\in D} \phi (\mathbf{x}) \phi (\mathbf{x})^T - m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T -m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \end{align} \tag{39}\]将式(39)带回式(35)：
\[\begin{align} \mathbf w^T \mathbf S_w^{\phi} \mathbf w &= \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) ^T \cdot \mathbf S_w^{\phi} \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \\&= \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) ^T \cdot \left( \sum_{\mathbf{x}\in D} \phi (\mathbf{x}) \phi (\mathbf{x})^T - m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T -m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \right) \cdot \sum^m_{i=1}\alpha_i \phi(\mathbf x_i) \\&= \sum^m_{i=1} \sum_{j=1}^m \sum_{\mathbf x \in D} \alpha_i \phi(\mathbf x_i) ^T \phi (\mathbf{x}) \phi (\mathbf{x})^T \alpha_j \phi(\mathbf x_j) - \sum^m_{i=1} \sum_{j=1}^m \alpha_i \phi(\mathbf x_i) ^T m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T \alpha_j \phi(\mathbf x_j) - \sum^m_{i=1} \sum_{j=1}^m \alpha_i \phi(\mathbf x_i) ^T m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \alpha_j \phi(\mathbf x_j) \end{align} \tag{40}\]其中，式(40)的第一项可化简为：
\[\begin{align} \sum^m_{i=1} \sum_{j=1}^m \sum_{\mathbf x \in D} \alpha_i \phi(\mathbf x_i) ^T \phi (\mathbf{x}) \phi (\mathbf{x})^T \alpha_j \phi(\mathbf x_j) &= \sum^m_{i=1} \sum_{j=1}^m \sum_{\mathbf x \in D} \alpha_i \alpha_j \kappa(\mathbf x_i,\mathbf x) \kappa(\mathbf x_j,\mathbf x) \\&= \mathbf \alpha^T \mathbf{KK}^T \mathbf{\alpha} \end{align}\tag{41}\]式(40)的第二项可化简为：
\[\begin{align} \sum^m_{i=1} \sum_{j=1}^m \alpha_i \phi(\mathbf x_i) ^T m_0 \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T \alpha_j \phi(\mathbf x_j) &= m_0 \sum^m_{i=1} \sum_{j=1}^m \alpha_i \alpha_j \phi(\mathbf x_i) ^T \mathbf{\mu}_0^{\phi} \left( \mathbf{\mu}_0^{\phi} \right)^T \phi(\mathbf x_j) \\&= m_0 \sum^m_{i=1} \sum_{j=1}^m \alpha_i \alpha_j \phi(\mathbf x_i) ^T \left[ \frac{1}{m_0} \sum_{\mathbf x \in X_0} \phi (\mathbf x) \right] \left[ \frac{1}{m_0} \sum_{\mathbf x \in X_0} \phi (\mathbf x) \right]^T \phi(\mathbf x_j) \\&= m_0 \sum^m_{i=1} \sum_{j=1}^m \alpha_i \alpha_j \left[ \frac{1}{m_0} \sum_{\mathbf x \in X_0} \phi(\mathbf x_i) ^T \phi (\mathbf x) \right] \left[ \frac{1}{m_0} \sum_{\mathbf x \in X_0} \phi (\mathbf x)^T \phi(\mathbf x_j) \right] \\&= m_0 \sum^m_{i=1} \sum_{j=1}^m \alpha_i \alpha_j \left[ \frac{1}{m_0} \sum_{\mathbf x \in X_0} \kappa (\mathbf x_i,\mathbf x) \right] \left[ \frac{1}{m_0} \sum_{\mathbf x \in X_0} \kappa (\mathbf x_j,\mathbf x) \right] \\&= m_0 \mathbf \alpha^T \hat{\mathbf \mu}_0 \hat{\mathbf \mu}_0^T \mathbf \alpha \end{align} \tag{42}\]同理，式(40)的第三项可化简为：
\[\sum^m_{i=1} \sum_{j=1}^m \alpha_i \phi(\mathbf x_i) ^T m_1 \mathbf{\mu}_1^{\phi} \left( \mathbf{\mu}_1^{\phi} \right)^T \alpha_j \phi(\mathbf x_j) = m_1 \mathbf \alpha^T \hat{\mathbf \mu}_1 \hat{\mathbf \mu}_1^T \mathbf \alpha \tag{43}\]将式(41)、式(42)、式(43)带回到式(40)：
\[\begin{align} \mathbf w^T \mathbf S_w^{\phi} \mathbf w &= \mathbf \alpha^T \mathbf{KK}^T \mathbf{\alpha} - m_0 \mathbf \alpha^T \hat{\mathbf \mu}_0 \hat{\mathbf \mu}_0^T \mathbf \alpha - m_1 \mathbf \alpha^T \hat{\mathbf \mu}_1 \hat{\mathbf \mu}_1^T \mathbf \alpha \\&= \mathbf \alpha^T \cdot \left( \mathbf{KK}^T - m_0 \hat{\mathbf \mu}_0 \hat{\mathbf \mu}_0^T- m_1 \hat{\mathbf \mu}_1 \hat{\mathbf \mu}_1^T \right) \cdot \mathbf \alpha \\&= \mathbf \alpha^T \cdot \left( \mathbf{KK}^T - \sum_{i=0}^1 m_i \hat{\mathbf \mu}_i \hat{\mathbf \mu}_i^T \right) \cdot \mathbf \alpha \\&= \mathbf \alpha^T \mathbf{N \alpha} \end{align} \tag{44}\]