矩阵求导笔记

1. ML中为什么需要矩阵求导

• 简洁
  用方程式表示如下：
  $$y_1=w_1{X}_{11}+w_2{X}_{12}\text{\hspace{1em}(1)}$$
  $$y_2=w_1{X}_{21}+w_2{X}_{22}\text{\hspace{1em}(2)}$$
  转换成矩阵表示如下：
  $$Y=XW\text{\hspace{1em}(3)}$$
  $$Y=\left[\begin{array}{c}{y}_1\\ \\ y_2\end{array}\right],X=\left[\begin{array}{ccc}{x}_{11}& & x_{12}\\ & & \\ x_{21}& & x_{22}\end{array}\right],W=\left[\begin{array}{c}{w}_1\\ \\ w_2\end{array}\right]\text{\hspace{1em}(4)}$$

• 快速
  当使用python 中的numpy 库时候，在相对于 for 循环，Numpy 本身的计算提速相当快

• 源代码

import time
import numpy as np

if __name__ == "__main__":
    N = 1000000
    a = np.random.rand(N)
    b = np.random.rand(N)
    start = time.time()
    c = np.dot(a,b)
    stop = time.time()
    print(f"c={c}")
    print("vectorized version: " + str(1000*(stop-start))+"ms")

    c = 0
    start1 = time.time()
    for i in range(N):
        c += a[i]*b[i]
    stop1 = time.time()

    print(f"c={c}")
    print("for loop: " + str(1000*(stop1-start1))+"ms")
    times1 = (stop1-start1)/(stop-start)
    print(f"times1={times1}")

• 结果

c=250071.8870070607
vectorized version: 6.549358367919922ms
c=250071.88700706122
for loop: 265.43641090393066ms
times1=40.52861303239898# 向量化居然比单独的for循环快40倍

2. 向量函数与矩阵求导初印象

• 标量函数:输出为标量的函数
  $$f(x)=x^2\Rightarrow x\in R\to x^2\in R$$
  $$f(x)=x_1^2+x_2^2\Rightarrow \left[\begin{array}{c}{x}_1\\ \\ x_2\end{array}\right]\in R^2\to x_1^2+x_2^2\in R$$
• 向量函数:输出为向量或矩阵的函数
  <1> 输入标量，输出向量
  $$f(x)=\left[\begin{array}{c}{f}_1(x)=x\\ \\ f_2(x)=x^2\end{array}\right]\Rightarrow x\in R,\left[\begin{array}{c}x\\ \\ x^2\end{array}\right]\in R^2$$
  <2> 输入标量，输出矩阵
  $$f(x)=\left[\begin{array}{ccc}{f}_{11}(x)=x& & f_{12}(x)=x^2\\ & & \\ f_{21}(x)=x^3& & f_{22}(x)=x^4\end{array}\right]\Rightarrow x\in R,\left[\begin{array}{ccc}x& & x^2\\ & & \\ x^3& & x^4\end{array}\right]\in R^{2\times 2}$$
  <3> 输入向量，输出矩阵
  $$f(x)=\left[\begin{array}{ccc}{f}_{11}(x)=x_1+x_2& & f_{12}(x)=x_1^2+x_2^2\\ & & \\ f_{21}(x)=x_1^3+x_2^3& & f_{22}(x)=x_1^4+x_2^4\end{array}\right]\Rightarrow \left[\begin{array}{c}{x}_1\\ \\ x_2\end{array}\right]\in R^2,\left[\begin{array}{ccc}{x}_1+x_2& & x_1^2+x_2^2\\ & & \\ x_1^3+x_2^3& & x_1^4+x_2^4\end{array}\right]\in R^{2\times 2}$$
• 总结
  矩阵求导的本质
  $$\frac{\mathrm{d}A}{\mathrm{d}B}=矩阵A中的每个元素对矩阵B中的每个元素求导$$

3. YX 拉伸术

3.1 f(x)为标量，X为列向量

• 标量不变，向量拉伸
• YX中，Y前面横向拉，X后面纵向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x},Y=f(x)为标量，X=\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]为列向量$$
  $$f(x)=f(x_1,x_2,....,x_n)为标量$$
• 标量$f(x)$不变，向量X 因为在YX拉伸术中在Y后面，所以向量X纵向拉伸，实际上就是将多元函数的偏导写在一个列向量中
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$

3.2 f(x)为列向量，X 为标量

$$f(x)=\left[\begin{array}{c}{f}_1(x)\\ \\ f_2(x)\\ \\ ⋮\\ \\ f_n(x)\end{array}\right];X为标量$$

• 标量不变，向量拉伸
• YX中，Y前面横向拉，X后面纵向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{ccccccc}\frac{\partial f_1(x)}{\partial x}& & \frac{\partial f_2(x)}{\partial x}& & \dots & & \frac{\partial f_n(x)}{\partial x}\end{array}\right]$$

3.3 f(x)为列向量，X 为列向量

$$f(x)=\left[\begin{array}{c}{f}_1(x)\\ \\ f_2(x)\\ \\ ⋮\\ \\ f_n(x)\end{array}\right];X=\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]为列向量$$

• 第一步先固定Y ，将 X 纵向拉
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$
• 第二步，看每一个项$\frac{\partial f(x)}{\partial x_1}$,其中f(x)为列向量，$x_1$为标量，那么可以看出要进行 Y 横向拉
  $$\frac{\partial f(x)}{\partial x_1}=\left[\begin{array}{ccccccc}\frac{\partial f_1(x)}{\partial x_1}& & \frac{\partial f_2(x)}{\partial x_1}& & \dots & & \frac{\partial f_n(x)}{\partial x_1}\end{array}\right]$$
• 第三步 ，将每项整合如下
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}x}=\left[\begin{array}{ccccccc}\frac{\partial f_1(x)}{\partial x_1}& & \frac{\partial f_2(x)}{\partial x_1}& & \dots & & \frac{\partial f_n(x)}{\partial x_1}\\ & & & & & & \\ \frac{\partial f_1(x)}{\partial x_2}& & \frac{\partial f_2(x)}{\partial x_2}& & \dots & & \frac{\partial f_n(x)}{\partial x_2}\\ & & & & & & \\ ⋮& & ⋮& & \dots & & ⋮\\ & & & & & & \\ \frac{\partial f_1(x)}{\partial x_n}& & \frac{\partial f_2(x)}{\partial x_n}& & \dots & & \frac{\partial f_n(x)}{\partial x_n}\end{array}\right]$$

4. 常见矩阵求导公式

4.1 $Y=A^{T}X$

$$f(x)=A^{T}X;A=[a_1,a_2,\dots ,a_n{]}^T;X=[x_1,x_2,\dots ,x_n{]}^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X}$$

• 由于$A^T=1\times n,X=n\times 1,那么f(x)为标量，即表示数值$，
• 标量不变，向量拉伸
• YX中，Y前面横向拉，X后面纵向拉
  $$f(x)=\sum _{i=1}^N{a}_i{x}_i$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$
• 可以计算$\frac{\partial f(x)}{\partial x_i}$
  $$\frac{\partial f(x)}{\partial x_i}=a_i$$
• 可得如下：
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}{a}_1\\ \\ a_2\\ \\ ⋮\\ \\ a_n\end{array}\right]=A$$
• 结论：
  $$当f(x)=A^{T}X$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=A$$

4.2 $Y=X^{T}AX$

$$f(x)=X^{T}AX;A=\left[\begin{array}{ccccccc}{a}_{11}& & a_{12}& & \dots & & a_{1n}\\ & & & & & & \\ a_{21}& & a_{22}& & \dots & & a_{2n}\\ & & & & & & \\ ⋮& & ⋮& & \dots & & ⋮\\ & & & & & & \\ a_{n1}& & a_{n2}& & \dots & & a_{nn}\end{array}\right];X=[x_1,x_2,\dots ,x_n{]}^T,求\frac{\mathrm{d}f(x)}{\mathrm{d}X}$$
$$f(x)=\sum _{i=1}^N\sum _{j=1}^N{a}_{ij}{x}_i{x}_j$$

• 标量不变，YX拉伸术，X纵向拉伸
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]$$
  $$\frac{\partial f(x)}{\partial x_i}=\left[\begin{array}{cccc}{a}_{i1}& a_{i2}& \dots & a_{in}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{1i}& a_{2i}& \dots & a_{ni}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]$$
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ & & & \\ a_{21}& a_{22}& \dots & a_{2n}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{n1}& a_{n2}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]+\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ & & & \\ a_{12}& a_{22}& \dots & a_{n2}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]$$
• 已知$A,A^T$表示如下：
  $$A=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \dots & a_{1n}\\ & & & \\ a_{21}& a_{22}& \dots & a_{2n}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{n1}& a_{n2}& \dots & a_{nn}\end{array}\right];A^T=\left[\begin{array}{cccc}{a}_{11}& a_{21}& \dots & a_{n1}\\ & & & \\ a_{12}& a_{22}& \dots & a_{n2}\\ & & & \\ ⋮& ⋮& \dots & ⋮\\ & & & \\ a_{1n}& a_{2n}& \dots & a_{nn}\end{array}\right]$$
• 综上所述如下：
  当 $f(x)=X^{T}AX$时
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=AX+A^{T}X=(A+A^T)X$$

5. 两种布局

5.1 概括

两种布局矩阵求导的本质是向量求导拉伸方向的区别，求导后元素排列不同

• 口诀：前面横向拉，后面纵向拉
• 分子布局 XY拉伸术$\frac{X}{Y}$，X横向拉，Y纵向拉
• 分母布局 YX拉伸术$\frac{Y}{X}$，Y横向拉，X纵向拉

5.2 举例

$$f(x)=X^{T}X,X={\left[\begin{array}{cccc}{x}_1& x_2& \dots & x_n\end{array}\right]}^T$$

• 分子布局，XY拉伸术，X横向拉，Y纵向拉,f(x)为标量，标量不变，X向量横向拉伸
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{cccc}\frac{\partial f(x)}{\partial x_1}& \frac{\partial f(x)}{\partial x_2}& \dots & \frac{\partial f(x)}{\partial x_n}\end{array}\right]=\left[\begin{array}{cccc}2x_1& 2x_2& \dots & 2x_n\end{array}\right]=2\left[\begin{array}{cccc}{x}_1& x_2& \dots & x_n\end{array}\right]=2X^T$$
• 分母布局，YX拉伸术，Y横向拉，X纵向拉,f(x)为标量，标量不变，X向量纵向拉伸
  $$\frac{\mathrm{d}f(x)}{\mathrm{d}X}=\left[\begin{array}{c}\frac{\partial f(x)}{\partial x_1}\\ \\ \frac{\partial f(x)}{\partial x_2}\\ \\ ⋮\\ \\ \frac{\partial f(x)}{\partial x_n}\end{array}\right]=\left[\begin{array}{c}2x_1\\ \\ 2x_2\\ \\ ⋮\\ \\ 2x_n\end{array}\right]=2\left[\begin{array}{c}{x}_1\\ \\ x_2\\ \\ ⋮\\ \\ x_n\end{array}\right]=2X$$
• 综上所述：$分子布局=(分母布局{)}^T$

6. 最小二乘法-分母布局

• 需要拟合一个线，使得线距离每个点的距离和最短。
  
  $$L(b)=\sum _{i=1}(y_i-x_i{b}_i{)}^2$$
• 为了方便计算，需将以上求和公式改为矩阵形式如下
  $$Y={\left[\begin{array}{cccc}{y}_1& y_2& \dots & y_n\end{array}\right]}^T;X={\left[\begin{array}{cccc}{x}_1^T& x_2^T& \dots & x_n^T\end{array}\right]}^T;x_i^T=\left[\begin{array}{cccc}{x}_{i1}& x_{i2}& \dots & x_{in}\end{array}\right]$$
  $L(b)=(Y-Xb{)}^T(Y-Xb)$
  $=(Y^T-b^T{X}^T)(Y-Xb)=Y^{T}Y-Y^{T}Xb-b^T{X}^{T}Y+b^T{X}^{T}Xb$
• 因为$Y^{T}Xb$为标量，所以 $Y^{T}Xb=b^T{X}^{T}Y$
  $$L(b)=Y^{T}Y-2Y^{T}Xb+b^T{X}^{T}Xb$$
• 求$L(b)$对b求导可得如下：
  $$\frac{\mathrm{d}{Y}^{T}Y}{\mathrm{d}(b)}={\left[\begin{array}{c}0\\ \\ 0\\ \\ ⋮\\ \\ 0\end{array}\right]}_{n\times 1}$$
• 因为$\frac{\mathrm{d}{A}^{T}X}{\mathrm{d}(X)}=A$可得如下：
  $$\frac{\mathrm{d}2Y^{T}Xb}{\mathrm{d}(b)}=(2Y^{T}X{)}^T=2X^{T}Y$$
• 因为 $\frac{\mathrm{d}{X}^{T}AX}{\mathrm{d}(X)}=(A+A^T)X,可得如下：$
  $$\frac{\mathrm{d}{b}^T{X}^{T}Xb}{\mathrm{d}(b)}=(X^{T}X+(X^{T}X{)}^T)b=2X^{T}Xb$$
• 综上所述可得如下：
  $$\frac{\mathrm{d}L(b)}{\mathrm{d}(b)}=-2X^{T}Y+2X^{T}Xb=0$$
  $$\hat{b}=(X^{T}X{)}^{-1}{X}^{T}Y$$