init version

13a86540 · 张锦伦 · a924ca10 · a924ca10 · 13a86540 · 13a86540
Commit 13a86540 authored 3 years ago by 张锦伦
--- a/.gitee/ISSUE_TEMPLATE.zh-CN.md
+++ b/.gitee/ISSUE_TEMPLATE.zh-CN.md
-### 该问题是怎么引起的？
-
-
-
-### 重现步骤
-
-
-
-### 报错信息
-
-
-
-
--- a/.gitignore
+++ b/.gitignore
@@ -30,3 +30,8 @@
 *.exe
 *.out
 *.app
+
+.cache
+build/*
+!build
+.vscode
--- a/.gitmodules
+++ b/.gitmodules
+[submodule "third_party/gtest"]
+	path = third_party/gtest
+	url = https://github.com/google/googletest.git
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
+cmake_minimum_required(VERSION 3.16.0)
+set(TARGET_NAME LZ-ml-engine)
+set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
+set(CMAKE_CXX_COMPILER /usr/bin/clang++)
+project(${TARGET_NAME} VERSION 1.0 LANGUAGES CXX)
+
+# enable c++ 17
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED True)
+
+# 添加头文件
+
+
+# 设置编译的可执行文件
+# 链接库必须放在后面
+# set(SOURCE test/main.cpp)
+# add_executable(${TARGET_NAME} ${SOURCE}) 
+add_subdirectory(src EXCLUDE_FROM_ALL)
+add_subdirectory(tools EXCLUDE_FROM_ALL)
+add_subdirectory(test)
--- a/README.en.md
+++ b/README.en.md
-# lz-ml-engine
-
-#### Description
-This is the machine learning engine writed by ourselves.
-
-#### Software Architecture
-Software architecture description
-
-#### Installation
-
-1.  xxxx
-2.  xxxx
-3.  xxxx
-
-#### Instructions
-
-1.  xxxx
-2.  xxxx
-3.  xxxx
-
-#### Contribution
-
-1.  Fork the repository
-2.  Create Feat_xxx branch
-3.  Commit your code
-4.  Create Pull Request
-
-
-#### Gitee Feature
-
-1.  You can use Readme\_XXX.md to support different languages, such as Readme\_en.md, Readme\_zh.md
-2.  Gitee blog [blog.gitee.com](https://blog.gitee.com)
-3.  Explore open source project [https://gitee.com/explore](https://gitee.com/explore)
-4.  The most valuable open source project [GVP](https://gitee.com/gvp)
-5.  The manual of Gitee [https://gitee.com/help](https://gitee.com/help)
-6.  The most popular members  [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/)
--- a/README.md
+++ b/README.md
-# lz-ml-engine
-
-#### 介绍
-This is the machine learning engine writed by ourselves.
-
-#### 软件架构
-软件架构说明
-
-
-#### 安装教程
-
-1.  xxxx
-2.  xxxx
-3.  xxxx
-
-#### 使用说明
-
-1.  xxxx
-2.  xxxx
-3.  xxxx
-
-#### 参与贡献
-
-1.  Fork 本仓库
-2.  新建 Feat_xxx 分支
-3.  提交代码
-4.  新建 Pull Request
-
-
-#### 特技
-
-1.  使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md
-2.  Gitee 官方博客 [blog.gitee.com](https://blog.gitee.com)
-3.  你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解 Gitee 上的优秀开源项目
-4.  [GVP](https://gitee.com/gvp) 全称是 Gitee 最有价值开源项目，是综合评定出的优秀开源项目
-5.  Gitee 官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help)
-6.  Gitee 封面人物是一档用来展示 Gitee 会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/)
+# LZ-ENGINE（一个玩具深度学习引擎）
+
+ example：使用例子
+ include：公共接口
+ src：引擎源码
+  + mm：内存分配代码
+  + gopt：图优化相关
+  + oprs：具体的算子实现
+    + naive：不考虑任何平台的算子实现
+    + x86：x86下的算子实现
+ tools：用于分析引擎的工具
+ test：测试代码
+
+## 目前计划
+
+ 先把一些nn定义好，并写一些基本的operator，比如conv，matmul，pool等。
+ 想好benchmarker的结构，做到能够测试算子的bandwidth
+ 写一个cpu_dispatch：非常简单，设一个任务为一个节点，构建任务图，将入度为0的任务节点分发给cpu执行，执行结束后，在任务图中删掉该节点，与它相连的其他节点入度减一，继续分发。
+
+### 必须实现的算子：
+
+ conv算子（采用 im2col + matmul）
+ matmul算子
+ matdot
+ matadd
+ pool
+
+benchmark
+
+下一步构建网络：
+
+## 编译运行
+
+```shell
+# 进入 third_party，执行
+sh ./prepare.sh
+# 然后创建build目录
+cd build
+cmake ..
+make
+./test/LZ-ml-engine-test
+```
--- a/document/blog/blog_1.md
+++ b/document/blog/blog_1.md
+# 深度学习引擎框架之计算图
+
+## 1. 计算图框架（理解版）
+
+网上的DNN和CNN的推导只是利于学习，尤其体现在反向推导上。但那种对层的推导不利于推广至所有的网络，尤其面对后续新型网络或者算子的出现，深度学习引擎框架更要采用精妙的设计。因此，我们需要计算图：
+
+在开始之前，笔者展示一下一个简单的计算图：
+
+![计算图](./picture/full_connect.svg)
+
+上图是一个简单的只有全连接层组成的神经网络，在这里，方框代表一个tensor，箭头代表数据流，椭圆框代表一个算子。因为全连接实际上就只是一个矩阵乘法，因此，上图用矩阵乘表示全连接算子。很明显，图中的 x 代表输入，w 代表权重，b 代表偏置，y 代表标签值。为了让图简单点，这里将矩阵乘和矩阵加算子合并成一个算子，方便观看，其实在优化中也是这么做的，将矩阵乘和矩阵加算子合并成该算子，有利于加速。 active 是一个激活函数算子，loss 是损失函数算子。
+
+上图的黑色箭头表示前向传播的数据流，红色箭头代表反向传播的数据流，红色箭头指向蓝色框代表传送残差进行权值更新，而指向算子的红色箭头只是传送残差到上一算子。由上图我们可以知道，对每一个算子而言，我们都必须实现该算子对任何一个输入的求导计算。**但不是所有的输入都需要算子计算对它的偏导**。比如上图中用绿色标注的 x 和 y 。pytorch 里采用这样的构造函数进行实现：
+
+```python
+tensor(data,requires_grad=True)
+```
+
+启动反向传播也非常简单，比如在 pytorch 里：
+
+```python
+optimizer.zero_grad()
+loss_value.backward()
+optimizer.step()
+```
+
+这条命令首先清空梯度累加器，接着启动反向传播，最后再将梯度加到权值矩阵上。
+
+其实所谓的自动求导，就是根据你构建的正向图，来进行反向图的构建，只要实现充足的算子，一个实用的自动求导框架是很容易实现的。
+
+为了将反向传播介绍的更加明白，这里我给上图引入一堆变量加以反向推导。
+
+![backward_infer](./picture/backward_infer.svg)
+
+为了方便推导，上图数据流上的小写字母代表输出，大写字母代表残差，ok，接下来开始推导。
+
+在这样一个神经网络中，我们计算出来的损失函数值为$g$，我们的目标是更新蓝色方框的权重值，也就是说，我们的目标是求以下梯度值：
+$$
+\frac{\partial g}{\partial w^1}、\frac{\partial g}{\partial b^1}、\frac{\partial g}{\partial w^2}、\frac{\partial g}{\partial b^2}
+$$
+就拿$\frac{\partial g}{\partial w_1}$举例子，通过上图的观察，我们很容易地通过链式求导法则进行计算：
+$$
+\frac{\partial g}{\partial w^1} =  \frac{\partial g}{\partial a^2} \frac{\partial a^2}{\partial z^2} \frac{\partial z^2}{\partial a^1} \frac{\partial a^1}{\partial z^1} \frac{\partial z^1}{\partial w^1}
+$$
+到这一步，我们可以定义上图反向传播的数据流符号了：
+$$
+G=1 \\ A^2=\frac{\partial g}{\partial a^2}=\frac{\partial g}{\partial a^2} G \\ Z^2=\frac{\partial g}{\partial z^2}=\frac{\partial a^2}{\partial z^2}\frac{\partial g}{\partial a^2}=\frac{\partial a^2}{\partial z^2}A^2 \\ A^1=\frac{\partial g}{\partial a^1}=\frac{\partial z^2}{\partial a^1}\frac{\partial g}{\partial z^2}=\frac{\partial z^2}{\partial a^1}Z^2 \\ \cdots 	\\ W^2=\frac{\partial g}{\partial w^2}=\frac{\partial z^2}{\partial w^2}\frac{\partial g}{\partial z^2}=\frac{\partial z^2}{\partial w^2}Z^2 \\ B^2=\frac{\partial g}{\partial b^2}=\frac{\partial z^2}{\partial b^2}\frac{\partial g}{\partial z^2}=\frac{\partial z^2}{\partial b^2}Z^2
+$$
+对了，这里要说明一下，因为算子的输入输出往往是张量形式，而且因为张量运算的复杂性，我上述的表示只是告诉诸位，算子之间的偏导数有着类似的相关性关系，上述的链式法则只对标量成立，对张量而言，上述的链式求导法则并不成立。
+
+不过总结上述公式我们不难发现一个规律，最后的残差 $G=1$，并且有：
+$$
+\delta^{input} = \frac{\partial output}{\partial input}\delta^{output}
+$$
+的关系存在，其中$\delta^{input}$代表 input 方向上的残差，$output$代表算子的输出，$input$代表算子的输入，从这里我们看到了独立算子实现的希望。
+
+但到这里为止，都还只是对计算图的感性认识，很多概念或者推导其实并不正确，但我们却对整个计算图有了基本的概念，在脑海中有了大致的轮廓。
+
+## 2. 计算图框架（实现版）
+
+真实的计算图框架实现不可能实现成下图所示：
+
+![计算图框架naive版](./picture/backward_infer.svg)
+
+因为这会遇到非常多的问题，比如：
+
+1. 如何统一搭建算子的架构，毕竟算子的输入可以任意多个（上图算子的输入有1，2，3个）
+2. 如何表示残差（梯度），毕竟它们是真实存在的，我们总得用一个tensor存放吧？
+3. 正向图和反向图一起构建会增大计算图设计复杂度，增大算子设计复杂度（尤其你要实现几百个算子的时候。。。都要同时考虑在算子里的的正向反向。
+4. 计算图细粒度太大，优化不能做充分。
+
+为了更好的解释理解计算图的反向传播，让我们对上述的计算图加以具象化，才能将理解版计算图架构中的缺点暴露出来。
+
+### 2.1 具象化计算图
+
+具象化该计算图的第一步就是具象化输入和输出，然后才是具象化算子。紧接着我们定义网络结构，根据具体的网络。我们再对正向传播和反向传播进行具象化，从而更加了解一个算子应该做什么事情，他是怎么做的。
+
+#### 2.1.1 具象化输入和输出
+
+通常我们的输入都是 `[N,C,H,W]`形式，依次表示batch，channel、height、width。输出通常的形式是 `[N,H]`，依次表示 batch 和分类结果（对于分类任务而言）。
+
+但就上图而言，输入的形式不可能是 `[N,C,H,W]`形式，因为我们上图的是全连接网络，因此我们需要把它展成 `[N,H]`形式，就是将原来的 `[N,C,H,W]`拍平。（我们可以添加一个用于拍平的算子，哈哈）这里，我们假定输入 x 的 tensor 张量为 `[N,H]`，y 的 tensor 张量也 为 `[N,H]`。
+
+#### 2.1.2 具像化算子
+
+在上图中我们总共有3个算子，分别是 `matrix_mul_add`、`active`、`loss`。
+
+1. **matrix_mul_add**
+   该算子是矩阵乘后加，数学形式是：$O=wx+b$
+2. **active**
+   该算子是激活函数算子，这里假定它的数学形式为：$O=f(x)$
+3. **loss**
+   该算子是损失函数算子，这里假定它的数学形式为：$O=g(x,y)$
+
+#### 2.1.3 具像化权值
+
+![](./picture/backward_infer.svg)
+
+神经网络最重要的一步就是构建网络，而网络的参数是是由用户敲定的，由他们决定每层网络的具体参数，比如层数，是否存在偏置等等。在上述的计算图中，我们假定用户采用三层全连接的神经网络。我们设输入 x 是一个$n*T^1$ 的矩阵，输出 y 是一个 $n*T^3$的矩阵，n 代表一个batch的数量，$T^1$ 代表输入的向量长度，$T^3$ 代表输出的向量长度，因此我们构建的神经网络架构图如下图所示：
+
+![full_net](./picture/full_connect_net.svg)
+
+三层的神经元数量分别为：$T^1,T^2,T^3$，因此三层中间的两个权值矩阵（阴影部分）的矩阵纬度分别为：$T^1*T^2,T^2*T^3$。
+
+#### 2.1.4 具象化正向过程
+
+ok，在上面我们将该具象的都具象化了，现在可以开始走一遍正向过程了。在走的时候，笔者会把每一个矩阵或者标量的维度通过下标进行标注（1*1即为标量）。
+
+![.](./picture/backward_infer.svg)
+$$
+z^1_{n*T^2}=x_{n*T^{1}}w^1_{T^1*T^2} + 1_{n*1}b^1_{1*T^2} \\
+a^1_{n*T^2}=f(z^1_{n*T^2}) \\
+z^2_{n*T^3}=a^1_{n*T^{2}}w^2_{T^2*T^3} + 1_{n*1}b^2_{1*T^3} \\
+a^2_{n*T^3}=f(z^2_{n*T^3}) \\
+g_{1*1} = g(a^2_{n*T^3},y_{n*T^3})
+$$
+
+#### 2.1.5 具象化反向过程
+
+在计算反向传播之前，插播一下矩阵求导的一些知识；
+
+标量f对矩阵X求导：一般定义为：$\frac{\partial f}{\partial \pmb{X}}=[\frac{\partial f}{\partial \pmb{X}_{ij}}]$，其实从微分的角度来看，有
+$$
+df = \sum_{i=1}^m\sum_{j=1}^n\frac{\partial f}{\partial x_{ij}}dx_{ij}=tr((\frac{\partial f}{\partial x})^Tdx)
+$$
+矩阵运算法则：
+$$
+d(X\pm Y)=dX \pm dY \\ d(XY)=(dX)Y+XdY \\ dtr(X)=tr(dX) \\ dX^{-1}=-X^{-1}dXX^{-1} \\ d(X \odot Y)=dX \odot Y + X \odot dY \\ 逐元素求导 : df(X)=f'(X) \odot dX
+$$
+迹技巧：
+$$
+标量:a = tr(a) \\ tr(A^T)=tr(A) \\ tr(A \pm B)=tr(A) \pm tr(B) \\ tr(AB)=tr(BA)，其中A和B^T维度相同 \\tr(A^T(B \odot C))=tr((A\odot B)^TC)
+$$
+这里说明一下，在实现算子的求导过程时，经常会遇到标量对标量求导，向量对标量求导，标量对向量求导，矩阵。。。等等，需要具体情况具体分析。其实反向还是蛮简单的，就是矩阵求导的过程，但肯定有一些对矩阵求导不熟悉的同学，对该过程神智无知，因此下面我将对每个算子进行逐一分析，恰好，这三个算子过于简单，很好分析。。。不像其他算子诸如卷积等。。。笔者暂时还不知道咋用数学符号表示。
+
+##### 2.1.5.1 loss算子
+
+![loss_kernal](./picture/loss_kernal.svg)
+
+设损失函数为$p(x,y)$，很明显，输出 $g$ 是一个标量，考虑 $x$ 为矩阵，假如该函数是逐元素计算（比如常用的欧氏距离）：
+$$
+dg = tr((\frac{\partial g}{\partial g})^Tdg)=tr((\frac{\partial g}{\partial g})^Tdp(x,y))\\=tr((\frac{\partial g}{\partial g})^T(\frac{\partial p(x,y)}{\partial x} \odot dx)) \\
+=tr((\frac{\partial g}{\partial g}\odot\frac{\partial p(x,y)}{\partial x})^Tdx)
+$$
+故有：
+$$
+X =\frac{\partial g}{\partial x} = \frac{\partial g}{\partial g}\odot\frac{\partial p(x,y)}{\partial x} = 1 \odot \frac{\partial p(x,y)}{\partial x} \\
+=1 \odot \begin{bmatrix}\frac{\partial p(x,y)}{\partial x_{11}}\cdots \frac{\partial p(x,y)}{\partial x_{1n}}\\ \frac{\partial p(x,y)}{\partial x_{21}}  \cdots \frac{\partial p(x,y)}{\partial x_{2n}} \\ \vdots \ddots \\\frac{\partial p(x,y)}{\partial x_{m1}} \cdots \frac{\partial p(x,y)}{\partial x_{mn}}\end{bmatrix}
+$$
+
+##### 2.1.5.2 active算子
+
+![active_kernal](./picture/active_kernal.svg)
+
+因为某个具体的激活函数也是一个具体的算子，并没有统一的模板，因此反向也不能够统一，这里设激活函数为 $y=f(x)=\frac{1}{1+e^{-x}}$，考虑 $ x $ 为矩阵，则有：
+$$
+y=f(x)=f(\begin{bmatrix}x_{11}\cdots x_{1n}\\ x_{21}  \cdots x_{2n} \\ \vdots \ddots \\x_{m1} \cdots x_{mn}\end{bmatrix})=\begin{bmatrix}f(x_{11})\cdots f(x_{1n})\\ f(x_{21})  \cdots f(x_{2n}) \\ \vdots \ddots \\f(x_{m1}) \cdots f(x_{mn})\end{bmatrix}
+$$
+明显该函数是逐元素计算，因此可以根据全微分进行偏导的推导，推导如下：
+$$
+dg = tr((\frac{\partial g}{\partial y})^Tdy) = tr((\frac{\partial g}{\partial y})^Td(f(x)))\\ 
+
+= tr((\frac{\partial g}{\partial y})^T (f'(x) \odot dx))= tr(((\frac{\partial g}{\partial y}) \odot f'(x))^Tdx)
+$$
+因此有：
+$$
+X=\frac{\partial g}{\partial y} \odot f'(x) = Y \odot f'(x) = Y \odot \begin{bmatrix}\frac{\partial f(x_{11})}{\partial x_{11}}\cdots \frac{\partial f(x_{1n})}{\partial x_{1n}}\\ \frac{\partial f(x_{21})}{\partial x_{21}}  \cdots \frac{\partial f(x_{2n})}{\partial x_{2n}} \\ \vdots \ddots \\\frac{\partial f(x_{m1})}{\partial x_{m1}} \cdots \frac{\partial f(x_{mn})}{\partial x_{mn}}\end{bmatrix}
+$$
+
+##### 2.1.5.3 matrix_mul_add算子
+
+![matrix_mul_dd](./picture/matrix_mul_add.svg)
+
+该算子其实就是实现以下正向过程：
+$$
+y=wx+b
+$$
+但实际上，因为我们将输入的tensor设置成 `[N,H]`的形式，因此，实际上，正向过程为：
+$$
+y=x_{t*n}w_{n*m}+1_{t*1}b_{1*m}
+$$
+因此反向过程为推导如下，我们不需要关注其他算子，我们只需要该算子的信息即可，因为最后的目标值肯定是标量，我们可以通过全微分推导得到偏导数（千万别漏看了表达式中的$1$，他是$1_{t*1}$全一矩阵，对于$b_{1*m}$）：
+$$
+dg=tr((\frac{\partial g}{\partial y})^Tdy)=tr((\frac{\partial g}{\partial y})^Td(xw+1b)) \\
+=tr((\frac{\partial g}{\partial y})^T1db) + tr((\frac{\partial g}{\partial y})^Tdxw) + tr((\frac{\partial g}{\partial y})^Txdw) \\
+=tr((\frac{\partial g}{\partial y})^T1db) + tr(w(\frac{\partial g}{\partial y})^Tdx) + tr((\frac{\partial g}{\partial y})^Txdw) \\
+=tr((1^T(\frac{\partial g}{\partial y}))^Tdb) + tr(w(\frac{\partial g}{\partial y})^Tdx) + tr((\frac{\partial g}{\partial y})^Txdw)
+$$
+通过上式第一项可知b对g的偏导数，第二项以此类推，故有：
+$$
+B =\frac{\partial g}{\partial b} = 1^T\frac{\partial g}{\partial y} =1^T Y \\
+X = \frac{\partial g}{\partial x} = \frac{\partial g}{\partial y} w^T= Yw^T \\
+W = \frac{\partial g}{\partial w} = x^T\frac{\partial g}{\partial y} = x^T Y \\
+$$
+那有人会说，全连接层的 $b$ 可以设置成 $1*1$ 的，那这个也行，没问题，来，注意下面的 $1$ 矩阵，我会标注清楚下标，它表示一个全1矩阵：
+$$
+dg=tr((\frac{\partial g}{\partial y})^Tdy)=tr((\frac{\partial g}{\partial y})^Td(xw+1_{t*1}b_{1*1}1_{1*m})) \\
+=tr((\frac{\partial g}{\partial y})^Td(1_{t*1}b_{1*1})1_{1*m}) + tr((\frac{\partial g}{\partial y})^Tdxw) + tr((\frac{\partial g}{\partial y})^Txdw) \\
+=tr(1_{1*m}(\frac{\partial g}{\partial y})^Td(1_{t*1}b_{1*1})) + tr(w(\frac{\partial g}{\partial y})^Tdx) + tr((\frac{\partial g}{\partial y})^Txdw) \\
+=tr(1_{1*m}(\frac{\partial g}{\partial y})^T1_{t*1}db_{1*1}) + tr(w(\frac{\partial g}{\partial y})^Tdx) + tr((\frac{\partial g}{\partial y})^Txdw)
+$$
+其他两项不变，而我们的 $B$ 有：
+$$
+B =\frac{\partial g}{\partial b} = 1_{1*t}\frac{\partial g}{\partial y}1_{m*1} =1_{1*t}Y1_{m*1}
+$$
+
+### 2.2 分析计算图
+
+走过一遍计算图的正向反向计算之后，大家会发现，其实所谓的正向或者反向只不过是一些常见的运算罢了，除了一些特殊的算子需要进行特殊的处理，其它的只不过是一些常见的运算罢了，而且正向过程和反向过程有很多的算子可以复用，比如矩阵乘算子。因此为了能复用算子，并且减小算子细粒度便于后续的优化，我们的计算图设计是这样的：
+
+
+
+
+
+## 3. 算子的架构
+
+## 4. 常用算子实现
+
+第二部分，我们实现了一个激活函数算子，损失函数算子，还有矩阵乘加算子。我们还需要添加亿点点通用算子。
+
+### 4.2 pooling 池化算子
+
+![pooling_kernal](./picture/pooling_kernal.svg)
+
+常用的池化层有 max 池化和 average 池化等等，因为池化层太过简单，而且没有具体的数学符号，因此我们直接写出 $X$ 。假设池化区域大小是 $2*2$，$stride_h = 2,stride_w = 2$ 。（最常用）
+
+下面讨论的都是默认输入和输出为矩阵，维度为$[H,W]$，当拓展到张量 $[N,C,H,W] $ 后，因为前两个维度和算子的运算并无关系，因此直接`for n in [1:N]:for c in [1:C]:`即可。设有：
+$$
+Y=\begin{bmatrix} \delta_{11} &  \delta_{12}  \\  \delta_{21} &  \delta_{22} \end{bmatrix}
+$$
+
+- max 池化
+
+  假如有：
+  $$
+  y=max\_pool(x) = max\_pool(\begin{bmatrix}x_{11} &  x_{12} & x_{13} &  x_{14} \\  x_{21} &  x_{22} & x_{23} &  x_{24} \\ x_{31} &  x_{32} & x_{33} &  x_{34} \\ x_{41} &  x_{42} & x_{43} &  x_{44}\end{bmatrix}) = \begin{bmatrix}x_{11} &  x_{23}  \\  x_{32} &  x_{44} \end{bmatrix}
+  $$
+  那么我们便可得：
+  $$
+  X =\begin{bmatrix}\delta_{11} &  0 & 0 &  0 \\  0 &  0 & \delta_{12} &  0 \\ 0 &  \delta_{21} & 0 &  0 \\ 0 &  0 & 0 &  \delta_{22}\end{bmatrix}
+  $$
+
+- average 池化
+
+  这个非常简单，我们可以直接得到：
+  $$
+  X =\begin{bmatrix}\frac{\delta_{11}}{4} &  \frac{\delta_{11}}{4} & \frac{\delta_{12}}{4} &  \frac{\delta_{12}}{4} \\  \frac{\delta_{11}}{4} &  \frac{\delta_{11}}{4} & \frac{\delta_{12}}{4} &  \frac{\delta_{12}}{4} \\ \frac{\delta_{21}}{4} &  \frac{\delta_{21}}{4} & \frac{\delta_{22}}{4} &  \frac{\delta_{22}}{4} \\ \frac{\delta_{21}}{4} &  \frac{\delta_{21}}{4} & \frac{\delta_{22}}{4} &  \frac{\delta_{22}}{4}\end{bmatrix}
+  $$
+
+我们将上述计算称为 $upsample$ 上采样，因此有：
+$$
+X = unsample(Y)
+$$
+
+### 4.1 conv 卷积算子
+
+![](./picture/conv_kernal.svg)
+
+在这里我们定义卷积符号为：$\circledast$，举个卷积的例子：
+$$
+\begin{bmatrix} 0 & 1 & 2 & 3\\
+1 & 2 & 3 & 4\\
+3 & 2 & 1 & 4\\4 & 3 & 2 & 1
+\end{bmatrix}
+\circledast
+\begin{bmatrix} 0 & 1 & 2\\
+3 & 2 & 1\\
+3 & 1 & 0 
+\end{bmatrix}
+=\begin{bmatrix} 26 & 31\\
+37 & 34
+\end{bmatrix}
+$$
+算子的输入和输出 tensor 的数据格式一般都是：`[N,C,H,W]`。假设 $x$ 的维度为：$[n,c_1,h_1,w_1]$，$y$ 的维度为：$[n,c_2,h_2,w_2]$，那么卷积 filter 的维度为：$[c_1,c_2,h,w]$。因为该卷积并没有工具可供偏导上的推导，因此，我们可以通过 $im2col$ 方法，重新组织输入和输出的数据排放，来实现求偏置。
+
+首先我们先来推导一下 `filter` 和输入输出的关系，设有 $stride_h,stride_w,padding_h,padding_w$ ，则有：
+$$
+h_2 = \frac{h_1 + 2 * padding_h - h}{stride_h} ＋ 1 \\
+w_2 = \frac{w_1 + 2 * padding_w - w}{stride_w} + 1
+$$
+如下图所示，首先将` input` 进行 `im2col` 操作：
+
+![img](./picture/im2col_corrected.png)
+
+我们将卷积核`kernel`组织成如下形式：
+
+![im2colmult_corrected](./picture/im2colmult_corrected1.png)
+
+在这种方法下，先忽略 `batch` 的信息，我们设 $x$ 为左边输入 `im2col` 后的矩阵，也就是 `Input Matrix`，$filter_i$为第 $i$ 个卷积核（包括通道数，三维），$y_i$ 为输出第 $i$ 个通道的矩阵，$b_i$ 为第 i 个卷积核的 $1*1$ 偏置，如下图所示：
+
+![](./picture/conv%E7%AE%97%E5%AD%90.jpg "conv算子")
+
+因此有：
+$$
+y_i = x * filter_i + 1_{h_2*1}b_i1_{1*w_2}
+$$
+因此接下来的 $X,W,B$ 计算便变得非常简单，参考上面 `matrix_mul_add` 的实现，有：
+$$
+B_i =\frac{\partial g}{\partial b_i} = 1_{1*h_2}\frac{\partial g}{\partial y_i}1_{w_2*1} =1_{1*h_2}Y_{h_2*w_2}1_{w_2*1}\\
+X_i = \frac{\partial g}{\partial x_i} = \frac{\partial g}{\partial y_i} filter_i^T= Y_ifilter_i^T \\
+W_i = \frac{\partial g}{\partial filter_i} = x_i^T\frac{\partial g}{\partial y_i} = x_i^T Y_i \\
+$$
+多个 `batch` 直接 `for n in [1:N]:` 即可。但我们将这些求出来之后，只需要将 $X_i$ 通过 `col2im` 加和成原来的 `shape` 即可。
\ No newline at end of file
--- a/document/blog/config.md
+++ b/document/blog/config.md
+# vscode 配置
+
+## 1. clangd的配置
+
+1. 首先安装`clangd`插件
+2. 然后修改`clangd`插件的`Arguments`，添加几项内容：
+   1. `-query-driver=[clang++的地址]`，这是为了让`clangd`找到标准库头文件。
+   2. `--compile-commands-dir=${workspaceFolder}/build`，用cmake的同学，添加这项内容，`clangd`会在该目录寻找`compile_commands.json`文件，该文件囊括了该项目所有调用关系，便于`clangd`进行自动补全和跳转。`compile_commands.json`文件可以在`CMakeFiles.txt`中自动生成，只需要我们在`CMakeFiles.txt`中加上一句：``set(CMAKE_EXPORT_COMPILE_COMMANDS ON)``，执行`cmake`的时候就会在对应`build`目录中生成`compile_commands.json`。
--- a/document/blog/drawio/active_kernal.drawio
+++ b/document/blog/drawio/active_kernal.drawio
+<mxfile host="65bd71144e">
+    <diagram id="n6amJyl6S3Y0V4u54FsE" name="第 1 页">
+        <mxGraphModel dx="766" dy="614" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
+            <root>
+                <mxCell id="0"/>
+                <mxCell id="1" parent="0"/>
+                <mxCell id="2" value="active" style="ellipse;whiteSpace=wrap;html=1;" vertex="1" parent="1">
+                    <mxGeometry x="346" y="243.5" width="120" height="80" as="geometry"/>
+                </mxCell>
+                <mxCell id="3" value="" style="endArrow=classic;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1" target="2">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="288" y="283.5" as="sourcePoint"/>
+                        <mxPoint x="349" y="284" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="4" value="" style="endArrow=classic;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="466" y="283" as="sourcePoint"/>
+                        <mxPoint x="524" y="283" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="5" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;entryX=1.003;entryY=0.685;entryDx=0;entryDy=0;exitX=0.017;exitY=0.625;exitDx=0;exitDy=0;entryPerimeter=0;exitPerimeter=0;" edge="1" parent="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="526.04" y="297" as="sourcePoint"/>
+                        <mxPoint x="463" y="297.54999999999995" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="6" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;entryX=1.003;entryY=0.685;entryDx=0;entryDy=0;exitX=0.017;exitY=0.625;exitDx=0;exitDy=0;entryPerimeter=0;exitPerimeter=0;" edge="1" parent="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="346" y="297" as="sourcePoint"/>
+                        <mxPoint x="282.96000000000004" y="297.54999999999995" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="7" value="&lt;span style=&quot;font-size: 10px&quot;&gt;x&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="307" y="258.5" width="20" height="20" as="geometry"/>
+                </mxCell>
+                <mxCell id="8" value="&lt;span style=&quot;font-size: 10px&quot;&gt;y&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="485" y="258.5" width="20" height="20" as="geometry"/>
+                </mxCell>
+                <mxCell id="9" value="&lt;span style=&quot;font-size: 10px&quot;&gt;Y&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="486" y="305" width="20" height="20" as="geometry"/>
+                </mxCell>
+                <mxCell id="10" value="&lt;span style=&quot;font-size: 10px&quot;&gt;X&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="307.5" y="302.5" width="20" height="20" as="geometry"/>
+                </mxCell>
+            </root>
+        </mxGraphModel>
+    </diagram>
+</mxfile>
\ No newline at end of file
--- a/document/blog/drawio/backward_infer.drawio
+++ b/document/blog/drawio/backward_infer.drawio
+<mxfile host="Electron" modified="2021-07-24T10:12:09.777Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" version="14.6.13" etag="QauPqEfhrBmr35lMVHWP" type="device"><diagram id="VRPE0WlrHYQ1c8PPHyiw" name="第 1 页">7VzLcqM6EP0aL8elJ49lXjN3kamaqizmsZkihtjcwcYX47E9X38FBoNacoKJwCTjbGIakE2fo9OtlsSI3sy3nxJvOfsc+0E0IsjfjujtiBBMEBf/Mstub7EZ2xumSejvTagyPIR/guLO0roO/WBV2PamNI6jNFzKxkm8WASTVLJ5SRJv5Mue4siXDEtvGiiGh4kXqdavoZ/O9laH2JX9nyCczspvxpa7PzP3youLJ1nNPD/e1Ez0bkRvkjhO95/m25sgypwn++XjkbOHH5YEi7TJDWR/w28vWhfPti1+WLornzaJ1ws/yG5AI3q9mYVp8LD0JtnZjcBX2GbpPBJHWHx8CqPoJo7iJL+XIuR4KLttlSbxr0A6w+38zFO8SGv2p/xP2NVHKZ7ud5CkwbZmKh7tUxDPgzTZiUuKs+XjFTQjvODdpgZawbxZDS9e4OgVNJkeGq48KT4UztQ7liqO3YyIFaWZK9ZL8XGafcSlTTRXMxuGgKPgSQ8Bvr2+6RYC7FgSBtSmvWHAFAxE/0vC7c/5Ovrp+b7iZnG7kJDgZRebcIzlyORkdKzSE5dCUfeNg17vG674RvXGwr/KxFIcTSJvtQonshPEsye7bwUn84Pv2YF4iuLwdls/ebsrj7Zh+q1sQ3yu3SWOqpuyg/Keow5fxetkEkhilnrJNEglCgS+JOgqKDWXc43HS1sSRF4a/pbDgA6G4hu+xKH4vQfMbS5BzggAcv8wxU111QbtYCI3RG3Q0N4FSkM5KQ5P3YgnlkmeVNT4Xj/3Ak8qbnyvU6M9T+jQeYIRA/haLYliQfUFDZkjiq0QxZvkLjinyFImP3/PIusY6DwtxLKtMDfqPEztPPagOo/Iw2XMKWvXeShz5YYc0JC5zuMqRNFli+TdZ4ucydj1mS2Wg8thpoucyCGBMLs/HcNYcc3bFLKBSBQDYZk4tJ1EKayADZmTKKyWCk7nQMtMcOh4cgfma7glns4hPymbInZXiG781f38aeWmyb334/Pd44f48/0HrqqgCrPw+kNxGCfpLJ7GCy+6q6zXcgSqrrmP42WB979Bmu6K0p63TmOdYgJ4X8xNsCazx1zPjMaQNxZJteQzgAzYRiAb6jVuqBWYtxk3JELRQWmPxWAIAILRVHtsxGFDpRZ1EE3U+lMUr1Zn7SpOSfmzdBW10LJTvPF2K/GOK1c7e03s1dLE6So0tBpWyRc11A1GmRziAsyddsrkgBE9ZZ0NxLGJks3Q6uLH06LBcMXGQB+gpjbliouw3BBUEINcUas2j5c5vpxvauDsStyJOl7RgUDePQiwdNYrCAbrQ3KkxAbUzx2U0okwOHbrfzYQvpaloMNsQzXV0ZXwleh2kVBVYfGUhMoATUohkYIkGhR1GAI667ac2WAcNNRdQqVZCJMN9H7uj18lvyYiF+Kg13BHM0HYmW6aqI8MIdscSAfBCIEhvGPBGkbjqXOFGmpbBruJgfU4IB3xCEKE69IR6yNCqFsx1Yw4tOVm5gyMQEwmEGUtq2mCPqCc5nY2EDGxSOcV5GlTmMWniVUj0p1CsAbU5PScTITFSIpbFk8wBppocdIZEQ1U2loQ8UCqMUJUJpblNI2DaIyIxGTLbsDlL0ESCgcFSe0KaGtE3qHNKrj22AKTmgi3Y6DFhG9rfwQ2O+asNh7qLsYaqO2dkZ3YlulJeqXnsEZCnCBITyW+Nh4MWRQ21Bch1QJin3FbUIhKlMoe9VlKadhTG8dbli3TG2H2Er11JD2JkPUwPqxCD5yYpazlSIS7YB7FAqmAOUbSJqs/upNIhUNCM8mLEnkSh+p8IQMTNRdW8FoqmgWWjyt1IoOEMVD5HTZhBkIOApd5MD7mdi1KtS0bMyZc5hzLzgR1RNpSRUOnO+khl+zslOxsILxUdli0TcOIQ8auVRsmKOldT1kZVSvW/WZl0koR4RP3zSRlw9pFJTL7sU2PDzwJH7esqWTiK7SwahluxyPa0x1wVTeBUEx+L72FRFrrv3W2qzyfg/6wyhfeXokLMFpuq5PlxPifUxYyVNb8O49MrafBNpV7iReF00XWhQSnMqJdZxM34cSLrooT89D39+uJA/GDvce8qYyNy8x3uTf59YjfZm2t03j/UHnTckdbxIvgOQ6fsKEMydO1hNua+SLdIsFSRV61qVxX6DUBt3eBWws3c/g54daVU7uCm1zgJg7DAG7dbHBncOvqk12J+QVuMU6Glb9+e7eu+mcC7ulfBSJGGIZkV7fH21JRLFeGveodI2rF7NM7dTRW0lx1Db+us8CZy1Zu1tWZTHSWq4s2HgmF4F0lLlbBxt0oY7mS1TjYPy5gNwqEFDm9hUGmK7d01bMvYxp1TEPLSlsfPbureoWuZ1/AzusVcEN5tt6iv96trmu8PgWq94gJgSVzrAmtmhXHJjapM7WCpMPjaFB8j3hYcONzn3ioJZ6vl/4BFItqApTdER5qDUaHx1/WP8jZ8Hh2TS/cU/LoTX5tvMRXYDjxLQKKazQOPD5wcuRcmpXLbft4i4B+IbOJ3Xmv3GVy+g4DE+9+eYY955qUtDmvb/+zwcCLgZJU4y3ymD3f0JE5RwG6t6tdVogMpJyJFwt1Nah/x5MZrxMi+JLYchlhD2n/MxsqNC9IGoZw22Bqt0vhFofVO973Pax6Uz69+x8=</diagram></mxfile>
\ No newline at end of file
--- a/document/blog/drawio/backward_use.drawio
+++ b/document/blog/drawio/backward_use.drawio
--- a/document/blog/drawio/conv_kernal.drawio
+++ b/document/blog/drawio/conv_kernal.drawio
+<mxfile host="Electron" modified="2021-07-24T08:37:56.652Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" etag="8d7QEdMF2TY4FIDZJ5SA" version="14.6.13" type="device"><diagram id="AKQbFceJ3HzJovVuOdc7" name="第 1 页">5Zpbc5s6EMc/jR/rkRAC8xjbTU7PnDOTaR5yeSMg20wx8gj51k9fyQiDJFJi4kvq+iGDVmix/vplVyvcQ6P55o6Fi9n/NCZpzwHxpofGPccZYCT+SsO2MGDHLwxTlsSFCVaGh+QnUUagrMskJrl2I6c05clCN0Y0y0jENVvIGF3rt01oqj91EU6JZXiIwtS2PiYxn6lplbOQ9n9IMp2VT4ZeUPTMw/JmNZN8FsZ0XTOhrz00YpTy4mq+GZFUalfqUoy7faN3/8UYyfh7Btyn09nj8t+HRfRyf/c9T7/9t5p/UV5WYbpUE96ob8u3pQSMLrOYSC+gh4brWcLJwyKMZO9arLmwzfg8FS0oLidJmo5oStluLAJgEAI5LOeM/iBaD/Z3PROa8Zp9svsIuz2/8ssSxsmmZlLzvSN0TjjbilvKXqy0V/A5GBftdW0p3cI0q60iVqsbKnime8+VvuJCSXyA3I4l9/rIcmNAJs1yw/FwdFq5EXQ1uZGPLis3suQWYWJlKS48iXBC2tU+gkZO4OlIuqhvQwnLoFGXaQBOJJNryWRLlMU3MpqKVpSGeZ5EujJCELZ9UszuGs+yIaammuNNvXO8LVubhD+VPsR1bZRoVYNkoxzz5irkdMki0h7teMimhLdzQ2ItN9hrWlsx3LBgpY2RNOTJSs8oTauonnBPEzGzChmANGRcxwChmLcaVc8ApiOsO0K+4agQxnK0g2o/7e6c4WNyVqH1XO9r4axi67mO1pE5c/5MzpBj4OF15Cww84Dh6MScBR/nzMiroQOAg5vyqncLANAJ8zTE+oHYEf4OM9m4JywRsyasDq/y5vl6SAUiy7bALlumy+40fy5Kke/2fSOOObg/6Iaq2KL0g2AA9h9skOs0dp8J5LIIuhTJIvHUSfZ82J1kqFPst+0MPsxra/SFn4prx9NrFbFj75rlDUfeebM8tGvKw9N8uS0U0U7fGroYHczgtYQ+FxnlrFkRvBcRDFscnRoRuw4+R1yrkmqAXSMcOUfNqX/Yzs/kAQHYDSzXPAEAYsPtn5ctu+jfWnCJYp3rOIVpMs0ka2Ip5foOZUmfRGF6ozrmSRzL4UNG8uRn+LpzJSFYyJns5oaHPTyWvpac5sVJJrSIzGhGjnOU4JqnW2Wgr0HTdI5glo9HO0eA9kHC898hPQJNpzjnFd+uru2j3KsQ39wpXZ57z5L+6e+Q/jNw71viP16n+Mgb6OK7NvforNIPLOnt1xnXIT1ql/5U1H/LbrfZy5dkdfOC4vTJH8EpaXhv92opf9EXSZa2DSvw7hdJENlyn+pFUqPcR6hXmk+gf3+g0msqElor2beBudjZxsDI2G7HwhVh8+3ZyQrXRhXt2uJgDg6vW99LweVOZM3VRR1X13AEg/Ourr2DtoPqZ0lnH4uvRjrb/9rmUunM3kAPr1R5H55LedGsflxU/ItUv9BCX38B</diagram></mxfile>
\ No newline at end of file
--- a/document/blog/drawio/full_connect.drawio
+++ b/document/blog/drawio/full_connect.drawio
+<mxfile host="Electron" modified="2021-07-24T10:01:10.321Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" version="14.6.13" etag="9OXYZSvp1rAgYjcOPzG5" type="device"><diagram id="P6mYd-6YqKYc33jBGSUm" name="第 1 页">5VtNc6M4EP01vrr0DRx3kpnZy1ZNVQ47OaVYI8fUYpPCJLb3169swKAW3mBFYGXWl6AW6ljdr5+6GzOjd+v99yJ+Wf2RJzKbEZTsZ/R+Rghmgqo/R8mhkkSoFjwXaVKJUCt4SP+R9cpG+pomclvLKlGZ51mZvujCRb7ZyEWpyeKiyHf6bcs8SzTBS/wsDcHDIs5M6Z9pUq4qaUiCVv67TJ9XzX/GIqpm1nFzc72T7SpO8l1HRL/O6F2R52V1td7fyexoPN0u3y7Mnr9YITflkAWkWvAWZ6/13vb1FysPzW6L/HWTyOMCNKNfdqu0lA8v8eI4u1P+VbJVuc7UCKvLZZpld3mWF6e1FKEwRsdl27LI/5baDA9OM8t8U3bky9NHyc2t1Lt7k0Up9x1RvbXvMl/LsjioW+rZZns1zAjn1XjXcRqrRKuOv3jtx7iGyfNZcWtJdVEbs9+w1DDszrFhOZLLfsPi+y934xoWh0KzLA3oZJZlhmVVVBXp/mn9mj3FSWKYWS1XxCDfN7ELw4hQhxyjcxN0uAn/rm1C9HHbcMM2pjU2yW9HClSjRRZvt+lCN4Lae3H4WWPyNHg8DtQu6uH9vjt5f2hG+7T82ehQ151VatQuOg6aNRcNvs1fi4XUKKqMi2dZahCQiUbTplM6Juc9Fm9khcziMn3Tyb3PDfV/+JGn6vuefR5wzeWMAEdWm6kXdbkY6MFEV0QDoKgygaHoBIrzrgfhRLjESQuNx+7cOzhpsfHYhYY9TqjvOMGIAf8KS6AIyL5AkTugBAZQ4sXJBLckWcr0/U9MsqGD4LEgS1tiHhQ8zAyewKvgUdm17nPK7IKHskhXFAJF7oInMoDyK+WAnOkemTIHbApBP5NATnSiJyyYjp0wNkzzOenJE+Jh4LAlIbUjHgMVUJE74sFmWX89BizzO9/9yUOYhWFLf4bnrKNRRYLRPGr2EzxIxAIEDuVJic5sBHxOousmXY2fPQkWoTLrqPMBzY0Q4H1o6ASIQ0VNKI0QOmZTJMu325sGTtgEwE0Cx6z+D4Y1Pm/TNwQonTQvNVP+yQ7e0RorDV40nuJe8VRIIuDz0I6ZQlBmUjZadUjMGuYDWPGkWYt7unCeYSXAgB8gpw7FSoSwrggyiEOsmEXdX44p26fHSZiax+FYlN1E3S9qWtilmdS0ZtFyPcP1nmr440zlWVmqjiwt044CQFKWXYdzu7rtlY9GUi4KsgvJDxp4oLmHCek50JBX0GEIsGdk2RpnHCgaMfnpL8qeqvGH6NfFeYQ4iBoe9jxhGo03nT6evWFm6EmAYIRAuR0K2G8Y/OzVgIapy2GYmM9fr0YCSEdighDhfemI+IYQ0jpceI4Q1fEjwqEIQhqCggFNtR+ySJVxZHEVQfMegvar4sCI6fijzLJxptAHOmfReDWHg16GBfbOBzkiVENQjbzLEFIDKwR5hhbYG6TYtpdBQH2KRkv9qINexs14SiFNB5oNV9mjz7OOfxTMBXhChrAdAAWN9JoGqp1z1pkd7RSlDh6J3xCdONDhSSaFp1+1DicIwtM4AgeXO4L8BzyV2qngaXagJj1oBThoj1u99qDt1O1CBDrYEWbvgd06+2vg2c3+Iq8QKxhgU2ZZeXDInwLkBQ4R6aBx9wHCNDCkGJS8S5hXYaiLl2ZzngCGw6fstsWCAL83NvpCDgHjoPXnN2A8AQcBbEIYn3PL+kDpUmYK0flDIFxU4tI5AUerNKmD11b+T/mZJ1g0foZvm4ipSnUeiRaHyEjwpsrEbtNuazMx7ZcbyibRp0nE/HrVRuX284CiS9RGCZ9bkyZXCXKHNOE7W8eawlVipobtq8DV7e0L1fTrvw==</diagram></mxfile>
\ No newline at end of file
--- a/document/blog/drawio/full_connect_net.drawio
+++ b/document/blog/drawio/full_connect_net.drawio
+<mxfile host="Electron" modified="2021-07-23T15:48:03.192Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" version="14.6.13" etag="Su_nRb-hbBTqCLtDo5in" type="device"><diagram id="fUrjSOwtcRa29wEhY1VY" name="第 1 页">5Zpdb5swFIZ/TS43xR8QctmvdZo0qVInVdqdC25gJTgyTpP0188OEILtLV5LXLJchRxsA8+xz3uOYYSu5utbThbpd5bQfATHyXqErkcQgjHE8kdZNpUlRLAyzHiW1I1aw332SpuetXWZJbTsNBSM5SJbdI0xKwoai46NcM5W3WZPLO9edUFm1DDcxyQ3rQ9ZItLKGsFJa/9Ks1naXBmE0+rMnDSN6ycpU5Kw1Z4J3YzQFWdMVEfz9RXNFbyGS9Xvyx/O7m6M00K4dEBVhxeSL+tnq+9LbJqH5WxZJFS1H4/Q5SrNBL1fkFidXUn3Slsq5rn8B+Shef36ll4oF3S9Z6rv55ayORV8I5s0Z1HNZtM4vf6/alHj2pTuUYZRbSS1d2e7oVsA8qBmYOeBD/OQHeQ8o4dZkHJRTb6nbK349QMn6MIBJhxkgYN6YBOcGpvAH5vw1NhM/bGZOLApkgsVl+W/ghWKUULKdBt0QBeOst8RISgvthY4RtJaCs6ed7EY/o0ZTTqx3SS2RySwEGlsnOZEZC9dRbBhqq9wxzJ5J61D8PRz1yUIa6xLtuQxrfvtx/CDQxmhUBA+o8IYauu53aM7OTM6sYm+m8EeJvr0MBvfYgrDw2IaHUtMG20a7GSB4cepKQCnBsejnAJ4anA86ilwSNrPT1DhpDdBNYc6oqCCoZcc+lT3qajAoebwLakY4e7Sx4HBwyqp4z6ADL3QwGiqqYZJ52izxaXSGBScqUc4Q8/cdTgQeYTjkLqfn6TiAHU8ghqP/Kug6gPBiTZQf3IKHdLqVjHA4aluuu0py/OHhjZWrUmcLjm9VV2vo9ZwUcy2l5QCUfW6Yjnj23tA4JEAqkYrn6mI0/7kaZeJNjEGmnJtmzNQz5beso4a2GdLH4XdmQ6CqRN90Ee9bdmF/DGCYS7Ugy4X8nCmDkFjk+PtmQ1HyUcWWtjPs5kKabEERCXKSwUmi0l+UZ+YZ0miul9yWmav5HE7lErLFmppbx8tuBwF12qspWBl9WandXTjojrA9rEctP2giVvyCvtwh5mO2NwBz8kdejHh0x1mAmRzBzondxgJqeMGWB/++Pkt+nWz+RF+4gWl4OE5KIKLT8ihvnlbbmpAsaD7c1gHmqgCMzeFXjk5lDpD4GSp/vxycqh6hsDJUgj65eRQAA2Ak60m9MoJu+zaH6lQ1KpCJ45+CkUEett7NYc64t6r3cUONYzzRsnu+57xfrEx1iqSjqC/b8GMoYbP9mIisKyY8GhLxpT0x/+gKnmnm6KOkzymwXYfmemEzUenVqq8y0c4Al2NbvbeP8xJLpuUQw1MeBJqgQlgx8jUx3sTK9BG/QaW9OBIe8EUmqC8Jj2BQ9IzAE7Q8m2LX04OmcMQOFkWnl9OH/gBw3CTaDxFehL95vctxlD9vXGRf9vP06vm7Uf+6OY3</diagram></mxfile>
\ No newline at end of file
--- a/document/blog/drawio/loss_kernal.drawio
+++ b/document/blog/drawio/loss_kernal.drawio
+<mxfile host="65bd71144e">
+    <diagram id="tm5XjXS1Pe2yocP4-5H7" name="第 1 页">
+        <mxGraphModel dx="766" dy="614" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
+            <root>
+                <mxCell id="0"/>
+                <mxCell id="1" parent="0"/>
+                <mxCell id="2" value="loss" style="ellipse;whiteSpace=wrap;html=1;" vertex="1" parent="1">
+                    <mxGeometry x="389" y="220" width="120" height="80" as="geometry"/>
+                </mxCell>
+                <mxCell id="3" value="y" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#008a00;strokeColor=#005700;fontColor=#ffffff;" vertex="1" parent="1">
+                    <mxGeometry x="407" y="346" width="84" height="57" as="geometry"/>
+                </mxCell>
+                <mxCell id="4" value="" style="endArrow=classic;html=1;entryX=0.5;entryY=1;entryDx=0;entryDy=0;exitX=0.5;exitY=0;exitDx=0;exitDy=0;" edge="1" parent="1" source="3" target="2">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="338" y="351" as="sourcePoint"/>
+                        <mxPoint x="391" y="317" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="5" value="" style="endArrow=classic;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;" edge="1" parent="1" target="2">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="331" y="260" as="sourcePoint"/>
+                        <mxPoint x="410" y="330" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="6" value="" style="endArrow=classic;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;" edge="1" parent="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="509" y="259.5" as="sourcePoint"/>
+                        <mxPoint x="567" y="259.5" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="7" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;entryX=1.003;entryY=0.685;entryDx=0;entryDy=0;exitX=0;exitY=0.75;exitDx=0;exitDy=0;entryPerimeter=0;" edge="1" parent="1" target="2">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="567" y="274.25" as="sourcePoint"/>
+                        <mxPoint x="564" y="270" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="8" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;exitX=0.023;exitY=0.685;exitDx=0;exitDy=0;exitPerimeter=0;" edge="1" parent="1" source="2">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="389" y="291" as="sourcePoint"/>
+                        <mxPoint x="330" y="276" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="9" value="&lt;span style=&quot;font-size: 10px&quot;&gt;x&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="350" y="232.5" width="20" height="20" as="geometry"/>
+                </mxCell>
+                <mxCell id="10" value="&lt;span style=&quot;font-size: 10px&quot;&gt;g&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="526" y="232.5" width="16" height="19" as="geometry"/>
+                </mxCell>
+                <mxCell id="11" value="G" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="528" y="281" width="20" height="18" as="geometry"/>
+                </mxCell>
+                <mxCell id="12" value="&lt;span style=&quot;font-size: 10px&quot;&gt;X&lt;/span&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="351" y="282" width="20" height="20" as="geometry"/>
+                </mxCell>
+            </root>
+        </mxGraphModel>
+    </diagram>
+</mxfile>
\ No newline at end of file
--- a/document/blog/drawio/matrix_mul_add.drawio
+++ b/document/blog/drawio/matrix_mul_add.drawio
+<mxfile host="65bd71144e">
+    <diagram id="lEVSFP9Le3XozFU6bgV3" name="第 1 页">
+        <mxGraphModel dx="438" dy="351" grid="0" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" math="0" shadow="0">
+            <root>
+                <mxCell id="0"/>
+                <mxCell id="1" parent="0"/>
+                <mxCell id="3" value="w" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#0050ef;strokeColor=#001DBC;fontColor=#ffffff;" parent="1" vertex="1">
+                    <mxGeometry x="436" y="373" width="84" height="57" as="geometry"/>
+                </mxCell>
+                <mxCell id="4" value="matrix_mul_add" style="ellipse;whiteSpace=wrap;html=1;" parent="1" vertex="1">
+                    <mxGeometry x="418" y="247" width="120" height="80" as="geometry"/>
+                </mxCell>
+                <mxCell id="5" value="" style="endArrow=classic;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" parent="1" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="360" y="283" as="sourcePoint"/>
+                        <mxPoint x="418" y="283" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="6" value="" style="endArrow=classic;html=1;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" parent="1" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="478" y="371" as="sourcePoint"/>
+                        <mxPoint x="477.5" y="327" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="7" value="b" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#0050ef;strokeColor=#001DBC;fontColor=#ffffff;" parent="1" vertex="1">
+                    <mxGeometry x="436" y="137" width="84" height="57" as="geometry"/>
+                </mxCell>
+                <mxCell id="8" value="" style="endArrow=classic;html=1;entryX=0.5;entryY=0;entryDx=0;entryDy=0;exitX=0.5;exitY=1;exitDx=0;exitDy=0;" parent="1" source="7" target="4" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="300" y="194" as="sourcePoint"/>
+                        <mxPoint x="350" y="144" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="9" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;entryX=1.003;entryY=0.685;entryDx=0;entryDy=0;exitX=0.017;exitY=0.625;exitDx=0;exitDy=0;entryPerimeter=0;exitPerimeter=0;" parent="1" source="4" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="414.64" y="297" as="sourcePoint"/>
+                        <mxPoint x="357" y="297.54999999999995" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="10" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;exitX=0.623;exitY=0.995;exitDx=0;exitDy=0;exitPerimeter=0;entryX=0.667;entryY=0.014;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" source="4" target="3" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="538" y="345" as="sourcePoint"/>
+                        <mxPoint x="493" y="368" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="11" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;entryX=0.667;entryY=1.025;entryDx=0;entryDy=0;entryPerimeter=0;" parent="1" target="7" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="492" y="247" as="sourcePoint"/>
+                        <mxPoint x="498" y="194" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="12" value="x" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#0050ef;strokeColor=#001DBC;fontColor=#ffffff;" parent="1" vertex="1">
+                    <mxGeometry x="276" y="261" width="84" height="57" as="geometry"/>
+                </mxCell>
+                <mxCell id="13" value="" style="endArrow=classic;html=1;fillColor=#a20025;strokeColor=#6F0000;" parent="1" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="604" y="300" as="sourcePoint"/>
+                        <mxPoint x="537" y="300" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="14" value="残差" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" parent="13" vertex="1" connectable="0">
+                    <mxGeometry x="0.2488" y="-1" relative="1" as="geometry">
+                        <mxPoint as="offset"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="16" value="" style="endArrow=classic;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" parent="1" edge="1">
+                    <mxGeometry width="50" height="50" relative="1" as="geometry">
+                        <mxPoint x="538" y="286.5" as="sourcePoint"/>
+                        <mxPoint x="596" y="286.5" as="targetPoint"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="17" value="y" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" parent="16" vertex="1" connectable="0">
+                    <mxGeometry x="0.3034" y="3" relative="1" as="geometry">
+                        <mxPoint x="-8" y="-7" as="offset"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="18" value="B" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="495" y="212" width="18" height="18" as="geometry"/>
+                </mxCell>
+                <mxCell id="19" value="Y" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="1">
+                    <mxGeometry x="568" y="312" as="geometry">
+                        <mxPoint y="-2" as="offset"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="20" value="W" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="495" y="341" width="21" height="18" as="geometry"/>
+                </mxCell>
+                <mxCell id="21" value="X" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;" vertex="1" parent="1">
+                    <mxGeometry x="383" y="298" width="18" height="18" as="geometry"/>
+                </mxCell>
+            </root>
+        </mxGraphModel>
+    </diagram>
+</mxfile>
\ No newline at end of file
--- a/document/blog/drawio/pooling_kernal.drawio
+++ b/document/blog/drawio/pooling_kernal.drawio
+<mxfile host="Electron" modified="2021-07-24T03:29:19.616Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" etag="2Se5a_g2G0A39Azb1jwc" version="14.6.13" type="device"><diagram id="hMD6IDgKTZdyEJsy110o" name="第 1 页">5ZhRb9owEMc/TR6HnDgJ8Fig3aR1UreqWvvoJSbxlsTIMSX0088mNoltOiiDdlp5IXeOL7n//TjbeHBaNh8ZWuRfaIoLLwBp48GZFwQ+CELxJT3r1hPDoHVkjKTqps5xS56wnqm8S5Li2riRU1pwsjCdCa0qnHDDhxijK/O2OS3Mpy5Qhh3HbYIK1/udpDxvvaNg2Pk/YZLl+sl+PG5HSqRvVpnUOUrpqueClx6cMkp5e1U2U1xI8bQu7byrZ0a3L8ZwxQ+ZMKuz8nqU/Pw8Xl1X3/Ad+3qXfFDFeETFUiXcqLflay0Bo8sqxTIK8OBklROObxcokaMrUXThy3lZCMsXl3NSFFNaULaZCwEYISCn1ZzRX9gYiYabkTmteM8/33yEX70ZZhw3z6bsb4UUBGJaYs7W4hY9IVLaK/iCKGrtVa+UCtC8V8VIVRcpeLJt5E5fcaEkfoHcoSP3QsBMqswRXQQTiOP9gp9ApmAcmzKFcOAK5WuQ+0qNwJmUihylXImq9EL+woWVFKiuSWIqI7Jn63uF7cZ4kIZITZmzpj84W2urIfxexxDXvVnC6iZJQ89pXw6nTjOxaiASoEuW4P0/SY5Yhvk+ltya9ioW7SiY9jFcIE4ezdfdVUX1hBtKRCIdMgAayISBBUKbpprV70p2oMgMBIdWoFYHJ9AGqm3ax3M2+nvOrJaHAgCCaFfLi68AAH3CwEDI2IcsHvp/xEwaN5gRkTVmfXi3vHaYD/dxfkJmwwOZDd6U2dhcDcQSdCyzVqD4dZkdn6A36h43AL7Z58IIvhjBU5P0VoSE0Nov2MvboYRE/p5AZyZEP/6V29p2wR2Mo9DqRsG+VVdax4P1by+XNg8Q+MeBFfqh3cMG9m713Gz5DltrBy6xGeUmTqggWSVZE5WT9Z3ILSsRB60LNVCSNJXTJwzX5An92ISSECxkJpvcookXzWSsJad1e1T0HSIrWuHT7ItD+/ig+3wPml2bYnsvdLJNse8e1x7eh/QQ7DqSvK740BHfPSv/F+LbG6W35949N9+/D+nPyr0wu7+f2uWh+xMPXv4G</diagram></mxfile>
\ No newline at end of file
--- a/document/blog/drawio/real_computer_graph.drawio
+++ b/document/blog/drawio/real_computer_graph.drawio
+<mxfile host="Electron" modified="2021-07-24T10:07:21.188Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.6.13 Chrome/89.0.4389.128 Electron/12.0.7 Safari/537.36" etag="rtFTTx2sm69Vo7nGtpzY" version="14.6.13" type="device"><diagram id="2mTccy_lQpPIlvz1nAHy" name="第 1 页">7VzLkps4FP0aL+PSG1j2I8ksZqoylUUemy5i1DY12DgYp+18/Ug8DJLoNqYBuR33pi0BMtx77rlXRzITfLfcfUz89eKfOODRBIFgN8H3E4QgYVj8kz37vMcDRcc8CYPipKrjc/ibF52g6N2GAd8oJ6ZxHKXhWu2cxasVn6VKn58k8ZN62mMcqd+69ufc6Pg88yOz90sYpIu810VO1f8XD+eL8psh8/IjS788uXiSzcIP4qdaF34/wXdJHKf5p+XujkfSeKVd8us+PHP0cGMJX6WtLsCPX34+fHS3N38/RD9v9uzf6Ps7TPJhfvnRtnjiXXG76b60QRJvVwGXw4AJvn1ahCn/vPZn8uiT8LroW6TLSLSg+PgYRtFdHMVJdi0GwPWBvGyTJvF/XDlCnezIY7xKa/2P2Z/oL+6MJynfPfvM8GBJAUEeL3ma7MUpxQXydHlFAT5Ead5+qrmysMCi5kVaeNcvwDM/DFzZV3woTHyKualh7qcJYlEqDbRdi49z+RGWfeJLat09O4YC/tjsGHh/ezesY6DLFM9gB1v2DDM8IyI4CXcPy2304AeBYXwxqCAhftzwfZiLuSqQCZ6aUIYl1dQt5oKhLOYYFjNttApuJAmL1izyN5twpppGPH6y/1rgN2t8kw3xbEXzflc/eL8vW7sw/VqOIT7XrhKt6iLZKK/Jb44HBt9rThAPEG+TGX/p0Qtkpn4y5+lRVJlurTmNNvis7Et45KfhL/WGmxxZfMOnOBSPckCNQxXQEKRBIX/O4qJ65tDGgUgdCDvaQLkdjIEyWB2e+hVIc/tEWgWub/VjR5BWoetbHVyDI42+CaRBQDSEsI5QY3pe0AYaGmqeATV/lhnGJv1jolrlHOifgB6CsgONd00ZfQYlaxuUns2gFLMUFTVl2jo1KDHx1IFcbaCBg5JAA2pNVTO6+KqZEtWj1qvm8n7Os2qmSM1KiDi2SRNfJmnaojiilQvIxd0ozoCKPtDQFGfqMKcDo2OJe+5Opq5eXMKOTnYPZVM5FHLGdbOp/5xBeekArVCwz5OmGnMRPNmiZswhYivWGNF5UAuQtrHmAKoPVMbeWLFm6lNRvNlYjTS3jJjziTRTW9kbJnq7CwCupwqn9itmU2AYLdFb1rJKqB2nQMcmBbrI0yDjdqNAV5t/YzLutJn2odC8TYGetJVN7ULNgRo76TTfFmoegOpAOn8NDTVToflxXdfMrGUm+FHzDTUVmibPoIv3jK6d2fdMj1qQmtHhWDRLreoCIjNPvfqfo5FpRy1I8LIyrqOOi8i40hAdUBqqEvcpFeN4+EJt0ziyCUQCNNL3Oi60EKoNNHbFaMpTcsr8kLdflQv6yK2AartgqNuwDDouifchVJ1hjW1vIwHQFBKX6bpR670EBl7MsYYOqB72SGlVlI8AQLSpimIfAACKigmnAGAVVMxtCyugwMppIZx+4kkoLMaTATLB25jQQUBU+GLSUT8V4NUEVG/kKV0Pm646QPdQcACEFQAWwH0egaIxIAAta1KaboxhV00KaUIBGLmY7UH+tEaIApMqJLuQ4tA4tbt85DlTpq3WAtgNqkynUgCnlNTmZeOmctaDmmoRudBRoYvOELpWJ3EUAR26RsptPY9jSFEQXH1Yq0A2tdpREzvTErt8/lMTe03PYMxRwwJAciwsBi5WS6XxeLFqda6lr/Vj0nGiRT1txYxp1cnQiEZWqdnAoOBqdJSaX4fBo9CidslUp7yu0yCm/bDAkNaGhlYPIv2bgpY1MV/feETolDq1JNlVzCdEmMwFhz+k40kUVlUydkdmrh60/T+4qLQFVuP3OF0LRTEfn3qsAicwClCrlaK5HDBupahsUxKG8i6sULT7oz0xTZk6GDxHjRjRaUeVSZK351XsrW+EcuT0qMbJI8O6ac2m2BGx9lcKvtnPrXw/Q7Yx4d0mezvFjTgBgvWuOljulvh9ypaXqjf7zmf2W6R8l6oB5UfhfCWjTaBNYvJWrqmFMz+6KQ4swyCQl98mXNyw/yMbSgJ3LQ2amZjeTui9HGubxvlDZUOrMbmKV3zSy8IdBtoKOnUa1u2atr2WhNP7uh1rWpjpAwP+FQONGCAuPTsMNK1wDIUBdMUAcssfcr64fj8uBprWI4bKBVcMiAl9OaM+Hx5wmpT9PjAw/6M8C+V0UHWt1/SWAma6FnpDudbUuj9erPU1ERa4rcJKX0nuz/ZNqmwfYXVzpdZW6bURAXBMYm0ST/tAwPcrAlol17YcMBwCmvTNoTjgOs0yp1m4VBStcUCThjgUB1wRkIkt+ksO5MYZyzxgSm63p/jvEh2F9EUECE0nNexlf0nofZ2TTE2syUnPJtpLdBLTX2Ng3UmmaPXlGkka4eGGpOeM6SRTVWpy0h8WSWgsJ4lm9VrtfNWoejk5fv8/</diagram></mxfile>
\ No newline at end of file
--- a/document/blog/picture/active_kernal.png
+++ b/document/blog/picture/active_kernal.png
--- a/document/blog/picture/active_kernal.svg
+++ b/document/blog/picture/active_kernal.svg
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="260px" height="83px" viewBox="-0.5 -0.5 260 83" content="&lt;mxfile&gt;&lt;diagram id=&quot;n6amJyl6S3Y0V4u54FsE&quot; name=&quot;第 1 页&quot;&gt;7VhNc5swEP01PtYjvmR8jJ2kvXQmMzk0OaqggKYyokKOcX99V0gyCJxpJrWTtikHW/ukXaT3HvYOs2i9aT9KUpefRU75LER5O4suZ2G4wBg+NbA3AA5iAxSS5QYKeuCW/aAWRBbdspw23kIlBFes9sFMVBXNlIcRKcXOX/YguH/XmhR0AtxmhE/RLyxXpUHTcNHjnygrSnfnAC/NzIa4xfYkTUlysRtA0dUsWkshlBlt2jXlmjvHi8m7fmL2sDFJK/WchNAkPBK+tWcjmWKP1O5O7d2RIQ3YhWC1K5mitzXJ9MwO9AWsVBsOUQBDW5BKRdsnNxUcjgoWoWJDldzDEpsQxdYf1h5hHM0Tg+x6ugPHYTmgOrUYsQoXh+I9CTCwPBznJJpwMmWjyi+0jyDKOGkalvkk0Japu8H4HsYIzmCiS31O5IK9CyrY590wGGTpsE/rIpdnNkfziWVHdMMzQmRB1UD4qQIDfpMj9DpMUk46m3hP1hHO7R1uBIO9HAQO09QXOD0I7Io0YiszavOGBh6ViuLlqFTsFzKHnhTqTHA4+LN8Ef+bvngrE8QYj03wMgskTpmnCp3OAsnvW+CBcb4WXMhueURChEItZaOk+EYHM/gawTU0QDBHKPJNgNNf2cBZDs1RsPBsh8NnGO+GSgYEUTlYMcb+dKclIZ6jkUeWi5eZLcbRuNA8iZf9lZzLe/i/9/5C7016mZcaD9rL+RKj/poY+pV8uDjiQ8yVtlFNKs+Q+PtWN7KrB1GpD03Xxl/AggDVbT8Jo0J/t64MbMBUMvjE5dBYKt/XhLOi0qYHzbUzVrr9ZNCvX9iJDctznb6SFLZBvnaltH1qTUhHUbKaJZe61lYJs9WutP9oVKKiJ+px0cLXL0mP9LjHWtzwBC1ueiYN9+9KwzhN3lDD5Zk0vH9nGvq/zxF6PQXd64yTS3j3riSEn9J5MhIxPN+DCGH/fsb8f/YvuaKrnw==&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g><ellipse cx="130" cy="40.5" rx="60" ry="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 41px; margin-left: 71px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">active</div></div></div></foreignObject><text x="130" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">active</text></switch></g><path d="M 12 40.5 L 63.63 40.5" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 68.88 40.5 L 61.88 44 L 63.63 40.5 L 61.88 37 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 190 40 L 241.63 40" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 246.88 40 L 239.88 43.5 L 241.63 40 L 239.88 36.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 250.04 54 L 193.37 54.49" fill="none" stroke="#6f0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 188.12 54.54 L 195.09 50.98 L 193.37 54.49 L 195.15 57.98 Z" fill="#6f0000" stroke="#6f0000" stroke-miterlimit="10" pointer-events="all"/><path d="M 70 54 L 13.33 54.49" fill="none" stroke="#6f0000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 8.08 54.54 L 15.05 50.98 L 13.33 54.49 L 15.11 57.98 Z" fill="#6f0000" stroke="#6f0000" stroke-miterlimit="10" pointer-events="all"/><rect x="31" y="15.5" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 26px; margin-left: 41px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; "><span style="font-size: 10px">x</span></div></div></div></foreignObject><text x="41" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">x</text></switch></g><rect x="209" y="15.5" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 26px; margin-left: 219px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; "><span style="font-size: 10px">y</span></div></div></div></foreignObject><text x="219" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">y</text></switch></g><rect x="210" y="62" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 72px; margin-left: 220px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; "><span style="font-size: 10px">Y</span></div></div></div></foreignObject><text x="220" y="76" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Y</text></switch></g><rect x="31.5" y="59.5" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 70px; margin-left: 42px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; "><span style="font-size: 10px">X</span></div></div></div></foreignObject><text x="42" y="73" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">X</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://www.diagrams.net/doc/faq/svg-export-text-problems" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file