RK3568 OpenHarmony运行KubeEdge.md

# RK3568 OpenHarmony运行KubeEdge

-----------------------------
以润和DAYU200为例，展示了OpenHarmony设备上运行Kubeedge的具体过程。这里，KubeEdge所用的运行时为docker。

## OpenHarmony运行Docker的关键步骤
-----
##### 1. 准备支持Docker容器的OpenHarmony内核
- 修改内核配置：
cggroup和namespace相关特性，主要修改的文件是openharmony3.1/kernel/linux/config/linux-5.10/arch/arm64/configs/rk3568_standard_defconfig 
- 修改network内核配置：
网络主要用的是bridge模式，所以要打开相应的配置以及支持网络包forward功能。
- 修改overlay filesystem
docker使用的overlay filesystem推荐使用未加密的f2fs作为backing filesystem。

##### 2. 编译并且在设备上安装运行Openharmony

##### 3. 安装必要的工具
docker需要用到的一些必要工具有iproutes、iptables和busybox。将这些工具的静态二进制文件拷贝到OpenHarmony设备/bin/目录下，并且添加执行权限。

##### 4. 安装Docker容器引擎组件
将docker静态二进制文件拷贝到Android设备 /bin/ 目录下，并添加执行权限

##### 5. 挂载OpenHarmony系统资源
创建OpenHarmony的/etc/cgroups.json文件，挂载所有的cgroup子系统。

##### 6. OpenHarmony环境设置
- 创建docker运行所需要的目录
```
# 开放root权限
mount -o rw,remount   -t  auto /

# 创建相关目录
mkdir /var
mkdir /run
mkdir /tmp
mkdir /opt
mkdir /usr
mkdir /data/var
mkdir /data/run
mkdir /data/tmp
mkdir /data/opt
mkdir /data/etc
mkdir /data/etc/docker
mkdir /data/usr
mkdir /data/bin
mkdir /data/root
```
- 挂载docker所需要的目录
```
mount tmpfs /sys/fs/cgroup -t tmpfs -o size=1G
mkdir /sys/fs/cgroup/blkio
mkdir /sys/fs/cgroup/cpu
mkdir /sys/fs/cgroup/cpuacct
mkdir /sys/fs/cgroup/cpuset
mkdir /sys/fs/cgroup/devices
mkdir /sys/fs/cgroup/freezer
mkdir /sys/fs/cgroup/hugetlb
mkdir /sys/fs/cgroup/memory
mkdir /sys/fs/cgroup/net_cls
mkdir /sys/fs/cgroup/net_prio
mkdir /sys/fs/cgroup/perf_event
mkdir /sys/fs/cgroup/pids
mkdir /sys/fs/cgroup/systemd
# mount --bind
mount --bind /data/etc/docker /etc/docker
mount --bind /data/var /var
mount --bind /data/run /run
mount --bind /data/tmp /tmp
mount --bind /data/opt /opt
mount --bind /data/usr /usr
mount --bind /data/bin /bin
mount --bind /data/root /root
#mount cgroup
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
mount -t cgroup -o blkio,nodev,noexec,nosuid cgroup /sys/fs/cgroup/blkio
mount -t cgroup -o cpu,nodev,noexec,nosuid cgroup /sys/fs/cgroup/cpu
mount -t cgroup -o cpuacct,nodev,noexec,nosuid cgroup /sys/fs/cgroup/cpuacct
mount -t cgroup -o cpuset,nodev,noexec,nosuid cgroup /sys/fs/cgroup/cpuset
mount -t cgroup -o devices,nodev,noexec,nosuid cgroup /sys/fs/cgroup/devices
mount -t cgroup -o freezer,nodev,noexec,nosuid cgroup /sys/fs/cgroup/freezer
mount -t cgroup -o hugetlb,nodev,noexec,nosuid cgroup /sys/fs/cgroup/hugetlb
mount -t cgroup -o memory,nodev,noexec,nosuid cgroup /sys/fs/cgroup/memory
mount -t cgroup -o net_cls,nodev,noexec,nosuid cgroup /sys/fs/cgroup/net_cls
mount -t cgroup -o net_prio,nodev,noexec,nosuid cgroup /sys/fs/cgroup/net_prio
mount -t cgroup -o perf_event,nodev,noexec,nosuid cgroup /sys/fs/cgroup/perf_event
mount -t cgroup -o pids,nodev,noexec,nosuid cgroup /sys/fs/cgroup/pids
```
- 添加路由规则
```
ip rule add pref 1 from all lookup main
ip rule add pref 2 from all lookup default
```
- 关闭selinux
```
setenforce 0
```
- 创建文件 /etc/docker/daemon.json 并且写入
```
{"registry-mirrors":["https://docker.mirrors.ustc.edu.cn"],"experimental":true}
```

##### 7. 运行docker
```
dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock & 
```

##### 8. 验证docker工作状态
```
# 确保OpenHarmony已经联网，执行
docker run hello-world

# 会看到
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
93288797bd35: Pull complete 
Digest: sha256:2498fce14358aa50ead0cc6c19990fc6ff866ce72aeb5546e1d59caac3d0d60f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/
```

## OpenHarmony运行KubeEdge的关键步骤
---------------------
利用KubeEdge打通云端和OpenHarmony边端交互。这边使用的KubeEdge是1.9.1版本（提醒：k8s与KubeEdge版本是有匹配的，有些高版本的k8s是不支持的。）

##### 1. 云端安装KubeEdge cloudcore
```
# 以centos7为例  部署k8s v1.21.0


# 1. 关闭防火墙和selinux
systemctl stop firewalld
systemctl disable firewalld
sed -i 's/enforcing/disabled/' /etc/selinux/config


# 最终的/etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


setenforce 0


# 2. 关闭swap分区
swapoff -a    # 临时关闭
vim /etc/fstab # 注释到swap那一行  永久关闭


# 3. 官方仓库无法使用，建议使用阿里源的仓库，执行以下命令添加kubernetes.repo仓库

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes 
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
 

# 4. 安装 kubectl kubeamd kubelet （选择21版本是因后面安装踩过雷）
yum install -y kubelet-1.21.0 kubeadm-1.21.0 kubectl-1.21.0
# k8s 如何降版本可参考 https://blog.csdn.net/u012069313/article/details/125561711


# 5. 然后在master服务器上启动kubelet