Newer
Older
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
##### 7. docker环境准备
运行脚本(如果脚本无法运行也可以按脚本一步一步来)
```
# mkdir on /system
if [ ! -d "/system/etc/docker" ]; then
mkdir /system/etc/docker
fi
# mkdir on /data
if [ ! -d "/data/var" ]; then
mkdir /data/var
else
rm -rf /data/var/run
fi
if [ ! -d "/data/run" ]; then
mkdir /data/run
fi
if [ ! -d "/data/tmp" ]; then
mkdir /data/tmp
fi
if [ ! -d "/data/opt" ]; then
mkdir /data/opt
fi
if [ ! -d "/data/etc" ]; then
mkdir /data/etc
mkdir /data/etc/docker
fi
if [ ! -d "/data/usr" ]; then
mkdir /data/usr
fi
if [ ! -d "/mnt/f2fs" ]; then
mkdir /mnt/f2fs
fi
# 这里对应刚刚已经被f2fs格式化的sd卡设备,可以用blkid查看到具体的名称
mount /dev/block/mmcblk0p1 /mnt/f2fs/
mount tmpfs /sys/fs/cgroup -t tmpfs -o size=1G
if [ ! -d "/sys/fs/cgroup/blkio" ]; then
mkdir /sys/fs/cgroup/blkio
mkdir /sys/fs/cgroup/cpu
mkdir /sys/fs/cgroup/cpuacct
mkdir /sys/fs/cgroup/cpuset
mkdir /sys/fs/cgroup/devices
mkdir /sys/fs/cgroup/freezer
mkdir /sys/fs/cgroup/hugetlb
mkdir /sys/fs/cgroup/memory
mkdir /sys/fs/cgroup/net_cls
mkdir /sys/fs/cgroup/net_prio
mkdir /sys/fs/cgroup/perf_event
mkdir /sys/fs/cgroup/pids
mkdir /sys/fs/cgroup/rdma
mkdir /sys/fs/cgroup/schedtune
mkdir /sys/fs/cgroup/systemd
fi
# mount --bind
mount --bind /data/etc/docker /etc/docker
mount --bind /data/var /var
mount --bind /data/run /run
mount --bind /data/tmp /tmp
mount --bind /data/opt /opt
mount --bind /data/usr /usr
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
mount -t cgroup -o blkio,nodev,noexec,nosuid cgroup /sys/fs/cgroup/blkio
mount -t cgroup -o cpu,nodev,noexec,nosuid cgroup /sys/fs/cgroup/cpu
mount -t cgroup -o cpuacct,nodev,noexec,nosuid cgroup /sys/fs/cgroup/cpuacct
mount -t cgroup -o cpuset,nodev,noexec,nosuid cgroup /sys/fs/cgroup/cpuset
mount -t cgroup -o devices,nodev,noexec,nosuid cgroup /sys/fs/cgroup/devices
mount -t cgroup -o freezer,nodev,noexec,nosuid cgroup /sys/fs/cgroup/freezer
mount -t cgroup -o hugetlb,nodev,noexec,nosuid cgroup /sys/fs/cgroup/hugetlb
mount -t cgroup -o memory,nodev,noexec,nosuid cgroup /sys/fs/cgroup/memory
mount -t cgroup -o net_cls,nodev,noexec,nosuid cgroup /sys/fs/cgroup/net_cls
mount -t cgroup -o net_prio,nodev,noexec,nosuid cgroup /sys/fs/cgroup/net_prio
mount -t cgroup -o perf_event,nodev,noexec,nosuid cgroup /sys/fs/cgroup/perf_event
mount -t cgroup -o pids,nodev,noexec,nosuid cgroup /sys/fs/cgroup/pids
mount -t cgroup -o rdma,nodev,noexec,nosuid cgroup /sys/fs/cgroup/rdma
mount -t cgroup -o schedtune,nodev,noexec,nosuid cgroup /sys/fs/cgroup/schedtune
# ip route
ip rule add pref 1 from all lookup main
ip rule add pref 2 from all lookup default
###
# setup dns nameserver and docker images registry
echo "{\"registry-mirrors\":[\"https://docker.mirrors.ustc.edu.cn\"],\"experimental\":false,\"storage-driver\": \"overlay2\",\"data-root\": \"/mnt/f2fs\"}" > /etc/docker/daemon.json
# open br_netfilter module
#modprobe br_netfilter
setenforce 0
```
```
# run dockerd
dockerd -D -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock &
```
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
```
# docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(arm64v8)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
# 查看docker的信息
docker version
docker info
docker ps -a
docker images
```
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
- Arm server上编译edgecore
```
# 匹配openhamrony的memory.stat格式, 修改kubeedge-1.9.1/vendor/github.com/opencontainers/runc/libcontainer/cgroups/fscommon/utils.go
func ParseKeyValue(t string) (string, uint64, error) {
tmp := strings.Replace(t, ":", "", -1)
tmpm := strings.Replace(tmp, "\t", "", -1)
tmpmt := strings.Replace(tmpm, " kB", "", -1)
count := strings.Count(tmpmt, " ")
tmpmts := strings.Replace(tmpmt, " ", "", count-1)
parts := strings.SplitN(tmpmts, " ", 3)
if len(parts) != 2 {
return "", 0, fmt.Errorf("line %q is not in key value format", t)
}
value, err := ParseUint(parts[1], 10, 64)
if err != nil {
return "", 0, fmt.Errorf("unable to convert to uint64: %v", err)
}
return parts[0], value, nil
}
# 由于openharmony安装docker用的是overlay2,所以需要修改kubeedge-1.9.1/edge/pkg/edged/edged.go
原:
DefaultRootDir = "/var/lib/edged"
// ContainerLogsDir is the location of container logs.
ContainerLogsDir = "/var/log/containers"
修改:(与docker overlay2的路径一样)
DefaultRootDir = "/mnt/f2fs/lib/edged"
// ContainerLogsDir is the location of container logs.
ContainerLogsDir = "/mnt/f2fs/log/containers"
# 编译edgecore
cd kubeedge-1.9.1
docker build -t kubeedge/edgecore:v1.9.1 -f build/edge/Dockerfile .
docker cp $(docker create --rm kubeedge/edgecore:v1.9.1):/usr/local/bin/edgecore ./edgecore.1.9.1
# 在kubeedge-1.9.1目录下有edgecore.1.9.1可执行文件
用hdc file send 拷贝到openharmony板子/system/bin上
```
- 修改openharmony运行时配置和启动edgecore
```
# 文件添加内容
echo "0" > /dev/cpuset/background/cpuset.mems
# 添加localhost的路径
/etc/hosts 要写入一个 127.0.0.1 localhost localhost
# 添加edgecore.yaml
mkdir -p /etc/kubeedge/config
cd /etc/kubeedge/config
edgecore --minconfig > edgecore.yaml
# 修改edgecore.yaml大体如下,注意cloud token mqtt的ip
# 启动edgecore
edgecore
```
edgecore.yaml参考
```
# With --minconfig , you can easily used this configurations as reference.
# It's useful to users who are new to KubeEdge, and you can modify/create your own configs accordingly.
# This configuration is suitable for beginners.
apiVersion: edgecore.config.kubeedge.io/v1alpha1
database:
dataSource: /var/lib/kubeedge/edgecore.db
kind: EdgeCore
modules:
edgeHub:
enable: true
heartbeat: 15
httpServer: https://8.142.141.36:10002
tlsCaFile: /etc/kubeedge/ca/rootCA.crt
tlsCertFile: /etc/kubeedge/certs/server.crt
tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
token: e8ceef8db0cd06fad0cc13c966527eb06b4c09a5f85818635b2a777ea915945d.eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTY5NDQ5MzV9.ff38T0LnXpJEdndfSvOuUUofG-5umKQY0IZ36_MFAOs
websocket:
enable: true
handshakeTimeout: 30
readDeadline: 15
server: 8.142.141.36:10000
writeDeadline: 15
edged:
cgroupDriver: cgroupfs
cgroupRoot: ""
cgroupsPerQOS: true
clusterDNS: 169.254.96.16
clusterDomain: cluster.local
devicePluginEnabled: false
dockerAddress: unix:///var/run/docker.sock
enable: true
gpuPluginEnabled: false
hostnameOverride: gandroid
customInterfaceName: wlan0
podSandboxImage: kubeedge/pause:3.1
remoteImageEndpoint: unix:///var/run/dockershim.sock
remoteRuntimeEndpoint: unix:///var/run/dockershim.sock
runtimeType: docker
eventBus:
enable: false
mqttMode: 2
mqttQOS: 0
mqttRetain: false
mqttServerExternal: tcp://127.0.0.1:1883
mqttServerInternal: tcp://127.0.0.1:1884
metaManager:
metaServer:
enable: true
```
最终效果是
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311

## 项目中可能遇到的问题及解决方案
- 问题一:二进制部署Docker流程
二进制部署Docker虽然官网有相应的静态二进制,但是要通过静态二进制运行起来Docker并不是一件容易的事。社区的老师真的提供了很多帮助,其中一位老师提供了他在Android运行Docker容器关键步骤教程让我来做参考。https://github.com/ThunderSoft001/kubeedgeOnAndroid
- 问题二:OpenHarmony许多开发板不适配无法运行
OpenHarmony还处于一个研发和推广阶段,所以支持OpenHarmony开发板并不是很多。在购入两块开发板不太适配OpenHarmony后,我联系了华为运营通过参与润和DAYU200体验适用活动申请到OpenHarmony官方用的开发板。DAYU200适配OpenHarmony各个标准版本,所以节省了研究开发板的烧录和运行时间。
- 问题三:OpenHarmony内核部分配置即使设置也无效
OpenHarmony 3.1 release使用的是5.10内核,有些配置是默认关闭的需要在deconfig去配置以下才能打开。然后有些配置即使在deconfig设置了,也无法打开。经过搜索和寻求帮助,发现有些配置是有依赖配置的,需要将依赖配置也打开才可以。查看具体配置的熟悉可以通过 https://cateee.net/lkddb/web-lkddb/CGROUPS.html 查看。
- 问题四:iptables静态交叉编译后无法在OpenHarmony上运行
运行docker需要一些基本软件工具,而OpenHarmony很多组件是需要自己去编译安装的。我所使用的DAYU200开发板是rk3568主板即arm64得。经过静态交叉编译iptables源码得到了arm64位的二进制文件,拷贝到OpenHarmony上一直无法运行。寻求社区老师帮助后,交叉编译iptables源码得到arm32位的静态二进制文件就可以在OpenHarmony上运行。解决方法是从64位的降到32位,这种情况平时安装软件的时候也有出现过。
- 问题五:OpenHarmony与KubeEdge的memory.stat格式不匹配
KubeEdge的edgecore在初始化的过程中会去访问系统memory.stat文件获取系统内存相关的信息。然而OpenHarmony在填充memory.stat时做了格式转换,导致edgecore无法读取memory.stat的最后四行。经过代码阅读和考虑,鉴于无法估量OpenHarmony对memory.stat数据的依赖程度,暂时修改了KubeEdge所使用的opencontainers第三方库的代码来解决这个问题。
- 问题六:OpenHarmony作为边缘端成功加入到云端后,云端无法下发任务。
在成功运行起来edgecore后,需要验证KubeEdge是否部署成功。需要从云端下发hello-world到边缘端并且运行起来,然而pod一直处于cannotrun状态。在请教过社区的老师后,发现是docker使用的overlay2模式下运行pod的路径与KubeEdge启动pod路径不同,需要将edgecore的defaultdir改成docker的pod路径即可。