kata-container磁盘镜像适配nfs，s3fs特性-天翼云开发者社区

环境信息：个人笔记本intel cpu，使用vmware搭建虚机（vmware workstation 要打开处理器的VT-x特性）

虚机使用ctyun23.01 iso镜像

配置好yum源

搭建kata-container环境

yum install containerd

containerd --version 测试时候安装成功

配置containerd的systemd服务：

cat /etc/systemd/system/containerd.service

[Unit]Description=containerd container runtime


After=network.target local-fs.target

[Service]ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead# in the kernel. We recommend using cgroups to do container-local accounting.LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.# Only systemd 226 and above support this version.TasksMax=infinity
OOMScoreAdjust=-999

[Install]WantedBy=multi-user.target

生成默认配置：

mkdir /etc/containerd

containerd config default > /etc/containerd/config.toml

将kata运行时添加到containerd，/etc/containerd/config.toml：

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"
       privileged_without_host_devices = false

重启containerd服务

systemctl daemon-reload

systemctl start containerd

systemctl enable containerd

systemctl status containerd

开始安装kata-containerd

从githua上下载kata-containerd二进制包，我使用的是kata-static-3.1.3-x86.tar.xz 这个版本

下载kata-containers，解压到/，并创建软链接：

tar xvf kata-static-3.1.3-x86.tar.xz -C /

ln -s /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2

ln -s /opt/kata/bin/kata-collect-data.sh /usr/local/bin/kata-collect-data.sh

ln -s /opt/kata/bin/kata-runtime /usr/local/bin/kata-runtime

查看版本

kata-runtime --version

kata-runtime kata-check

使用kata-runtime kata-check命令检查时候，如果vmware没有打开VT-x特性开关，检查会报错，会提示当前系统不支持，

kata-containerd需要cpu支持VT-x硬件虚拟化的特性，因为我是使用的vmware搭建的虚机，虚机里面的cpu默认是不支持VT-x的，vmware打开VT-x特性后，会把给cpu特性透传给虚机

创建kata容器

1、ctr image import busybox.tar 这个busybox.tar是docker使用的镜像，如果不能在线pull的话，需要手动下载并导入，busybox是比较常用的测试镜像

2、ctr run --runtime "io.containerd.kata.v2" --rm -t docker.io/library/busybox:latest test-kata uname –r

3、启动容器不退出：

ctr run --runtime "io.containerd.kata.v2" --rm -t docker.io/library/busybox:latest test-kata sleep infinity

4、启动一个已经创建的容器

ctr task start containerd_name

5、进入容器

ctr tasks exec -t --exec-id 1 test-kata sh

6、删除虚机

ctr tasks kill test-kata

7、进入虚机

kata-runtime exec test-kata

注意，进入虚机需要在/etc/kata-containers/configuration.toml 配置文件中打开debug_console_enabled开关

8、替换的kata-containers.img 也需要在/etc/kata-containers/configuration.toml 中修改路径或者文件名

[hypervisor.qemu]
path = "/opt/kata/bin/qemu-system-x86_64"
kernel = "/opt/kata/share/kata-containers/vmlinux.container"
image = "/opt/kata/share/kata-containers/kata-containers.img"
# initrd = "/opt/kata/share/kata-containers/kata-containers-initrd.img"

以上是kata-containers环境的构建及测试，下面是编译带nfs及s3fsrpm包的rootfs镜像文件过程

下载kata-containers源码

从git hub上就可以下载，我下载的是3.6.0的版

[root@localhost image-builder]# ls /home/code/
kata-containers-3.6.0  kata-containers-3.6.0.zip

进入kata-containers-3.6.0/tools/osbuilder/rootfs-builder目录进行rootfs构建

使用该目录下的rootfs脚本进行rootfs目录的构建，目前该脚本只支持有限的操作系统，如下所示

[root@localhost rootfs-builder]# ./rootfs.sh -l
alpine
cbl-mariner
centos
clearlinux
ctyun
debian
ubuntu

对于其它的操作系统，提供了脚本生成自定义操作系统配置，以ctyunos为例，使用以下命令生成ctyunos配置

make -f template/Makefile ROOTFS_BASE_NAME=ctyun

执行后会在该目录下生成一个ctyun的目录，里面有config.sh Dockerfile.in root_lib.sh 三个文件

[root@localhost ctyun]# ls
config.sh  Dockerfile.in  rootfs_lib.sh

需改config.sh 及 rootfs_lib.sh 两个脚本。

cong.sh中，需要指定AGENT_INIT=yes, 表示使用kata-agent作为init进程,还有例如OS_NAME, OS_VERSION等参数，我这里使用的是ctyunos23.01的版本，所有OS_VERSION就设置成23.01，由于此次需要适配nfs，s3fs文件系统，在config.sh 中配置EXTRA_PKGS="nfs-utils rpcbind" 用于支持nfs，其它参数可以使用rootfs.sh -h 查询

OS_VERSION=${OS_VERSION:-23.01}
EXTRA_PKGS="nfs-utils rpcbind"

PACKAGES="iptables"

AGENT_INIT=yes
# Init process must be one of {systemd,kata-agent}
INIT_PROCESS=kata-agent
# List of zero or more architectures to exclude from build,
# as reported by  `uname -m`
ARCH_EXCLUDE_LIST=()
# [When uncommented,] Allow the build to fail without generating an error
# For more info see: https://github.com/kata-containers/osbuilder/issues/190
#BUILD_CAN_FAIL=1

rootfs_lib.sh中需要实现build_rootfs函数，也就是具体的构建rootfs目录的函数，如果不定义的话，需要把这个配置文件删掉，系统会默认使用上一层script目录的lib.sh脚本，但是这个脚本在23.01上使用依照config.sh中的配置自动生成的dnf的配置文件，不知道为何不生效，可能是不识别--config参数，只有在ctyun目录下的rootfs_lib.sh 中指定releasever及指定installroot目录后才能下载安装rootfs安装包

build_rootfs() {
        # Mandatory
        local ROOTFS_DIR=$1

        #Name of the Kata-Agent binary
        local BIN_AGENT=${BIN_AGENT}

        # In case of support EXTRA packages, use it to allow
        # users add more packages to the base rootfs
        local EXTRA_PKGS=${EXTRA_PKGS:-}

        #In case rootfs is created usign repositories allow user to modify
        # the default URL
        local REPO_URL=${REPO_URL:-https://repo.ctyun.cn/hostos/ctyunos-23.01/everything/$basearch/}

        #PATH where files this script is placed
        #Use it to refer to files in the same directory
        #Exmaple: ${CONFIG_DIR}/foo
        local CONFIG_DIR=${CONFIG_DIR}

        [ -z "$ROOTFS_DIR" ] && die "need rootfs"

        check_root
        mkdir -p "${ROOTFS_DIR}"
        PKG_MANAGER="dnf"

        DNF="${PKG_MANAGER} -y --installroot=${ROOTFS_DIR} --noplugins --releasever=${OS_VERSION}"

        info "install packages for rootfs"
        $DNF install ${EXTRA_PKGS} ${PACKAGES}

        rm -rf ${ROOTFS_DIR}/usr/share/{bash-completion,cracklib,doc,info,locale,man,misc,pixmaps,terminfo,zoneinfo,zsh}


        # Populate ROOTFS_DIR
        # Must provide /sbin/init and /bin/${BIN_AGENT}
}

由于s3fs在23.01版本上没有yum源

使用ctyunos2.0.1版本的src.rpm包进行编译，编译好后，在本地进行安装

安装好s3fs后，将s3fs及其依赖mailcap二进制程序cp到生成的rootfs对应的目录下

cp /etc/mailcap /home/ctyun_rootfs/etc/
cp /usr/bin/s3fs /home/ctyun_rootfs/usr/bin/

使用rootfs.sh构建rootfs目录

使用rootfs.sh -r /home/ctyun_rootfs ctyun 这个命令，会在/home/ctyun_rootfs目录下构建rootfs

如果之前环境们没有配置好相应的编译工具链，在执行过程中会报错，按照报错提示逐个安装对应rpm包即可

创建rootfs目录成功后，进入kata-containers-3.6.0/tools/osbuilder/image-builder 目录，使用image_builder.sh脚本构建磁盘镜像文件

输入如下：

[root@localhost image-builder]# ./image_builder.sh /home/ctyun_rootfs/
[OK] init is installed
[OK] Agent installed
INFO: Creating raw disk with size 382M
[OK] Image file created
INFO: Creating partitions
[OK] Partitions created
INFO: Set filesystem reserved blocks percentage to 3%
tune2fs 1.46.4 (18-Aug-2021)
将保留块所占百分比设置为 3%（2926 个块）
INFO: Mounting root partition
[OK] root partition mounted
INFO: Copying content from rootfs to root partition
INFO: Setup SELinux
[OK] rootfs copied
INFO: Setup systemd
INFO: Removing unneeded systemd services and sockets
INFO: Removing unneeded systemd files
INFO: Creating empty machine-id to allow systemd to bind-mount it
INFO: Unmounting root partition
[OK] Root partition unmounted
e2fsck 1.46.4 (18-Aug-2021)
/dev/loop0p1：没有问题，10731/97536 文件，86016/97536 块
INFO: Creating raw disk with size 384M
[OK] Image file created
INFO: Creating partitions
[OK] Partitions created
INFO: Set DAX metadata
Opening file 'kata-containers.img.header'
Writing metadata
OK!
记录了1+0 的读入
记录了1+0 的写出
2097152字节（2.1 MB，2.0 MiB）已复制，0.00904466 s，232 MB/s
记录了782336+0 的读入
记录了782336+0 的写出
400556032字节（401 MB，382 MiB）已复制，7.04719 s，56.8 MB/s

最后生成kata-containers.img文件

[root@localhost image-builder]# ll -h
总用量 385M
-rw-r--r--. 1 root root  447  6月 18 03:59 Dockerfile
-rwxr-xr-x. 1 root root  19K  7月 31 14:11 image_builder.sh
-rw-r--r--. 1 root root 384M  8月  2 17:09 kata-containers.img
-rw-r--r--. 1 root root 3.7K  6月 18 03:59 nsdax.gpl.c
-rw-r--r--. 1 root root  680  6月 18 03:59 README.md

[Unit]Description=containerd container runtime After=network.target local-fs.target [Service]ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 # Having non-zero Limit*s causes performance problems due to accounting overhead# in the kernel. We recommend using cgroups to do container-local accounting.LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not supports it.# Only systemd 226 and above support this version.TasksMax=infinity OOMScoreAdjust=-999 [Install]WantedBy=multi-user.target

[hypervisor.qemu] path = "/opt/kata/bin/qemu-system-x86_64" kernel = "/opt/kata/share/kata-containers/vmlinux.container" image = "/opt/kata/share/kata-containers/kata-containers.img" # initrd = "/opt/kata/share/kata-containers/kata-containers-initrd.img"

OS_VERSION=${OS_VERSION:-23.01} EXTRA_PKGS="nfs-utils rpcbind" PACKAGES="iptables" AGENT_INIT=yes # Init process must be one of {systemd,kata-agent} INIT_PROCESS=kata-agent # List of zero or more architectures to exclude from build, # as reported by `uname -m` ARCH_EXCLUDE_LIST=() # [When uncommented,] Allow the build to fail without generating an error # For more info see: https://github.com/kata-containers/osbuilder/issues/190 #BUILD_CAN_FAIL=1

build_rootfs() { # Mandatory local ROOTFS_DIR=$1 #Name of the Kata-Agent binary local BIN_AGENT=${BIN_AGENT} # In case of support EXTRA packages, use it to allow # users add more packages to the base rootfs local EXTRA_PKGS=${EXTRA_PKGS:-} #In case rootfs is created usign repositories allow user to modify # the default URL local REPO_URL=${REPO_URL:-https://repo.ctyun.cn/hostos/ctyunos-23.01/everything/$basearch/} #PATH where files this script is placed #Use it to refer to files in the same directory #Exmaple: ${CONFIG_DIR}/foo local CONFIG_DIR=${CONFIG_DIR} [ -z "$ROOTFS_DIR" ] && die "need rootfs" check_root mkdir -p "${ROOTFS_DIR}" PKG_MANAGER="dnf" DNF="${PKG_MANAGER} -y --installroot=${ROOTFS_DIR} --noplugins --releasever=${OS_VERSION}" info "install packages for rootfs" $DNF install ${EXTRA_PKGS} ${PACKAGES} rm -rf ${ROOTFS_DIR}/usr/share/{bash-completion,cracklib,doc,info,locale,man,misc,pixmaps,terminfo,zoneinfo,zsh} # Populate ROOTFS_DIR # Must provide /sbin/init and /bin/${BIN_AGENT} }

[root@localhost image-builder]# ./image_builder.sh /home/ctyun_rootfs/ [OK] init is installed [OK] Agent installed INFO: Creating raw disk with size 382M [OK] Image file created INFO: Creating partitions [OK] Partitions created INFO: Set filesystem reserved blocks percentage to 3% tune2fs 1.46.4 (18-Aug-2021) 将保留块所占百分比设置为 3%（2926 个块） INFO: Mounting root partition [OK] root partition mounted INFO: Copying content from rootfs to root partition INFO: Setup SELinux [OK] rootfs copied INFO: Setup systemd INFO: Removing unneeded systemd services and sockets INFO: Removing unneeded systemd files INFO: Creating empty machine-id to allow systemd to bind-mount it INFO: Unmounting root partition [OK] Root partition unmounted e2fsck 1.46.4 (18-Aug-2021) /dev/loop0p1：没有问题，10731/97536 文件，86016/97536 块 INFO: Creating raw disk with size 384M [OK] Image file created INFO: Creating partitions [OK] Partitions created INFO: Set DAX metadata Opening file 'kata-containers.img.header' Writing metadata OK! 记录了1+0 的读入记录了1+0 的写出 2097152字节（2.1 MB，2.0 MiB）已复制，0.00904466 s，232 MB/s 记录了782336+0 的读入记录了782336+0 的写出 400556032字节（401 MB，382 MiB）已复制，7.04719 s，56.8 MB/s

[root@localhost image-builder]# ll -h 总用量 385M -rw-r--r--. 1 root root 447 6月 18 03:59 Dockerfile -rwxr-xr-x. 1 root root 19K 7月 31 14:11 image_builder.sh -rw-r--r--. 1 root root 384M 8月 2 17:09 kata-containers.img -rw-r--r--. 1 root root 3.7K 6月 18 03:59 nsdax.gpl.c -rw-r--r--. 1 root root 680 6月 18 03:59 README.md

智算服务

应用商城

定价

合作伙伴

开发者

支持与服务

了解天翼云

kata-container磁盘镜像适配nfs，s3fs特性

kata-container磁盘镜像适配nfs，s3fs特性

活动

智算服务

应用商城

定价

合作伙伴

开发者

支持与服务

了解天翼云

kata-container磁盘镜像适配nfs，s3fs特性

kata-container磁盘镜像适配nfs，s3fs特性