安装2个节点的集群 master节点是x86 agent是arm64的，agent的网卡插件起不来svclb-traefik-mm9fh

K3s
1

环境信息:
K3s 版本: v1.21.14+k3s1

节点 CPU 架构、操作系统和版本：:

master是x86的 agent是arm64 系统都是ubuntu20.04
集群配置:

1 server 1agent
问题描述:

agent上的svclb-traefik-mm9fh起不来
复现步骤:

预期结果:
kubectl get pods -A
kube-system svclb-traefik-kpz88 2/2 Running

实际结果:
kube-system coredns-79b69bd849-qkbxw 1/1 Running 0 30m
kube-system helm-install-traefik-crd-2pqvp 0/1 Completed 0 30m
kube-system local-path-provisioner-54db69c4b6-8wg7g 1/1 Running 0 30m
kube-system svclb-traefik-kpz88 2/2 Running 0 30m
kube-system helm-install-traefik-wsk68 0/1 Completed 1 30m
kube-system metrics-server-848c4d7648-bvxxv 1/1 Running 0 30m
kube-system traefik-79687485bf-85vg5 1/1 Running 0 30m
kube-system svclb-traefik-mm9fh 0/2 ContainerCreating 0 29m

附加上下文/日志:

日志 
  Normal   Scheduled               37m                     default-scheduler  Successfully assigned kube-system/svclb-traefik-mm9fh to mss
  Warning  FailedCreatePodSandBox  37m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "42ebcfd998fbe4559a2faff3c05610b730e23040ad08d9209388a853e5c1f748" network for pod "svclb-traefik-mm9fh": networkPlugin cni failed to set up pod "svclb-traefik-mm9fh_kube-system" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "42ebcfd998fbe4559a2faff3c05610b730e23040ad08d9209388a853e5c1f748"
  Warning  FailedCreatePodSandBox  37m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "cafd4a3e56e5ffdcd8b4aad6599d211d82de28a6d8dd2d369e29596937d4a990" network for pod "svclb-traefik-mm9fh": networkPlugin cni failed to set up pod "svclb-traefik-mm9fh_kube-system" network: failed to Statfs "/proc/45873/ns/net": no such file or directory
  Warning  FailedCreatePodSandBox  37m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "839413be5438ed237195e21f0a82d2b0f19427fb824f8fda832b0af7fc2f5b65" network for pod "svclb-traefik-mm9fh": networkPlugin cni failed to set up pod "svclb-traefik-mm9fh_kube-system" network: failed to Statfs "/proc/46022/ns/net": no such file or directory
  Warning  FailedCreatePodSandBox  37m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "a5d31f5ec73ec12cb3662e69122dde33efd65f5e42b6771173304242ef065944" network for pod "svclb-traefik-mm9fh": networkPlugin cni failed to set up pod "svclb-traefik-mm9fh_kube-system" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "a5d31f5ec73ec12cb3662e69122dde33efd65f5e42b6771173304242ef065944"
  Warning  FailedCreatePodSandBox  37m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "be0f1085090e98471df4c7fc14ed6d04d8bac4ed763ea81bde4f465604e9c04d" network for pod "svclb-traefik-mm9fh": networkPlugin cni failed to set up pod "svclb-traefik-mm9fh_kube-system" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "be0f1085090e98471df4c7fc14ed6d04d8bac4ed763ea81bde4f465604e9c04d"
  Warning  FailedCreatePodSandBox  37m                     kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b045b950837a1562c444d30b2edaa53f2bcea09bad9cbcdb052faff05fffbcd0" network for pod "svclb-traefik-mm9fh": networkPlugin cni failed to set up pod "svclb-traefik-mm9fh_kube-system" network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "b045b950837a1562c444d30b2edaa53f2bcea09bad9cbcdb052faff05fffbcd0"
2

无法复现，重现步骤如下：
OS ubuntu 20.04.6 clouding image

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Linux ubuntu20 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

server安装：

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.21.14+k3s1 INSTALL_K3S_EXEC='server --flannel-backend=host-gw --docker --system-default-registry=registry.cn-hangzhou.aliyuncs.com' sh -

agent安装：

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_VERSION=v1.21.14+k3s1 INSTALL_K3S_EXEC='agent --docker' K3S_URL=https://192.168.31.203:6443 K3S_TOKEN='<token>' sh -

从你提供的kubectl get po -A看出来，只有一个svclb的pod运行失败，应该与安装方式和架构无关

3

可以尝试 docker system prune 清理磁盘空间看下
另外查看一下磁盘占用 df -h

4

和我系统默认已经安装了docker有没有关系？

5

你使用了 --docker 来让k3s使用cri-docker，应该是无关的，其他的Pod都已经启动，所以我认为环境与网络都应该没有问题。
网上搜索出现这个问题时，有可能是磁盘空间不足导致，从而无法建立 podSandBox，所以先按照这个方向排查

6

Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 234G 18G 206G 8% /
none 3.1G 0 3.1G 0% /dev
tmpfs 3.2G 52K 3.2G 1% /dev/shm
tmpfs 644M 75M 569M 12% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 3.2G 0 3.2G 0% /sys/fs/cgroup
tmpfs 644M 16K 644M 1% /run/user/124
tmpfs 644M 0 644M 0% /run/user/0
我的磁盘空间是够的，我现在全部改用2个都是arm64的 并且物理机上之前已经暗转过了docker 都会报类似的错误
root@zhiguo-o:~# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-54db69c4b6-j6c6f 0/1 ContainerCreating 0 6m31s
kube-system metrics-server-848c4d7648-ldfrm 0/1 ContainerCreating 0 6m31s
kube-system coredns-79b69bd849-t5hhk 0/1 ContainerCreating 0 6m31s
kube-system helm-install-traefik-crd-56h2j 0/1 ContainerCreating 0 6m30s
kube-system helm-install-traefik-jwdmx 0/1 ContainerCreating 0 6m30s

错误如下
Warning FailedCreatePodSandBox 7m7s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “95d6bae58ff9909743f7a450853122b615ca4b112416dc9a96874ca902243101” network for pod “local-path-provisioner-54db69c4b6-j6c6f”: networkPlugin cni failed to set up pod “local-path-provisioner-54db69c4b6-j6c6f_kube-system” network: failed to Statfs “/proc/7975/ns/net”: no such file or directory
Warning FailedCreatePodSandBox 7m5s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “9ba0e03d83d8e70812f422a5cf7464c1b146ec1aa89ea7d56078f14af90762d2” network for pod “local-path-provisioner-54db69c4b6-j6c6f”: networkPlugin cni failed to set up pod “local-path-provisioner-54db69c4b6-j6c6f_kube-system” network: failed to Statfs “/proc/8537/ns/net”: no such file or directory
Warning FailedCreatePodSandBox 7m4s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “a73305fefcc9a9e7197cdab5aa8b7d4d714215978e0d2dd370861b7a683a9f00” network for pod “local-path-provisioner-54db69c4b6-j6c6f”: networkPlugin cni failed to set up pod “local-path-provisioner-54db69c4b6-j6c6f_kube-system” network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container “a73305fefcc9a9e7197cdab5aa8b7d4d714215978e0d2dd370861b7a683a9f00”
Warning FailedCreatePodSandBox 7m3s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “bba49b75ec99cee154023ac9a710bea35a46cb4bb38fdfd52c0bffc0b9b59689” network for pod “local-path-provisioner-54db69c4b6-j6c6f”: networkPlugin cni failed to set up pod “local-path-provisioner-54db69c4b6-j6c6f_kube-system” network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container “bba49b75ec99cee154023ac9a710bea35a46cb4bb38fdfd52c0bffc0b9b59689”
Normal SandboxChanged 6m59s (x12 over 7m11s) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedCreatePodSandBox 2m14s (x210 over 6m55s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “4ffcd21b22db83bee23c4837cfb9b39327ed59a2ee186db5799f4effb0c26df6” network for pod “local-path-provisioner-54db69c4b6-j6c6f”: networkPlugin cni failed to set up pod “local-path-provisioner-54db69c4b6-j6c6f_kube-system” network: CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "4ffcd21b22db83bee23c4837cfb9b39327ed59a2ee186db5799f4effb0c26df6

7

下载一下k3s的journalctl的日志，journalctl -u k3s > k3s_logs.txt