环境信息:
K3s 版本:
k3s version v1.28.7+k3s1 (051b14b2)
go version go1.21.7
节点 CPU 架构、操作系统和版本：:
Linux master 5.15.0-1029-mtk #33-Ubuntu SMP PREEMPT Tue Oct 3 10:53:53 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
集群配置:
单一 Server
问题描述:
安装完毕后， Node 正常
ubuntu@master:/proc$ sudo kubectl get node -A
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 30m v1.28.7+k3s1
但是kube-system 各个Pod 没有正常工作
ubuntu@master:/proc$ sudo kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-58c9946f4-8ddvr 0/1 CrashLoopBackOff 11 (15s ago) 30m
kube-system local-path-provisioner-7f4c755b68-qv755 0/1 CrashLoopBackOff 10 (18s ago) 30m
kube-system metrics-server-595fb6fd99-6h5hv 0/1 CrashLoopBackOff 10 (2m50s ago) 30m
针对某个具体的pod, 查看它的具体状态：
ubuntu@master:/proc$ sudo kubectl describe pod coredns-58c9946f4-8ddvr -n kube-system
发现错误信息：
Type Reason Age From Message
Normal Scheduled 32m default-scheduler Successfully assigned kube-system/coredns-58c9946f4-8ddvr to master
Normal Pulling 32m kubelet Pulling image “registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.10.1”
Normal Pulled 32m kubelet Successfully pulled image “registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.10.1” in 9.564s (9.564s including waiting)
Warning Failed 32m kubelet Error: failed to get sandbox container task: no running task found: task ecbe6de97942a2b16a96612e3882d04bda42735d00403191b5de055fa19f30c9 not found: not found
Warning Failed 32m kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to create new parent process: namespace path: lstat /proc/3987/ns/ipc: no such file or directory: unknown
- 安装 K3s 的命令:
sudo curl –sfL
https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh |
INSTALL_K3S_MIRROR=cn sh -s -
–system-default-registry “registry.cn-hangzhou.aliyuncs.com”
–write-kubeconfig ~/.kube/config
–write-kubeconfig-mode 666
–disable traefik
预期结果:
实际结果:
ubuntu@master:/proc$ sudo kubectl get pod -A -owide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system local-path-provisioner-7f4c755b68-qv755 0/1 CrashLoopBackOff 11 (16s ago) 35m 10.42.0.26 master
kube-system metrics-server-595fb6fd99-6h5hv 0/1 CrashLoopBackOff 11 (2m47s ago) 35m 10.42.0.29 master
kube-system coredns-58c9946f4-8ddvr 0/1 CrashLoopBackOff 12 (12s ago) 35m 10.42.0.30 master
附加上下文/日志:
日志
ubuntu@master:/proc$ sudo kubectl describe pod coredns-58c9946f4-8ddvr -n kube-system
Name: coredns-58c9946f4-8ddvr
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: coredns
Node: master/172.16.30.64
Start Time: Tue, 26 Mar 2024 16:11:17 +0800
Labels: k8s-app=kube-dns
pod-template-hash=58c9946f4
Annotations: <none>
Status: Running
IP: 10.42.0.186
IPs:
IP: 10.42.0.186
Controlled By: ReplicaSet/coredns-58c9946f4
Containers:
coredns:
Container ID: containerd://c0e2db41ee89729eb1056b609d3e10f0e71ef29f6f240147248617357d5f42b8
Image: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.10.1
Image ID: registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns@sha256:be7652ce0b43b1339f3d14d9b14af9f588578011092c1f7893bd55432d83a378
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: StartError
Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to create new parent process: namespace path: lstat /proc/2216998/ns/ipc: no such file or directory: unknown
Exit Code: 128
Started: Thu, 01 Jan 1970 08:00:00 +0800
Finished: Tue, 26 Mar 2024 16:41:53 +0800
Ready: False
Restart Count: 11
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8181/ready delay=0s timeout=1s period=2s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/etc/coredns/custom from custom-config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cqs2s (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
custom-config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns-custom
Optional: true
kube-api-access-cqs2s:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector k8s-app=kube-dns
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 32m default-scheduler Successfully assigned kube-system/coredns-58c9946f4-8ddvr to master
Normal Pulling 32m kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.10.1"
Normal Pulled 32m kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.10.1" in 9.564s (9.564s including waiting)
Warning Failed 32m kubelet Error: failed to get sandbox container task: no running task found: task ecbe6de97942a2b16a96612e3882d04bda42735d00403191b5de055fa19f30c9 not found: not found
Warning Failed 32m kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to create new parent process: namespace path: lstat /proc/3987/ns/ipc: no such file or directory: unknown
Normal Pulled 32m (x2 over 32m) kubelet Container image "registry.cn-hangzhou.aliyuncs.com/rancher/mirrored-coredns-coredns:1.10.1" already present on machine
Normal Created 32m (x2 over 32m) kubelet Created container coredns
Warning Failed 32m kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to create new parent process: namespace path: lstat /proc/4397/ns/ipc: no such file or directory: unknown
Warning BackOff 7m54s (x941 over 32m) kubelet Back-off restarting failed container coredns in pod coredns-58c9946f4-8ddvr_kube-system(de804fc1-a2cd-4a0a-b69a-75d80932f732)
Normal SandboxChanged 2m54s (x1008 over 32m) kubelet Pod sandbox changed, it will be killed and re-created.
/var/lib/rancher/k3s/agent/containerd/containerd.log
map[string]interface {}{"cniVersion":"1.0.0", "forceAddress":true, "hairpinMode":true, "ipMasq":false, "ipam":map[string]interface {}{"ranges":[][]map[string]interface {}{[]map[string]interface {}{map[string]interface {}{"subnet":"10.42.0.0/24"}}}, "routes":[]types.Route{types.Route{Dst:net.IPNet{IP:net.IP{0xa, 0x2a, 0x0, 0x0}, Mask:net.IPMask{0xff, 0xff, 0x0, 0x0}}, GW:net.IP(nil)}}, "type":"host-local"}, "isDefaultGateway":true, "isGateway":true, "mtu":(*uint)(0x400000e880), "name":"cbr0", "type":"bridge"}
delegateAdd: netconf sent to delegate plugin:
{"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.42.0.0/24"}]],"routes":[{"dst":"10.42.0.0/16"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}time="2024-03-26T16:50:03.482896167+08:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-03-26T16:50:03.483118851+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-03-26T16:50:03.483164157+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-03-26T16:50:03.483448147+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-03-26T16:50:03.669637486+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:coredns-58c9946f4-8ddvr,Uid:de804fc1-a2cd-4a0a-b69a-75d80932f732,Namespace:kube-system,Attempt:1294,} returns sandbox id \"09fb3daecef4009e187c7f0133792b5500b86399abeff5f7318274c5a8148403\""
time="2024-03-26T16:50:03.815101205+08:00" level=info msg="StopPodSandbox for \"183c43e0d3933eace0d29fe17e03cadf8e04cad57d871115f6c51cc7dbc7d5e4\""
time="2024-03-26T16:50:03.852025038+08:00" level=info msg="shim disconnected" id=09fb3daecef4009e187c7f0133792b5500b86399abeff5f7318274c5a8148403 namespace=k8s.io
time="2024-03-26T16:50:03.852166802+08:00" level=warning msg="cleaning up after shim disconnected" id=09fb3daecef4009e187c7f0133792b5500b86399abeff5f7318274c5a8148403 namespace=k8s.io
time="2024-03-26T16:50:03.852200493+08:00" level=info msg="cleaning up dead shim" namespace=k8s.io
time="2024-03-26T16:50:03.940569485+08:00" level=info msg="StopPodSandbox for \"498b32d8a584f508dfd5fc6a5b8f1cdd419b29bebc0092d4c82423029f7394bd\""
time="2024-03-26T16:50:03.978978957+08:00" level=info msg="TearDown network for sandbox \"183c43e0d3933eace0d29fe17e03cadf8e04cad57d871115f6c51cc7dbc7d5e4\" successfully"
time="2024-03-26T16:50:03.979050724+08:00" level=info msg="StopPodSandbox for \"183c43e0d3933eace0d29fe17e03cadf8e04cad57d871115f6c51cc7dbc7d5e4\" returns successfully"
time="2024-03-26T16:50:03.980080379+08:00" level=info msg="StopPodSandbox for \"5257a268781af454ac6ed7c0e19b199de136e99d65bfeffad87cc269474a0b1f\""
time="2024-03-26T16:50:04.017417429+08:00" level=info msg="TearDown network for sandbox \"5257a268781af454ac6ed7c0e19b199de136e99d65bfeffad87cc269474a0b1f\" successfully"
time="2024-03-26T16:50:04.017453812+08:00" level=info msg="StopPodSandbox for \"5257a268781af454ac6ed7c0e19b199de136e99d65bfeffad87cc269474a0b1f\" returns successfully"
time="2024-03-26T16:50:04.018168786+08:00" level=info msg="StopPodSandbox for \"b9d4ca74d5bde8f4d5db7cac26bcaedf51d3b1324ea6bf248d4d8ac22523c757\""
time="2024-03-26T16:50:04.036818581+08:00" level=info msg="TearDown network for sandbox \"b9d4ca74d5bde8f4d5db7cac26bcaedf51d3b1324ea6bf248d4d8ac22523c757\" successfully"
time="2024-03-26T16:50:04.036897348+08:00" level=info msg="StopPodSandbox for \"b9d4ca74d5bde8f4d5db7cac26bcaedf51d3b1324ea6bf248d4d8ac22523c757\" returns successfully"
time="2024-03-26T16:50:04.037615476+08:00" level=info msg="StopPodSandbox for \"78acf7e27f8b411094755d6a003310b43939c7c2b43a0072f0a7252431cd2291\""
time="2024-03-26T16:50:04.040993509+08:00" level=info msg="TearDown network for sandbox \"498b32d8a584f508dfd5fc6a5b8f1cdd419b29bebc0092d4c82423029f7394bd\" successfully"
time="2024-03-26T16:50:04.041087275+08:00" level=info msg="StopPodSandbox for \"498b32d8a584f508dfd5fc6a5b8f1cdd419b29bebc0092d4c82423029f7394bd\" returns successfully"
time="2024-03-26T16:50:04.041798019+08:00" level=info msg="StopPodSandbox for \"648b3652ecda8624f600102c1d6a1db99db7f22cf64cfb711a1eefe76bd917a6\""
time="2024-03-26T16:50:04.064817811+08:00" level=info msg="TearDown network for sandbox \"648b3652ecda8624f600102c1d6a1db99db7f22cf64cfb711a1eefe76bd917a6\" successfully"
time="2024-03-26T16:50:04.064896808+08:00" level=info msg="StopPodSandbox for \"648b3652ecda8624f600102c1d6a1db99db7f22cf64cfb711a1eefe76bd917a6\" returns successfully"
time="2024-03-26T16:50:04.065046803+08:00" level=info msg="TearDown network for sandbox \"78acf7e27f8b411094755d6a003310b43939c7c2b43a0072f0a7252431cd2291\" successfully"
time="2024-03-26T16:50:04.065089032+08:00" level=info msg="StopPodSandbox for \"78acf7e27f8b411094755d6a003310b43939c7c2b43a0072f0a7252431cd2291\" returns successfully"
time="2024-03-26T16:50:04.065797622+08:00" level=info msg="StopPodSandbox for \"3d8668ecb864edce88fa89504c14cf534ab3091a921259ae74af3eab4448629c\""
time="2024-03-26T16:50:04.065800237+08:00" level=info msg="StopPodSandbox for \"d469676de41457f6a1fd871d20dcc3f7392f674d8fa6c8a82530f69fedbb2f94\""
time="2024-03-26T16:50:04.090114984+08:00" level=info msg="TearDown network for sandbox \"d469676de41457f6a1fd871d20dcc3f7392f674d8fa6c8a82530f69fedbb2f94\" successfully"
time="2024-03-26T16:50:04.090209750+08:00" level=info msg="StopPodSandbox for \"d469676de41457f6a1fd871d20dcc3f7392f674d8fa6c8a82530f69fedbb2f94\" returns successfully"
time="2024-03-26T16:50:04.091263173+08:00" level=info msg="StopPodSandbox for \"635be05e8270e4bb4a799081fb374fcfc986f4f6e7d09b59566e687537b0759b\""
time="2024-03-26T16:50:04.100702067+08:00" level=info msg="TearDown network for sandbox \"3d8668ecb864edce88fa89504c14cf534ab3091a921259ae74af3eab4448629c\" successfully"
time="2024-03-26T16:50:04.100793064+08:00" level=info msg="StopPodSandbox for \"3d8668ecb864edce88fa89504c14cf534ab3091a921259ae74af3eab4448629c\" returns successfully"
time="2024-03-26T16:50:04.101794951+08:00" level=info msg="StopPodSandbox for \"8fda8d40084077f151e95a680c40e9cf47dc31ba49762a832b6bfe223dba23ed\""
time="2024-03-26T16:50:04.114900175+08:00" level=info msg="TearDown network for sandbox \"635be05e8270e4bb4a799081fb374fcfc986f4f6e7d09b59566e687537b0759b\" successfully"
time="2024-03-26T16:50:04.114989018+08:00" level=info msg="StopPodSandbox for \"635be05e8270e4bb4a799081fb374fcfc986f4f6e7d09b59566e687537b0759b\" returns successfully"
time="2024-03-26T16:50:04.115751375+08:00" level=info msg="StopPodSandbox for \"eda9e113c7063de95f94aec3aeafd04bb47ccd2998214f1d1fbd97c0152a25f8\""
time="2024-03-26T16:50:04.123850778+08:00" level=info msg="TearDown network for sandbox \"8fda8d40084077f151e95a680c40e9cf47dc31ba49762a832b6bfe223dba23ed\" successfully"
time="2024-03-26T16:50:04.123927852+08:00" level=info msg="StopPodSandbox for \"8fda8d40084077f151e95a680c40e9cf47dc31ba49762a832b6bfe223dba23ed\" returns successfully"
time="2024-03-26T16:50:04.124764053+08:00" level=info msg="StopPodSandbox for \"a015f007deec5aa3f6c047f367302bb18109985bd77b9100338c91cb28ef1aad\""
time="2024-03-26T16:50:04.137440754+08:00" level=info msg="TearDown network for sandbox \"eda9e113c7063de95f94aec3aeafd04bb47ccd2998214f1d1fbd97c0152a25f8\" successfully"
time="2024-03-26T16:50:04.137522058+08:00" level=info msg="StopPodSandbox for \"eda9e113c7063de95f94aec3aeafd04bb47ccd2998214f1d1fbd97c0152a25f8\" returns successfully"
time="2024-03-26T16:50:04.138486947+08:00" level=info msg="StopPodSandbox for \"85f54fd9a7ab8eb0ea1710463d7a850216756ae2a33c19198195bc038b7660e7\""