K3s服务metrics-server异常

barfi · 2025 年6 月 3 日 01:48

环境信息:
Rancher Server 设置
Rancher 版本：2.7.9
安装选项 (Docker install/Helm Chart): Helm Chart
k3s版本：v1.26.9+k3s1
在线或离线部署：离线部署
**主机操作系统：三台ubtuntu22.04.6，两台server，一台agent，都已关闭防火墙

问题描述:
安装后metrics-server异常，具体看下方日志

安装 K3s 的命令:

已经有离线文件，拷贝离线数据
mkdir -p /var/lib/rancher/k3s/agent/images/
cp ./k3s-airgap-images-amd64.tar /var/lib/rancher/k3s/agent/images/

主：
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_VERSION=v1.26.9+k3s1 INSTALL_K3S_MIRROR=cn INSTALL_K3S_SKIP_DOWNLOAD=true sh -s - server --cluster-init

副主
curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_VERSION=v1.26.9+k3s1 K3S_URL=https://k3s-master:6443 sh -s - server --cluster-init --token “xxxxx”

代理

curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_SKIP_DOWNLOAD=true K3S_URL=https://k3s-master:6443 INSTALL_K3S_VERSION=v1.26.9+k3s1 K3S_TOKEN=“xxx” sh -

修改离线镜像库为本地
vi /etc/rancher/k3s/registries.yaml

拷贝配置文件，并对127.0.0.1的server为自定义的局域域名
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

查看状态
kubectl --kubeconfig ~/.kube/config get pods --all-namespaces

重启服务
systemctl daemon-reload
systemctl restart k3s
systemctl restart k3s-agent.service

预期结果:

实际结果:

附加上下文/日志:

metrics-server提示日志错误如下
I0530 02:26:56.498718 1 dynamic_serving_content.go:131] “Starting controller” name=“serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key”
I0530 02:26:56.499112 1 secure_serving.go:267] Serving securely on [::]:10250
I0530 02:26:56.499138 1 tlsconfig.go:240] “Starting DynamicServingCertificateController”
W0530 02:26:56.499174 1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0530 02:26:56.598376 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0530 02:26:56.598475 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0530 02:26:56.598484 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0530 02:26:56.646320 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:26:57.266337 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:26:57.649189 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:26:59.267256 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:27:01.267261 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:27:03.267763 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:27:05.267548 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:27:07.268040 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:27:09.266372 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:27:11.267207 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
E0530 02:33:37.167549 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: the server is currently unable to handle the request (get configmaps)
E0530 02:33:37.291417 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.Node: the server is currently unable to handle the request (get nodes)
E0530 02:33:38.368690 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: the server is currently unable to handle the request (get configmaps)
W0530 02:33:38.435646 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.Node: apiserver not ready
E0530 02:33:38.435666 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: apiserver not ready
W0530 02:33:38.552644 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:38.552668 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:39.528753 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.PartialObjectMetadata: the server is currently unable to handle the request
W0530 02:33:39.715704 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:39.715729 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:39.763661 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: the server is currently unable to handle the request (get configmaps)
W0530 02:33:40.724990 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.PartialObjectMetadata: apiserver not ready
E0530 02:33:40.725012 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: apiserver not ready
W0530 02:33:40.842728 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:40.842755 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:41.502625 1 scraper.go:140] “Failed to scrape node” err=“Get "[https://172.18.11.247:10250/metrics/resource\](https://172.18.11.247:10250/metrics/resource%5C)”: dial tcp 172.18.11.247:10250: connect: connection refused" node=“k3s-master”
W0530 02:33:41.603593 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.Node: apiserver not ready
E0530 02:33:41.603615 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.Node: failed to list *v1.Node: apiserver not ready
W0530 02:33:41.629420 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:41.629440 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver not ready
W0530 02:33:42.800177 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:42.800200 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver not ready
W0530 02:33:43.432121 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.PartialObjectMetadata: apiserver not ready
E0530 02:33:43.432142 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: apiserver not ready
W0530 02:33:43.581099 1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: failed to list *v1.ConfigMap: apiserver not ready
E0530 02:33:43.581123 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.17/tools/cache/reflector.go:167: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: apiserver not ready
I0530 02:33:48.420493 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:33:48.866288 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:33:50.421035 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:33:52.421704 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:33:54.420761 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”
I0530 02:33:56.421157 1 server.go:187] “Failed probe” probe=“metric-storage-ready” err=“no metrics to serve”



查看k3s日志是
May 30 13:33:46 k3s-master systemd[1177]: data-k3s-k3s-containerd-io.containerd.grpc.v1.cri-sandboxes-c96bc9af7813c64b32926962c8c9efb304bd9be94e15a943937837e7942c3d9f-shm.mount: Succeeded.
May 30 13:33:46 k3s-master systemd[1]: data-k3s-k3s-containerd-io.containerd.grpc.v1.cri-sandboxes-c96bc9af7813c64b32926962c8c9efb304bd9be94e15a943937837e7942c3d9f-shm.mount: Succeeded.
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.695857 1682 job_controller.go:514] enqueueing job kube-system/helm-install-traefik
May 30 13:33:46 k3s-master systemd[1]: cri-containerd-c96bc9af7813c64b32926962c8c9efb304bd9be94e15a943937837e7942c3d9f.scope: Succeeded.
May 30 13:33:46 k3s-master systemd[1]: data-k3s-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-c96bc9af7813c64b32926962c8c9efb304bd9be94e15a943937837e7942c3d9f-rootfs.mount: Succeeded.
May 30 13:33:46 k3s-master systemd[1177]: data-k3s-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-c96bc9af7813c64b32926962c8c9efb304bd9be94e15a943937837e7942c3d9f-rootfs.mount: Succeeded.
May 30 13:33:46 k3s-master kernel: [ 167.100434] cni0: port 1(vethf247339a) entered disabled state
May 30 13:33:46 k3s-master kernel: [ 167.103374] device vethf247339a left promiscuous mode
May 30 13:33:46 k3s-master kernel: [ 167.103377] cni0: port 1(vethf247339a) entered disabled state
May 30 13:33:46 k3s-master systemd-networkd[887]: vethf247339a: Link DOWN
May 30 13:33:46 k3s-master systemd-networkd[887]: vethf247339a: Lost carrier
May 30 13:33:46 k3s-master systemd-networkd[887]: rtnl: received neighbor for link ‘6’ we don’t know about, ignoring.
May 30 13:33:46 k3s-master systemd-networkd[887]: rtnl: received neighbor for link ‘6’ we don’t know about, ignoring.
May 30 13:33:46 k3s-master systemd[1177]: run-netns-cni\x2d99762788\x2d7640\x2d8156\x2d3a5c\x2d4b324baa1f10.mount: Succeeded.
May 30 13:33:46 k3s-master systemd[1]: run-netns-cni\x2d99762788\x2d7640\x2d8156\x2d3a5c\x2d4b324baa1f10.mount: Succeeded.
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.919355 1682 job_controller.go:514] enqueueing job kube-system/helm-install-traefik
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.924097 1682 reconciler_common.go:169] “operationExecutor.UnmountVolume started for volume "content" (UniqueName: "kubernetes.io/configmap/28ab06aa-cd4d-4312-a466-a577db6f04a4-content") pod "28ab06aa-cd4d-4312-a466-a577db6f04a4" (UID: "28ab06aa-cd4d-4312-a466-a577db6f04a4") "
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.924296 1682 reconciler_common.go:169] “operationExecutor.UnmountVolume started for volume "values" (UniqueName: "kubernetes.io/secret/28ab06aa-cd4d-4312-a466-a577db6f04a4-values") pod "28ab06aa-cd4d-4312-a466-a577db6f04a4" (UID: "28ab06aa-cd4d-4312-a466-a577db6f04a4") "
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.924437 1682 reconciler_common.go:169] “operationExecutor.UnmountVolume started for volume "kube-api-access-p4dtc" (UniqueName: "kubernetes.io/projected/28ab06aa-cd4d-4312-a466-a577db6f04a4-kube-api-access-p4dtc") pod "28ab06aa-cd4d-4312-a466-a577db6f04a4" (UID: "28ab06aa-cd4d-4312-a466-a577db6f04a4") "
May 30 13:33:46 k3s-master k3s[1682]: W0530 13:33:46.924307 1682 empty_dir.go:525] Warning: Failed to clear quota on /var/lib/kubelet/pods/28ab06aa-cd4d-4312-a466-a577db6f04a4/volumes/kubernetes.io~configmap/content: clearQuota called, but quotas disabled
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.924805 1682 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume “[kubernetes.io/configmap/28ab06aa-cd4d-4312-a466-a577db6f04a4-content](http://kubernetes.io/configmap/28ab06aa-cd4d-4312-a466-a577db6f04a4-content)” (OuterVolumeSpecName: “content”) pod “28ab06aa-cd4d-4312-a466-a577db6f04a4” (UID: “28ab06aa-cd4d-4312-a466-a577db6f04a4”). InnerVolumeSpecName “content”. PluginName “[kubernetes.io/configmap](http://kubernetes.io/configmap)”, VolumeGidValue “”
May 30 13:33:46 k3s-master systemd[1177]: data-k3s-kubelet-pods-28ab06aa\x2dcd4d\x2d4312\x2da466\x2da577db6f04a4-volumes-kubernetes.io\x7esecret-values.mount: Succeeded.
May 30 13:33:46 k3s-master systemd[1]: data-k3s-kubelet-pods-28ab06aa\x2dcd4d\x2d4312\x2da466\x2da577db6f04a4-volumes-kubernetes.io\x7esecret-values.mount: Succeeded.
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.927666 1682 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume “[kubernetes.io/secret/28ab06aa-cd4d-4312-a466-a577db6f04a4-values](http://kubernetes.io/secret/28ab06aa-cd4d-4312-a466-a577db6f04a4-values)” (OuterVolumeSpecName: “values”) pod “28ab06aa-cd4d-4312-a466-a577db6f04a4” (UID: “28ab06aa-cd4d-4312-a466-a577db6f04a4”). InnerVolumeSpecName “values”. PluginName “[kubernetes.io/secret](http://kubernetes.io/secret)”, VolumeGidValue “”
May 30 13:33:46 k3s-master k3s[1682]: I0530 13:33:46.929313 1682 operation_generator.go:900] UnmountVolume.TearDown succeeded for volume “[kubernetes.io/projected/28ab06aa-cd4d-4312-a466-a577db6f04a4-kube-api-access-p4dtc](http://kubernetes.io/projected/28ab06aa-cd4d-4312-a466-a577db6f04a4-kube-api-access-p4dtc)” (OuterVolumeSpecName: “kube-api-access-p4dtc”) pod “28ab06aa-cd4d-4312-a466-a577db6f04a4” (UID: “28ab06aa-cd4d-4312-a466-a577db6f04a4”). InnerVolumeSpecName “kube-api-access-p4dtc”. PluginName “[kubernetes.io/projected](http://kubernetes.io/projected)”, VolumeGidValue “”
May 30 13:33:46 k3s-master systemd[1177]: data-k3s-kubelet-pods-28ab06aa\x2dcd4d\x2d4312\x2da466\x2da577db6f04a4-volumes-kubernetes.io\x7eprojected-kube\x2dapi\x2daccess\x2dp4dtc.mount: Succeeded.
May 30 13:33:46 k3s-master systemd[1]: data-k3s-kubelet-pods-28ab06aa\x2dcd4d\x2d4312\x2da466\x2da577db6f04a4-volumes-kubernetes.io\x7eprojected-kube\x2dapi\x2daccess\x2dp4dtc.mount: Succeeded.
May 30 13:33:47 k3s-master k3s[1682]: I0530 13:33:47.025574 1682 reconciler_common.go:295] “Volume detached for volume "values" (UniqueName: "kubernetes.io/secret/28ab06aa-cd4d-4312-a466-a577db6f04a4-values") on node "k3s-master" DevicePath ""”
May 30 13:33:47 k3s-master k3s[1682]: I0530 13:33:47.025821 1682 reconciler_common.go:295] “Volume detached for volume "kube-api-access-p4dtc" (UniqueName: "kubernetes.io/projected/28ab06aa-cd4d-4312-a466-a577db6f04a4-kube-api-access-p4dtc") on node "k3s-master" DevicePath ""”
May 30 13:33:47 k3s-master k3s[1682]: I0530 13:33:47.025956 1682 reconciler_common.go:295] “Volume detached for volume "content" (UniqueName: "kubernetes.io/configmap/28ab06aa-cd4d-4312-a466-a577db6f04a4-content") on node "k3s-master" DevicePath ""”
May 30 13:33:47 k3s-master k3s[1682]: I0530 13:33:47.682836 1682 pod_container_deletor.go:80] “Container not found in pod’s containers” containerID=“c96bc9af7813c64b32926962c8c9efb304bd9be94e15a943937837e7942c3d9f”
May 30 13:33:47 k3s-master systemd[1]: Removed slice libcontainer container kubepods-besteffort-pod28ab06aa_cd4d_4312_a466_a577db6f04a4.slice.
May 30 13:33:47 k3s-master systemd[1]: kubepods-besteffort-pod28ab06aa_cd4d_4312_a466_a577db6f04a4.slice: Consumed 1.756s CPU time.
May 30 13:33:47 k3s-master k3s[1682]: I0530 13:33:47.694191 1682 job_controller.go:514] enqueueing job kube-system/helm-install-traefik
May 30 13:33:48 k3s-master k3s[1682]: I0530 13:33:48.697967 1682 job_controller.go:514] enqueueing job kube-system/helm-install-traefik
May 30 13:33:48 k3s-master k3s[1682]: I0530 13:33:48.705250 1682 job_controller.go:514] enqueueing job kube-system/helm-install-traefik
May 30 13:33:48 k3s-master k3s[1682]: I0530 13:33:48.708182 1682 job_controller.go:514] enqueueing job kube-system/helm-install-traefik
May 30 13:33:48 k3s-master k3s[1682]: I0530 13:33:48.708274 1682 event.go:294] “Event occurred” object=“kube-system/helm-install-traefik” fieldPath=”” kind=“Job” apiVersion=“batch/v1” type=“Normal” reason=“Completed” message=“Job completed”
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.313082 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [tlsstores.traefik.containo.us](http://tlsstores.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.313947 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [middlewares.traefik.containo.us](http://middlewares.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.314231 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [tlsoptions.traefik.containo.us](http://tlsoptions.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.314489 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [ingressrouteudps.traefik.containo.us](http://ingressrouteudps.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.314712 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [serverstransports.traefik.containo.us](http://serverstransports.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.314940 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [middlewaretcps.traefik.containo.us](http://middlewaretcps.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.315159 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [traefikservices.traefik.containo.us](http://traefikservices.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.315372 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [ingressroutes.traefik.containo.us](http://ingressroutes.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.315585 1682 resource_quota_monitor.go:218] QuotaMonitor created object count evaluator for [ingressroutetcps.traefik.containo.us](http://ingressroutetcps.traefik.containo.us/)
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.315841 1682 shared_informer.go:270] Waiting for caches to sync for resource quota
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.417086 1682 shared_informer.go:277] Caches are synced for resource quota
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.750403 1682 shared_informer.go:270] Waiting for caches to sync for garbage collector
May 30 13:34:06 k3s-master k3s[1682]: I0530 13:34:06.750465 1682 shared_informer.go:277] Caches are synced for garbage collector
May 30 13:34:24 k3s-master k3s[1682]: E0530 13:34:24.447259 1682 remote_runtime.go:415] “ContainerStatus from runtime service failed” err=“rpc error: code = NotFound desc = an error occurred when try to find container "8e666644e1742941b3ffa3748ebe08db080e66933f2528e4501feaae55323a49": not found” containerID=“8e666644e1742941b3ffa3748ebe08db080e66933f2528e4501feaae55323a49”
May 30 13:34:24 k3s-master k3s[1682]: I0530 13:34:24.447293 1682 kuberuntime_gc.go:362] “Error getting ContainerStatus for containerID” containerID=“8e666644e1742941b3ffa3748ebe08db080e66933f2528e4501feaae55323a49” err=“rpc error: code = NotFound desc = an error occurred when try to find container "8e666644e1742941b3ffa3748ebe08db080e66933f2528e4501feaae55323a49": not found”
May 30 13:34:24 k3s-master k3s[1682]: E0530 13:34:24.448268 1682 remote_runtime.go:415] “ContainerStatus from runtime service failed” err=“rpc error: code = NotFound desc = an error occurred when try to find container "4e01d9b8a7c07c63cca4dfe382a1fa5d5cb3abd62cce5eeda5e383ef391e6d3d": not found” containerID=“4e01d9b8a7c07c63cca4dfe382a1fa5d5cb3abd62cce5eeda5e383ef391e6d3d”
May 30 13:34:24 k3s-master k3s[1682]: I0530 13:34:24.448606 1682 kuberuntime_gc.go:362] “Error getting ContainerStatus for containerID” containerID=“4e01d9b8a7c07c63cca4dfe382a1fa5d5cb3abd62cce5eeda5e383ef391e6d3d” err=“rpc error: code = NotFound desc = an error occurred when try to find container "4e01d9b8a7c07c63cca4dfe382a1fa5d5cb3abd62cce5eeda5e383ef391e6d3d": not found”
May 30 13:34:24 k3s-master k3s[1682]: E0530 13:34:24.449192 1682 remote_runtime.go:415] “ContainerStatus from runtime service failed” err=“rpc error: code = NotFound desc = an error occurred when try to find container "5513f2a905ba74ff8f47aa2c35a62be96300fcf3897e4a67041d5ce4cd234aa2": not found” containerID=“5513f2a905ba74ff8f47aa2c35a62be96300fcf3897e4a67041d5ce4cd234aa2”
May 30 13:34:24 k3s-master k3s[1682]: I0530 13:34:24.449219 1682 kuberuntime_gc.go:362] “Error getting ContainerStatus for containerID” containerID=“5513f2a905ba74ff8f47aa2c35a62be96300fcf3897e4a67041d5ce4cd234aa2” err=“rpc error: code = NotFound desc = an error occurred when try to find container "5513f2a905ba74ff8f47aa2c35a62be96300fcf3897e4a67041d5ce4cd234aa2": not found”
May 30 13:34:24 k3s-master k3s[1682]: E0530 13:34:24.449574 1682 remote_runtime.go:415] “ContainerStatus from runtime service failed” err=“rpc error: code = NotFound desc = an error occurred when try to find container "3ea05c6ed6ca763aa01642cb810050df3d06bfdd9ca32b04d7bcd9c5b7b2008c": not found” containerID=“3ea05c6ed6ca763aa01642cb810050df3d06bfdd9ca32b04d7bcd9c5b7b2008c”
May 30 13:34:24 k3s-master k3s[1682]: I0530 13:34:24.449599 1682 kuberuntime_gc.go:362] “Error getting ContainerStatus for containerID” containerID=“3ea05c6ed6ca763aa01642cb810050df3d06bfdd9ca32b04d7bcd9c5b7b2008c” err=“rpc error: code = NotFound desc = an error occurred when try to find container "3ea05c6ed6ca763aa01642cb810050df3d06bfdd9ca32b04d7bcd9c5b7b2008c": not found”
May 30 13:36:14 k3s-master dbus-daemon[904]: [system] Activating via systemd: service name=‘org.freedesktop.timedate1’ unit=‘dbus-org.freedesktop.timedate1.service’ requested by ‘:1.11’ (uid=0 pid=917 comm=”/usr/lib/snapd/snapd " label=“unconfined”)
May 30 13:36:14 k3s-master systemd[1]: Starting Time & Date Service…
May 30 13:36:14 k3s-master dbus-daemon[904]: [system] Successfully activated service ‘org.freedesktop.timedate1’
May 30 13:36:14 k3s-master systemd[1]: Started Time & Date Service.
May 30 13:36:21 k3s-master snapd[917]: storehelpers.go:916: cannot refresh: snap has no updates available: “core20”, “lxd”, “snapd”
May 30 13:36:44 k3s-master systemd[1]: systemd-timedated.service: Succeeded.
May 30 13:39:48 k3s-master k3s[1682]: I0530 13:39:48.046279 1682 trace.go:236] Trace[2058330633]: “Get” accept:application/json, */* ,audit-id:0605e3c2-6341-44c1-ac99-a439d1b3c1f4,client:127.0.0.1,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/traefik-57c84cf78d-tkdkk/log,user-agent:kubectl/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728,verb:GET (30-May-2025 13:39:46.353) (total time: 1692ms):
May 30 13:39:48 k3s-master k3s[1682]: Trace[2058330633]: —“Writing http response done” 1691ms (13:39:48.046)
May 30 13:39:48 k3s-master k3s[1682]: Trace[2058330633]: [1.692864526s] [1.692864526s] END
May 30 13:40:23 k3s-master k3s[1682]: I0530 13:40:23.480609 1682 trace.go:236] Trace[835957571]: “Get” accept:application/json, */* ,audit-id:0369ff25-8aa0-4749-a530-0a99f59f051e,client:127.0.0.1,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/metrics-server-68cf49699b-6lf9c/log,user-agent:kubectl/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728,verb:GET (30-May-2025 13:40:00.670) (total time: 22810ms):
May 30 13:40:23 k3s-master k3s[1682]: Trace[835957571]: —“Writing http response done” 22808ms (13:40:23.480)
May 30 13:40:23 k3s-master k3s[1682]: Trace[835957571]: [22.81003241s] [22.81003241s] END
May 30 13:43:18 k3s-master k3s[1682]: {“level”:“info”,“ts”:“2025-05-30T13:43:18.420884+0800”,“caller”:“mvcc/index.go:214”,“msg”:“compact tree index”,“revision”:1723}
May 30 13:43:18 k3s-master k3s[1682]: {“level”:“info”,“ts”:“2025-05-30T13:43:18.436526+0800”,“caller”:“mvcc/kvstore_compaction.go:66”,“msg”:“finished scheduled compaction”,“compact-revision”:1723,“took”:“14.421645ms”,“hash”:3835480941}
May 30 13:43:18 k3s-master k3s[1682]: {“level”:“info”,“ts”:“2025-05-30T13:43:18.436689+0800”,“caller”:“mvcc/hash.go:137”,“msg”:“storing new hash”,“hash”:3835480941,“revision”:1723,“compact-revision”:-1}

[/details]

ksd · 2025 年6 月 3 日 05:41

看下 kubectl --kubeconfig ~/.kube/config get pods --all-namespaces 的结果

如果您在生产环境中使用 Rancher，希望获得更专业、及时的技术支持，也欢迎了解一下我们的商业订阅服务。可以点击论坛右上角聊天（）图标，私信联系我了解详细信息，我们有中文支持团队为您服务

barfi · 2025 年6 月 3 日 07:15

重新试着装了几次，现在就在一台机子上跑还是上面的问题，下面是命令返回内容

root@k3s-master:~# kubectl --kubeconfig ~/.kube/config get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-59b4f5bbd5-jrrk5 1/1 Running 0 3d23h
kube-system local-path-provisioner-76d776f6f9-9bm45 1/1 Running 0 3d23h
kube-system helm-install-traefik-crd-czh9t 0/1 Completed 0 3d23h
kube-system svclb-traefik-c0d227fa-g7ddw 2/2 Running 0 3d23h
kube-system traefik-57c84cf78d-klxw4 1/1 Running 0 3d23h
kube-system helm-install-traefik-hgg9x 0/1 Completed 1 3d23h
kube-system metrics-server-68cf49699b-fhmzk 1/1 Running 0 3d23h

ksd · 2025 年6 月 3 日 07:20

kubectl --kubeconfig ~/.kube/config top node 看看结果

如果您在生产环境中使用 Rancher，希望获得更专业、及时的技术支持，也欢迎了解一下我们的商业订阅服务。可以点击论坛右上角聊天（）图标，私信联系我了解详细信息，我们有中文支持团队为您服务

barfi · 2025 年6 月 3 日 07:27

root@k3s-master:~# kubectl --kubeconfig ~/.kube/config top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k3s-master 49m 1% 1654Mi 20%

ksd · 2025 年6 月 3 日 07:27

那也没问题啊…… 一切正常

如果您在生产环境中使用 Rancher，希望获得更专业、及时的技术支持，也欢迎了解一下我们的商业订阅服务。可以点击论坛右上角聊天（）图标，私信联系我了解详细信息，我们有中文支持团队为您服务

barfi · 2025 年6 月 3 日 07:37

目前metrics-server服务算是正常？看提示不是说没有服务吗。我之前继续安装到cer-manager时就进行不下去，那我继续安装把cer-manager问题反馈上来吧

ksd · 2025 年6 月 3 日 07:38

我感觉是正常的，pod 是 running 状态，而且 kubeclt top 能获取到指标。

如果您在生产环境中使用 Rancher，希望获得更专业、及时的技术支持，也欢迎了解一下我们的商业订阅服务。可以点击论坛右上角聊天（）图标，私信联系我了解详细信息，我们有中文支持团队为您服务