Harvester Cluster 1.4.2不能添加节点，提示Management URL: Unavailable，Status: NotReady

Harvester
, ,
1

在已有harvester集群添加节点，之前已经添加过没问题，最近不能添加节点了，对比了/oem/90_custom.yaml和其他配置没有发现问题

系统安装失败截图：

harvester-10-12
harvester-10-121919×1080 80 KB

预期结果：
成功添加节点

环境信息:

  • Harvester ISO 版本: 1.4.2
  • rancher: 2.10.1
  • rke2: v1.31.4+rke2r1

请教下排查方向

目前查询的信息：
在所需添加的节点查看rancherd日志Probe [kubelet] is unhealthy

journalctl -u rancherd -f 
Jul 22 03:58:30 hci-**-** rancherd[2955]: time="2026-07-22T03:58:30Z" level=info msg="[stdout]: [INFO]  systemd: Creating service file"
Jul 22 03:58:30 hci-**-** rancherd[2955]: time="2026-07-22T03:58:30Z" level=info msg="[stdout]: [INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env"
Jul 22 03:58:30 hci-**-** rancherd[2955]: time="2026-07-22T03:58:30Z" level=info msg="[stdout]: [INFO]  /usr/local is unsuitable for installation: adding fallback path to systemd unit env file."
Jul 22 03:58:31 hci-**-** rancherd[2955]: time="2026-07-22T03:58:31Z" level=info msg="[stdout]: [INFO]  Enabling rancher-system-agent.service"
Jul 22 03:58:31 hci-**-** rancherd[2955]: time="2026-07-22T03:58:31Z" level=info msg="[stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service."
Jul 22 03:58:31 hci-**-** rancherd[2955]: time="2026-07-22T03:58:31Z" level=info msg="[stdout]: [INFO]  Starting/restarting rancher-system-agent.service"
Jul 22 03:58:31 hci-**-** rancherd[2955]: time="2026-07-22T03:58:31Z" level=info msg="No image provided, creating empty working directory /var/lib/rancher/rancherd/plan/work/20260722-035829-applied.plan/_2"
Jul 22 03:58:31 hci-**-** rancherd[2955]: time="2026-07-22T03:58:31Z" level=info msg="Running command: /usr/bin/rancherd [probe]"
Jul 22 03:58:31 hci-**-** rancherd[2955]: time="2026-07-22T03:58:31Z" level=info msg="[stderr]: time=\"2026-07-22T03:58:31Z\" level=info msg=\"Running probes defined in /var/lib/rancher/rancherd/plan/plan.json\""
Jul 22 03:58:32 hci-**-** rancherd[2955]: time="2026-07-22T03:58:32Z" level=info msg="[stderr]: time=\"2026-07-22T03:58:32Z\" level=info msg=\"Probe [kubelet] is unhealthy\""

rancher-system-agent.service 有error日志

Jul 22 09:30:35 hci-**-** rancher-system-agent[3155]: W0722 09:30:35.308171    3155 reflector.go:462] pkg/mod/github.com/rancher/client-go@v1.29.3-rancher1/tools/cache/reflector.go:229: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 65; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
Jul 22 09:31:48 hci-**-** rancher-system-agent[3155]: W0722 09:31:48.153805    3155 reflector.go:462] pkg/mod/github.com/rancher/client-go@v1.29.3-rancher1/tools/cache/reflector.go:229: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 69; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
Jul 22 09:34:31 hci-**-** rancher-system-agent[3155]: W0722 09:34:31.043918    3155 reflector.go:462] pkg/mod/github.com/rancher/client-go@v1.29.3-rancher1/tools/cache/reflector.go:229: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 73; INTERNAL_ERROR; received from peer") has prevented the request from succeeding
Jul 22 09:36:32 hci-**-** rancher-system-agent[3155]: W0722 09:36:32.151503    3155 reflector.go:462] pkg/mod/github.com/rancher/client-go@v1.29.3-rancher1/tools/cache/reflector.go:229: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 77; INTERNAL_ERROR; received from peer") has prevented the request from succeeding

主节点查看rke2-ingress-nginx pod日志upstream timed out (110: Connection timed out)：

kubectl -n kube-system logs -f --tail 200 -l app.kubernetes.io/name=rke2-ingress-nginx --max-log-requests 20 | grep <node-ip>
<node-ip> - - [22/Jul/2026:09:42:10 +0000] "GET /api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833796250&timeout=9m0s&timeoutSeconds=540&watch=true HTTP/2.0" 200 0 "-" "rancher-system-agent/v0.0.0 (linux/amd64) kubernetes/$Format" 148 60.001 [cattle-system-rancher-80] [] 10.52.13.232:80 0 60.002 200 16e2e96b5d44022d4fa91c3104781bc7
2026/07/22 09:42:10 [error] 35080#35080: *198985750 upstream timed out (110: Connection timed out) while reading upstream, client: <node-ip>, server: _, request: "GET /api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833796250&timeout=9m0s&timeoutSeconds=540&watch=true HTTP/2.0", upstream: "http://10.52.13.232:80/api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833796250&timeout=9m0s&timeoutSeconds=540&watch=true", host: "<master-vip>"
<node-ip> - - [22/Jul/2026:09:42:40 +0000] "GET /api/v1/namespaces/fleet-local/secrets?fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833796250 HTTP/2.0" 200 2650 "-" "rancher-system-agent/v0.0.0 (linux/amd64) kubernetes/$Format" 98 0.003 [cattle-system-rancher-80] [] 10.52.11.90:80 2650 0.003 200 2f2a893dbb35195e064574283fbf4cec
2026/07/22 09:43:40 [error] 35080#35080: *198985750 upstream timed out (110: Connection timed out) while reading upstream, client: <node-ip>, server: _, request: "GET /api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833798126&timeout=9m16s&timeoutSeconds=556&watch=true HTTP/2.0", upstream: "http://10.52.4.106:80/api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833798126&timeout=9m16s&timeoutSeconds=556&watch=true", host: "<master-vip>"
<node-ip> - - [22/Jul/2026:09:43:40 +0000] "GET /api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833798126&timeout=9m16s&timeoutSeconds=556&watch=true HTTP/2.0" 200 0 "-" "rancher-system-agent/v0.0.0 (linux/amd64) kubernetes/$Format" 149 60.001 [cattle-system-rancher-80] [] 10.52.4.106:80 0 60.001 200 a56a5876e376983e4fb0de4da3a5d4c6
<node-ip> - - [22/Jul/2026:09:44:14 +0000] "GET /api/v1/namespaces/fleet-local/secrets?fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833798126 HTTP/2.0" 200 2650 "-" "rancher-system-agent/v0.0.0 (linux/amd64) kubernetes/$Format" 98 0.003 [cattle-system-rancher-80] [] 10.52.13.232:80 2650 0.003 200 672999c238efb6489cfb2d2e26c92ae8
2026/07/22 09:45:14 [error] 35080#35080: *198985750 upstream timed out (110: Connection timed out) while reading upstream, client: <node-ip>, server: _, request: "GET /api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833800054&timeout=6m35s&timeoutSeconds=395&watch=true HTTP/2.0", upstream: "http://10.52.11.90:80/api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833800054&timeout=6m35s&timeoutSeconds=395&watch=true", host: "<master-vip>"
<node-ip> - - [22/Jul/2026:09:45:14 +0000] "GET /api/v1/namespaces/fleet-local/secrets?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcustom-e711b305e5c5-machine-plan&resourceVersion=833800054&timeout=6m35s&timeoutSeconds=395&watch=true HTTP/2.0" 200 0 "-" "rancher-system-agent/v0.0.0 (linux/amd64) kubernetes/$Format" 149 60.001 [cattle-system-rancher-80] [] 10.52.11.90:80 0 60.001 200 059f6e4458e8bc1a785bfcba7b021b55

查看fleet-local machines.cluster.x-k8s.io 一直Provisioning：

kubectl -n fleet-local  get machine
NAME                  CLUSTER   NODENAME    PROVIDERID         PHASE          AGE     VERSION
custom-129b79bbaa9d   local                                    Provisioning   6h12m   
custom-185fa781d934   local     hci-**-**    rke2://hci-**-**    Running        253d    
.
.
.

kubectl -n fleet-local get secret | grep custom-e711b305e5c5
custom-e711b305e5c5-machine-plan的data是空

custom-e711b305e5c5-machine-bootstrap               rke.cattle.io/bootstrap               1      6h22m
custom-e711b305e5c5-machine-bootstrap-token-ktvwb   kubernetes.io/service-account-token   3      6h22m
custom-e711b305e5c5-machine-plan                    rke.cattle.io/machine-plan            0      6h22m
custom-e711b305e5c5-machine-plan-token-8rm2z        kubernetes.io/service-account-token   3      6h22m

kubectl -n cattle-system logs -f --tail 200 -l app=rancher
有error日志

2026/07/23 02:19:27 [INFO] [systemcharts] feature ManagedSystemUpgradeController = false, toEnable = true
2026/07/23 02:19:28 [INFO] Downloading repo index from http://harvester-cluster-repo.cattle-system/charts/index.yaml
2026/07/23 02:20:45 [ERROR] rkecluster fleet-local/local: error while retrieving management cluster from cache: management cluster cache was nil
2026/07/23 02:20:45 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-b413012cd086: waiting for probes: kube-controller-manager, kube-scheduler
2026/07/23 02:22:05 [ERROR] rkecluster fleet-local/local: error while retrieving management cluster from cache: management cluster cache was nil
2026/07/23 02:22:05 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-b413012cd086: waiting for probes: kube-controller-manager, kube-scheduler
W0723 02:25:07.525367      39 warnings.go:70] kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2.
2026/07/23 02:25:10 [ERROR] rkecluster fleet-local/local: error while retrieving management cluster from cache: management cluster cache was nil
2026/07/23 02:25:10 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-b413012cd086: waiting for probes: kube-controller-manager, kube-scheduler
2026/07/23 02:30:46 [ERROR] rkecluster fleet-local/local: error while retrieving management cluster from cache: management cluster cache was nil
2026/07/23 02:30:46 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-b413012cd086: waiting for probes: kube-controller-manager, kube-scheduler
2026/07/23 02:32:06 [ERROR] rkecluster fleet-local/local: error while retrieving management cluster from cache: management cluster cache was nil
2026/07/23 02:32:06 [INFO] [planner] rkecluster fleet-local/local: configuring bootstrap node(s) custom-b413012cd086: waiting for probes: kube-controller-manager, kube-scheduler
W0723 02:32:45.537714      39 warnings.go:70] kubevirt.io/v1 VirtualMachineInstancePresets is now deprecated and will be removed in v2.
2026/07/23 02:35:11 [ERROR] rkecluster fleet-local/local: error while retrieving management cluster from cache: management cluster cache was nil
2

还需要检查如下内容：

# 1. 检查所有节点状态
kubectl get nodes -o wide

# 2. 检查静态 Pod（kube-controller-manager、kube-scheduler、etcd）是否正常运行
kubectl -n kube-system get pods -l tier=control-plane

# 3. 查看具体哪台节点的 Pod 探针不健康
kubectl -n kube-system describe pod -l component=kube-controller-manager
kubectl -n kube-system describe pod -l component=kube-scheduler
3

进行了检查没发现异常，检查信息如下
kubectl get nodes -o wide

NAME        STATUS   ROLES                       AGE    VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION                 CONTAINER-RUNTIME
hci-10-1    Ready    worker                      251d   v1.31.4+rke2r1   10.5.10.1     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-11   Ready    worker                      70d    v1.31.4+rke2r1   10.5.10.11    <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-2    Ready    worker                      231d   v1.31.4+rke2r1   10.5.10.2     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-3    Ready    control-plane,etcd,master   492d   v1.31.4+rke2r1   10.5.10.3     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-4    Ready    worker                      401d   v1.31.4+rke2r1   10.5.10.4     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-5    Ready    control-plane,etcd,master   471d   v1.31.4+rke2r1   10.5.10.5     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-6    Ready    control-plane,etcd,master   471d   v1.31.4+rke2r1   10.5.10.6     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-7    Ready    worker                      405d   v1.31.4+rke2r1   10.5.10.7     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-8    Ready    worker                      251d   v1.31.4+rke2r1   10.5.10.8     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-10-9    Ready    worker                      251d   v1.31.4+rke2r1   10.5.10.9     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-11-1    Ready    worker                      471d   v1.31.4+rke2r1   10.5.11.1     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-11-2    Ready    worker                      471d   v1.31.4+rke2r1   10.5.11.2     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-11-3    Ready    worker                      471d   v1.31.4+rke2r1   10.5.11.3     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2
hci-11-4    Ready    worker                      471d   v1.31.4+rke2r1   10.5.11.4     <none>        Harvester v1.4.2   5.14.21-150500.55.94-default   containerd://1.7.23-k3s2

kubectl -n kube-system get pods -l tier=control-plane

NAME                                READY   STATUS    RESTARTS   AGE
cloud-controller-manager-hci-10-3   1/1     Running   0          16d
cloud-controller-manager-hci-10-5   1/1     Running   0          232d
cloud-controller-manager-hci-10-6   1/1     Running   0          251d
etcd-hci-10-3                       1/1     Running   0          16d
etcd-hci-10-5                       1/1     Running   0          232d
etcd-hci-10-6                       1/1     Running   4          251d
kube-apiserver-hci-10-3             1/1     Running   0          16d
kube-apiserver-hci-10-5             1/1     Running   0          232d
kube-apiserver-hci-10-6             1/1     Running   0          251d
kube-controller-manager-hci-10-3    1/1     Running   0          16d
kube-controller-manager-hci-10-5    1/1     Running   0          232d
kube-controller-manager-hci-10-6    1/1     Running   0          16d
kube-proxy-hci-10-1                 1/1     Running   0          232d
kube-proxy-hci-10-11                1/1     Running   0          16d
kube-proxy-hci-10-2                 1/1     Running   0          231d
kube-proxy-hci-10-3                 1/1     Running   0          16d
kube-proxy-hci-10-4                 1/1     Running   0          232d
kube-proxy-hci-10-5                 1/1     Running   0          232d
kube-proxy-hci-10-6                 1/1     Running   1          251d
kube-proxy-hci-10-7                 1/1     Running   0          55d
kube-proxy-hci-10-8                 1/1     Running   0          142d
kube-proxy-hci-10-9                 1/1     Running   0          142d
kube-proxy-hci-11-1                 1/1     Running   0          142d
kube-proxy-hci-11-2                 1/1     Running   0          55d
kube-proxy-hci-11-3                 1/1     Running   0          55d
kube-proxy-hci-11-4                 1/1     Running   0          51d
kube-scheduler-hci-10-3             1/1     Running   0          16d
kube-scheduler-hci-10-5             1/1     Running   0          232d
kube-scheduler-hci-10-6             1/1     Running   0          16d

hci-10-6:~ # 
hci-10-6:~ # kubectl -n kube-system describe pod -l component=kube-controller-manager
Name:                 kube-controller-manager-hci-10-3
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hci-10-3/10.5.10.3
Start Time:           Thu, 09 Jul 2026 10:12:17 +0000
Labels:               component=kube-controller-manager
                      tier=control-plane
Status:               Running
IP:                   10.5.10.3
IPs:
  IP:           10.5.10.3
Controlled By:  Node/hci-10-3
Containers:
  kube-controller-manager:
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.31.4-rke2r1-build20241212
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-controller-manager
    Args:
      --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins
      --terminated-pod-gc-threshold=1000
      --permit-port-sharing=true
      --allocate-node-cidrs=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --bind-address=127.0.0.1
      --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager
      --cluster-cidr=10.52.0.0/16
      --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
      --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
      --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
      --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
      --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --configure-cloud-routes=false
      --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle
      --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --profiling=false
      --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --secure-port=10257
      --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key
      --service-cluster-ip-range=10.53.0.0/16
      --use-service-account-credentials=true
    State:          Running
      Started:      Tue, 07 Jul 2026 03:05:05 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     200m
      memory:  256Mi
    Liveness:  http-get https://localhost:10257/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10257/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  d8f5f1c7e58c995604153a8a3a1cf43f170f845549af3fb9f5eb93f5e0aa0b9a
      NO_PROXY:   .svc,.cluster.local,10.52.0.0/16,10.53.0.0/16
    Mounts:
      /etc/ca-certificates from dir1 (rw)
      /etc/ssl/certs from dir0 (rw)
      /var/lib/rancher/rke2/server/cred/controller.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.key from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.nochain.crt from file3 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.crt from file4 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.key from file5 (ro)
      /var/lib/rancher/rke2/server/tls/kube-controller-manager from extra-mount-0 (rw)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file6 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.key from file7 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.nochain.crt from file8 (ro)
      /var/lib/rancher/rke2/server/tls/service.current.key from file9 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  dir0:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  dir1:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
  file0:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/db/etcd/name
    HostPathType:  File
  file1:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/cred/controller.kubeconfig
    HostPathType:  File
  file2:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-ca.key
    HostPathType:  File
  file3:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
    HostPathType:  File
  file4:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-controller.crt
    HostPathType:  File
  file5:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/client-controller.key
    HostPathType:  File
  file6:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/server-ca.crt
    HostPathType:  File
  file7:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/server-ca.key
    HostPathType:  File
  file8:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
    HostPathType:  File
  file9:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/service.current.key
    HostPathType:  File
  extra-mount-0:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rancher/rke2/server/tls/kube-controller-manager
    HostPathType:  Directory
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute op=Exists
Events:            <none>


Name:                 kube-controller-manager-hci-10-5
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hci-10-5/10.5.10.5
Start Time:           Wed, 03 Dec 2025 06:14:27 +0000
Labels:               component=kube-controller-manager
                      tier=control-plane
Status:               Running
IP:                   10.5.10.5
IPs:
  IP:           10.5.10.5
Controlled By:  Node/hci-10-5
Containers:
  kube-controller-manager:
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.31.4-rke2r1-build20241212
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-controller-manager
    Args:
      --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins
      --terminated-pod-gc-threshold=1000
      --permit-port-sharing=true
      --allocate-node-cidrs=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --bind-address=127.0.0.1
      --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager
      --cluster-cidr=10.52.0.0/16
      --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
      --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
      --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
      --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
      --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --configure-cloud-routes=false
      --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle
      --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --profiling=false
      --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --secure-port=10257
      --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key
      --service-cluster-ip-range=10.53.0.0/16
      --use-service-account-credentials=true
    State:          Running
      Started:      Wed, 03 Dec 2025 06:14:28 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     200m
      memory:  256Mi
    Liveness:  http-get https://localhost:10257/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10257/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  ff2d859c5c07e016b6e5a2356e0493aec3c7792c44b68d3b20fd680303e7845c
      NO_PROXY:   .svc,.cluster.local,10.52.0.0/16,10.53.0.0/16
    Mounts:
      /etc/ca-certificates from dir1 (rw)
      /etc/ssl/certs from dir0 (rw)
      /var/lib/rancher/rke2/server/cred/controller.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.key from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.nochain.crt from file3 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.crt from file4 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.key from file5 (ro)
      /var/lib/rancher/rke2/server/tls/kube-controller-manager from extra-mount-0 (rw)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file6 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.key from file7 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.nochain.crt from file8 (ro)
      /var/lib/rancher/rke2/server/tls/service.current.key from file9 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 

Name:                 kube-controller-manager-hci-10-6
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hci-10-6/10.5.10.6
Start Time:           Mon, 06 Jul 2026 08:42:27 +0000
Labels:               component=kube-controller-manager
                      tier=control-plane
Status:               Running
IP:                   10.5.10.6
IPs:
  IP:           10.5.10.6
Controlled By:  Node/hci-10-6
Containers:
  kube-controller-manager:
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.31.4-rke2r1-build20241212
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-controller-manager
    Args:
      --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins
      --terminated-pod-gc-threshold=1000
      --permit-port-sharing=true
      --allocate-node-cidrs=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --bind-address=127.0.0.1
      --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager
      --cluster-cidr=10.52.0.0/16
      --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
      --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
      --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key
      --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
      --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
      --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key
      --configure-cloud-routes=false
      --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle
      --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig
      --profiling=false
      --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --secure-port=10257
      --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key
      --service-cluster-ip-range=10.53.0.0/16
      --use-service-account-credentials=true
    State:          Running
      Started:      Mon, 06 Jul 2026 08:42:28 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     200m
      memory:  256Mi
    Liveness:  http-get https://localhost:10257/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10257/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  ee718985f7f992f94710653ac13e74d0fe354299bfab7a1a29456b763f5e24c9
      NO_PROXY:   .svc,.cluster.local,10.52.0.0/16,10.53.0.0/16
    Mounts:
      /etc/ca-certificates from dir1 (rw)
      /etc/ssl/certs from dir0 (rw)
      /var/lib/rancher/rke2/server/cred/controller.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.key from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.nochain.crt from file3 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.crt from file4 (ro)
      /var/lib/rancher/rke2/server/tls/client-controller.key from file5 (ro)
      /var/lib/rancher/rke2/server/tls/kube-controller-manager from extra-mount-0 (rw)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file6 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.key from file7 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.nochain.crt from file8 (ro)
      /var/lib/rancher/rke2/server/tls/service.current.key from file9 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True

kubectl -n kube-system describe pod -l component=kube-scheduler

Name:                 kube-scheduler-hci-10-3
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hci-10-3/10.5.10.3
Start Time:           Thu, 09 Jul 2026 10:12:17 +0000
Labels:               component=kube-scheduler
                      tier=control-plane
Status:               Running
IP:                   10.5.10.3
IPs:
  IP:           10.5.10.3
Controlled By:  Node/hci-10-3
Containers:
  kube-scheduler:
    Container ID:  containerd://72b980726cdf00066b0ab704c19ab0f2af008801f118502fcaa6d15b5fa3aa6a
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.31.4-rke2r1-build20241212
    Image ID:      sha256:7a2b010c3c72662a92480174efcd3be3dfea00167237526ad869304f8731efac
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
    Args:
      --permit-port-sharing=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --bind-address=127.0.0.1
      --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler
      --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --profiling=false
      --secure-port=10259
    State:          Running
      Started:      Tue, 07 Jul 2026 03:05:05 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  128Mi
    Liveness:  http-get https://localhost:10259/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10259/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  bcf98a21aa520d6e38b24b66d5e07a1bb2383fc61d91b4efdf19b3091622bf7c
      NO_PROXY:   .svc,.cluster.local,10.52.0.0/16,10.53.0.0/16
    Mounts:
      /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-scheduler.crt from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-scheduler.key from file3 (ro)
      /var/lib/rancher/rke2/server/tls/kube-scheduler from extra-mount-0 (rw)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file4 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 

Name:                 kube-scheduler-hci-10-5
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hci-10-5/10.5.10.5
Start Time:           Wed, 03 Dec 2025 06:14:27 +0000
Labels:               component=kube-scheduler
                      tier=control-plane
Status:               Running
IP:                   10.5.10.5
IPs:
  IP:           10.5.10.5
Controlled By:  Node/hci-10-5
Containers:
  kube-scheduler:
    Container ID:  containerd://6dbb98d51a9591c9dcaad25bdca7f4e7b85e64087235a8cca43629e0be87d927
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.31.4-rke2r1-build20241212
    Image ID:      sha256:7a2b010c3c72662a92480174efcd3be3dfea00167237526ad869304f8731efac
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
    Args:
      --permit-port-sharing=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --bind-address=127.0.0.1
      --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler
      --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --profiling=false
      --secure-port=10259
    State:          Running
      Started:      Wed, 03 Dec 2025 06:14:28 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  128Mi
    Liveness:  http-get https://localhost:10259/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10259/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  55b57715c1d53fb0c72ff7d2157a019330a23c9be0ab296d91fb0bacd5e31c37
      NO_PROXY:   .svc,.cluster.local,10.52.0.0/16,10.53.0.0/16
    Mounts:
      /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-scheduler.crt from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-scheduler.key from file3 (ro)
      /var/lib/rancher/rke2/server/tls/kube-scheduler from extra-mount-0 (rw)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file4 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 

Name:                 kube-scheduler-hci-10-6
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 hci-10-6/10.5.10.6
Start Time:           Mon, 06 Jul 2026 08:42:27 +0000
Labels:               component=kube-scheduler
                      tier=control-plane
Status:               Running
IP:                   10.5.10.6
IPs:
  IP:           10.5.10.6
Controlled By:  Node/hci-10-6
Containers:
  kube-scheduler:
    Container ID:  containerd://4f4acfbd7bc1130860502bb08e770a17d7ce3d807346b0c3125c0b891885adc2
    Image:         index.docker.io/rancher/hardened-kubernetes:v1.31.4-rke2r1-build20241212
    Image ID:      sha256:7a2b010c3c72662a92480174efcd3be3dfea00167237526ad869304f8731efac
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-scheduler
    Args:
      --permit-port-sharing=true
      --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --bind-address=127.0.0.1
      --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler
      --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig
      --profiling=false
      --secure-port=10259
    State:          Running
      Started:      Mon, 06 Jul 2026 08:42:28 +0000
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  128Mi
    Liveness:  http-get https://localhost:10259/healthz delay=10s timeout=15s period=10s #success=1 #failure=8
    Startup:   http-get https://localhost:10259/healthz delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  7a31169352b5dc0be858eb2aeae93166c16a6c5a279061efdf6f9d20c875c5ac
      NO_PROXY:   .svc,.cluster.local,10.52.0.0/16,10.53.0.0/16
    Mounts:
      /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/tls/client-scheduler.crt from file2 (ro)
      /var/lib/rancher/rke2/server/tls/client-scheduler.key from file3 (ro)
      /var/lib/rancher/rke2/server/tls/kube-scheduler from extra-mount-0 (rw)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file4 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True
4

我对 harvester 排查故障不是特别专业，目前没看出来啥问题。

如果希望获得更专业、及时的技术支持，可以了解一下 Harvester 商业订阅服务，如果有需要，可以联系我，我转给对应的负责的同事。

5

OK 3Q

6

问题记录
类似问题：

https://github.com/harvester/harvester/issues/9880

查看下边的最新状态有异常，重启suc pod没有恢复，不清楚怎么触发更新的status
kubectl -n fleet-local describe rkecontrolplanes local

    Last Update Time:   2026-07-09T10:18:22Z
    Message:            configuring bootstrap node(s) custom-b413012cd086: waiting for probes: kube-controller-manager, kube-scheduler
    Reason:             Waiting
    Status:             Unknown
    Type:               Ready
    Last Update Time:   2026-07-09T10:18:22Z
    Message:            RKEControlPlane has not been fully reconciled yet
    Reason:             Waiting
    Status:             Unknown
    Type:               Reconciled

kubectl -n fleet-local describe machine custom-b413012cd086

   Last Transition Time:  2026-07-09T10:12:16Z
    Message:               waiting for probes: kube-controller-manager, kube-scheduler
    Reason:                Waiting
    Status:                Unknown
    Type:                  Reconciled
7

问题：

https://github.com/harvester/harvester/issues/3863

结论：
kube-controller-manager、kube-scheduler证书过期了，之前重启rke2这俩俩证书并没有更新，kube-controller-manager、kube-scheduler还没异常日志。

echo "Rotating kube-controller-manager certificate"
rm /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.{crt,key}
crictl rm -f $(crictl ps -q --name kube-controller-manager)

echo "Rotating kube-scheduler certificate"
rm /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.{crt,key}
crictl rm -f $(crictl ps -q --name kube-scheduler)