rancher2.15重启下游集群的node节点后agent就连接不上Rancher。

myrancher · 2026 年8 月 10 日 07:13

具体agent的报错如下：
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs -n cattle-system cattle-cluster-agent-5dd9cddd47-5ndqh
INFO: Environment: CATTLE_ADDRESS=10.42.0.36 CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.43.120.218:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.43.120.218:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.43.120.218 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.43.120.218:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.43.120.218 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.43.120.218 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CREDENTIAL_NAME=cattle-credentials-4b950b21b8 CATTLE_FEATURES=fleet=false,managed-system-upgrade-controller=true,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningprebootstrap=false,provisioningv2=false,rke2=false,turtles=false,ui-sql-cache=true CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=8503a0b9-937c-4cc3-af13-e81866756fb9 CATTLE_INTERNAL_ADDRESS= CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-5dd9cddd47-5ndqh CATTLE_RANCHER_TURTLES_VERSION=110.0.0+up0.27.0 CATTLE_RANCHER_WEBHOOK_VERSION=110.0.0+up0.11.0 CATTLE_SERVER=https://ali-rancher-test.ich.tech CATTLE_SERVER_VERSION=v2.15.0 CATTLE_SUC_APP_NAME_OVERRIDE=mcc-aliyun-managed-system-upgrade-controller CATTLE_SYSTEM_DEFAULT_REGISTRY=harbor.ich.cn
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local openstacklocal nameserver 10.43.0.10 options ndots:5
INFO: https://ali-rancher-test.ich.tech/ping is accessible
INFO: ali-rancher-test.ich.tech resolves to 10.72.2.189
time=“2026-08-10T02:56:02Z” level=info msg=“starting cattle-credential-cleanup goroutine in the background”
time=“2026-08-10T02:56:02Z” level=info msg=“Listening on /tmp/log.sock”
time=“2026-08-10T02:56:02Z” level=info msg=“Rancher agent version v2.15.0 is starting”
time=“2026-08-10T02:56:02Z” level=error msg=“unable to read CA file from /etc/kubernetes/ssl/certs/serverca: open /etc/kubernetes/ssl/certs/serverca: no such file or directory”
time=“2026-08-10T02:56:03Z” level=info msg=“Connecting to wss://ali-rancher-test.ich.tech/v3/connect/register with token starting with htc9mx8c4fqcpv9485fv665rfkw”
time=“2026-08-10T02:56:03Z” level=info msg=“Connecting to proxy” url=“wss://ali-rancher-test.ich.tech/v3/connect/register”
time=“2026-08-10T02:56:03Z” level=error msg=“Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: cluster not found” error=“websocket: bad handshake”
time=“2026-08-10T02:56:03Z” level=error msg=“Remotedialer proxy error” error=“websocket: bad handshake”
time=“2026-08-10T02:56:13Z” level=info msg=“Connecting to wss://ali-rancher-test.ich.tech/v3/connect/register with token starting with htc9mx8c4fqcpv9485fv665rfkw”
time=“2026-08-10T02:56:13Z” level=info msg=“Connecting to proxy” url=“wss://ali-rancher-test.ich.tech/v3/connect/register”
time=“2026-08-10T02:56:13Z” level=error msg=“Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: cluster not found” error=“websocket: bad handshake”
time=“2026-08-10T02:56:13Z” level=error msg=“Remotedialer proxy error” error=“websocket: bad handshake”
time=“2026-08-10T02:56:23Z” level=info msg=“Connecting to wss://ali-rancher-test.ich.tech/v3/connect/register with token starting with htc9mx8c4fqcpv9485fv665rfkw”
time=“2026-08-10T02:56:23Z” level=info msg=“Connecting to proxy” url=“wss://ali-rancher-test.ich.tech/v3/connect/register”
time=“2026-08-10T02:56:23Z” level=error msg=“Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: cluster not found” error=“websocket: bad handshake”

我在Rancher控制台删除这个集群，重新导入后就可以。但是只要导入集群的node重启，就会复现。

myrancher · 2026 年8 月 10 日 07:14

我们在Rancher 2.14版本随意重启下游k8s集群的机器都不会出现。

ksd · 2026 年8 月 11 日 06:40

你在下游集群执行 kubectl get deployment -n cattle-system 和 kubectl get pod -n cattle-system 看下一共有几个 rancher agent 的 deployment 和 pod

myrancher · 2026 年8 月 11 日 08:45

root@wanglei:~# /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get deploy -n cattle-system
NAME READY UP-TO-DATE AVAILABLE AGE
cattle-cluster-agent 1/1 1 1 27m
rancher-webhook 1/1 1 1 25m
system-upgrade-controller 1/1 1 1 25m

root@wanglei:~# /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get pods -n cattle-system
NAME READY STATUS RESTARTS AGE
cattle-cluster-agent-5f785bffcc-prvb5 1/1 Running 1 (2m46s ago) 3m29s
helm-operation-8dpr4 0/2 Completed 0 25m
helm-operation-dt9h2 0/2 Completed 0 24m
helm-operation-msglq 0/2 Completed 0 25m
helm-operation-sj2pn 0/2 Completed 0 25m
rancher-webhook-76579775b8-jwvvk 1/1 Running 1 (5m10s ago) 25m
system-upgrade-controller-7df76fb9bb-dz8b6 1/1 Running 1 (5m10s ago) 25m

这些POD都在的，而且我还发现，这种导入的rke2集群，配置管理即使开了，执行etcd备份和换秘钥操作也是失败的。

myrancher · 2026 年8 月 11 日 08:46

这是我在重启node出问题之前执行的一次etcd备份，一直pending。

myrancher · 2026 年8 月 11 日 08:48

journalctl -u rancher-system-agent --no-pager -a
Aug 11 16:38:17 wanglei systemd[1]: Started rancher-system-agent.service - Rancher System Agent.
Aug 11 16:38:18 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:18+08:00” level=info msg=“Rancher System Agent version v0.15.0 (a396bb4) is starting”
Aug 11 16:38:18 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:18+08:00” level=info msg=“Using directory /var/lib/rancher/agent/work for work”
Aug 11 16:38:18 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:18+08:00” level=info msg=“Starting remote watch of plans”
Aug 11 16:38:19 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:19+08:00” level=info msg=“Starting /v1, Kind=Secret controller”
Aug 11 16:38:19 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:19+08:00” level=info msg=“Detected first start, force-applying one-time instruction set”
Aug 11 16:38:19 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:19+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:24 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:24+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:29 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:29+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:34 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:34+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:39 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:39+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:44 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:44+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:49 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:49+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:54 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:54+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:38:59 wanglei rancher-system-agent[808]: time=“2026-08-11T16:38:59+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:04 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:04+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:09 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:09+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:14 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:14+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:19 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:19+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:24 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:24+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:29 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:29+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:34 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:34+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:39 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:39+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:44 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:44+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”
Aug 11 16:39:49 wanglei rancher-system-agent[808]: time=“2026-08-11T16:39:49+08:00” level=error msg=“[K8s] Maximum failure threshold exceeded for plan with checksum value of 10405428a216903b96cce03e6c7c5ea48af1e5cb9d2fe348d14579a69fb9476b, (failures: 1, threshold: 1)”

上面是rancher-system-agent的日志，我今天又出现了这个问题。跟上次一样。 100%的。

myrancher · 2026 年8 月 11 日 09:07

另外我还想咨询一个问题。完全使用rke2安装的集群导入到Rancher，和先用Rancher安装一个rke2集群，然后删除再导入，是一样的吗？？？

ksd · 2026 年8 月 12 日 01:28

首先确定下你的下游K8s 集群的版本和 rancher 的版本是否兼容，这个可以通过本论坛的右上角支持矩阵连接去确认

另外，感觉像是有垃圾数据比如 token，导致重启 node 时候重新触发，你可以这样试试：

在 rancher ui 删除这个集群（你得确定这个是个导入集群，UI 删除导入集群不会卸载下游集群）
通过下游 rke2 的 kubectl 去移除对应的 rancher cluster deployment，还有 cattle-system namespace。
从新在 rancher ui 上创建导入集群，然后从新导入

myrancher · 2026 年8 月 12 日 02:09

我详细的操作步骤是这样的：

1、我使用Rancher安装一个 k8s集群，当然这个集群是基于rke2的，这个集群也使用正常。

2、我在Rancher上删除这个集群，此时Rancher的界面上已经不存在这个集群的UI了

3、我登录到这个集群，kubectl都可以使用，集群脱离Rancher后也是正常的

4、我通过官方的脚本清理这个集群跟Rancher相关的组件。
https://raw.githubusercontent.com/rancher/rancher/main/cleanup/user-cluster.sh

https://raw.githubusercontent.com/rancher/rancher/release/v2.15/cleanup/user-cluster.yml

5、此时我认为集群是干净的，之后执行导入这个集群到Rancher

6、导入后的集群是可以正常使用的，没有问题。

7、我重启刚才导入集群的node节点机器

8、之后就报上面的错误（如上），agent连接过来失败。就是不能重启机器。一旦重启就不行。

ksd · 2026 年8 月 12 日 05:20

我上面说的你在看一下

myrancher · 2026 年8 月 12 日 06:37

首先确定下你的下游K8s 集群的版本和 rancher 的版本是否兼容，这个可以通过本论坛的右上角支持矩阵连接去确认

–回复：如我的描述，下游集群其实是以前这个Rancher创建的，所以版本肯定是兼容的，因为集群就是用以前Rancher提供的版本创新的。只不过创建完删除再导入。

另外，感觉像是有垃圾数据比如 token，导致重启 node 时候重新触发，你可以这样试试：

在 rancher ui 删除这个集群（你得确定这个是个导入集群，UI 删除导入集群不会卸载下游集群）

–回复：如上描述，我确实是在Rancher UI上删除了，这个非常确定，删除后再UI上执行导入。

通过下游 rke2 的 kubectl 去移除对应的 rancher cluster deployment，还有 cattle-system namespace。

–回复：清理后如下，使用的官方的脚本清理的。
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get ns
NAME STATUS AGE
cilium-secrets Active 84m
default Active 84m
external-dns Active 84m
kube-node-lease Active 84m
kube-public Active 84m
kube-system Active 84m
local Active 82m

从新在 rancher ui 上创建导入集群，然后从新导入
回复：是的，我的导入也是创建的新的导入。导入后是可以跟Rancher成功连接的。但是部分功能无法使用，重启node节点，就不行了，出现上面的情况。

ksd · 2026 年8 月 13 日 02:49

经过排查，问题的原因：
当导入集群，重启下游集群节点之后，/var/lib/rancher/rke2/server/manifests/ 目录有之前卸载集群遗留的 rancher yaml，面的 YAML 会被 RKE2 的 Helm Controller / Addon 机制自动启动一个 rancher-cluster-agent 和 rancher 连接，因为 rancher 中并没有这个 cluster-agent 对应的集群，所以连接不上。

处理方法：手动删除 /var/lib/rancher/rke2/server/manifests/rancher 目录，然后删除 cattle-system namespace，重新导入即可