Rancher 2.5.16 etcd集群崩溃

Rancher 2.x
1

Rancher Server 设置

  • Rancher 版本：2.5.16
  • 安装选项 (Docker install/Helm Chart): docker install
    • 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：RKE1
  • 在线或离线部署： 在线

下游集群信息

  • Kubernetes 版本: v1.20.15
  • Cluster Type (Local/Downstream):
    • 如果 Downstream，是什么类型的集群?(自定义/导入或为托管 等): 自定义

用户信息

  • 登录用户的角色是什么？ （管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
    • 如果自定义，自定义权限集：

主机操作系统： centos7.9

**问题描述：*在 API 准备就绪之前，直接与 API 交互的功能将不可用。

Failed to ensure monitoring project name: failed to find “cattle-prometheus” Namespace: Get “https://172.23.1.252:6443/api/v1/namespaces/cattle-prometheus”: x509: certificate signed by unknown authority*
master 1 台
etcd 1 台

**重现步骤：重启 etcd、重启 rancher 无法恢复，后重启 etcd 服务器，结果 etcd 服务器 docker ps 查看无任何容器。
尝试页面恢复 etcd 也没效果，提示failed to start backup server on all etcd nodes: [Failed to run backup server container, container logs: time=“2024-08-01T08:37:34Z” level=fatal msg=“stat /backup/c-md7bg-rl-vpmcz_2024-07-31T19:19:59Z: no such file or directory” ]

*结果：
image

image
image1066×851 276 KB

*

预期结果：

*截图：

1722502268884
17225022688841471×701 92.9 KB

*

**其他上下文信息：*2024/08/01 08:37:20 [WARNING] [clean-catalog-orphan-bindings] Error when deleting role global-catalog, roles.rbac.authorization.k8s.io “global-catalog” not found
2024/08/01 08:37:20 [WARNING] [CleanupOrphanCatalogBindingsDone] error during orphan binding cleanup: roles.rbac.authorization.k8s.io “global-catalog” not found
2024/08/01 08:37:20 [ERROR] failed to cleanup orphan catalog bindings
2024/08/01 08:37:20 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
2024/08/01 08:37:21 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.5/data.json
2024/08/01 08:37:21 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2024/08/01 08:37:25 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:33770
2024/08/01 08:37:25 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:44626
2024/08/01 08:37:25 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:40074
2024/08/01 08:37:25 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2024/08/01 08:37:25 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2024/08/01 08:37:25 [INFO] kontainerdriver azurekubernetesservice stopped
2024/08/01 08:37:25 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2024/08/01 08:37:25 [INFO] kontainerdriver googlekubernetesengine stopped
2024/08/01 08:37:25 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2024/08/01 08:37:28 [INFO] kontainerdriver rancherkubernetesengine listening on address 127.0.0.1:39617
2024/08/01 08:37:28 [INFO] Not checking if state file is included in snapshot file for [c-md7bg-rl-vpmcz_2024-07-31T19:19:59Z], using local state file [management-state/rke/rke-555605321/cluster.rkestate]
2024/08/01 08:37:28 [INFO] Restoring etcd snapshot c-md7bg-rl-vpmcz_2024-07-31T19:19:59Z
2024/08/01 08:37:28 [INFO] Successfully Deployed state file at [management-state/rke/rke-555605321/cluster.rkestate]
2024/08/01 08:37:28 [INFO] [dialer] Setup tunnel for host [172.23.0.45]
2024/08/01 08:37:28 [INFO] [dialer] Setup tunnel for host [172.23.1.252]
2024/08/01 08:37:28 [WARNING] Failed to set up SSH tunneling for host [172.23.1.252]: Can’t retrieve Docker Info: error during connect: Get “http://%!F(MISSING)var%!F(MISSING)run%!F(MISSING)docker.sock/v1.24/info”: can not build dialer to [c-md7bg:m-a226e9fbdfc1]
2024/08/01 08:37:28 [WARNING] Removing host [172.23.1.252] from node lists
2024/08/01 08:37:28 [INFO] Checking if container [cert-deployer] is running on host [172.23.0.45], try #1
2024/08/01 08:37:28 [INFO] Image [rancher/rke-tools:v0.1.80] exists on host [172.23.0.45]
2024/08/01 08:37:29 [INFO] Starting container [cert-deployer] on host [172.23.0.45], try #1
2024/08/01 08:37:29 [INFO] Checking if container [cert-deployer] is running on host [172.23.0.45], try #1
2024/08/01 08:37:34 [INFO] Checking if container [cert-deployer] is running on host [172.23.0.45], try #1
2024/08/01 08:37:34 [INFO] Removing container [cert-deployer] on host [172.23.0.45], try #1
2024/08/01 08:37:34 [INFO] [etcd] etcd snapshot configuration found and no s3 backup configuration found, will use local as source
2024/08/01 08:37:34 [INFO] Stopping container [etcd] on host [172.23.0.45] with stopTimeoutDuration [5s], try #1
2024/08/01 08:37:34 [WARNING] Can’t stop Docker container [etcd] for host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:34 [INFO] Stopping container [etcd] on host [172.23.0.45] with stopTimeoutDuration [5s], try #2
2024/08/01 08:37:34 [WARNING] Can’t stop Docker container [etcd] for host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:34 [INFO] Stopping container [etcd] on host [172.23.0.45] with stopTimeoutDuration [5s], try #3
2024/08/01 08:37:34 [WARNING] Can’t stop Docker container [etcd] for host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:34 [WARNING] failed to stop etcd container on host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:34 [INFO] [etcd] starting backup server on host [172.23.0.45]
2024/08/01 08:37:34 [INFO] Image [rancher/rke-tools:v0.1.80] exists on host [172.23.0.45]
2024/08/01 08:37:34 [INFO] Starting container [etcd-Serve-backup] on host [172.23.0.45], try #1
2024/08/01 08:37:34 [INFO] [etcd] Successfully started [etcd-Serve-backup] container on host [172.23.0.45]
2024/08/01 08:37:39 [INFO] Removing container [etcd-Serve-backup] on host [172.23.0.45], try #1
2024/08/01 08:37:39 [WARNING] failed to start backup server on host [172.23.0.45]: Failed to run backup server container, container logs: time=“2024-08-01T08:37:34Z” level=fatal msg=“stat /backup/c-md7bg-rl-vpmcz_2024-07-31T19:19:59Z: no such file or directory”

2024/08/01 08:37:39 [INFO] Starting container [etcd] on host [172.23.0.45], try #1
2024/08/01 08:37:39 [WARNING] Can’t start Docker container [etcd] on host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:39 [INFO] Starting container [etcd] on host [172.23.0.45], try #2
2024/08/01 08:37:39 [WARNING] Can’t start Docker container [etcd] on host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:39 [INFO] Starting container [etcd] on host [172.23.0.45], try #3
2024/08/01 08:37:39 [WARNING] Can’t start Docker container [etcd] on host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:39 [WARNING] failed to start etcd container on host [172.23.0.45]: Error response from daemon: No such container: etcd
2024/08/01 08:37:39 [INFO] kontainerdriver rancherkubernetesengine stopped
2024/08/01 08:37:42 [ERROR] error syncing ‘c-md7bg’: handler cluster-deploy: Get “https://172.23.1.252:6443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent”: x509: certificate signed by unknown authority, handler cluster-provisioner-controller: failed to start backup server on all etcd nodes: [Failed to run backup server container, container logs: time=“2024-08-01T08:37:34Z” level=fatal msg=“stat /backup/c-md7bg-rl-vpmcz_2024-07-31T19:19:59Z: no such file or directory”
], requeuing
2024/08/01 08:37:42 [ERROR] error syncing ‘c-md7bg’: handler cluster-deploy: Get “https://172.23.1.252:6443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent”: x509: certificate signed by unknown authority, requeuing
2024/08/01 08:37:54 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:37:54 [ERROR] failed to start cluster controllers c-md7bg: context canceled
2024-08-01 08:39:03.807401 W | etcdserver: read-only range request "key:"/registry/configmaps/fleet-system/fleet-controller-lock" " with result “range_response_count:1 size:579” took too long (193.403537ms) to execute
2024-08-01 08:39:03.807495 W | etcdserver: read-only range request "key:"/registry/management.cattle.io/clusterroletemplatebindings/" range_end:"/registry/management.cattle.io/clusterroletemplatebindings0" count_only:true " with result “range_response_count:0 size:9” took too long (328.53212ms) to execute
2024/08/01 08:39:29 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:39:29 [ERROR] failed to start cluster controllers c-md7bg: context canceled
2024/08/01 08:41:34 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:41:34 [ERROR] failed to start cluster controllers c-md7bg: context canceled
W0801 08:43:13.882912 8 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
2024/08/01 08:43:39 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:43:39 [ERROR] failed to start cluster controllers c-md7bg: context canceled
W0801 08:45:24.594263 8 warnings.go:80] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
2024/08/01 08:45:44 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:45:44 [ERROR] failed to start cluster controllers c-md7bg: context canceled

2024-08-01 08:47:06.559159 I | mvcc: store.index: compact 176662265
2024-08-01 08:47:06.645040 I | mvcc: finished scheduled compaction at 176662265 (took 85.263375ms)
2024/08/01 08:47:49 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:47:49 [ERROR] failed to start cluster controllers c-md7bg: context canceled
2024/08/01 08:49:54 [INFO] Stopping cluster agent for c-md7bg
2024/08/01 08:49:54 [ERROR] failed to start cluster controllers c-md7bg: context canceled
W0801 08:51:25.883772 8 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
2024-08-01 08:52:06.562403 I | mvcc: store.index: compact 176664124
2024-08-01 08:52:06.615217 I | mvcc: finished scheduled compaction at 176664124 (took 52.220552ms)*

日志