Docker安装的rancher导入k3s集群崩溃

hxp0176 · 2025 年12 月 23 日 02:36

Rancher Server 设置

Rancher 版本：2.13.0
安装选项 (Docker install/Helm Chart): docker
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
在线或离线部署：离线部署

下游集群信息

Kubernetes 版本: 1.34.2+k3s1，单节点集群
Cluster Type (Local/Downstream): Local
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等):

用户信息*

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：管理员
- 如果自定义，自定义权限集：

主机操作系统：**openEuler 20.03SP4

问题描述：在rancher内操作导入本机的k3s集群(容器外)后，rancher崩溃，回滚k3s的部署可以恢复rancher正常运行。

结果：

截图：

日志


2025/12/23 02:15:20 [INFO] initialized required info for telemetry manager
2025/12/23 02:15:20 [INFO] telemetry manager info not available yet, re-queing check...
2025/12/23 02:15:20 [INFO] starting telemetry gathering
2025/12/23 02:15:20 [INFO] telemetry manager started
2025/12/23 02:15:22 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:21.894800931 +0000 UTC
2025/12/23 02:15:22 [ERROR] http: TLS handshake error from 192.168.22.243:7647: remote error: tls: unknown certificate
2025/12/23 02:15:22 [ERROR] http: TLS handshake error from 192.168.22.243:7071: remote error: tls: unknown certificate
2025/12/23 02:15:22 [ERROR] http: TLS handshake error from 192.168.22.243:11622: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:1503: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:6255: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:14034: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:2509: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:8385: remote error: tls: unknown certificate
2025/12/23 02:15:24 [ERROR] http: TLS handshake error from 192.168.22.243:9772: remote error: tls: unknown certificate
2025/12/23 02:15:24 [ERROR] http: TLS handshake error from 192.168.22.243:4735: remote error: tls: unknown certificate
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for cluster.x-k8s.io/v1beta1, Kind=Machine
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=Node
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for cluster.x-k8s.io_v1beta1_Machine (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=NodePool
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_Node (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_NodePool (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for provisioning.cattle.io/v1, Kind=Cluster
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for provisioning.cattle.io_v1_Cluster (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for cluster.x-k8s.io/v1beta1, Kind=Machine (took 104.016841ms)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=Node (took 107.548633ms)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=NodePool (took 113.518524ms)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for provisioning.cattle.io/v1, Kind=Cluster (took 120.708018ms)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=Project
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for ui.cattle.io/v1, Kind=NavLink
2025/12/23 02:15:26 [ERROR] http: TLS handshake error from 192.168.22.243:4299: remote error: tls: unknown certificate
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_Project (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for ui.cattle.io_v1_NavLink (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=Project (took 153.035674ms)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for ui.cattle.io/v1, Kind=NavLink (took 190.045798ms)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for /v1, Kind=Node
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for /v1, Kind=Endpoints
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for _v1_Node (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for _v1_Endpoints (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for apps/v1, Kind=Deployment
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for apps_v1_Deployment (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for /v1, Kind=Event
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for _v1_Event (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Node (took 120.004356ms)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Endpoints (took 130.62556ms)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for apps/v1, Kind=Deployment (took 106.730526ms)
2025/12/23 02:15:26 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1766455414,O=dynamiclistener-org: notBefore=2025-12-23 01:15:26 +0000 UTC notAfter=2026-12-23 01:15:26 +0000 UTC
2025/12/23 02:15:26 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 3): map[field.cattle.io/projectId:local:p-wsm4g listener.cattle.io/cn-10.43.137.219:10.43.137.219 listener.cattle.io/cn-172.17.0.2:172.17.0.2 listener.cattle.io/fingerprint:SHA1=4291D253BB7E42A0D666900C6F10FA64536E5711]
2025/12/23 02:15:27 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=5817) (count 3): map[field.cattle.io/projectId:local:p-wsm4g listener.cattle.io/cn-10.43.137.219:10.43.137.219 listener.cattle.io/cn-172.17.0.2:172.17.0.2 listener.cattle.io/fingerprint:SHA1=4291D253BB7E42A0D666900C6F10FA64536E5711]
2025/12/23 02:15:27 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Event (took 509.472107ms)
2025/12/23 02:15:28 [INFO] CacheFor STARTS creating informer for /v1, Kind=Pod
2025/12/23 02:15:28 [INFO] Started SQL cache garbage collection for _v1_Pod (interval=15m0s, keep=1000)
2025/12/23 02:15:28 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Pod (took 108.743925ms)
2025/12/23 02:15:28 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:28.800995067 +0000 UTC
2025/12/23 02:15:31 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wsm4g to namespace=cattle-fleet-local-system
2025/12/23 02:15:31 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:15:32 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wsm4g to namespace=cattle-fleet-local-system
2025/12/23 02:15:32 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:32.491706613 +0000 UTC
2025/12/23 02:15:36 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:36.770695864 +0000 UTC
2025/12/23 02:15:42 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lqn5r': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lqn5r), requeuing
2025/12/23 02:15:42 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lqn5r': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lqn5r), requeuing
2025/12/23 02:15:43 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lqn5r': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lqn5r), requeuing
2025/12/23 02:15:43 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:43.180401666 +0000 UTC
2025/12/23 02:15:55 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:55.701815103 +0000 UTC
2025/12/23 02:16:06 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:06.765226066 +0000 UTC
2025/12/23 02:16:17 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:17.835008275 +0000 UTC
2025/12/23 02:16:20 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:20.124055872 +0000 UTC
2025/12/23 02:16:27 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:27.958805985 +0000 UTC
I1223 02:17:34.714379      43 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I1223 02:18:22.247202      43 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
2025/12/23 02:18:38 [INFO] CacheFor STARTS creating informer for apps/v1, Kind=StatefulSet
2025/12/23 02:18:38 [INFO] Started SQL cache garbage collection for apps_v1_StatefulSet (interval=15m0s, keep=1000)
2025/12/23 02:18:38 [INFO] CacheFor IS DONE creating informer for apps/v1, Kind=StatefulSet (took 106.043578ms)
2025/12/23 02:18:59 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:19:00 [INFO] CacheFor STARTS creating informer for catalog.cattle.io/v1, Kind=ClusterRepo
2025/12/23 02:19:00 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=NodeDriver
2025/12/23 02:19:00 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=KontainerDriver
2025/12/23 02:19:00 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_NodeDriver (interval=15m0s, keep=1000)
2025/12/23 02:19:00 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_KontainerDriver (interval=15m0s, keep=1000)
2025/12/23 02:19:00 [INFO] Started SQL cache garbage collection for catalog.cattle.io_v1_ClusterRepo (interval=15m0s, keep=1000)
2025/12/23 02:19:00 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=NodeDriver (took 111.539623ms)
2025/12/23 02:19:00 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=KontainerDriver (took 121.984476ms)
2025/12/23 02:19:00 [INFO] CacheFor IS DONE creating informer for catalog.cattle.io/v1, Kind=ClusterRepo (took 130.065962ms)
2025/12/23 02:19:03 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=User
2025/12/23 02:19:03 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=RoleTemplate
2025/12/23 02:19:03 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_User (interval=15m0s, keep=1000)
2025/12/23 02:19:03 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_RoleTemplate (interval=15m0s, keep=1000)
2025/12/23 02:19:03 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=User (took 104.694146ms)
2025/12/23 02:19:03 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=RoleTemplate (took 113.311396ms)
2025/12/23 02:19:29 [INFO] [mgmt-cluster-rbac-delete] Creating namespace c-xmmmh
2025/12/23 02:19:29 [INFO] [mgmt-cluster-rbac-delete] Creating Default project for cluster c-xmmmh
2025/12/23 02:19:29 [INFO] [mgmt-project-rbac-create] Creating namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-64mlb for project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Creating System project for cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Setting InitialRolesPopulated condition on project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Creating namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-cd6bx-projectowner
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Creating creator clusterRoleTemplateBinding for user user-64mlb for cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-64mlb for project p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-cd6bx for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Setting InitialRolesPopulated condition on project p-qxrrr
2025/12/23 02:19:30 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-xmmmh-p-cd6bx, err=Operation cannot be fulfilled on namespaces "c-xmmmh-p-cd6bx": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole c-xmmmh-clustermember
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating role/clusterRole c-xmmmh-clusterowner
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-qxrrr-projectowner
2025/12/23 02:19:30 [INFO] CacheFor STARTS creating informer for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment
2025/12/23 02:19:30 [INFO] CacheFor STARTS creating informer for rke.cattle.io/v1, Kind=ETCDSnapshot
2025/12/23 02:19:30 [ERROR] http: TLS handshake error from 192.168.22.243:13133: remote error: tls: unknown certificate
2025/12/23 02:19:30 [INFO] Started SQL cache garbage collection for cluster.x-k8s.io_v1beta1_MachineDeployment (interval=15m0s, keep=1000)
2025/12/23 02:19:30 [INFO] Started SQL cache garbage collection for rke.cattle.io_v1_ETCDSnapshot (interval=15m0s, keep=1000)
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-qxrrr for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Setting InitialRolesPopulated condition on cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-xmmmh
2025/12/23 02:19:30 [INFO] CacheFor IS DONE creating informer for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment (took 109.268168ms)
2025/12/23 02:19:30 [INFO] CacheFor IS DONE creating informer for rke.cattle.io/v1, Kind=ETCDSnapshot (took 159.320007ms)
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-64mlb with role cluster-owner in namespace c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role project-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding crb-d3p2gfxhv2 for cluster membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-64mlb with role cluster-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role admin in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-64mlb with role cluster-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role project-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role admin in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-xmmmh
2025/12/23 02:19:52 [ERROR] http: TLS handshake error from 192.168.22.214:35914: local error: tls: bad record MAC
2025/12/23 02:20:02 [ERROR] http: TLS handshake error from 192.168.22.214:35934: local error: tls: bad record MAC
2025/12/23 02:20:22 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:23 [ERROR] http: TLS handshake error from 192.168.22.243:3908: remote error: tls: unknown certificate
2025/12/23 02:20:23 [ERROR] http: TLS handshake error from 192.168.22.243:6024: remote error: tls: unknown certificate
2025/12/23 02:20:30 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:31 [ERROR] http: TLS handshake error from 192.168.22.243:4261: remote error: tls: unknown certificate
2025/12/23 02:20:31 [ERROR] http: TLS handshake error from 192.168.22.243:13651: remote error: tls: unknown certificate
2025/12/23 02:20:32 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:32 [ERROR] http: TLS handshake error from 192.168.22.243:9992: read tcp 172.17.0.2:443->192.168.22.243:9992: read: connection reset by peer
2025/12/23 02:20:32 [ERROR] http: TLS handshake error from 192.168.22.243:12850: remote error: tls: unknown certificate
2025/12/23 02:20:37 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:38 [INFO] Starting cluster controllers for c-xmmmh
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=ResourceQuota controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=LimitRange controller
2025/12/23 02:20:38 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Namespace controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Node controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Secret controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=ServiceAccount controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Secret controller
2025/12/23 02:20:38 [INFO] Starting cluster controllers for c-xmmmh
2025/12/23 02:20:38 [INFO] Starting cluster agent for c-xmmmh [owner=true]
2025/12/23 02:20:38 [INFO] RDPClient: certificate updated successfully
2025/12/23 02:20:38 [INFO] Creating clusterRole for roleTemplate Project Owner (project-owner).
2025/12/23 02:20:38 [INFO] Creating clusterRole for roleTemplate Cluster Owner (cluster-owner).
2025/12/23 02:20:38 [INFO] Creating clusterRole for roleTemplate Create Namespaces (create-ns).
2025/12/23 02:20:38 [INFO] Creating clusterRole project-owner-promoted for project access to global resource.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in cattle-system
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding User user-64mlb Role cluster-owner
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role create-ns.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in cattle-system
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role p-cd6bx-namespaces-edit.
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role project-owner-promoted.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in default
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in default
2025/12/23 02:20:38 [INFO] Updating clusterRoleBinding crb-wsih4kgxli for project access to global resource for subject user-64mlb role create-ns.
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role p-qxrrr-namespaces-edit.
2025/12/23 02:20:38 [INFO] Updating clusterRoleBinding crb-y2dnlx2rne for project access to global resource for subject user-64mlb role project-owner-promoted.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in kube-system
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in kube-system
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in kube-public
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in kube-public
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in kube-node-lease
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in kube-node-lease
2025/12/23 02:20:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-system:cattle-impersonation-user-64mlb-token-wpj2k] for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] to be populated with token
2025/12/23 02:20:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-system:cattle-impersonation-user-64mlb-token-wpj2k] for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] to be populated with token
2025/12/23 02:20:38 [INFO] Rolling back ServiceAccount secret for [cattle-impersonation-system:cattle-impersonation-user-64mlb-token-7glml]
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: got the service account token for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] in 8.739718ms
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: got the service account token for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] in 16.99499ms
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in cattle-impersonation-system
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in cattle-impersonation-system
2025/12/23 02:20:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:20:39 [INFO] Created machine for node [fzdt03]
2025/12/23 02:20:40 [INFO] Creating user for principal system://c-xmmmh
2025/12/23 02:20:40 [INFO] Creating globalRoleBindings for u-wavpw5e27b
2025/12/23 02:20:40 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:20:40.747498022 +0000 UTC
2025/12/23 02:20:40 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-p848m
2025/12/23 02:20:40 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-p848m for user u-wavpw5e27b with role cattle-globalrole-user
2025/12/23 02:20:41 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:41 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:41 [ERROR] http: TLS handshake error from 192.168.22.243:5359: remote error: tls: unknown certificate
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject u-wavpw5e27b
2025/12/23 02:20:41 [ERROR] http: TLS handshake error from 192.168.22.243:6226: remote error: tls: unknown certificate
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-wavpw5e27b with role cluster-owner in namespace c-xmmmh
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-wavpw5e27b with role cluster-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-wavpw5e27b with role cluster-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:20:42 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-system:cattle-impersonation-u-wavpw5e27b-token-7pxvg] for service account [cattle-impersonation-system:cattle-impersonation-u-wavpw5e27b] to be populated with token
2025/12/23 02:20:42 [INFO] EnsureSecretForServiceAccount: got the service account token for service account [cattle-impersonation-system:cattle-impersonation-u-wavpw5e27b] in 7.544772ms
2025/12/23 02:20:42 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:42 [INFO] Creating clusterRoleBinding User u-wavpw5e27b Role cluster-owner
2025/12/23 02:20:44 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:46 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:47 [ERROR] http: TLS handshake error from 192.168.22.243:8679: remote error: tls: unknown certificate
2025/12/23 02:20:47 [ERROR] http: TLS handshake error from 192.168.22.243:8408: remote error: tls: unknown certificate
2025/12/23 02:20:48 [INFO] Redeploy Rancher Agents is needed for c-xmmmh: forceDeploy=false, agent/auth image changed=false, agent features changed=true
2025/12/23 02:20:48 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:48 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:50 [INFO] Redeploy Rancher Agents is needed for c-xmmmh: forceDeploy=false, agent/auth image changed=false, agent features changed=true
2025/12/23 02:20:50 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:51 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:20:51.422861612 +0000 UTC
2025/12/23 02:20:53 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:54 [INFO] error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
2025/12/23 02:20:55 [INFO] Creating roleBinding User user-64mlb Role project-owner in cattle-local-user-passwords
2025/12/23 02:20:55 [INFO] Creating roleBinding User user-64mlb Role admin in cattle-local-user-passwords
2025/12/23 02:20:55 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-local-user-passwords, err=Operation cannot be fulfilled on namespaces "cattle-local-user-passwords": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:21:06 httputil: ReverseProxy read error during body copy: unexpected EOF
2025/12/23 02:21:06 [ERROR] Lost connection to pod: lost connection to pod, retrying in 1 secs.
E1223 02:21:06.173703      43 leaderelection.go:441] Failed to update lock optimistically: Put "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers?timeout=15m0s": read tcp 127.0.0.1:41406->127.0.0.1:6443: read: connection reset by peer, falling back to slow path
2025/12/23 02:21:06 [WARNING] [2] encountered error "write tcp 127.0.0.1:52154->127.0.0.1:5555: write: broken pipe" while writing error "tunnel disconnect" to close remotedialer
2025/12/23 02:21:06 [ERROR] Remotedialer proxy error
2025/12/23 02:21:06 [WARNING] [2] encountered error "write tcp 127.0.0.1:52154->127.0.0.1:5555: write: broken pipe" while writing error "writeto tcp 127.0.0.1:39188->127.0.0.1:6666: read tcp 127.0.0.1:39188->127.0.0.1:6666: use of closed network connection" to close remotedialer
2025/12/23 02:21:06 [WARNING] [7] encountered error "write tcp 127.0.0.1:52154->127.0.0.1:5555: write: broken pipe" while writing error "writeto tcp 127.0.0.1:39194->127.0.0.1:6666: read tcp 127.0.0.1:39194->127.0.0.1:6666: use of closed network connection" to close remotedialer
2025/12/23 02:21:06 [ERROR] auth: Error updating lastUsedAt for token token-vz89q: Patch "https://127.0.0.1:6443/apis/management.cattle.io/v3/tokens/token-vz89q": read tcp 127.0.0.1:41406->127.0.0.1:6443: read: connection reset by peer
2025/12/23 02:21:06 [FATAL] k3s exited with: exit status 1

ksd · 2025 年12 月 23 日 09:55

得 exec 到rancher 容器中，然后查看 k3s.log 日志，直到最终崩溃，可查看到具体的 K3s 退出的原因

hxp0176 · 2026 年1 月 8 日 08:10

W0108 08:05:42.706164     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.706228     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
I0108 08:05:42.706780     189 event.go:389] "Event occurred" object="kube-system/traefik" fieldPath="" kind="Addon" apiVersion="k3s.cattle.io/v1" type="Normal" reason="DeletingManifest" message="Deleting manifest at \"/var/lib/rancher/k3s/server/manifests/traefik.yaml\""
W0108 08:05:42.797504     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.797566     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
{"level":"warn","ts":"2026-01-08T08:05:42.948225Z","caller":"txn/util.go:93","msg":"apply request took too long","took":"146.528014ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/management.cattle.io/features/unsupported-storage-drivers\" limit:1 ","response":"range_response_count:1 size:782"}
{"level":"info","ts":"2026-01-08T08:05:42.948345Z","caller":"traceutil/trace.go:172","msg":"trace[966080967] range","detail":"{range_begin:/registry/management.cattle.io/features/unsupported-storage-drivers; range_end:; response_count:1; response_revision:10658; }","duration":"146.66077ms","start":"2026-01-08T08:05:42.801668Z","end":"2026-01-08T08:05:42.948328Z","steps":["trace[966080967] 'range keys from in-memory index tree'  (duration: 146.42067ms)"],"step_count":1}
W0108 08:05:42.953184     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.953250     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
W0108 08:05:42.960820     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.960873     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
W0108 08:05:42.966925     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.966971     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
W0108 08:05:42.974430     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.974472     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
W0108 08:05:42.981006     189 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0108 08:05:42.981037     189 dispatcher.go:213] "Unhandled Error" err="failed calling webhook \"rancher.cattle.io.features.management.cattle.io\": failed to call webhook: Post \"https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s\": no endpoints available for service \"rancher-webhook\""
I0108 08:05:43.276646     189 scope.go:117] "RemoveContainer" containerID="e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062"
E0108 08:05:43.277453     189 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062\": not found" containerID="e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062"
I0108 08:05:43.277508     189 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062"} err="failed to get container status \"e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062\": rpc error: code = NotFound desc = an error occurred when try to find container \"e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062\": not found"
I0108 08:05:43.277542     189 scope.go:117] "RemoveContainer" containerID="daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6"
E0108 08:05:43.279855     189 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to set removing state for container \"daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6\": container is already in removing state" containerID="daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6"
I0108 08:05:43.279913     189 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6"} err="rpc error: code = Unknown desc = failed to set removing state for container \"daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6\": container is already in removing state"
I0108 08:05:43.279946     189 scope.go:117] "RemoveContainer" containerID="daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6"
E0108 08:05:43.281924     189 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to set removing state for container \"daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6\": container is already in removing state" containerID="daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6"
I0108 08:05:43.281972     189 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6"} err="rpc error: code = Unknown desc = failed to set removing state for container \"daeda098d7164bbf7e9070bb93996d58496de6f08aeeb5132ff7cc60c98a30e6\": container is already in removing state"
I0108 08:05:43.281999     189 scope.go:117] "RemoveContainer" containerID="d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"
{"level":"info","ts":"2026-01-08T08:05:43.284918Z","caller":"traceutil/trace.go:172","msg":"trace[1903324248] transaction","detail":"{read_only:false; response_revision:10660; number_of_response:1; }","duration":"214.234035ms","start":"2026-01-08T08:05:43.070662Z","end":"2026-01-08T08:05:43.284896Z","steps":["trace[1903324248] 'process raft request'  (duration: 186.159923ms)","trace[1903324248] 'compare'  (duration: 27.937221ms)"],"step_count":2}
{"level":"warn","ts":"2026-01-08T08:05:43.506593Z","caller":"txn/util.go:93","msg":"apply request took too long","took":"121.209096ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/apiextensions.k8s.io/customresourcedefinitions/dynamicschemas.management.cattle.io\" limit:1 ","response":"range_response_count:1 size:18492"}
{"level":"info","ts":"2026-01-08T08:05:43.506714Z","caller":"traceutil/trace.go:172","msg":"trace[1718458946] range","detail":"{range_begin:/registry/apiextensions.k8s.io/customresourcedefinitions/dynamicschemas.management.cattle.io; range_end:; response_count:1; response_revision:10660; }","duration":"121.347065ms","start":"2026-01-08T08:05:43.385331Z","end":"2026-01-08T08:05:43.506678Z","steps":["trace[1718458946] 'range keys from in-memory index tree'  (duration: 121.022887ms)"],"step_count":1}
I0108 08:05:43.973684     189 scope.go:117] "RemoveContainer" containerID="8b3b9b2cc1e0d06689027496ab01ac2dd56c6abeb10a17ad7553008f233cf543"
E0108 08:05:43.975171     189 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"8b3b9b2cc1e0d06689027496ab01ac2dd56c6abeb10a17ad7553008f233cf543\": rpc error: code = NotFound desc = an error occurred when try to find container \"8b3b9b2cc1e0d06689027496ab01ac2dd56c6abeb10a17ad7553008f233cf543\": not found" containerID="8b3b9b2cc1e0d06689027496ab01ac2dd56c6abeb10a17ad7553008f233cf543"
I0108 08:05:43.975217     189 scope.go:117] "RemoveContainer" containerID="7a17f6a390576912fcd3dc1b2b74a6011e6e5f3e485aca3a407a8425c4d764fd"
E0108 08:05:43.975672     189 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"7a17f6a390576912fcd3dc1b2b74a6011e6e5f3e485aca3a407a8425c4d764fd\": rpc error: code = NotFound desc = an error occurred when try to find container \"7a17f6a390576912fcd3dc1b2b74a6011e6e5f3e485aca3a407a8425c4d764fd\": not found" containerID="7a17f6a390576912fcd3dc1b2b74a6011e6e5f3e485aca3a407a8425c4d764fd"
I0108 08:05:43.975771     189 scope.go:117] "RemoveContainer" containerID="d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"
E0108 08:05:43.980059     189 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to set removing state for container \"d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde\": container is already in removing state" containerID="d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"
E0108 08:05:43.980132     189 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to set removing state for container \"d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde\": container is already in removing state" containerID="d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"
I0108 08:05:43.980178     189 scope.go:117] "RemoveContainer" containerID="bdeba547feb49a128459726e2bbab2ef1dc4f8769c75b437fd01ce2094eb8228"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49610: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:39748: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:39760: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49614: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:39752: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:39746: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49604: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:39758: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:39750: dial tcp 10.42.0.41:6666: connect: no route to host"
time="2026-01-08T08:05:44Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49608: dial tcp 10.42.0.41:6666: connect: no route to host"
{"level":"info","ts":"2026-01-08T08:05:44.505777Z","caller":"traceutil/trace.go:172","msg":"trace[924466158] transaction","detail":"{read_only:false; response_revision:10661; number_of_response:1; }","duration":"175.906601ms","start":"2026-01-08T08:05:44.329855Z","end":"2026-01-08T08:05:44.505762Z","steps":["trace[924466158] 'process raft request'  (duration: 175.738799ms)"],"step_count":1}
W0108 08:05:44.507019     189 handler_proxy.go:99] no RequestInfo found in the context
E0108 08:05:44.507059     189 remote_available_controller.go:462] "Unhandled Error" err="v1.ext.cattle.io failed with: failing or missing response from https://10.42.0.41:6666/apis/ext.cattle.io/v1: Get \"https://10.42.0.41:6666/apis/ext.cattle.io/v1\": proxy error from 127.0.0.1:6443 while dialing 10.42.0.41:6666, code 502: 502 Bad Gateway"
E0108 08:05:44.507116     189 controller.go:146] "Unhandled Error" err=<
        Error updating APIService "v1.ext.cattle.io" with err: failed to download v1.ext.cattle.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 >
I0108 08:05:44.923169     189 scope.go:117] "RemoveContainer" containerID="d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"
E0108 08:05:44.923644     189 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde\": not found" containerID="d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"
I0108 08:05:44.923683     189 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde"} err="failed to get container status \"d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde\": rpc error: code = NotFound desc = an error occurred when try to find container \"d6792ee84406a8dced9fc8afa80d2e2d065225fdabbe1c9807d48eacd4eb7cde\": not found"
I0108 08:05:44.923712     189 scope.go:117] "RemoveContainer" containerID="b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884"
E0108 08:05:44.924066     189 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = an error occurred when try to find container \"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884\": not found" containerID="b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884"
I0108 08:05:44.924098     189 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884"} err="failed to get container status \"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884\": rpc error: code = NotFound desc = an error occurred when try to find container \"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884\": not found"
I0108 08:05:44.924121     189 scope.go:117] "RemoveContainer" containerID="b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884"
I0108 08:05:44.924450     189 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"containerd","ID":"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884"} err="failed to get container status \"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884\": rpc error: code = NotFound desc = an error occurred when try to find container \"b838fc5bea472ba07dc1c1560aab6cb6a3b374a6526e9f8586aa041e3886e884\": not found"
I0108 08:05:44.924480     189 scope.go:117] "RemoveContainer" containerID="13478a2b24c1b08def0155af7396ff40c959bd84ae7806c77b6c5bac1e716720"
W0108 08:05:45.509105     189 handler_proxy.go:99] no RequestInfo found in the context
W0108 08:05:45.509170     189 handler_proxy.go:99] no RequestInfo found in the context
E0108 08:05:45.509177     189 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for \"v1.ext.cattle.io\" failed with: Error, could not get list of group versions for APIService"
I0108 08:05:45.509302     189 controller.go:126] OpenAPI AggregationController: action for item v1.ext.cattle.io: Rate Limited Requeue.
E0108 08:05:45.509305     189 controller.go:102] "Unhandled Error" err=<
        loading OpenAPI spec for "v1.ext.cattle.io" failed with: failed to download v1.ext.cattle.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
 >
{"level":"info","ts":"2026-01-08T08:05:45.509305Z","caller":"traceutil/trace.go:172","msg":"trace[919991017] transaction","detail":"{read_only:false; response_revision:10664; number_of_response:1; }","duration":"219.873902ms","start":"2026-01-08T08:05:45.289410Z","end":"2026-01-08T08:05:45.509284Z","steps":["trace[919991017] 'process raft request'  (duration: 219.673402ms)"],"step_count":1}
I0108 08:05:45.510381     189 controller.go:109] OpenAPI AggregationController: action for item v1.ext.cattle.io: Rate Limited Requeue.
I0108 08:05:46.040576     189 scope.go:117] "RemoveContainer" containerID="15618b91ed36997373ead0cb9e2d6e5936727e41afdd3b5591899626653e6ad8"
E0108 08:05:46.041175     189 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"15618b91ed36997373ead0cb9e2d6e5936727e41afdd3b5591899626653e6ad8\": rpc error: code = NotFound desc = an error occurred when try to find container \"15618b91ed36997373ead0cb9e2d6e5936727e41afdd3b5591899626653e6ad8\": not found" containerID="15618b91ed36997373ead0cb9e2d6e5936727e41afdd3b5591899626653e6ad8"
I0108 08:05:46.041226     189 scope.go:117] "RemoveContainer" containerID="13478a2b24c1b08def0155af7396ff40c959bd84ae7806c77b6c5bac1e716720"
E0108 08:05:46.044000     189 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to set removing state for container \"13478a2b24c1b08def0155af7396ff40c959bd84ae7806c77b6c5bac1e716720\": container is already in removing state" containerID="13478a2b24c1b08def0155af7396ff40c959bd84ae7806c77b6c5bac1e716720"
E0108 08:05:46.044044     189 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to set removing state for container \"13478a2b24c1b08def0155af7396ff40c959bd84ae7806c77b6c5bac1e716720\": container is already in removing state" containerID="13478a2b24c1b08def0155af7396ff40c959bd84ae7806c77b6c5bac1e716720"
I0108 08:05:46.044066     189 scope.go:117] "RemoveContainer" containerID="e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062"
E0108 08:05:46.044437     189 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062\": rpc error: code = NotFound desc = an error occurred when try to find container \"e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062\": not found" containerID="e2c7861dd43bacf0c3fa2e03d4a8a0eb623d6b9ae28282e66c800bfdc9a66062"
W0108 08:05:46.914191     189 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.namespaces.create-non-kubesystem: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": no endpoints available for service "rancher-webhook"

这是容器内k3s-log的最后部分，实在很困惑，现在启动时看着很正常，几个小时后就崩溃，导入本机上的另外一个k3s时也很正常，但导入成功后就会直接崩溃，然后重启后会再崩溃一次，然后就引起整个服务崩溃无法正常启动

hxp0176 · 2026 年1 月 8 日 08:13

目前已经进入几乎无法排查的地步了，rancher容器不断地在崩溃，内置的k3s只要一起来就会崩溃，想不明白

ksd · 2026 年1 月 8 日 12:52

这个是崩溃前的日志？没看出来 k3s 退出的相关说明啊