Rancher Server 设置
- Rancher 版本：2.13.0
- 安装选项 (Docker install/Helm Chart): docker
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
- 在线或离线部署：离线部署
下游集群信息
- Kubernetes 版本: 1.34.2+k3s1，单节点集群
- Cluster Type (Local/Downstream): Local
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管 等):
用户信息*
- 登录用户的角色是什么？ （管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：管理员
- 如果自定义，自定义权限集：
主机操作系统：**openEuler 20.03SP4
问题描述：在rancher内操作导入本机的k3s集群(容器外)后，rancher崩溃，回滚k3s的部署可以恢复rancher正常运行。
结果：
截图：
日志
2025/12/23 02:15:20 [INFO] initialized required info for telemetry manager
2025/12/23 02:15:20 [INFO] telemetry manager info not available yet, re-queing check...
2025/12/23 02:15:20 [INFO] starting telemetry gathering
2025/12/23 02:15:20 [INFO] telemetry manager started
2025/12/23 02:15:22 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:21.894800931 +0000 UTC
2025/12/23 02:15:22 [ERROR] http: TLS handshake error from 192.168.22.243:7647: remote error: tls: unknown certificate
2025/12/23 02:15:22 [ERROR] http: TLS handshake error from 192.168.22.243:7071: remote error: tls: unknown certificate
2025/12/23 02:15:22 [ERROR] http: TLS handshake error from 192.168.22.243:11622: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:1503: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:6255: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:14034: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:2509: remote error: tls: unknown certificate
2025/12/23 02:15:23 [ERROR] http: TLS handshake error from 192.168.22.243:8385: remote error: tls: unknown certificate
2025/12/23 02:15:24 [ERROR] http: TLS handshake error from 192.168.22.243:9772: remote error: tls: unknown certificate
2025/12/23 02:15:24 [ERROR] http: TLS handshake error from 192.168.22.243:4735: remote error: tls: unknown certificate
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for cluster.x-k8s.io/v1beta1, Kind=Machine
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=Node
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for cluster.x-k8s.io_v1beta1_Machine (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=NodePool
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_Node (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_NodePool (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] CacheFor STARTS creating informer for provisioning.cattle.io/v1, Kind=Cluster
2025/12/23 02:15:24 [INFO] Started SQL cache garbage collection for provisioning.cattle.io_v1_Cluster (interval=15m0s, keep=1000)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for cluster.x-k8s.io/v1beta1, Kind=Machine (took 104.016841ms)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=Node (took 107.548633ms)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=NodePool (took 113.518524ms)
2025/12/23 02:15:24 [INFO] CacheFor IS DONE creating informer for provisioning.cattle.io/v1, Kind=Cluster (took 120.708018ms)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=Project
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for ui.cattle.io/v1, Kind=NavLink
2025/12/23 02:15:26 [ERROR] http: TLS handshake error from 192.168.22.243:4299: remote error: tls: unknown certificate
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_Project (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for ui.cattle.io_v1_NavLink (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=Project (took 153.035674ms)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for ui.cattle.io/v1, Kind=NavLink (took 190.045798ms)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for /v1, Kind=Node
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for /v1, Kind=Endpoints
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for _v1_Node (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for _v1_Endpoints (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for apps/v1, Kind=Deployment
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for apps_v1_Deployment (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor STARTS creating informer for /v1, Kind=Event
2025/12/23 02:15:26 [INFO] Started SQL cache garbage collection for _v1_Event (interval=15m0s, keep=1000)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Node (took 120.004356ms)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Endpoints (took 130.62556ms)
2025/12/23 02:15:26 [INFO] CacheFor IS DONE creating informer for apps/v1, Kind=Deployment (took 106.730526ms)
2025/12/23 02:15:26 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1766455414,O=dynamiclistener-org: notBefore=2025-12-23 01:15:26 +0000 UTC notAfter=2026-12-23 01:15:26 +0000 UTC
2025/12/23 02:15:26 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 3): map[field.cattle.io/projectId:local:p-wsm4g listener.cattle.io/cn-10.43.137.219:10.43.137.219 listener.cattle.io/cn-172.17.0.2:172.17.0.2 listener.cattle.io/fingerprint:SHA1=4291D253BB7E42A0D666900C6F10FA64536E5711]
2025/12/23 02:15:27 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=5817) (count 3): map[field.cattle.io/projectId:local:p-wsm4g listener.cattle.io/cn-10.43.137.219:10.43.137.219 listener.cattle.io/cn-172.17.0.2:172.17.0.2 listener.cattle.io/fingerprint:SHA1=4291D253BB7E42A0D666900C6F10FA64536E5711]
2025/12/23 02:15:27 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Event (took 509.472107ms)
2025/12/23 02:15:28 [INFO] CacheFor STARTS creating informer for /v1, Kind=Pod
2025/12/23 02:15:28 [INFO] Started SQL cache garbage collection for _v1_Pod (interval=15m0s, keep=1000)
2025/12/23 02:15:28 [INFO] CacheFor IS DONE creating informer for /v1, Kind=Pod (took 108.743925ms)
2025/12/23 02:15:28 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:28.800995067 +0000 UTC
2025/12/23 02:15:31 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wsm4g to namespace=cattle-fleet-local-system
2025/12/23 02:15:31 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-local-system": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:15:32 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-wsm4g to namespace=cattle-fleet-local-system
2025/12/23 02:15:32 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:32.491706613 +0000 UTC
2025/12/23 02:15:36 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:36.770695864 +0000 UTC
2025/12/23 02:15:42 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lqn5r': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lqn5r), requeuing
2025/12/23 02:15:42 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lqn5r': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lqn5r), requeuing
2025/12/23 02:15:43 [ERROR] error syncing 'cattle-fleet-system/helm-operation-lqn5r': handler helm-operation: an error on the server ("container not found (\"proxy\")") has prevented the request from succeeding (get pods helm-operation-lqn5r), requeuing
2025/12/23 02:15:43 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:43.180401666 +0000 UTC
2025/12/23 02:15:55 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:15:55.701815103 +0000 UTC
2025/12/23 02:16:06 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:06.765226066 +0000 UTC
2025/12/23 02:16:17 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:17.835008275 +0000 UTC
2025/12/23 02:16:20 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:20.124055872 +0000 UTC
2025/12/23 02:16:27 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:16:27.958805985 +0000 UTC
I1223 02:17:34.714379 43 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I1223 02:18:22.247202 43 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
2025/12/23 02:18:38 [INFO] CacheFor STARTS creating informer for apps/v1, Kind=StatefulSet
2025/12/23 02:18:38 [INFO] Started SQL cache garbage collection for apps_v1_StatefulSet (interval=15m0s, keep=1000)
2025/12/23 02:18:38 [INFO] CacheFor IS DONE creating informer for apps/v1, Kind=StatefulSet (took 106.043578ms)
2025/12/23 02:18:59 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:19:00 [INFO] CacheFor STARTS creating informer for catalog.cattle.io/v1, Kind=ClusterRepo
2025/12/23 02:19:00 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=NodeDriver
2025/12/23 02:19:00 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=KontainerDriver
2025/12/23 02:19:00 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_NodeDriver (interval=15m0s, keep=1000)
2025/12/23 02:19:00 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_KontainerDriver (interval=15m0s, keep=1000)
2025/12/23 02:19:00 [INFO] Started SQL cache garbage collection for catalog.cattle.io_v1_ClusterRepo (interval=15m0s, keep=1000)
2025/12/23 02:19:00 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=NodeDriver (took 111.539623ms)
2025/12/23 02:19:00 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=KontainerDriver (took 121.984476ms)
2025/12/23 02:19:00 [INFO] CacheFor IS DONE creating informer for catalog.cattle.io/v1, Kind=ClusterRepo (took 130.065962ms)
2025/12/23 02:19:03 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=User
2025/12/23 02:19:03 [INFO] CacheFor STARTS creating informer for management.cattle.io/v3, Kind=RoleTemplate
2025/12/23 02:19:03 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_User (interval=15m0s, keep=1000)
2025/12/23 02:19:03 [INFO] Started SQL cache garbage collection for management.cattle.io_v3_RoleTemplate (interval=15m0s, keep=1000)
2025/12/23 02:19:03 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=User (took 104.694146ms)
2025/12/23 02:19:03 [INFO] CacheFor IS DONE creating informer for management.cattle.io/v3, Kind=RoleTemplate (took 113.311396ms)
2025/12/23 02:19:29 [INFO] [mgmt-cluster-rbac-delete] Creating namespace c-xmmmh
2025/12/23 02:19:29 [INFO] [mgmt-cluster-rbac-delete] Creating Default project for cluster c-xmmmh
2025/12/23 02:19:29 [INFO] [mgmt-project-rbac-create] Creating namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-64mlb for project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Creating System project for cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Setting InitialRolesPopulated condition on project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Creating namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-cd6bx-projectowner
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Creating creator clusterRoleTemplateBinding for user user-64mlb for cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Creating creator projectRoleTemplateBinding for user user-64mlb for project p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-cd6bx for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Setting InitialRolesPopulated condition on project p-qxrrr
2025/12/23 02:19:30 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=c-xmmmh-p-cd6bx, err=Operation cannot be fulfilled on namespaces "c-xmmmh-p-cd6bx": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole c-xmmmh-clustermember
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating role/clusterRole c-xmmmh-clusterowner
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role/clusterRole p-qxrrr-projectowner
2025/12/23 02:19:30 [INFO] CacheFor STARTS creating informer for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment
2025/12/23 02:19:30 [INFO] CacheFor STARTS creating informer for rke.cattle.io/v1, Kind=ETCDSnapshot
2025/12/23 02:19:30 [ERROR] http: TLS handshake error from 192.168.22.243:13133: remote error: tls: unknown certificate
2025/12/23 02:19:30 [INFO] Started SQL cache garbage collection for cluster.x-k8s.io_v1beta1_MachineDeployment (interval=15m0s, keep=1000)
2025/12/23 02:19:30 [INFO] Started SQL cache garbage collection for rke.cattle.io_v1_ETCDSnapshot (interval=15m0s, keep=1000)
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for membership in project p-qxrrr for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Setting InitialRolesPopulated condition on cluster c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-xmmmh
2025/12/23 02:19:30 [INFO] CacheFor IS DONE creating informer for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment (took 109.268168ms)
2025/12/23 02:19:30 [INFO] CacheFor IS DONE creating informer for rke.cattle.io/v1, Kind=ETCDSnapshot (took 159.320007ms)
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:30 [INFO] [mgmt-project-rbac-create] Updating project p-qxrrr
2025/12/23 02:19:30 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-64mlb with role cluster-owner in namespace c-xmmmh
2025/12/23 02:19:30 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role project-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Updating clusterRoleBinding crb-d3p2gfxhv2 for cluster membership in cluster c-xmmmh for subject user-64mlb
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-64mlb with role cluster-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role admin in namespace c-xmmmh-p-cd6bx
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating role admin in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating role project-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject user-64mlb with role cluster-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role project-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject user-64mlb with role admin in namespace c-xmmmh-p-qxrrr
2025/12/23 02:19:31 [INFO] [mgmt-cluster-rbac-delete] Updating cluster c-xmmmh
2025/12/23 02:19:52 [ERROR] http: TLS handshake error from 192.168.22.214:35914: local error: tls: bad record MAC
2025/12/23 02:20:02 [ERROR] http: TLS handshake error from 192.168.22.214:35934: local error: tls: bad record MAC
2025/12/23 02:20:22 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:23 [ERROR] http: TLS handshake error from 192.168.22.243:3908: remote error: tls: unknown certificate
2025/12/23 02:20:23 [ERROR] http: TLS handshake error from 192.168.22.243:6024: remote error: tls: unknown certificate
2025/12/23 02:20:30 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:31 [ERROR] http: TLS handshake error from 192.168.22.243:4261: remote error: tls: unknown certificate
2025/12/23 02:20:31 [ERROR] http: TLS handshake error from 192.168.22.243:13651: remote error: tls: unknown certificate
2025/12/23 02:20:32 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:32 [ERROR] http: TLS handshake error from 192.168.22.243:9992: read tcp 172.17.0.2:443->192.168.22.243:9992: read: connection reset by peer
2025/12/23 02:20:32 [ERROR] http: TLS handshake error from 192.168.22.243:12850: remote error: tls: unknown certificate
2025/12/23 02:20:37 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:38 [INFO] Starting cluster controllers for c-xmmmh
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=ResourceQuota controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=LimitRange controller
2025/12/23 02:20:38 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Namespace controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Node controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2025/12/23 02:20:38 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Secret controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=ServiceAccount controller
2025/12/23 02:20:38 [INFO] Starting /v1, Kind=Secret controller
2025/12/23 02:20:38 [INFO] Starting cluster controllers for c-xmmmh
2025/12/23 02:20:38 [INFO] Starting cluster agent for c-xmmmh [owner=true]
2025/12/23 02:20:38 [INFO] RDPClient: certificate updated successfully
2025/12/23 02:20:38 [INFO] Creating clusterRole for roleTemplate Project Owner (project-owner).
2025/12/23 02:20:38 [INFO] Creating clusterRole for roleTemplate Cluster Owner (cluster-owner).
2025/12/23 02:20:38 [INFO] Creating clusterRole for roleTemplate Create Namespaces (create-ns).
2025/12/23 02:20:38 [INFO] Creating clusterRole project-owner-promoted for project access to global resource.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in cattle-system
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding User user-64mlb Role cluster-owner
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role create-ns.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in cattle-system
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role p-cd6bx-namespaces-edit.
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role project-owner-promoted.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in default
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in default
2025/12/23 02:20:38 [INFO] Updating clusterRoleBinding crb-wsih4kgxli for project access to global resource for subject user-64mlb role create-ns.
2025/12/23 02:20:38 [INFO] Creating clusterRoleBinding for project access to global resource for subject user-64mlb role p-qxrrr-namespaces-edit.
2025/12/23 02:20:38 [INFO] Updating clusterRoleBinding crb-y2dnlx2rne for project access to global resource for subject user-64mlb role project-owner-promoted.
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in kube-system
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in kube-system
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in kube-public
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in kube-public
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in kube-node-lease
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in kube-node-lease
2025/12/23 02:20:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-system:cattle-impersonation-user-64mlb-token-wpj2k] for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] to be populated with token
2025/12/23 02:20:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-system:cattle-impersonation-user-64mlb-token-wpj2k] for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] to be populated with token
2025/12/23 02:20:38 [INFO] Rolling back ServiceAccount secret for [cattle-impersonation-system:cattle-impersonation-user-64mlb-token-7glml]
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: got the service account token for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] in 8.739718ms
2025/12/23 02:20:38 [INFO] EnsureSecretForServiceAccount: got the service account token for service account [cattle-impersonation-system:cattle-impersonation-user-64mlb] in 16.99499ms
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role project-owner in cattle-impersonation-system
2025/12/23 02:20:38 [INFO] Creating roleBinding User user-64mlb Role admin in cattle-impersonation-system
2025/12/23 02:20:38 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:20:39 [INFO] Created machine for node [fzdt03]
2025/12/23 02:20:40 [INFO] Creating user for principal system://c-xmmmh
2025/12/23 02:20:40 [INFO] Creating globalRoleBindings for u-wavpw5e27b
2025/12/23 02:20:40 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:20:40.747498022 +0000 UTC
2025/12/23 02:20:40 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-p848m
2025/12/23 02:20:40 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-p848m for user u-wavpw5e27b with role cattle-globalrole-user
2025/12/23 02:20:41 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:41 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:41 [ERROR] http: TLS handshake error from 192.168.22.243:5359: remote error: tls: unknown certificate
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster c-xmmmh for subject u-wavpw5e27b
2025/12/23 02:20:41 [ERROR] http: TLS handshake error from 192.168.22.243:6226: remote error: tls: unknown certificate
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-wavpw5e27b with role cluster-owner in namespace c-xmmmh
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-wavpw5e27b with role cluster-owner in namespace c-xmmmh-p-qxrrr
2025/12/23 02:20:41 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-wavpw5e27b with role cluster-owner in namespace c-xmmmh-p-cd6bx
2025/12/23 02:20:42 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-system:cattle-impersonation-u-wavpw5e27b-token-7pxvg] for service account [cattle-impersonation-system:cattle-impersonation-u-wavpw5e27b] to be populated with token
2025/12/23 02:20:42 [INFO] EnsureSecretForServiceAccount: got the service account token for service account [cattle-impersonation-system:cattle-impersonation-u-wavpw5e27b] in 7.544772ms
2025/12/23 02:20:42 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:42 [INFO] Creating clusterRoleBinding User u-wavpw5e27b Role cluster-owner
2025/12/23 02:20:44 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:46 [ERROR] Error during subscribe websocket: close sent
2025/12/23 02:20:47 [ERROR] http: TLS handshake error from 192.168.22.243:8679: remote error: tls: unknown certificate
2025/12/23 02:20:47 [ERROR] http: TLS handshake error from 192.168.22.243:8408: remote error: tls: unknown certificate
2025/12/23 02:20:48 [INFO] Redeploy Rancher Agents is needed for c-xmmmh: forceDeploy=false, agent/auth image changed=false, agent features changed=true
2025/12/23 02:20:48 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:48 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:50 [INFO] Redeploy Rancher Agents is needed for c-xmmmh: forceDeploy=false, agent/auth image changed=false, agent features changed=true
2025/12/23 02:20:50 [INFO] Creating system token for u-wavpw5e27b, token: agent-u-wavpw5e27b
2025/12/23 02:20:51 [INFO] Skipping handler for clusterrepo rancher-charts. NumberOfRetries is 0, MaxRetry is 3, ClusterRepo Generation is 1, ObservedGeneration is 1, LastUpdated plus interval is 2025-12-23 03:03:51 +0000 UTC, now is 2025-12-23 02:20:51.422861612 +0000 UTC
2025/12/23 02:20:53 [INFO] Handling backend connection request [c-xmmmh]
2025/12/23 02:20:54 [INFO] error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF
2025/12/23 02:20:55 [INFO] Creating roleBinding User user-64mlb Role project-owner in cattle-local-user-passwords
2025/12/23 02:20:55 [INFO] Creating roleBinding User user-64mlb Role admin in cattle-local-user-passwords
2025/12/23 02:20:55 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-local-user-passwords, err=Operation cannot be fulfilled on namespaces "cattle-local-user-passwords": the object has been modified; please apply your changes to the latest version and try again
2025/12/23 02:21:06 httputil: ReverseProxy read error during body copy: unexpected EOF
2025/12/23 02:21:06 [ERROR] Lost connection to pod: lost connection to pod, retrying in 1 secs.
E1223 02:21:06.173703 43 leaderelection.go:441] Failed to update lock optimistically: Put "https://127.0.0.1:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers?timeout=15m0s": read tcp 127.0.0.1:41406->127.0.0.1:6443: read: connection reset by peer, falling back to slow path
2025/12/23 02:21:06 [WARNING] [2] encountered error "write tcp 127.0.0.1:52154->127.0.0.1:5555: write: broken pipe" while writing error "tunnel disconnect" to close remotedialer
2025/12/23 02:21:06 [ERROR] Remotedialer proxy error
2025/12/23 02:21:06 [WARNING] [2] encountered error "write tcp 127.0.0.1:52154->127.0.0.1:5555: write: broken pipe" while writing error "writeto tcp 127.0.0.1:39188->127.0.0.1:6666: read tcp 127.0.0.1:39188->127.0.0.1:6666: use of closed network connection" to close remotedialer
2025/12/23 02:21:06 [WARNING] [7] encountered error "write tcp 127.0.0.1:52154->127.0.0.1:5555: write: broken pipe" while writing error "writeto tcp 127.0.0.1:39194->127.0.0.1:6666: read tcp 127.0.0.1:39194->127.0.0.1:6666: use of closed network connection" to close remotedialer
2025/12/23 02:21:06 [ERROR] auth: Error updating lastUsedAt for token token-vz89q: Patch "https://127.0.0.1:6443/apis/management.cattle.io/v3/tokens/token-vz89q": read tcp 127.0.0.1:41406->127.0.0.1:6443: read: connection reset by peer
2025/12/23 02:21:06 [FATAL] k3s exited with: exit status 1