K3s中rancher安装后出现大量helm-operation-xxxx，随后rancher无法进入

Rancher 2.x
,
1

Rancher Server 设置

  • Rancher 版本：2.7.9
  • 安装选项 (Docker install/Helm Chart): Helm Chart
    k3s版本：v1.26.9+k3s1
  • 在线或离线部署：离线部署

**主机操作系统：三台ubtuntu22.04.6，两台master一台agent，都已关闭防火墙

**问题描述：k3s中rancher安装后出现大量helm-operation-xxxx，随后rancher无法进入

**重现步骤：逐步安装k3s，安装cert-manager，安装rancher，进入rancher的web界面以切正常，但是查看pod信息发现随后出现大量helm-operation-xxxx，然后rancher的web界面无法进入

截图：

6513d641f0751b9eff24cbb4035bc85
6513d641f0751b9eff24cbb4035bc85861×435 50.5 KB

其他上下文信息：

日志

rancher中错误

2023/12/28 08:12:25 [INFO] Rancher version v2.7.9 (0c020f0c6) is starting
2023/12/28 08:12:25 [INFO] Rancher arguments {ACMEDomains: AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2023/12/28 08:12:25 [INFO] Listening on /tmp/log.sock
2023/12/28 08:12:34 [FATAL] 1 error occurred:
* Get “https://10.43.0.1:443/apis/management.cattle.io/v3/clusters/local?timeout=15m0s”: dial tcp 10.43.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF

helm-operation中报错

Defaulted container “helm” out of: helm, proxy
Error from server: Get “https://172.18.11.249:10250/containerLogs/cattle-system/helm-operation-gvt79/helm?follow=true&sinceSeconds=3600”: proxy error from 127.0.0.1:6443 while dialing 172.18.11.249:10250, code 502: 502 Bad Gateway

2

感觉像是网络的问题，你可以检查下：

  1. ubuntu 的 ufw 防火墙
  2. 如果是公有云，检查下安全组
3

三台主机防火墙都处于关闭状态，目前环境是我的三台虚拟机，相互间网络没有问题

4

这是其中一个helm-operation的pod描述

root@k3s-master:/data/install# kubectl describe pod helm-operation-t6cpm -n cattle-system
Name: helm-operation-t6cpm
Namespace: cattle-system
Priority: 0
Service Account: default
Node: k3s-agent1/172.18.11.248
Start Time: Thu, 28 Dec 2023 15:15:24 +0800
Labels: pod-impersonation.cattle.io/token=6w6klv4bs7tmnxc545pk7w878klm88nc7wmfghvzlvfjz6wnzb6ld4
Annotations: pod-impersonation.cattle.io/cluster-role: pod-impersonation-helm-op-q2bhd
Status: Running
IP: 10.42.1.14
IPs:
IP: 10.42.1.14
Containers:
helm:
Container ID: containerd://1f2e81206ee597fd6c713d3bc5ea3f1550940d99f440cb74e8e9654b9951291a
Image: Harbor
Image ID: Harbor
Port:
Host Port:
Command:
helm-cmd
State: Terminated
Reason: Error
Exit Code: 123
Started: Thu, 28 Dec 2023 15:15:24 +0800
Finished: Thu, 28 Dec 2023 15:16:14 +0800
Ready: False
Restart Count: 0
Environment:
KUBECONFIG: /home/shell/.kube/config
Mounts:
/home/shell/.kube/config from user-kubeconfig (ro,path=“config”)
/home/shell/helm from data (ro)
proxy:
Container ID: containerd://96bad668853cfdd8b9bb867fba736abc6fe9fe3fe67287b0d49988c5db6602db
Image: Harbor
Image ID: Harbor
Port:
Host Port:
Command:
sh
-c
kubectl proxy --disable-filter || true
State: Running
Started: Thu, 28 Dec 2023 15:15:25 +0800
Ready: True
Restart Count: 0
Environment:
KUBECONFIG: /root/.kube/config
Mounts:
/root/.kube/config from admin-kubeconfig (ro,path=“config”)
/var/run/secrets/kubernetes.io/serviceaccount from pod-impersonation-helm-op-z4gp4-token (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: Secret (a volume populated by a Secret)
SecretName: helm-operation-pm7w7
Optional: false
admin-kubeconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: impersonation-helm-op-admin-kubeconfig-246xw
Optional: false
user-kubeconfig:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: impersonation-helm-op-user-kubeconfig-j7jm5
Optional: false
pod-impersonation-helm-op-z4gp4-token:
Type: Secret (a volume populated by a Secret)
SecretName: pod-impersonation-helm-op-z4gp4-token
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: cattle.io/os=linux:NoSchedule
node-role.kubernetes.io/controlplane=true:NoSchedule
node-role.kubernetes.io/etcd=true:NoExecute
node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

Normal Scheduled 91m default-scheduler Successfully assigned cattle-system/helm-operation-t6cpm to k3s-agent1
Normal Pulled 91m kubelet Container image “Harbor” already present on machine
Normal Created 91m kubelet Created container helm
Normal Started 91m kubelet Started container helm
Normal Pulled 91m kubelet Container image “Harbor” already present on machine
Normal Created 91m kubelet Created container proxy
Normal Started 91m kubelet Started container proxy

5

一个目前处于运行状态的rancher-webhook-xxx的日志

I1228 08:44:09.039000 1 trace.go:219] Trace[2134911446]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:43:57.143) (total time: 11895ms):
Trace[2134911446]: —“Objects listed” error:clusters.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “clusters” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found] 11895ms (08:44:09.038)
Trace[2134911446]: [11.895158764s] [11.895158764s] END
E1228 08:44:09.039008 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.Cluster: failed to list *v3.Cluster: clusters.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “clusters” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found]
W1228 08:44:09.039532 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v3.GlobalRole: globalroles.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “globalroles” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found, clusterrole.rbac.authorization.k8s.io “system:discovery” not found]
I1228 08:44:09.039569 1 trace.go:219] Trace[2114485477]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:43:58.629) (total time: 10409ms):
Trace[2114485477]: —“Objects listed” error:globalroles.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “globalroles” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found, clusterrole.rbac.authorization.k8s.io “system:discovery” not found] 10409ms (08:44:09.039)
Trace[2114485477]: [10.409725631s] [10.409725631s] END
E1228 08:44:09.039577 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.GlobalRole: failed to list *v3.GlobalRole: globalroles.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “globalroles” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found, clusterrole.rbac.authorization.k8s.io “system:discovery” not found]
time=“2023-12-28T08:44:09Z” level=info msg=“Sleeping for 15 seconds then applying webhook config”
time=“2023-12-28T08:44:10Z” level=info msg=“Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=7D805E999D30F2F1D17A323F8AF10DF3B0618837]”
E1228 08:45:53.701579 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ProjectRoleTemplateBinding: unknown (get projectroletemplatebindings.meta.k8s.io)
E1228 08:45:53.701647 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.CustomResourceDefinition: unknown (get customresourcedefinitions.meta.k8s.io)
E1228 08:45:53.701799 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.ClusterRoleBinding: unknown (get clusterrolebindings.meta.k8s.io)
E1228 08:45:53.701885 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ClusterRoleTemplateBinding: unknown (get clusterroletemplatebindings.meta.k8s.io)
E1228 08:45:53.701948 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.ClusterRole: unknown (get clusterroles.meta.k8s.io)
E1228 08:45:53.701962 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.Cluster: unknown (get clusters.meta.k8s.io)
E1228 08:45:53.701973 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.Node: unknown (get nodes.meta.k8s.io)
E1228 08:45:53.701984 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.RoleBinding: unknown (get rolebindings.meta.k8s.io)
time=“2023-12-28T08:45:59Z” level=info msg=“Sleeping for 15 seconds then applying webhook config”
time=“2023-12-28T08:45:59Z” level=info msg=“Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=7D805E999D30F2F1D17A323F8AF10DF3B0618837]”
W1228 08:46:08.713035 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v3.PodSecurityAdmissionConfigurationTemplate: podsecurityadmissionconfigurationtemplates.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “podsecurityadmissionconfigurationtemplates” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found]
I1228 08:46:08.713118 1 trace.go:219] Trace[839515773]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:45:52.151) (total time: 16561ms):
Trace[839515773]: —“Objects listed” error:podsecurityadmissionconfigurationtemplates.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “podsecurityadmissionconfigurationtemplates” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found] 16561ms (08:46:08.713)
Trace[839515773]: [16.561926669s] [16.561926669s] END
E1228 08:46:08.713134 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.PodSecurityAdmissionConfigurationTemplate: failed to list *v3.PodSecurityAdmissionConfigurationTemplate: podsecurityadmissionconfigurationtemplates.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “podsecurityadmissionconfigurationtemplates” in API group “management.cattle.io” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found]
time=“2023-12-28T08:47:23Z” level=info msg=“Sleeping for 15 seconds then applying webhook config”
time=“2023-12-28T08:47:23Z” level=info msg=“Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=7D805E999D30F2F1D17A323F8AF10DF3B0618837]”
E1228 08:48:46.424230 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Secret: unknown (get secrets.meta.k8s.io)
E1228 08:48:46.424301 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Cluster: unknown (get clusters.meta.k8s.io)
E1228 08:48:47.050437 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.RoleTemplate: unknown (get roletemplates.meta.k8s.io)
E1228 08:48:47.340030 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.Cluster: unknown (get clusters.meta.k8s.io)
E1228 08:48:48.671264 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.CustomResourceDefinition: unknown (get customresourcedefinitions.meta.k8s.io)
E1228 08:48:49.412708 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.APIService: unknown (get apiservices.meta.k8s.io)
E1228 08:48:51.275043 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Role: unknown (get roles.meta.k8s.io)
E1228 08:48:51.321987 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.Node: unknown (get nodes.meta.k8s.io)
E1228 08:48:51.917898 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.ClusterRole: unknown (get clusterroles.meta.k8s.io)
E1228 08:48:52.599332 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.RoleBinding: unknown (get rolebindings.meta.k8s.io)
E1228 08:48:54.483268 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ProjectRoleTemplateBinding: unknown (get projectroletemplatebindings.meta.k8s.io)
E1228 08:48:55.282033 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.PodSecurityAdmissionConfigurationTemplate: unknown (get podsecurityadmissionconfigurationtemplates.meta.k8s.io)
E1228 08:48:58.175971 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.GlobalRole: unknown (get globalroles.meta.k8s.io)
W1228 08:49:16.426957 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v3.ClusterRoleTemplateBinding: Get “https://10.43.0.1:443/apis/management.cattle.io/v3/clusterroletemplatebindings?resourceVersion=29105”: dial tcp 10.43.0.1:443: connect: connection refused
I1228 08:49:16.427074 1 trace.go:219] Trace[102884900]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:15.350) (total time: 61076ms):
Trace[102884900]: —“Objects listed” error:Get “https://10.43.0.1:443/apis/management.cattle.io/v3/clusterroletemplatebindings?resourceVersion=29105”: dial tcp 10.43.0.1:443: connect: connection refused 61076ms (08:49:16.426)
Trace[102884900]: [1m1.076174807s] [1m1.076174807s] END
E1228 08:49:16.427091 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ClusterRoleTemplateBinding: failed to list *v3.ClusterRoleTemplateBinding: Get “https://10.43.0.1:443/apis/management.cattle.io/v3/clusterroletemplatebindings?resourceVersion=29105”: dial tcp 10.43.0.1:443: connect: connection refused
W1228 08:49:16.427537 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1.ClusterRoleBinding: Get “https://10.43.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?resourceVersion=28996”: dial tcp 10.43.0.1:443: connect: connection refused
I1228 08:49:16.427605 1 trace.go:219] Trace[483428901]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:21.719) (total time: 54707ms):
Trace[483428901]: —“Objects listed” error:Get “https://10.43.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?resourceVersion=28996”: dial tcp 10.43.0.1:443: connect: connection refused 54707ms (08:49:16.427)
Trace[483428901]: [54.707779425s] [54.707779425s] END
E1228 08:49:16.427617 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.ClusterRoleBinding: failed to list *v1.ClusterRoleBinding: Get “https://10.43.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?resourceVersion=28996”: dial tcp 10.43.0.1:443: connect: connection refused
W1228 08:49:16.428275 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1.Secret: secrets is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “secrets” in API group “” at the cluster scope
I1228 08:49:16.428324 1 trace.go:219] Trace[2136853271]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:48.709) (total time: 27718ms):
Trace[2136853271]: —“Objects listed” error:secrets is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “secrets” in API group “” at the cluster scope 27718ms (08:49:16.428)
Trace[2136853271]: [27.718883753s] [27.718883753s] END
E1228 08:49:16.428335 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “secrets” in API group “” at the cluster scope
W1228 08:49:16.428854 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1.Cluster: clusters.provisioning.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “clusters” in API group “provisioning.cattle.io” at the cluster scope
I1228 08:49:16.428886 1 trace.go:219] Trace[1327279978]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:49.100) (total time: 27327ms):
Trace[1327279978]: —“Objects listed” error:clusters.provisioning.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “clusters” in API group “provisioning.cattle.io” at the cluster scope 27327ms (08:49:16.428)
Trace[1327279978]: [27.327946381s] [27.327946381s] END
E1228 08:49:16.428905 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Cluster: failed to list *v1.Cluster: clusters.provisioning.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “clusters” in API group “provisioning.cattle.io” at the cluster scope
W1228 08:49:16.429443 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v3.RoleTemplate: roletemplates.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “roletemplates” in API group “management.cattle.io” at the cluster scope
I1228 08:49:16.429478 1 trace.go:219] Trace[1733148284]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:49.505) (total time: 26923ms):
Trace[1733148284]: —“Objects listed” error:roletemplates.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “roletemplates” in API group “management.cattle.io” at the cluster scope 26923ms (08:49:16.429)
Trace[1733148284]: [26.92357597s] [26.92357597s] END
E1228 08:49:16.429492 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.RoleTemplate: failed to list *v3.RoleTemplate: roletemplates.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “roletemplates” in API group “management.cattle.io” at the cluster scope
W1228 08:49:16.429999 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “apiservices” in API group “apiregistration.k8s.io” at the cluster scope
I1228 08:49:16.430026 1 trace.go:219] Trace[1439100011]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:52.416) (total time: 24013ms):
Trace[1439100011]: —“Objects listed” error:apiservices.apiregistration.k8s.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “apiservices” in API group “apiregistration.k8s.io” at the cluster scope 24013ms (08:49:16.429)
Trace[1439100011]: [24.013193573s] [24.013193573s] END
E1228 08:49:16.430043 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.APIService: failed to list *v1.APIService: apiservices.apiregistration.k8s.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “apiservices” in API group “apiregistration.k8s.io” at the cluster scope
W1228 08:49:16.430550 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “roles” in API group “rbac.authorization.k8s.io” at the cluster scope
I1228 08:49:16.430583 1 trace.go:219] Trace[431061159]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:48:54.135) (total time: 22294ms):
Trace[431061159]: —“Objects listed” error:roles.rbac.authorization.k8s.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “roles” in API group “rbac.authorization.k8s.io” at the cluster scope 22294ms (08:49:16.430)
Trace[431061159]: [22.294863913s] [22.294863913s] END
E1228 08:49:16.430597 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Role: failed to list *v1.Role: roles.rbac.authorization.k8s.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “roles” in API group “rbac.authorization.k8s.io” at the cluster scope
W1228 08:49:16.431075 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v3.GlobalRole: globalroles.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “globalroles” in API group “management.cattle.io” at the cluster scope
I1228 08:49:16.431101 1 trace.go:219] Trace[1099471533]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:49:00.452) (total time: 15979ms):
Trace[1099471533]: —“Objects listed” error:globalroles.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “globalroles” in API group “management.cattle.io” at the cluster scope 15979ms (08:49:16.431)
Trace[1099471533]: [15.979070702s] [15.979070702s] END
E1228 08:49:16.431116 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.GlobalRole: failed to list *v3.GlobalRole: globalroles.management.cattle.io is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “globalroles” in API group “management.cattle.io” at the cluster scope
time=“2023-12-28T08:49:19Z” level=info msg=“Sleeping for 15 seconds then applying webhook config”
time=“2023-12-28T08:49:20Z” level=info msg=“Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=7D805E999D30F2F1D17A323F8AF10DF3B0618837]”
time=“2023-12-28T08:51:10Z” level=info msg=“Sleeping for 15 seconds then applying webhook config”
time=“2023-12-28T08:51:12Z” level=info msg=“Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=7D805E999D30F2F1D17A323F8AF10DF3B0618837]”
E1228 08:51:13.072873 1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E1228 08:51:13.078906 1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
time=“2023-12-28T08:52:02Z” level=info msg=“Sleeping for 15 seconds then applying webhook config”
time=“2023-12-28T08:52:02Z” level=info msg=“Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=7D805E999D30F2F1D17A323F8AF10DF3B0618837]”
I1228 08:52:32.389075 1 trace.go:219] Trace[1375563519]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:52:21.064) (total time: 11324ms):
Trace[1375563519]: —“Objects listed” error: 11324ms (08:52:32.388)
Trace[1375563519]: [11.324630897s] [11.324630897s] END
E1228 08:53:03.309770 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ClusterRoleTemplateBinding: unknown (get clusterroletemplatebindings.meta.k8s.io)
E1228 08:53:03.309914 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.PodSecurityAdmissionConfigurationTemplate: unknown (get podsecurityadmissionconfigurationtemplates.meta.k8s.io)
E1228 08:53:03.309935 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.ClusterRoleBinding: unknown (get clusterrolebindings.meta.k8s.io)
E1228 08:53:03.309953 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.RoleBinding: unknown (get rolebindings.meta.k8s.io)
E1228 08:53:03.310070 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ProjectRoleTemplateBinding: unknown (get projectroletemplatebindings.meta.k8s.io)
E1228 08:53:03.310320 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.ClusterRole: unknown (get clusterroles.meta.k8s.io)
E1228 08:53:03.310337 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.CustomResourceDefinition: unknown (get customresourcedefinitions.meta.k8s.io)
E1228 08:53:03.310357 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.Cluster: unknown (get clusters.meta.k8s.io)
W1228 08:53:18.313819 1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1.Secret: secrets is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “secrets” in API group “” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found]
I1228 08:53:18.313978 1 trace.go:219] Trace[533549750]: “Reflector ListAndWatch” name:pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170 (28-Dec-2023 08:52:59.313) (total time: 19000ms):
Trace[533549750]: —“Objects listed” error:secrets is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “secrets” in API group “” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found] 18999ms (08:53:18.313)
Trace[533549750]: [19.000022428s] [19.000022428s] END
E1228 08:53:18.313999 1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User “system:serviceaccount:cattle-system:rancher-webhook” cannot list resource “secrets” in API group “” at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io “system:service-account-issuer-discovery” not found, clusterrole.rbac.authorization.k8s.io “system:public-info-viewer” not found, clusterrole.rbac.authorization.k8s.io “system:discovery” not found, clusterrole.rbac.authorization.k8s.io “system:basic-user” not found, clusterrole.rbac.authorization.k8s.io “cluster-admin” not found]

6

你在微信群里么？如果在微信群里，@下 ksd

7 
cacher (apiservices.apiregistration.k8s.io): unexpected ListAndWatch error: failed to list *apiregistration.APIService: rpc error: code = Unknown desc = Error 1040: Too many connections; reinitializing...

这个错我没见过，但基本是因为连接数 过多导致的

看了下你的 K3s 主机，2C2G 的虚拟机，上面还跑 rancher 啥的，估计是因为配置引起的

可以增加主机的配置，然后继续观察

8

好的，我试试