Rke2集群上通过helm 高可用部署rancher2.8.0，集群正常，但是rancher 老是不知名的重启,求解！

ju · 2024 年2 月 19 日 07:01

环境信息:
RKE2 版本:

rke2 version v1.27.10+rke2r1 (915672bd6cab658edb974d0aedb33ec5a32c239a)
go version go1.20.13 X:boringcrypto

节点 CPU 架构，操作系统和版本：

Linux rke2-m1 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
集群配置:

问题描述:

重现步骤:

安装 RKE2 的命令:
curl -sfL https://rancher-mirror.rancher.cn/rke2/install.sh | INSTALL_RKE2_TYPE=server INSTALL_RKE2_MIRROR=cn sh -

预期结果:
rancher 访问正常，没得不断的重启

实际结果:

日志

jacie · 2024 年2 月 20 日 06:56

Rancher是否是离线环境安装，建议提供下 helm 部署命令以及Rancher pod的运行日志

orangedeng · 2024 年2 月 20 日 07:18

看日志有两个可能

集群资源不足，请问3个节点的cpu memory配置是多少？
在describe pod的时候，有提示无法创建临时 volume，apiserver 无法被访问，可能是rke2集群本身启动异常导致的，可以kubectl get po -A截图上来看看，以及看下 systemctl status rke2-server 是否正常

ju · 2024 年2 月 21 日 00:52

Rancher 是在线安装，部署命令helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rke2-rancher.com --set bootstrapPassword=admin123456 --set tls=external --set global.cattle.psp.enabled=false --set rancherImageTag=v2.8.0
日志如下其中一个pod日志
2024/02/20 11:01:00 [INFO] Rancher version v2.8.0 (72f58378b) is starting
2024-02-20T19:01:00.790980936+08:00 2024/02/20 11:01:00 [INFO] Listening on /tmp/log.sock
2024-02-20T19:01:00.790994931+08:00 2024/02/20 11:01:00 [INFO] Rancher arguments {ACMEDomains: AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2024-02-20T19:01:18.095618950+08:00 2024/02/20 11:01:18 [INFO] Running in clustered mode with ID 10.42.4.4, monitoring endpoint cattle-system/rancher
2024-02-20T19:01:32.328202856+08:00 2024/02/20 11:01:32 [INFO] Applying CRD features.management.cattle.io
2024-02-20T19:01:45.769208880+08:00 2024/02/20 11:01:45 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
2024-02-20T19:01:45.807824082+08:00 2024/02/20 11:01:45 [INFO] Updating embedded CRD globalroles.management.cattle.io
2024-02-20T19:01:45.825772956+08:00 2024/02/20 11:01:45 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
2024-02-20T19:01:46.010893440+08:00 2024/02/20 11:01:46 [INFO] Updating embedded CRD projects.management.cattle.io
2024-02-20T19:01:46.132336635+08:00 2024/02/20 11:01:46 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
2024-02-20T19:01:46.238744738+08:00 2024/02/20 11:01:46 [INFO] Updating embedded CRD roletemplates.management.cattle.io
2024-02-20T19:01:47.889648813+08:00 2024/02/20 11:01:47 [INFO] Applying CRD navlinks.ui.cattle.io
2024-02-20T19:01:48.535255464+08:00 2024/02/20 11:01:48 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2024-02-20T19:01:49.063888427+08:00 2024/02/20 11:01:49 [INFO] Applying CRD clusters.management.cattle.io
2024-02-20T19:01:50.102531203+08:00 2024/02/20 11:01:50 [INFO] Applying CRD apiservices.management.cattle.io
2024-02-20T19:
01:50.778780881+08:00 2024/02/20 11:01:50 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2024-02-20T19:01:51.236044458+08:00 2024/02/20 11:01:51 [INFO] Applying CRD settings.management.cattle.io
2024-02-20T19:01:52.077200934+08:00 2024/02/20 11:01:52 [INFO] Applying CRD preferences.management.cattle.io
2024-02-20T19:01:52.269656461+08:00 2024/02/20 11:01:52 [INFO] Applying CRD features.management.cattle.io
2024-02-20T19:01:52.659993341+08:00 2024/02/20 11:01:52 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2024-02-20T19:01:52.935027079+08:00 2024/02/20 11:01:52 [INFO] Applying CRD operations.catalog.cattle.io
2024-02-20T19:01:53.172097454+08:00 2024/02/20 11:01:53 [INFO] Applying CRD apps.catalog.cattle.io
2024-02-20T19:01:53.819572841+08:00 2024/02/20 11:01:53 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2024-02-20T19:01:54.526044290+08:00 2024/02/20 11:01:54 [INFO] Applying CRD managedcharts.management.cattle.io
2024-02-20T19:01:55.018755841+08:00 2024/02/20 11:01:55 [INFO] Applying CRD clusters.provisioning.cattle.io
2024-02-20T19:01:55.378506784+08:00 2024/02/20 11:01:55 [INFO] Applying CRD clusters.provisioning.cattle.io
2024-02-20T19:01:55.527965260+08:00 2024/02/20 11:01:55 [INFO] Applying CRD rkeclusters.rke.cattle.io
2024-02-20T19:01:55.916518444+08:00 2024/02/20 11:01:55 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024-02-20T19:01:56.168023921+08:00 2024/02/20 11:01:56 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2024-02-20T19:01:56.275111093+08:00 2024/02/20 11:01:56 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2024-02-20T19:01:56.349130430+08:00 2024/02/20 11:01:56 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024-02-20T19:01:56.455351271+08:00 2024/02/20 11:01:56 [INFO] Applying CRD custommachines.rke.cattle.io
2024-02-20T19:01:56.627997647+08:00 2024/02/20 11:01:56 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
2024-02-20T19:01:56.779078966+08:00 2024/02/20 11:01:56 [INFO] Applying CRD clusters.cluster.x-k8s.io
2024-02-20T19:01:57.1821
42572+08:00 2024/02/20 11:01:57 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
2024-02-20T19:01:57.534101853+08:00 2024/02/20 11:01:57 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
2024-02-20T19:01:57.836723432+08:00 2024/02/20 11:01:57 [INFO] Applying CRD machines.cluster.x-k8s.io
2024-02-20T19:01:58.083977691+08:00 2024/02/20 11:01:58 [INFO] Applying CRD machinesets.cluster.x-k8s.io
2024-02-20T19:02:40.970191903+08:00 2024/02/20 11:02:40 [INFO] Starting API controllers
2024-02-20T19:02:40.970868121+08:00 2024/02/20 11:02:40 [INFO] Starting /v1, Kind=Secret controller
2024-02-20T19:02:40.971334073+08:00 2024/02/20 11:02:40 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2024-02-20T19:02:40.971609241+08:00 2024/02/20 11:02:40 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2024-02-20T19:02:40.971894356+08:00 2024/02/20 11:02:40 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2024-02-20T19:02:40.971903201+08:00 2024/02/20 11:02:40 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2024-02-20T19:02:40.972648584+08:00 2024/02/20 11:02:40 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2024-02-20T19:02:40.977483953+08:00 2024/02/20 11:02:40 [INFO] Starting project.cattle.io/v3, Kind=App controller
2024-02-20T19:02:41.026806025+08:00 I0220 11:02:40.981041 34 leaderelection.go:245] attempting to acquire leader lease kube-system/cattle-controllers…
2024-02-20T19:02:41.026854127+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2024-02-20T19:02:41.026857382+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2024-02-20T19:02:41.026859453+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2024-02-20T19:02:41.026861011+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2024-02-20T19:02:41.0268624
41+08:00 2024/02/20 11:02:41 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2024-02-20T19:02:41.026864047+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=User controller
2024-02-20T19:02:41.026866070+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2024-02-20T19:02:41.026868763+08:00 2024/02/20 11:02:41 [INFO] Starting /v1, Kind=Namespace controller
2024-02-20T19:02:41.026870178+08:00 2024/02/20 11:02:41 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2024-02-20T19:02:41.026871594+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2024-02-20T19:02:41.026873027+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2024-02-20T19:02:41.026874381+08:00 2024/02/20 11:02:41 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2024-02-20T19:02:41.026875714+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2024-02-20T19:02:41.026877040+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2024-02-20T19:02:41.026878396+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2024-02-20T19:02:41.026879694+08:00 2024/02/20 11:02:41 [INFO] Starting /v1, Kind=ConfigMap controller
2024-02-20T19:02:41.026881079+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2024-02-20T19:02:41.026895745+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2024-02-20T19:02:41.026897894+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=APIService controller
2024-02-20T19:02:41.026899363+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2024-02-20T19:02:41.026900722+08:00 2024/02/20 11:02:41 [INFO
] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2024-02-20T19:02:41.026902389+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2024-02-20T19:02:41.026903813+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2024-02-20T19:02:41.026905142+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2024-02-20T19:02:41.026906476+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2024-02-20T19:02:41.026907824+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2024-02-20T19:02:41.026909217+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2024-02-20T19:02:41.026910593+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2024-02-20T19:02:41.070500665+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2024-02-20T19:02:41.070530793+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2024-02-20T19:02:41.070533275+08:00 2024/02/20 11:02:41 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2024-02-20T19:02:41.070534878+08:00 2024/02/20 11:02:41 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Machine controller
2024-02-20T19:02:41.070536771+08:00 2024/02/20 11:02:41 [INFO] Starting /v1, Kind=ServiceAccount controller
2024-02-20T19:02:41.070538938+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2024-02-20T19:02:41.070540408+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2024-02-20T19:02:41.070541781+08:00 2024/02/20 11:02:41 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2024-02-20T19:02:41.070543615+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Clus
terRegistrationToken controller
2024-02-20T19:02:41.070544989+08:00 2024/02/20 11:02:41 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2024-02-20T19:02:41.070546360+08:00 2024/02/20 11:02:41 [INFO] Starting /v1, Kind=Endpoints controller
2024-02-20T19:02:41.070547816+08:00 2024/02/20 11:02:41 [INFO] Adding peer wss://10.42.1.23/v3/connect, 10.42.1.23
2024-02-20T19:02:41.070549310+08:00 2024/02/20 11:02:41 [INFO] Adding peer wss://10.42.2.28/v3/connect, 10.42.2.28
2024-02-20T19:02:41.127099973+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2024-02-20T19:02:41.127121821+08:00 2024/02/20 11:02:41 [INFO] Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller
2024-02-20T19:02:41.127124510+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2024-02-20T19:02:41.127126218+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2024-02-20T19:02:41.127127723+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2024-02-20T19:02:41.127129138+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2024-02-20T19:02:41.127138473+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2024-02-20T19:02:41.127140809+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2024-02-20T19:02:41.127142380+08:00 2024/02/20 11:02:41 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2024-02-20T19:02:41.281355783+08:00 2024/02/20 11:02:41 [ERROR] Failed to connect to peer wss://10.42.1.23/v3/connect [local ID=10.42.4.4]: websocket: bad handshake
2024-02-20T19:02:41.352309794+08:00 2024/02/20 11:02:41 [ERROR] Failed to connect to peer wss://10.42.2.28/v3/connect [local ID=10.42.4.4]: websocket: bad handshake
2024-02-20T19:02:41.550397499+08:00 2024/02/20 11:02:41 [INFO] Starting cluster controllers f
or local
2024-02-20T19:02:42.062539851+08:00 2024/02/20 11:02:42 [INFO] Starting /v1, Kind=ConfigMap controller
2024-02-20T19:02:42.062570350+08:00 2024/02/20 11:02:42 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2024-02-20T19:02:42.062572812+08:00 2024/02/20 11:02:42 [INFO] Starting management.cattle.io/v3, Kind=User controller
2024-02-20T19:02:42.062574375+08:00 2024/02/20 11:02:42 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2024-02-20T19:02:42.062575793+08:00 2024/02/20 11:02:42 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2024-02-20T19:02:42.062577811+08:00 2024/02/20 11:02:42 [INFO] Starting /v1, Kind=Secret controller
2024-02-20T19:02:42.062579273+08:00 2024/02/20 11:02:42 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2024-02-20T19:02:42.062581144+08:00 2024/02/20 11:02:42 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2024-02-20T19:02:42.073359640+08:00 2024/02/20 11:02:42 [INFO] Starting cluster agent for local [owner=false]
2024-02-20T19:02:42.073387355+08:00 2024/02/20 11:02:42 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2024-02-20T19:02:42.073389663+08:00 2024/02/20 11:02:42 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2024-02-20T19:02:42.077170933+08:00 2024/02/20 11:02:42 [INFO] Starting /v1, Kind=Secret controller
2024-02-20T19:02:42.077201512+08:00 2024/02/20 11:02:42 [INFO] Starting /v1, Kind=ServiceAccount controller
2024-02-20T19:02:42.077204562+08:00 2024/02/20 11:02:42 [INFO] Starting /v1, Kind=Namespace controller
2024-02-20T19:02:42.078087576+08:00 2024/02/20 11:02:42 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2024-02-20T19:02:42.078748372+08:00 2024/02/20 11:02:42 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2024-02-20T19:02:42.228328105+08:00 2024/02/20 11:02:42 [INFO] Listening on :443
2024-02-20T19:02:42.228356654+08:00 2024/02/20 11:02:42 [INFO] certificate CN=dynamic
,O=dynamic signed by CN=dynamiclistener-ca@1708305993,O=dynamiclistener-org: notBefore=2024-02-19 01:26:33 +0000 UTC notAfter=2025-02-19 11:02:42 +0000 UTC
2024-02-20T19:02:42.250791289+08:00 2024/02/20 11:02:42 [INFO] Active TLS secret cattle-system/serving-cert (ver=1038898) (count 15): map[field.cattle.io/projectId:local:p-zk72c listener.cattle.io/cn-10.42.0.49:10.42.0.49 listener.cattle.io/cn-10.42.1.12:10.42.1.12 listener.cattle.io/cn-10.42.1.14:10.42.1.14 listener.cattle.io/cn-10.42.1.19:10.42.1.19 listener.cattle.io/cn-10.42.1.23:10.42.1.23 listener.cattle.io/cn-10.42.2.18:10.42.2.18 listener.cattle.io/cn-10.42.2.27:10.42.2.27 listener.cattle.io/cn-10.42.2.28:10.42.2.28 listener.cattle.io/cn-10.42.4.3:10.42.4.3 listener.cattle.io/cn-10.42.4.4:10.42.4.4 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/cn-rke2-rancher.com:rke2-rancher.com listener.cattle.io/fingerprint:SHA1=9F585DDBEFCBD4E5B8B3FDBEE6315573E0825A92]
2024-02-20T19:02:42.252160092+08:00 2024/02/20 11:02:42 [WARNING] dynamiclistener [::]:443: no cached certificate available for preload - deferring certificate load until storage initialization or first client request
2024-02-20T19:02:42.253165029+08:00 2024/02/20 11:02:42 [INFO] Listening on :80
2024-02-20T19:02:42.323901120+08:00 2024/02/20 11:02:42 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=151376) (count 5): map[field.cattle.io/projectId:local:p-zk72c listener.cattle.io/cn-10.42.0.49:10.42.0.49 listener.cattle.io/cn-10.42.2.18:10.42.2.18 listener.cattle.io/cn-10.43.150.198:10.43.150.198 listener.cattle.io/cn-10.43.77.82:10.43.77.82 listener.cattle.io/fingerprint:SHA1=A43DE85D6AFEA81F7B198BC8C5B866186992F2A1]
2024-02-20T19:02:42.326457549+08:00 2024/02/20 11:02:42 [INFO] Listening on :444
2024-02-20T19:02:42.541157854+08:00 2024/02/20 11:02:42 [INFO] Starting /v1, Kind=Secret controller
2024-02-20T19:02:42.550586075+08:00 2024/02/20 11
:02:42 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 5): map[field.cattle.io/projectId:local:p-zk72c listener.cattle.io/cn-10.42.0.49:10.42.0.49 listener.cattle.io/cn-10.42.2.18:10.42.2.18 listener.cattle.io/cn-10.43.150.198:10.43.150.198 listener.cattle.io/cn-10.43.77.82:10.43.77.82 listener.cattle.io/fingerprint:SHA1=A43DE85D6AFEA81F7B198BC8C5B866186992F2A1]
2024-02-20T19:02:42.555356588+08:00 2024/02/20 11:02:42 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 15): map[field.cattle.io/projectId:local:p-zk72c listener.cattle.io/cn-10.42.0.49:10.42.0.49 listener.cattle.io/cn-10.42.1.12:10.42.1.12 listener.cattle.io/cn-10.42.1.14:10.42.1.14 listener.cattle.io/cn-10.42.1.19:10.42.1.19 listener.cattle.io/cn-10.42.1.23:10.42.1.23 listener.cattle.io/cn-10.42.2.18:10.42.2.18 listener.cattle.io/cn-10.42.2.27:10.42.2.27 listener.cattle.io/cn-10.42.2.28:10.42.2.28 listener.cattle.io/cn-10.42.4.3:10.42.4.3 listener.cattle.io/cn-10.42.4.4:10.42.4.4 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/cn-rke2-rancher.com:rke2-rancher.com listener.cattle.io/fingerprint:SHA1=9F585DDBEFCBD4E5B8B3FDBEE6315573E0825A92]
2024-02-20T19:02:45.738157096+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2024-02-20T19:02:45.738187687+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for rke.cattle.io/v1, Kind=CustomMachine
2024-02-20T19:02:45.738191425+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineHealthCheck
2024-02-20T19:02:45.738193441+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2024-02-20T19:02:45.738194935+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=HarvesterMachineTemplate
2024-02-20T19:02:45.738196380+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.
cattle.io/v3, Kind=Catalog
2024-02-20T19:02:45.738197937+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2024-02-20T19:02:45.738199680+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2024-02-20T19:02:45.738201145+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2024-02-20T19:02:45.738202506+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2024-02-20T19:02:45.738203899+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2024-02-20T19:02:45.738205251+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for /v1, Kind=ConfigMap
2024-02-20T19:02:45.738206609+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrap
2024-02-20T19:02:45.738219412+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=Amazonec2Machine
2024-02-20T19:02:45.776914186+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2024-02-20T19:02:45.776943726+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIDriver
2024-02-20T19:02:45.776946672+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2024-02-20T19:02:45.776948233+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=ManagedChart
2024-02-20T19:02:45.776949659+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2024-02-20T19:02:45.776950989+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for provisioning.cattle.io/v1, Kind=Cluster
2024-02-20T19:02:45.776952650+08:00 2024/02/20 11:02:45 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta3, Kind=FlowSchema
2024-02-20T19:02:45.776954364+08:00 2024/02/20 11:02:45 [INFO] Watching me

ju · 2024 年2 月 21 日 00:56

集群资源配置4核8G的master,work节点是2核4G的

systemctl status rke2-server

orangedeng · 2024 年2 月 22 日 02:03

除了Rancher之外，其他依赖 apiserver 的组件都有重启的情况，所以是搭建的rke2集群有问题，可以查看etcd/apiserver 日志进一步排查。
注意到 uname 返回的内核版本较低，最好看看rke2安装需求，Requirements | RKE2 ，对比一下环境情况，避免由于环境带来的影响因素

ju · 2024 年2 月 22 日 07:08

内核版本rke2-m2 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux 操作系统版本是centons7.9
rke2 版本： v1.27.10+rke2r1

etcd 日志

apiserver 日志

—“Request completed” 946ms (06:55:43.613)]
2024-02-22T14:55:44.224307735+08:00 Trace[2874213]: —“Write to database call failed” len:324,err:secrets “rke2-m1.node-password.rke2” already exists 66ms (06:55:44.223)
2024-02-22T14:55:44.224309411+08:00 Trace[2874213]: [1.578045837s] [1.578045837s] END
2024-02-22T14:55:44.228375775+08:00 I0222 06:55:44.224264 1 trace.go:219] Trace[130616123]: “Create” accept:application/json, /,audit-id:2ad3ab11-70dd-4397-b447-a0a892ae5234,client:127.0.0.1,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/kube-system/secrets,user-agent:rke2-supervisor@rke2-m1/v1.27.10+rke2r1 (linux/amd64) rke2/915672bd6cab658edb974d0aedb33ec5a32c239a,verb:POST (22-Feb-2024 06:55:42.933) (total time: 1290ms):
2024-02-22T14:55:44.228424715+08:00 Trace[130616123]: [“Call mutating webhook” configuration:rancher.cattle.io,webhook:rancher.cattle.io.secrets,resource:/v1, Resource=secrets,subresource:,operation:CREATE,UID:75b18863-3be3-4548-bd02-caac3e9f8d7a 1290ms (06:55:42.933)
2024-02-22T14:55:44.228430089+08:00 Trace[130616123]: —“Request completed” 672ms (06:55:43.613)]
2024-02-22T14:55:44.228433748+08:00 Trace[130616123]: —“Write to database call failed” len:3282,err:secrets “rke2-serving” already exists 70ms (06:55:44.224)
Write to database call failed" len:1199,err:podsecurityadmissionconfigurationtemplates.management.cattle.io “rancher-restricted” already exists 387ms (06:55:18.825)
“Write to database call failed” len:653,err:podsecurityadmissionconfigurationtemplates.management.cattle.io "

rancher-privileged" already exists 230ms (06:55:17.857)
“Write to database call failed” len:803,err:podsecuritypolicytemplates.management.cattle.io “restricted-noroot” already exists 768ms (06:55:15.503)
—“Write to database call failed” len:297,err:authconfigs.management.cattle.io “local” already exists 559ms (06:55:06.999)
2024-02-22T14:55:07.003991661+08:00 Trace[747193942]: [1.196855681s] [1.196855681s] END
“Write to database call failed” len:296,err:authconfigs.management.cattle.io “keycloakoidc” already exists 648ms (06:55:05.468)
“Write to database call failed” len:280,err:authconfigs.management.cattle.io “oidc” already exists 580ms (06:55:03.039)
2024-02-22T14:55:03.106584815+08:00 Trace[1984042289]: [994.430866ms] [994.430866ms] END

orangedeng · 2024 年2 月 26 日 07:55

看起来是网络问题导致etcd无法组成集群，请检查节点上的网络是否正常，如果有防火墙或者网络策略配置，请参考这个表中的端口开放列表配置 Requirements | RKE2