Rancher UI总是莫名刷新不出来

benny · 2024 年6 月 27 日 06:43

Rancher Server 设置

Rancher 版本：2.7.1
安装选项 (Docker install/Helm Chart):
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：rke
在线或离线部署：

下游集群信息

Kubernetes 版本: 1.18
Cluster Type (Local/Downstream): Local
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等):

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
- 如果自定义，自定义权限集：

主机操作系统： ubuntu 20.04 4c8g

问题描述： rancher 在启动一段时间后，总会莫名地打不开所有页面。圈圈一直在刷新却打不开，F12看到控制台的报错是：GET https://rancher.xxxx.com/api/v1/namespaces/cattle-ui-plugin-system/services/http:ui-plugin-operator:80/proxy/index.json 401 (Unauthorized)。这种情况下 rancher 服务的pod没有被重启，资源使用正常，没有设置cpu/memory限制，其他磁盘io监控看着挺正常。
每次都需要重启 rancher pod 才能恢复，但隔一段时间又会出现。

重现步骤： 页面无法刷新的情况无法重现，不知道什么时候突然就出现

结果：

预期结果： 解决刷新不出来的问题

截图：

其他上下文信息：

日志

2024/06/27 04:28:30 [ERROR] Error during subscribe write tcp 10.42.0.125:80->172.16.1.12:39442: write: connection reset by peer
E0627 04:33:00.110510      34 upgradeaware.go:426] Error proxying data from client to backend: read tcp 10.42.0.125:80->172.16.1.12:47242: read: connection reset by peer
W0627 04:33:23.040536      34 transport.go:288] Unable to cancel request for *client.addQuery
2024/06/27 05:10:30 [ERROR] Error during subscribe websocket: close sent
2024/06/27 05:35:48 [ERROR] Error during subscribe websocket: close sent
W0627 05:38:12.006434      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 05:38:12.006422      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 05:38:12.006893      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 05:38:12.007216      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 05:38:12.007326      34 transport.go:288] Unable to cancel request for *client.addQuery
2024/06/27 06:02:08 [ERROR] Error during subscribe websocket: close sent
2024/06/27 06:06:00 [ERROR] Unknown error: Operation cannot be fulfilled on preferences.management.cattle.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2024/06/27 06:06:00 [ERROR] Unknown error: Operation cannot be fulfilled on preferences.management.cattle.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2024/06/27 06:06:03 [ERROR] Unknown error: Operation cannot be fulfilled on preferences.management.cattle.io "cluster": the object has been modified; please apply your changes to the latest version and try again
2024/06/27 06:06:17 [ERROR] Error during subscribe websocket: close sent
2024/06/27 06:06:37 [ERROR] Error during subscribe websocket: close sent
W0627 06:08:17.020422      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:08:17.020422      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:08:17.020472      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:08:17.020670      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:08:17.021368      34 transport.go:288] Unable to cancel request for *client.addQuery
2024/06/27 06:15:52 [ERROR] Error during subscribe websocket: close sent
2024/06/27 06:17:00 [ERROR] Error during subscribe websocket: close sent
2024/06/27 06:17:15 [ERROR] Error during subscribe websocket: close sent
W0627 06:17:25.293139      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:25.427725      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:25.563713      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:25.563727      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:25.711629      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:25.721453      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:25.753490      34 transport.go:288] Unable to cancel request for *client.addQuery
W0627 06:17:27.962544      34 transport.go:288] Unable to cancel request for *client.addQuery
2024/06/27 06:17:32 [ERROR] Error during subscribe websocket: close sent
2024/06/27 06:17:33 [ERROR] Error during subscribe websocket: close sent
2024/06/27 06:17:54 [ERROR] Error during subscribe websocket: close sent

liuyan · 2024 年6 月 28 日 02:30

如果出现一直显示 loading 页面的情况下，能否尝试刷新一下浏览器页面，看看是否跳转到登录页面？这样来做进一步的判断

benny · 2024 年6 月 28 日 02:34

你好，刷新浏览器页面是停留在当前页面的，没有跳转到登录页面呢。通过改url跳转到登录页面也还是一直 loading

liuyan · 2024 年6 月 28 日 02:52

能看看浏览器控制台，除了 index.json 加载异常，是否有其他接口都正常加载了，比如/v3/users?me=true 请求是否正常返回

benny · 2024 年6 月 28 日 03:13

你好，由于现在rancher 服务还在正常运行，需要等这个问题复现了我再回复其他接口是否正常加载。
但有个奇怪的问题，现在rancher正常运行中，但是浏览器控制台也是index.json 加载异常，可是cattle-ui-plugin-system空间和ui-plugin-operator服务在rancher集群是没有的啊？这是正常的吗？

$ kubectl get all -A
NAMESPACE                   NAME                                        READY   STATUS    RESTARTS   AGE
cattle-fleet-local-system   pod/fleet-agent-55bf958767-hbdpv            1/1     Running   0          15d
cattle-fleet-system         pod/fleet-controller-6dcd9c8c6f-6k76z       1/1     Running   0          15d
cattle-fleet-system         pod/gitjob-b9447f999-ldt56                  1/1     Running   0          15d
cattle-resources-system     pod/rancher-backup-7c4ddf5d89-jdc7n         1/1     Running   0          112d
cattle-system               pod/rancher-997d64b77-78szp                 1/1     Running   1          43h
cattle-system               pod/rancher-webhook-747ff4b44c-l5dx5        1/1     Running   0          14d
ingress-nginx               pod/default-http-backend-598b7d7dbd-9b97p   1/1     Running   0          112d
ingress-nginx               pod/nginx-ingress-controller-gtgqc          1/1     Running   1          112d
kube-system                 pod/canal-fj9xf                             2/2     Running   0          3d
kube-system                 pod/coredns-849545576b-c2rfr                1/1     Running   0          112d
kube-system                 pod/coredns-autoscaler-5dcd676cbd-zqnn6     1/1     Running   0          112d
kube-system                 pod/metrics-server-697746ff48-nm5m2         1/1     Running   0          112d

NAMESPACE             NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
cattle-fleet-system   service/gitjob                 ClusterIP   10.43.129.244   <none>        80/TCP                   111d
cattle-system         service/rancher                ClusterIP   10.43.104.205   <none>        80/TCP,443/TCP           111d
cattle-system         service/rancher-webhook        ClusterIP   10.43.19.195    <none>        443/TCP                  14d
cattle-system         service/webhook-service        ClusterIP   10.43.80.236    <none>        443/TCP                  14d
default               service/kubernetes             ClusterIP   10.43.0.1       <none>        443/TCP                  112d
ingress-nginx         service/default-http-backend   ClusterIP   10.43.140.174   <none>        80/TCP                   112d
kube-system           service/kube-dns               ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP   112d
kube-system           service/metrics-server         ClusterIP   10.43.221.117   <none>        443/TCP                  112d

NAMESPACE       NAME                                      DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
ingress-nginx   daemonset.apps/nginx-ingress-controller   1         1         1       1            1           <none>                   112d
kube-system     daemonset.apps/canal                      1         1         1       1            1           kubernetes.io/os=linux   112d

NAMESPACE                   NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
cattle-fleet-local-system   deployment.apps/fleet-agent            1/1     1            1           15d
cattle-fleet-system         deployment.apps/fleet-controller       1/1     1            1           111d
cattle-fleet-system         deployment.apps/gitjob                 1/1     1            1           111d
cattle-resources-system     deployment.apps/rancher-backup         1/1     1            1           112d
cattle-system               deployment.apps/rancher                1/1     1            1           111d
cattle-system               deployment.apps/rancher-webhook        1/1     1            1           14d
ingress-nginx               deployment.apps/default-http-backend   1/1     1            1           112d
kube-system                 deployment.apps/coredns                1/1     1            1           112d
kube-system                 deployment.apps/coredns-autoscaler     1/1     1            1           112d
kube-system                 deployment.apps/metrics-server         1/1     1            1           112d

NAMESPACE                   NAME                                              DESIRED   CURRENT   READY   AGE
cattle-fleet-local-system   replicaset.apps/fleet-agent-55bf958767            1         1         1       15d
cattle-fleet-system         replicaset.apps/fleet-controller-5c88485f74       0         0         0       15d
cattle-fleet-system         replicaset.apps/fleet-controller-6dcd9c8c6f       1         1         1       15d
cattle-fleet-system         replicaset.apps/fleet-controller-75dc8d5b64       0         0         0       111d
cattle-fleet-system         replicaset.apps/fleet-controller-c7f6d9459        0         0         0       15d
cattle-fleet-system         replicaset.apps/fleet-controller-d56d6bc95        0         0         0       111d
cattle-fleet-system         replicaset.apps/gitjob-6954fd87b9                 0         0         0       111d
cattle-fleet-system         replicaset.apps/gitjob-7644ddfcd7                 0         0         0       15d
cattle-fleet-system         replicaset.apps/gitjob-764665fd95                 0         0         0       15d
cattle-fleet-system         replicaset.apps/gitjob-7b77766cb7                 0         0         0       111d
cattle-fleet-system         replicaset.apps/gitjob-b9447f999                  1         1         1       15d
cattle-resources-system     replicaset.apps/rancher-backup-7c4ddf5d89         1         1         1       112d
cattle-resources-system     replicaset.apps/rancher-backup-d4495bfd           0         0         0       112d
cattle-system               replicaset.apps/rancher-997d64b77                 1         1         1       111d
cattle-system               replicaset.apps/rancher-webhook-66d569489d        0         0         0       14d
cattle-system               replicaset.apps/rancher-webhook-747ff4b44c        1         1         1       14d
ingress-nginx               replicaset.apps/default-http-backend-598b7d7dbd   1         1         1       112d
kube-system                 replicaset.apps/coredns-849545576b                1         1         1       112d
kube-system                 replicaset.apps/coredns-autoscaler-5dcd676cbd     1         1         1       112d
kube-system                 replicaset.apps/metrics-server-697746ff48         1         1         1       112d

NAMESPACE     NAME                                          COMPLETIONS   DURATION   AGE
kube-system   job.batch/rke-coredns-addon-deploy-job        1/1           2s         112d
kube-system   job.batch/rke-ingress-controller-deploy-job   1/1           2s         112d
kube-system   job.batch/rke-metrics-addon-deploy-job        1/1           2s         112d
kube-system   job.batch/rke-network-plugin-deploy-job       1/1           13s        112d

liuyan · 2024 年6 月 28 日 03:23

404 问题，是因为没启用 UI Extension 导致的，UI 全局菜单中应该有个Extension 菜单，点击后会提示用户是否启用 UI Extension ，默认是不启用的，不启用的话，是不会部署ui-plugin-operator服务端

benny · 2024 年6 月 28 日 03:28

好的，明白！那我待loading问题复现了再根据你的指引排查下其他接口是否正常加载再来回复。感谢你的回复～

benny · 2024 年7 月 10 日 02:22

大佬你好，今天rancher又出现一直loading不出来的问题了，控制台看了下接口，好像卡在了/v1/management.cattle.io.setting 请求

/v3/users?me=true请求好像正常

rancher 服务输出日志

W0710 02:16:50.967328      33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *unstructured.Unstructured: the server could not find the requested resource (get aliyunecsconfigs.meta.k8s.io)
E0710 02:16:50.967364      33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource (get aliyunecsconfigs.meta.k8s.io)
W0710 02:17:03.710076      33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: the server could not find the requested resource
E0710 02:17:03.710108      33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: the server could not find the requested resource
W0710 02:17:03.799076      33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *unstructured.Unstructured: the server could not find the requested resource (get aliyunecsmachinetemplates.meta.k8s.io)
E0710 02:17:03.799107      33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: the server could not find the requested resource (get aliyunecsmachinetemplates.meta.k8s.io)
W0710 02:17:07.893717      33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: the server could not find the requested resource
E0710 02:17:07.893744      33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: the server could not find the requested resource
W0710 02:17:21.072001      33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: the server could not find the requested resource
E0710 02:17:21.072028      33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: the server could not find the requested resource

liuyan · 2024 年7 月 19 日 07:42

你看浏览器控制台是否有报错信息？
出现这种情况时，你访问的是哪个页面？