更新证书后，rancher启动报错

雄赳赳 · 2023 年3 月 2 日 05:30

Rancher Server 设置

Rancher 版本：v2.5.11
安装选项 (Docker install/Helm Chart):
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
在线或离线部署：在线安装

下游集群信息

Kubernetes 版本: v1.20.14+k3s1
Cluster Type (Local/Downstream):
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等):

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
- 如果自定义，自定义权限集：default admin

主机操作系统：
centos 7.6
问题描述：
证书到期：

证书问题已参考官网指导完成，但rancher无法日志一直报错，rancher ui无法访问自定义的k3s集群，如下：

重现步骤：

结果：

预期结果：

截图：
参考官网2.5.x版本的证书轮换指导后，证书显示有效：

其他上下文信息：

日志

2023/03/02 05:24:37 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
2023-03-02 05:25:13.896346 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:501" took too long (363.876698ms) to execute
2023-03-02 05:25:18.928801 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:498" took too long (688.334318ms) to execute
2023-03-02 05:25:18.928862 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/\" range_end:\"/registry/services/endpoints0\" count_only:true " with result "range_response_count:0 size:9" took too long (402.500599ms) to execute
2023-03-02 05:25:18.928933 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-scheduler\" " with result "range_response_count:1 size:480" took too long (687.703045ms) to execute
2023-03-02 05:25:18.928969 W | etcdserver: read-only range request "key:\"/registry/configmaps/kube-system/k3s\" " with result "range_response_count:1 size:509" took too long (359.038393ms) to execute
2023-03-02 05:25:18.928989 W | etcdserver: read-only range request "key:\"/registry/events/\" range_end:\"/registry/events0\" count_only:true " with result "range_response_count:0 size:10" took too long (318.138584ms) to execute
2023/03/02 05:25:19 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, handler helm-clusterrepo-download: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
2023-03-02 05:25:19.904775 W | etcdserver: read-only range request "key:\"/registry/configmaps/\" range_end:\"/registry/configmaps0\" count_only:true " with result "range_response_count:0 size:9" took too long (463.591099ms) to execute
2023-03-02 05:25:20.904975 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/projects/\" range_end:\"/registry/management.cattle.io/projects0\" count_only:true " with result "range_response_count:0 size:9" took too long (452.946773ms) to execute
2023-03-02 05:25:20.905073 W | etcdserver: read-only range request "key:\"/registry/roles/\" range_end:\"/registry/roles0\" count_only:true " with result "range_response_count:0 size:10" took too long (706.597216ms) to execute
2023-03-02 05:25:20.905424 W | etcdserver: request "header:<ID:7587868972250305573 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/local-node\" mod_revision:194700298 > success:<request_put:<key:\"/registry/leases/kube-node-lease/local-node\" value_size:538 >> failure:<request_range:<key:\"/registry/leases/kube-node-lease/local-node\" > >>" with result "size:20" took too long (441.519089ms) to execute
2023-03-02 05:25:20.906711 W | etcdserver: read-only range request "key:\"/registry/leases/\" range_end:\"/registry/leases0\" count_only:true " with result "range_response_count:0 size:9" took too long (170.841545ms) to execute
2023/03/02 05:25:23 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
2023-03-02 05:25:27.934691 W | etcdserver: read-only range request "key:\"/registry/leases/\" range_end:\"/registry/leases0\" count_only:true " with result "range_response_count:0 size:9" took too long (824.976657ms) to execute
2023-03-02 05:25:27.934717 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/kube-controller-manager\" " with result "range_response_count:1 size:499" took too long (968.091215ms) to execute
2023-03-02 05:25:27.934799 W | etcdserver: read-only range request "key:\"/registry/management.cattle.io/multiclusterapps/\" range_end:\"/registry/management.cattle.io/multiclusterapps0\" count_only:true " with result "range_response_count:0 size:7" took too long (841.747008ms) to execute
2023-03-02 05:25:27.934850 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:979" took too long (885.629183ms) to execute
2023-03-02 05:25:28.922660 W | etcdserver: read-only range request "key:\"/registry/leases/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:501" took too long (957.262544ms) to execute
2023-03-02 05:25:30.900465 W | etcdserver: read-only range request "key:\"/registry/validatingwebhookconfigurations/\" range_end:\"/registry/validatingwebhookconfigurations0\" count_only:true " with result "range_response_count:0 size:9" took too long (550.459625ms) to execute
2023-03-02 05:25:30.900533 W | etcdserver: read-only range request "key:\"/registry/csinodes/\" range_end:\"/registry/csinodes0\" count_only:true " with result "range_response_count:0 size:9" took too long (593.709981ms) to execute
2023/03/02 05:25:32 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, handler helm-clusterrepo-download: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
2023-03-02 05:25:47.914260 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/cloud-controller-manager\" " with result "range_response_count:1 size:597" took too long (923.396323ms) to execute
2023-03-02 05:25:47.914934 W | etcdserver: read-only range request "key:\"/registry/namespaces/default\" " with result "range_response_count:1 size:979" took too long (865.744569ms) to execute
2023/03/02 05:26:06 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
2023/03/02 05:26:09 [INFO] Stopping cluster agent for c-mftj7
2023/03/02 05:26:09 [ERROR] failed to start cluster controllers c-mftj7: context canceled
2023/03/02 05:26:55 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
2023/03/02 05:27:42 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
2023/03/02 05:28:12 [INFO] Stopping cluster agent for c-mftj7
2023/03/02 05:28:12 [ERROR] failed to start cluster controllers c-mftj7: context canceled
2023/03/02 05:28:17 [ERROR] error syncing 'rancher-partner-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, handler helm-clusterrepo-download: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974 reset --hard HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-partner-charts/8f17acdce9bffd6e05a58a3798840e408c4ea71783381ecd2e9af30baad65974/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, requeuing
2023/03/02 05:28:27 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
W0302 05:28:43.319654       9 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
2023/03/02 05:29:48 [ERROR] error syncing 'c-mftj7': handler cluster-deploy: Get "https://10.43.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent": waiting for cluster [c-mftj7] agent to connect, requeuing
2023/03/02 05:30:12 [INFO] Stopping cluster agent for c-mftj7
2023/03/02 05:30:12 [ERROR] failed to start cluster controllers c-mftj7: context canceled
2023/03/02 05:30:18 [ERROR] error syncing 'rancher-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard FETCH_HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
, handler helm-clusterrepo-download: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721 reset --hard HEAD error: exit status 128, detail: fatal: Unable to create '/var/lib/rancher-data/local-catalogs/v2/rancher-charts/4b40cac650031b74776e87c1a726b0484d0877c3ec137da0872547ff9b73a721/.git/index.lock': File exists.

ksd · 2023 年3 月 2 日 06:06

请提供完整的更新证书的操作步骤

雄赳赳 · 2023 年3 月 2 日 06:13

参考官网指导手册：轮换证书 | Rancher文档

ksd · 2023 年3 月 2 日 06:27

既然你这样说，我只能说我按照官网的步骤更新证书，没出现任何问题……

我是要你真实在主机上执行的命令！

雄赳赳 · 2023 年3 月 2 日 06:33

在宿主机执行如下命令进入到rancher容器内：
docker exec -it rancher bash

进入到rancher容器内，执行如下命令：
kubectl --insecure-skip-tls-verify -n kube-system delete secrets k3s-serving
kubectl --insecure-skip-tls-verify delete secret serving-cert -n cattle-system
rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json

执行命令 exit 退出rancher容器

执行重启命令：docker restart rancher
再执行命令：curl --insecure -sfL https://192.168.1.216:30000/v3
再执行重启命令：docker restart rancher

雄赳赳 · 2023 年3 月 2 日 06:40

倒数第二条命令里的ip，我试过内网ip、公网ip、域名、容器ip，都试过

ksd · 2023 年3 月 2 日 06:43

执行这个命令的时候，返回正确的结果了么

ksd · 2023 年3 月 2 日 06:44

而且，你的 server url 默认设置的是啥，就是加油节点通过哪个地址去连接的 rancher

雄赳赳 · 2023 年3 月 2 日 06:47

我执行后什么结果也没有我就直接重启容器了

雄赳赳 · 2023 年3 月 2 日 06:49

server url 我也写过域名方式的您的意思是下图中的地址么？

ksd · 2023 年3 月 2 日 06:49

你的下游集群是通过哪个地址链接的 rancher

雄赳赳 · 2023 年3 月 2 日 06:56

您指的是我的k3s集群么？我是通过rancher页面导入进来的

ksd · 2023 年3 月 2 日 06:58

那你试试 curl 域名，然后观察 rancher server 和 agent 的日志再看看情况

雄赳赳 · 2023 年3 月 2 日 07:06

我 curl 域名后如下图，底部区域执行命令，上部区域 docker logs -f rancher

雄赳赳 · 2023 年3 月 2 日 07:07

您说的观察agent日志请问是如何操作？

ksd · 2023 年3 月 2 日 07:08

看 cluster-agent 日志

雄赳赳 · 2023 年3 月 2 日 07:12

看到了些信息，图中红色线上的ip是我rancher的公网ip

ksd · 2023 年3 月 2 日 07:29

curl 这个公网 IP 试试

雄赳赳 · 2023 年3 月 2 日 07:59

按照您说的我用那个公网IP 试了下 k3s集群好使了
但是日志中报了 not rancher-server，我尝试curl rancher-server，结果没任何反应，以前一直使用的是rancher-server

ksd · 2023 年3 月 2 日 08:04

我不知道你的 rancher-server 这个地址是哪来的，没上下文