证书没更新

乐哈哈 · 2023 年8 月 21 日 01:59

rke版本：rke2 version v1.22.17+rke2r1

问题描述：
在使用：rke2 certificate rotate更新证书后，证书目录：/var/lib/rancher/rke2/server/tls，kube-scheduler，kube-controller-manager2个目录的证书没有更新。

日志提示证书已过期，日志信息：
rancher-system-agent: time=“2023-08-21T09:58:43+08:00” level=debug msg=“Probe output was Get “https://127.0.0.1:10259/healthz”: x509: certificate has expired or is not yet valid: current time 2023-08-21T09:58:43+08:00 is after 2023-06-23T01:05:06Z”

如何解决？

ksd · 2023 年8 月 23 日 01:31

青提供更多的重现步骤帮助我去重现

乐哈哈 · 2023 年9 月 5 日 07:14

rancher采用2.6.3，docker部署，3个master节点，若干node节点，rke2版本：v1.21.9+rke2r1 ，k8s集群快到期了，赶上公司停电，服务器重启之后，k8s证书更新。“如果证书已经过期或剩余的时间不足 90 天，则在 RKE2 重启时轮换证书。” 证书目录：/var/lib/rancher/rke2/server/tls，kube-scheduler，kube-controller-manager2个目录的证书没有更新。没有其他特殊操作

乐哈哈 · 2023 年9 月 5 日 07:16

后期用rancher部署k8s，使用命令：systemctl stop rke2-server
rke2 certificate rotate
systemctl start rke2-server 进行证书更新操作，目录：/var/lib/rancher/rke2/server/tls，kube-scheduler，kube-controller-manager2个目录的证书一样没有更新。

ksd · 2023 年9 月 5 日 09:36

我自己测试了一下 rke2 certificate rotate ，没任何问题，你可以自己比对下证书时间。

安装 rke2

root@ip-172-31-12-145:~# curl -sfL https://get.rke2.io | sh -

[INFO]  finding release for channel stable
[INFO]  using v1.25.12+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.25.12+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.25.12+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@ip-172-31-12-145:~#
root@ip-172-31-12-145:~#
root@ip-172-31-12-145:~# systemctl enable rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@ip-172-31-12-145:~#
root@ip-172-31-12-145:~# systemctl start rke2-server.service

查询证书过期时间

root@ip-172-31-12-145:~# for i in `ls /var/lib/rancher/rke2/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
/var/lib/rancher/rke2/server/tls/client-admin.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-ca.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/client-controller.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-kube-proxy.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-controller.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-scheduler.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/client-supervisor.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/request-header-ca.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt
notAfter=Sep  4 09:30:13 2024 GMT

手动轮换证书

root@ip-172-31-12-145:~# rke2 certificate rotate
INFO[0000] Server detected, rotating server certificates
INFO[0000] Rotating certificates for admin service
INFO[0000] Rotating certificates for etcd service
INFO[0000] Rotating certificates for api-server service
INFO[0000] Rotating certificates for controller-manager service
INFO[0000] Rotating certificates for cloud-controller service
INFO[0000] Rotating certificates for scheduler service
INFO[0000] Rotating certificates for rke2-server service
INFO[0000] Rotating dynamic listener certificate
INFO[0000] Rotating certificates for rke2-controller service
INFO[0000] Rotating certificates for auth-proxy service
INFO[0000] Rotating certificates for kubelet service
INFO[0000] Rotating certificates for kube-proxy service
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/rke2/server/tls-1693906466, please restart rke2 server or agent to rotate certificates
root@ip-172-31-12-145:~# systemctl restart rke2-server.service

再次查看证书有效期

root@ip-172-31-12-145:~# for i in `ls /var/lib/rancher/rke2/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
/var/lib/rancher/rke2/server/tls/client-admin.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-ca.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/client-controller.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-kube-proxy.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-controller.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-scheduler.crt
notAfter=Sep  4 09:35:09 2024 GMT
/var/lib/rancher/rke2/server/tls/client-supervisor.crt
notAfter=Sep  4 09:30:13 2024 GMT
/var/lib/rancher/rke2/server/tls/request-header-ca.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
notAfter=Sep  2 09:30:13 2033 GMT
/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt
notAfter=Sep  4 09:35:09 2024 GMT

chqj · 2023 年9 月 6 日 02:58

大佬，工作节点怎么轮换证书呢？手动设置过期时间，master节点轮换后乱换工作节点失败。

ksd · 2023 年9 月 6 日 06:12

理论上来说，乱换证书只轮换 rke2 master 节点即可，和 agent 节点没啥关系，你这个报错，得看看具体的 agent 日志。

下面是我的测试：

安装 rke2

安装 rke2 masster 节点

root@ip-172-31-2-226:~# curl -sfL https://get.rke2.io | sh -

[INFO]  finding release for channel stable
[INFO]  using v1.25.12+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.25.12+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.25.12+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@ip-172-31-2-226:~#
root@ip-172-31-2-226:~# systemctl enable rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@ip-172-31-2-226:~# systemctl start rke2-server.service

安装 rke2 worker 节点

root@ip-172-31-12-130:~# curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -
[INFO]  finding release for channel stable
[INFO]  using v1.25.12+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.25.12+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.25.12+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@ip-172-31-12-130:~#
root@ip-172-31-12-130:~# systemctl enable rke2-agent.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-agent.service → /usr/local/lib/systemd/system/rke2-agent.service.

root@ip-172-31-12-130:~#
root@ip-172-31-12-130:~#
root@ip-172-31-12-130:~# mkdir -p /etc/rancher/rke2/
root@ip-172-31-12-130:~# cat /etc/rancher/rke2/config.yaml
server: https://172.31.2.226:9345
token: a514b9a2e3f6187954c0956d4b116cf7
root@ip-172-31-12-130:~#
root@ip-172-31-12-130:~# systemctl start rke2-agent.service

查看集群状态

root@ip-172-31-2-226:~# kubectl get nodes
NAME               STATUS   ROLES                       AGE   VERSION
ip-172-31-12-130   Ready    <none>                      63s   v1.25.12+rke2r1
ip-172-31-2-226    Ready    control-plane,etcd,master   10m   v1.25.12+rke2r1

查看证书有效期


root@ip-172-31-2-226:~# for i in `ls /var/lib/rancher/rke2/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
/var/lib/rancher/rke2/server/tls/client-admin.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/client-controller.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-kube-proxy.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-controller.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-scheduler.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/client-supervisor.crt
notAfter=Sep  5 05:35:44 2024 GMT
/var/lib/rancher/rke2/server/tls/request-header-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt
notAfter=Sep  5 05:35:44 2024 GMT

修改系统时间为一年以后

master 节点

root@ip-172-31-2-226:~# date
Wed Sep  6 05:51:39 UTC 2023
root@ip-172-31-2-226:~#
root@ip-172-31-2-226:~# timedatectl set-ntp no

root@ip-172-31-2-226:~#
root@ip-172-31-2-226:~# date -s 20240920
Fri Sep 20 00:00:00 UTC 2024
root@ip-172-31-2-226:~#
root@ip-172-31-2-226:~# date
Fri Sep 20 00:00:03 UTC 2024

worker 节点

root@ip-172-31-12-130:/var/lib/rancher/rke2# timedatectl set-ntp no
root@ip-172-31-12-130:/var/lib/rancher/rke2# date -s 20240920
Fri Sep 20 00:00:00 UTC 2024
root@ip-172-31-12-130:/var/lib/rancher/rke2# date
Fri Sep 20 00:00:01 UTC 2024

查看证书有效期

本地磁盘上的证书均已轮换

root@ip-172-31-2-226:~# for i in `ls /var/lib/rancher/rke2/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
/var/lib/rancher/rke2/server/tls/client-admin.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/client-controller.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-kube-proxy.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-controller.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-scheduler.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-supervisor.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/request-header-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt
notAfter=Sep 20 00:00:44 2025 GMT

rke2-serving 未轮换

root@ip-172-31-2-226:~# kubectl get secret -n kube-system rke2-serving --template='{{index .data "tls.crt"}}' | base64 -d | openssl x509 -text | grep -E "After|Before"
            Not Before: Sep  6 05:35:44 2023 GMT
            Not After : Sep  5 05:35:44 2024 GMT

rke2 server 和 agent 均报证书过期的日志：

Sep 20 00:12:27 ip-172-31-2-226 rke2[22111]: time="2024-09-20T00:12:27Z" level=error msg="CA cert validation failed: Get \"https://127.0.0.1:9345/cacerts\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-09-20T00:12:27Z is after 2024-09-05T05:35:44Z"

手动轮换 rke2-serving

root@ip-172-31-2-226:~# rm /var/lib/rancher/rke2/server/tls/dynamic-cert.json
root@ip-172-31-2-226:~# systemctl restart rke2-server.service

再次查看证书有效期

root@ip-172-31-2-226:~# for i in `ls /var/lib/rancher/rke2/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
/var/lib/rancher/rke2/server/tls/client-admin.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/client-controller.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-kube-proxy.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-cloud-controller.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-rke2-controller.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-scheduler.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/client-supervisor.crt
notAfter=Sep 20 00:00:44 2025 GMT
/var/lib/rancher/rke2/server/tls/request-header-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt
notAfter=Sep  3 05:35:44 2033 GMT
/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt
notAfter=Sep 20 00:00:44 2025 GMT
root@ip-172-31-2-226:~#
root@ip-172-31-2-226:~#
root@ip-172-31-2-226:~# kubectl get secret -n kube-system rke2-serving --template='{{index .data "tls.crt"}}' | base64 -d | openssl x509 -text | grep -E "After|Before"
            Not Before: Sep  6 05:35:44 2023 GMT
            Not After : Sep 20 00:16:12 2025 GMT

chqj · 2023 年9 月 6 日 07:16

大佬，手动轮换rke2-serving执行一样的操作，重启rke2-server.service ，再次查看时间没变

ksd · 2023 年9 月 6 日 07:29

那你手动删除 rke2-serving 和 dynamic-cert.json 试试，但最好提前找个测试环境试试，我不确定删除 rke2-serving 会不会有影响

ksd · 2023 年9 月 6 日 07:29

如果再不行，可以升级到我使用的版本

chqj · 2023 年9 月 6 日 07:56

手动删除rke2-serving后重启,时间更新了

chqj · 2023 年9 月 6 日 07:59

大佬，牛逼

乐哈哈 · 2023 年9 月 6 日 09:56

集群证书没问题，还有一个证书，目录：/var/lib/rancher/rke2/server/tls/kube-scheduler，/var/lib/rancher/rke2/server/tls/kube-controller-manager 这里面的证书没有更新，这个证书没更新的话，rancher集群会提示“ Provisioning bootstrap node(s) custom-66b25a09a714: waiting on probes: kube-controller-manager, kube-scheduler ”，检测失败。

乐哈哈 · 2023 年9 月 6 日 10:02

更新完证书，将时间调到一年后，登录rancher，看

这个是否正常，看日志：”https://127.0.0.1:10259/healthz“ 检测是否正常。

乐哈哈 · 2023 年9 月 6 日 10:03

ksd，您回复的没问题，只是问题点不是一个。

乐哈哈 · 2023 年9 月 7 日 01:05

[root@redis45 server]# ls -lhtr tls

总用量 112K

-rw------- 1 root root 227 9月 6 2023 client-ca.key

-rw-r–r-- 1 root root 570 9月 6 2023 client-ca.crt

-rw------- 1 root root 227 9月 6 2023 server-ca.key

-rw-r–r-- 1 root root 570 9月 6 2023 server-ca.crt

-rw------- 1 root root 227 9月 6 2023 request-header-ca.key

-rw-r–r-- 1 root root 595 9月 6 2023 request-header-ca.crt

-rw------- 1 root root 1.7K 9月 6 2023 service.key

drwx------ 2 root root 6 9月 6 2023 temporary-certs

drwxr-xr-x 2 root root 76 9月 6 2023 kube-controller-manager

drwxr-xr-x 2 root root 58 9月 6 2023 kube-scheduler

-rw------- 1 root root 227 8月 7 08:58 client-admin.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-admin.crt

-rw------- 1 root root 227 8月 7 08:58 client-controller.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-controller.crt

-rw------- 1 root root 227 8月 7 08:58 client-scheduler.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-scheduler.crt

-rw------- 1 root root 227 8月 7 08:58 client-kube-apiserver.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-kube-apiserver.crt

-rw------- 1 root root 227 8月 7 08:58 client-kube-proxy.key

-rw------- 1 root root 227 8月 7 08:58 client-rke2-controller.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-kube-proxy.crt

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-rke2-controller.crt

-rw------- 1 root root 227 8月 7 08:58 client-rke2-cloud-controller.key

-rw------- 1 root root 227 8月 7 08:58 client-kubelet.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-rke2-cloud-controller.crt

-rw------- 1 root root 227 8月 7 08:58 serving-kube-apiserver.key

-rw------- 1 root root 227 8月 7 08:58 serving-kubelet.key

-rw-r–r-- 1 root root 1.4K 8月 7 08:58 serving-kube-apiserver.crt

-rw------- 1 root root 227 8月 7 08:58 client-auth-proxy.key

-rw-r–r-- 1 root root 1.2K 8月 7 08:58 client-auth-proxy.crt

drwxr-xr-x 2 root root 232 8月 7 08:58 etcd

-rw-r–r-- 1 root root 2.9K 8月 7 08:58 dynamic-cert.json

[root@redis45 server]# date

2024年 08月 07日星期三 09:00:01 CST

[root@redis45 server]# ls -lhtr tls/kube-scheduler/

总用量 8.0K

-rw------- 1 root root 1.7K 9月 6 2023 kube-scheduler.key

-rw-r–r-- 1 root root 2.3K 9月 6 2023 kube-scheduler.crt

[root@redis45 server]# ls -lhtr tls/kube-controller-manager/

总用量 8.0K

-rw------- 1 root root 1.7K 9月 6 2023 kube-controller-manager.key

-rw-r–r-- 1 root root 2.3K 9月 6 2023 kube-controller-manager.crt

[root@redis45 server]# date

2024年 10月 07日星期一 08:57:37 CST

[root@redis45 server]#

查看日志 tail -f /var/log/message

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg="[Prober] (calico) writing probe status to map"

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg=“Probe output was Get “https://127.0.0.1:10259/healthz”: x509: certificate has expired or is not yet valid: current time 2024-10-07T08:59:06+08:00 is after 2024-09-05T07:47:14Z”

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg=“Setting success threshold to 1”

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg=“Setting failure threshold to 2”

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg=“Probe failed”

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg="[Prober] (kube-scheduler) writing probe status to map"

Oct 7 08:59:06 redis45 rancher-system-agent[12862]: time=“2024-10-07T08:59:06+08:00” level=debug msg=“Probe output was Get “https://127.0.0.1:10257/healthz”: x509: certificate has expired or is not yet valid: current time 2024-10-07T08:59:06+08:00 is after 2024-09-05T07:47:14Z”

chenrizhi · 2023 年11 月 21 日 12:21

github.com/rancher/rancher

[BUG] Rancher managed RKE2 clusters stuck in "Waiting for probes: kube-controller-manager, kube-scheduler"

opened 10:52AM - 11 Apr 23 UTC

closed 04:08PM - 24 May 23 UTC

AlexisDucastel

kind/bug area/provisioning-v2 team/area2 JIRA team/area2/sc

As the issue occurs on quite every cluster we own, I try to sum up version infor…mation for all of them : **Rancher Server Setup** - Rancher version: 2.6.9 and 2.7.1 - Installation option (Docker install/Helm Chart): - HeIm chart on RKE2 cluster version 1.24 - Proxy/Cert Details: **Information about the Cluster** - Kubernetes version: - Cluster Type (Local/Downstream): - Downstream RKE2 Custom (version 1.24.7+rek2r1, 1.24.8+rke2r1, 1.24.9+rke2r2) **User Information** - What is the role of the user logged in? - Admin **Describe the bug** On all our clusters, upgrades are frozen with one/multiple controlplane node(s) stuck in state `Waiting for probes: kube-controller-manager, kube-scheduler`. **To Reproduce** Not sure how to reproduce it, but all our clusters have hundreds days of run time, and they did not have any problem prior to Rancher 2.6.9 / Kubernetes version 1.24.7, but we have mix of cluster with Rancher 2.6.9 and 2.7.1 with clusters in 1.24.7 to 1.24.9. The problem appears after we edit the cluster to upgrade to a newer version. **Result** Upgrade is stuck, waiting for on or more controlpane stuck in state `Waiting for probes: kube-controller-manager, kube-scheduler`. **Expected Result** We expect upgrades to work without being stuck. **Screenshots** Ex cluster 1 (on controlplane impacted) : ![image](https://user-images.githubusercontent.com/1911945/231137741-921b1bcc-f939-4ff1-ad36-8120c4b2c860.png) Ex cluster 2 (all controlplanes impacted) : ![image](https://user-images.githubusercontent.com/1911945/231137652-98f3edd2-25ac-49dc-a483-3d4afd7344a4.png) **Additional context** We tried to upgrade a stuck cluster that was stuck going from 1.24.7 to 1.24.9, by upgrading to 1.24.11, nothing happened. When we restarted `rancher-system-agent` the node was effectively in 1.24.11, but still stuck in state `Waiting for probes: kube-controller-manager, kube-scheduler`. (SURE-6264)