Rancher导入自建k8s集群失败，证书问题导致agent无法启动

Borning · 2023 年6 月 1 日 09:44

Rancher Server 设置

Rancher 版本：2.7.3
安装选项:
- Helm Chart 安装，RKE 版本1.4.5：
在线或离线部署：
- 离线部署

下游集群信息

Kubernetes 版本: 1.25.9
Cluster Type::
- Downstream，导入

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
- 管理员

主机操作系统：
Ubuntu 22

问题描述：
通过RKE自建k8s集群，再注册到rancher时报错

重现步骤：

在三台服务器上搭建k8s集群（使用RKE离线安装）
在Rancher UI上导入新搭建的集群
在k8s节点上直接使用kubectl命令注册会提示"证书由未知机构签名"的错误，因此采用官网curl --insecure -sfL | kubectl apply -f -方式注册可以成功
但注册完成后，集群始终处于Pending状态
在k8s节点上检查发现Cluster-agent未启动
检查日志发现是证书认证错误（见日志）

结果：
导入集群失败（Pending）

预期结果：
导入集群成功（Active）

截图：

其他上下文信息：

日志

INFO: Environment: CATTLE_ADDRESS=10.42.0.12 CATTLE_CA_CHECKSUM=4376fe42b8bc49199fea4ae5f7c363c764db6dd9f498473979c9626c22666ac3 CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=tcp://10.43.72.136:80 CATTLE_CLUSTER_AGENT_PORT_443_TCP=tcp://10.43.72.136:443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.43.72.136 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=tcp://10.43.72.136:80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.43.72.136 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.43.72.136 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY=10.30.40.38:5000 CATTLE_INGRESS_IP_DOMAIN=sslip.io CATTLE_INSTALL_UUID=4f8ea810-3568-4bd4-b18a-15f928210e4e CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-56cd4d6cd6-x4mt6 CATTLE_SERVER=https://10.30.40.38:31515 CATTLE_SERVER_VERSION=v2.7.3
INFO: Using resolv.conf: nameserver 10.43.0.10 search cattle-system.svc.cluster.local svc.cluster.local cluster.local options ndots:5
INFO: https://10.30.40.38:31515/ping is accessible
INFO: Value from https://10.30.40.38:31515/v3/settings/cacerts is an x509 certificate
time="2023-06-01T09:37:48Z" level=info msg="Listening on /tmp/log.sock"
time="2023-06-01T09:37:48Z" level=info msg="Rancher agent version v2.7.3 is starting"
time="2023-06-01T09:37:48Z" level=info msg="Certificate details from https://10.30.40.38:31515"
time="2023-06-01T09:37:48Z" level=info msg="Certificate #0 (https://10.30.40.38:31515)"
time="2023-06-01T09:37:48Z" level=info msg="Subject: CN=dynamic,O=dynamic"
time="2023-06-01T09:37:48Z" level=info msg="Issuer: CN=dynamiclistener-ca@1684900228,O=dynamiclistener-org"
time="2023-06-01T09:37:48Z" level=info msg="IsCA: false"
time="2023-06-01T09:37:48Z" level=info msg="DNS Names: <none>"
time="2023-06-01T09:37:48Z" level=info msg="IPAddresses: [10.30.40.36 10.30.40.38 10.42.0.22 10.42.0.29 10.42.0.30 10.42.1.23 10.42.1.37 10.42.2.10 10.42.2.16 10.43.151.44]"
time="2023-06-01T09:37:48Z" level=info msg="NotBefore: 2023-05-24 03:50:28 +0000 UTC"
time="2023-06-01T09:37:48Z" level=info msg="NotAfter: 2024-05-31 03:38:03 +0000 UTC"
time="2023-06-01T09:37:48Z" level=info msg="SignatureAlgorithm: ECDSA-SHA256"
time="2023-06-01T09:37:48Z" level=info msg="PublicKeyAlgorithm: ECDSA"
time="2023-06-01T09:37:48Z" level=info msg="Certificate #1 (https://10.30.40.38:31515)"
time="2023-06-01T09:37:48Z" level=info msg="Subject: CN=dynamiclistener-ca@1684900228,O=dynamiclistener-org"
time="2023-06-01T09:37:48Z" level=info msg="Issuer: CN=dynamiclistener-ca@1684900228,O=dynamiclistener-org"
time="2023-06-01T09:37:48Z" level=info msg="IsCA: true"
time="2023-06-01T09:37:48Z" level=info msg="DNS Names: <none>"
time="2023-06-01T09:37:48Z" level=info msg="IPAddresses: <none>"
time="2023-06-01T09:37:48Z" level=info msg="NotBefore: 2023-05-24 03:50:28 +0000 UTC"
time="2023-06-01T09:37:48Z" level=info msg="NotAfter: 2033-05-21 03:50:28 +0000 UTC"
time="2023-06-01T09:37:48Z" level=info msg="SignatureAlgorithm: ECDSA-SHA256"
time="2023-06-01T09:37:48Z" level=info msg="PublicKeyAlgorithm: ECDSA"
time="2023-06-01T09:37:48Z" level=info msg="Certificate details for /etc/kubernetes/ssl/certs/serverca"
time="2023-06-01T09:37:48Z" level=info msg="Certificate #0 (/etc/kubernetes/ssl/certs/serverca)"
time="2023-06-01T09:37:48Z" level=info msg="Subject: CN=dynamiclistener-ca@1684900228,O=dynamiclistener-org"
time="2023-06-01T09:37:48Z" level=info msg="Issuer: CN=dynamiclistener-ca@1684900228,O=dynamiclistener-org"
time="2023-06-01T09:37:48Z" level=info msg="IsCA: true"
time="2023-06-01T09:37:48Z" level=info msg="DNS Names: <none>"
time="2023-06-01T09:37:48Z" level=info msg="IPAddresses: <none>"
time="2023-06-01T09:37:48Z" level=info msg="NotBefore: 2023-05-24 03:50:28 +0000 UTC"
time="2023-06-01T09:37:48Z" level=info msg="NotAfter: 2033-05-21 03:50:28 +0000 UTC"
time="2023-06-01T09:37:48Z" level=info msg="SignatureAlgorithm: ECDSA-SHA256"
time="2023-06-01T09:37:48Z" level=info msg="PublicKeyAlgorithm: ECDSA"
time="2023-06-01T09:37:48Z" level=fatal msg="Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get \"https://10.30.40.38:31515\": x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"dynamiclistener-ca@1684900228\")"

ksd · 2023 年6 月 2 日 02:50

你安装 rancher 使用的证书有问题，你还是把你安装 rancher 的完整步骤列下吧

rzzh · 2024 年2 月 27 日 09:14

**重现步骤：
./helm install cert-manager jetstack/cert-manager
–namespace cert-manager --create-namespace
–set installCRDs=true
–version v1.8.0

./helm install rancher rancher-stable/rancher --namespace cattle-system --create-namespace --set hostname=zsm.prod.com --set replicas=1 --set systemDefaultRegistry:58443=registry.harbor.com --version 2.6.14

**结果：
e54313ae1b86 registry.harbor.com:58443/rancher/rancher-agent:v2.6.14 "run.sh --server htt?? 30 minutes ago Restarting (1) 34 seconds ago

time=“2024-02-27T08:52:54Z” level=info msg=“Option customConfig=map[address:134.175.220.170 internalAddress: label:map roles:[worker] taints:]”
time=“2024-02-27T08:52:54Z” level=info msg=“Option etcd=false”
time=“2024-02-27T08:52:54Z” level=info msg=“Option controlPlane=false”
time=“2024-02-27T08:52:54Z” level=info msg=“Certificate details from https://134.175.220.122:32176”
time=“2024-02-27T08:52:54Z” level=info msg=“Certificate #0 (https://134.175.220.122:32176)”
time=“2024-02-27T08:52:54Z” level=info msg=“Subject: CN=dynamic,O=dynamic”
time=“2024-02-27T08:52:54Z” level=info msg=“Issuer: CN=dynamiclistener-ca@1709022118,O=dynamiclistener-org”
time=“2024-02-27T08:52:54Z” level=info msg=“IsCA: false”
time=“2024-02-27T08:52:54Z” level=info msg=“DNS Names: ”
time=“2024-02-27T08:52:54Z” level=info msg=“IPAddresses: [10.104.135.186 10.244.202.3 134.175.220.122]”
time=“2024-02-27T08:52:54Z” level=info msg=“NotBefore: 2024-02-27 08:21:58 +0000 UTC”
time=“2024-02-27T08:52:54Z” level=info msg=“NotAfter: 2025-02-26 08:32:22 +0000 UTC”
time=“2024-02-27T08:52:54Z” level=info msg=“SignatureAlgorithm: ECDSA-SHA256”
time=“2024-02-27T08:52:54Z” level=info msg=“PublicKeyAlgorithm: ECDSA”
time=“2024-02-27T08:52:54Z” level=info msg=“Certificate #1 (https://134.175.220.122:32176)”
time=“2024-02-27T08:52:54Z” level=info msg=“Subject: CN=dynamiclistener-ca@1709022118,O=dynamiclistener-org”
time=“2024-02-27T08:52:54Z” level=info msg=“Issuer: CN=dynamiclistener-ca@1709022118,O=dynamiclistener-org”
time=“2024-02-27T08:52:54Z” level=info msg=“IsCA: true”
time=“2024-02-27T08:52:54Z” level=info msg=“DNS Names: ”
time=“2024-02-27T08:52:54Z” level=info msg=“IPAddresses: ”
time=“2024-02-27T08:52:54Z” level=info msg=“NotBefore: 2024-02-27 08:21:58 +0000 UTC”
time=“2024-02-27T08:52:54Z” level=info msg=“NotAfter: 2034-02-24 08:21:58 +0000 UTC”
time=“2024-02-27T08:52:54Z” level=info msg=“SignatureAlgorithm: ECDSA-SHA256”
time=“2024-02-27T08:52:54Z” level=info msg=“PublicKeyAlgorithm: ECDSA”
time=“2024-02-27T08:52:54Z” level=info msg=“Certificate details for /etc/kubernetes/ssl/certs/serverca”
time=“2024-02-27T08:52:54Z” level=info msg=“Certificate #0 (/etc/kubernetes/ssl/certs/serverca)”
time=“2024-02-27T08:52:54Z” level=info msg=“Subject: CN=dynamiclistener-ca@1709022118,O=dynamiclistener-org”
time=“2024-02-27T08:52:54Z” level=info msg=“Issuer: CN=dynamiclistener-ca@1709022118,O=dynamiclistener-org”
time=“2024-02-27T08:52:54Z” level=info msg=“IsCA: true”
time=“2024-02-27T08:52:54Z” level=info msg=“DNS Names: ”
time=“2024-02-27T08:52:54Z” level=info msg=“IPAddresses: ”
time=“2024-02-27T08:52:54Z” level=info msg=“NotBefore: 2024-02-27 08:21:58 +0000 UTC”
time=“2024-02-27T08:52:54Z” level=info msg=“NotAfter: 2034-02-24 08:21:58 +0000 UTC”
time=“2024-02-27T08:52:54Z” level=info msg=“SignatureAlgorithm: ECDSA-SHA256”
time=“2024-02-27T08:52:54Z” level=info msg=“PublicKeyAlgorithm: ECDSA”
time=“2024-02-27T08:52:54Z” level=fatal msg=“Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get “https://134.175.220.122:32176”: x509: certificate signed by unknown authority (possibly because of “x509: ECDSA verification failure” while trying to verify candidate authority certificate “dynamiclistener-ca@1709022118”)”

rzzh · 2024 年2 月 27 日 09:15

大佬我也出现了这个问题发了一个帖子。 rancher server 2.6.14 helm rke k8s:1.23.17 执行步骤回复了麻烦指点下哦尝试了很多遍了还是解决不了