Rancher在添加节点时 x509

LT · 2024 年7 月 25 日 09:45

Rancher Server 设置

Rancher 版本： 2.6.14
安装选项 (Docker install/Helm Chart): helm
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
在线或离线部署：在线

下游集群信息

Kubernetes 版本: v1.24.11+k3s1
Cluster Type (Local/Downstream):
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等):

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
- 如果自定义，自定义权限集：

主机操作系统：
centos7
问题描述：
2.8.5也是 x509，求助
重现步骤：

结果：

预期结果：

截图：

其他上下文信息：

日志


INFO: Arguments: --server https://xxx.com --token REDACTED --ca-checksum 9b3272304b48e95ed652e571c17e3094ac566da1c6ec13288c645b79f8900cc2 --etcd --controlplane --worker
INFO: Environment: CATTLE_ADDRESS=172.16.11.201 CATTLE_INTERNAL_ADDRESS= CATTLE_NODE_NAME=master02 CATTLE_ROLE=,etcd,worker,controlplane CATTLE_SERVER=https://xxx.com CATTLE_TOKEN=REDACTED
INFO: Using resolv.conf: nameserver 223.5.5.5
INFO: https://xxx.com/ping is accessible
INFO: xxx.com resolves to 172.16.11.220
INFO: Value from https://xxx.com/v3/settings/cacerts is an x509 certificate
time="2024-07-25T09:40:24Z" level=info msg="Listening on /tmp/log.sock"
time="2024-07-25T09:40:24Z" level=info msg="Rancher agent version v2.6.14 is starting"
time="2024-07-25T09:40:24Z" level=info msg="Option dockerInfo={4f5e4c1c-413b-4579-832f-75150a4d5286 1 1 0 0 1 overlay2 [[Backing Filesystem xfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] [] {[local] [bridge host ipvlan macvlan null overlay] [] [awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} true true true true true true true true true true true true false 30 true 45 2024-07-25T17:40:24.953602738+08:00 json-file systemd 1 0 3.10.0-1160.el7.x86_64 CentOS Linux 7 (Core) 7 linux x86_64 https://index.docker.io/v1/ 0xc000bcd880 8 16705785856 [] /data/docker_data    master02 [] false 26.1.4   map[io.containerd.runc.v2:{runc [] <nil>} runc:{runc [] <nil>}] runc {  inactive false  [] 0 0 <nil> []} false  docker-init {d2d58213f83a351ca8f528a95fbd145f5654e957 d2d58213f83a351ca8f528a95fbd145f5654e957} {v1.1.12-0-g51d5e94 v1.1.12-0-g51d5e94} {de40ad0 de40ad0} [name=seccomp,profile=builtin]  [] []}"
time="2024-07-25T09:40:24Z" level=info msg="Option customConfig=map[address:172.16.11.201 internalAddress: label:map[] roles:[etcd worker controlplane] taints:[]]"
time="2024-07-25T09:40:24Z" level=info msg="Option etcd=true"
time="2024-07-25T09:40:24Z" level=info msg="Option controlPlane=true"
time="2024-07-25T09:40:24Z" level=info msg="Option worker=true"
time="2024-07-25T09:40:24Z" level=info msg="Option requestedHostname=master02"
time="2024-07-25T09:40:24Z" level=info msg="Certificate details from https://xxx.com"
time="2024-07-25T09:40:24Z" level=info msg="Certificate #0 (https://xxx.com)"
time="2024-07-25T09:40:24Z" level=info msg="Subject: CN=xxx.com,C=CN"
time="2024-07-25T09:40:24Z" level=info msg="Issuer: CN=cattle-ca,C=CN"
time="2024-07-25T09:40:24Z" level=info msg="IsCA: false"
time="2024-07-25T09:40:24Z" level=info msg="DNS Names: [xxx.com xxx.com]"
time="2024-07-25T09:40:24Z" level=info msg="IPAddresses: [172.16.8.205 172.16.8.206]"
time="2024-07-25T09:40:24Z" level=info msg="NotBefore: 2024-07-25 10:28:38 +0000 UTC"
time="2024-07-25T09:40:24Z" level=info msg="NotAfter: 2034-07-23 10:28:38 +0000 UTC"
time="2024-07-25T09:40:24Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2024-07-25T09:40:24Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2024-07-25T09:40:24Z" level=info msg="Certificate #1 (https://xxx.com)"
time="2024-07-25T09:40:24Z" level=info msg="Subject: CN=cattle-ca,C=CN"
time="2024-07-25T09:40:24Z" level=info msg="Issuer: CN=cattle-ca,C=CN"
time="2024-07-25T09:40:24Z" level=info msg="IsCA: true"
time="2024-07-25T09:40:24Z" level=info msg="DNS Names: <none>"
time="2024-07-25T09:40:24Z" level=info msg="IPAddresses: <none>"
time="2024-07-25T09:40:24Z" level=info msg="NotBefore: 2024-07-25 10:28:38 +0000 UTC"
time="2024-07-25T09:40:24Z" level=info msg="NotAfter: 2034-07-23 10:28:38 +0000 UTC"
time="2024-07-25T09:40:24Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2024-07-25T09:40:24Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2024-07-25T09:40:24Z" level=info msg="Certificate details for /etc/kubernetes/ssl/certs/serverca"
time="2024-07-25T09:40:24Z" level=info msg="Certificate #0 (/etc/kubernetes/ssl/certs/serverca)"
time="2024-07-25T09:40:24Z" level=info msg="Subject: CN=cattle-ca,C=CN"
time="2024-07-25T09:40:24Z" level=info msg="Issuer: CN=cattle-ca,C=CN"
time="2024-07-25T09:40:24Z" level=info msg="IsCA: true"
time="2024-07-25T09:40:24Z" level=info msg="DNS Names: <none>"
time="2024-07-25T09:40:24Z" level=info msg="IPAddresses: <none>"
time="2024-07-25T09:40:24Z" level=info msg="NotBefore: 2024-07-25 10:28:38 +0000 UTC"
time="2024-07-25T09:40:24Z" level=info msg="NotAfter: 2034-07-23 10:28:38 +0000 UTC"
time="2024-07-25T09:40:24Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2024-07-25T09:40:24Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2024-07-25T09:40:24Z" level=fatal msg="Server certificate is not valid, please check if the host has the correct time configured and if the server certificate has a notAfter date and time in the future. Certificate information is displayed above. error: Get \"https://xxx.com\": x509: certificate has expired or is not yet valid: current time 2024-07-25T09:40:24Z is before 2024-07-25T10:28:38Z"

ksd · 2024 年7 月 25 日 13:13

你得把你的安装步骤详细的写下来

LT · 2024 年7 月 26 日 00:07

K3S

postgresql数据库

docker run -d --name postgis -p 5432:5432 -e POSTGRES_PASSWORD='postgres' -v /opt/pg_data:/var/lib/postgresql/data  postgis/postgis:11.5

harbor

将所有镜像导入到harbor中
bash rancher-load-images.sh --image-list ./rancher-images.txt --registry 172.16.8.204:11111

下载安装包并安装

mkdir -p /var/lib/rancher/k3s/agent/images/
cd /var/lib/rancher/k3s/agent/images/
wget https://rancher-mirror.rancher.cn/k3s/v1.26.15-k3s1/k3s
wget https://rancher-mirror.rancher.cn/k3s/v1.26.15-k3s1/k3s-airgap-images-amd64.tar.gz
wget https://rancher-mirror.rancher.cn/k3s/k3s-install.sh

chmod +x k3s-install.sh k3s
mv k3s /usr/local/bin/

wget https://get.helm.sh/helm-v3.15.2-linux-amd64.tar.gz
tar -zxf helm-v3.15.2-linux-amd64.tar.gz
cd linux-amd64/ && mv helm /usr/bin/
cd .. && rm -rf linux-amd64/ helm-v3.15.2-linux-amd64.tar.gz 
chmod +x /usr/bin/helm

安装

#第一个节点
INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='server' K3S_DATASTORE_ENDPOINT='postgres://postgres:postgres@172.16.8.204:6666/k3s?sslmode=disable' ./k3s-install.sh
# 第一节点安装完成后，查看token：
cat /var/lib/rancher/k3s/server/token

#第二个节点
K3S_TOKEN='K10da8d7ca44025951f83aca705d6355a98ec579501754eb2f8e73b11f40c7d6ff3::server:9db97f019c608e0ea34ff656ccb8740d' INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC='server' K3S_DATASTORE_ENDPOINT='postgres://postgres:postgres@172.16.8.204:6666/k3s?sslmode=disable' ./k3s-install.sh

#####
cat >>/etc/profile<<'EOF'
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
EOF
source /etc/profile

镜像仓库

所有节点执行

mkdir -p /etc/rancher/k3s/
cat >/etc/rancher/k3s/registries.yaml<<'EOF'
mirrors:
  docker.io:
    endpoint:
      - "http://172.16.8.204:11111"
  "172.16.8.204:11111":
    endpoint:
      - "http://172.16.8.204:11111"
configs:
  "172.16.8.204:11111":
    auth:
      username: admin
      password: xxxxx
EOF
systemctl restart k3s

证书

# 创建证书
cat >create_self-signed-cert.sh<<'EOF'
#!/bin/bash -e

help ()
{
    echo  ' ================================================================ '
    echo  ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；'
    echo  ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；'
    echo  ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；'
    echo  ' --ssl-size: ssl加密位数，默认2048；'
    echo  ' --ssl-cn: 国家代码(2个字母的代号),默认CN;'
    echo  ' 使用示例:'
    echo  ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ '
    echo  ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650'
    echo  ' ================================================================'
}

case "$1" in
    -h|--help) help; exit;;
esac

if [[ $1 == '' ]];then
    help;
    exit;
fi

CMDOPTS="$*"
for OPTS in $CMDOPTS;
do
    key=$(echo ${OPTS} | awk -F"=" '{print $1}' )
    value=$(echo ${OPTS} | awk -F"=" '{print $2}' )
    case "$key" in
        --ssl-domain) SSL_DOMAIN=$value ;;
        --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;;
        --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;;
        --ssl-size) SSL_SIZE=$value ;;
        --ssl-date) SSL_DATE=$value ;;
        --ca-date) CA_DATE=$value ;;
        --ssl-cn) CN=$value ;;
    esac
done

# CA相关配置
CA_DATE=${CA_DATE:-3650}
CA_KEY=${CA_KEY:-cakey.pem}
CA_CERT=${CA_CERT:-cacerts.pem}
CA_DOMAIN=cattle-ca

# ssl相关配置
SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf}
SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'}
SSL_DATE=${SSL_DATE:-3650}
SSL_SIZE=${SSL_SIZE:-2048}

## 国家代码(2个字母的代号),默认CN;
CN=${CN:-CN}

SSL_KEY=$SSL_DOMAIN.key
SSL_CSR=$SSL_DOMAIN.csr
SSL_CERT=$SSL_DOMAIN.crt

echo -e "\033[32m ---------------------------- \033[0m"
echo -e "\033[32m       | 生成 SSL Cert |       \033[0m"
echo -e "\033[32m ---------------------------- \033[0m"

if [[ -e ./${CA_KEY} ]]; then
    echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m"
    mv ${CA_KEY} "${CA_KEY}"-bak
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
else
    echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m"
    openssl genrsa -out ${CA_KEY} ${SSL_SIZE}
fi

if [[ -e ./${CA_CERT} ]]; then
    echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m"
    mv ${CA_CERT} "${CA_CERT}"-bak
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
else
    echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m"
    openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}"
fi

echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m"
cat > ${SSL_CONFIG} <<EOM
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOM

if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} || -n ${SSL_DOMAIN} ]]; then
    cat >> ${SSL_CONFIG} <<EOM
subjectAltName = @alt_names
[alt_names]
EOM
    IFS=","
    dns=(${SSL_TRUSTED_DOMAIN})
    dns+=(${SSL_DOMAIN})
    for i in "${!dns[@]}"; do
      echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG}
    done

    if [[ -n ${SSL_TRUSTED_IP} ]]; then
        ip=(${SSL_TRUSTED_IP})
        for i in "${!ip[@]}"; do
          echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG}
        done
    fi
fi

echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m"
openssl genrsa -out ${SSL_KEY} ${SSL_SIZE}

echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m"
openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG}

echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m"
openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \
    -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \
    -days ${SSL_DATE} -extensions v3_req \
    -extfile ${SSL_CONFIG}

echo -e "\033[32m ====> 7. 证书制作完成 \033[0m"
echo
echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m"
echo "----------------------------------------------------------"
echo "ca_key: |"
cat $CA_KEY | sed 's/^/  /'
echo
echo "ca_cert: |"
cat $CA_CERT | sed 's/^/  /'
echo
echo "ssl_key: |"
cat $SSL_KEY | sed 's/^/  /'
echo
echo "ssl_csr: |"
cat $SSL_CSR | sed 's/^/  /'
echo
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m"
cat ${CA_CERT} >> ${SSL_CERT}
echo "ssl_cert: |"
cat $SSL_CERT | sed 's/^/  /'
echo

echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m"
echo "cp ${SSL_DOMAIN}.key tls.key"
cp ${SSL_DOMAIN}.key tls.key
echo "cp ${SSL_DOMAIN}.crt tls.crt"
cp ${SSL_DOMAIN}.crt tls.crt
EOF
chmod +x create_self-signed-cert.sh

./create_self-signed-cert.sh \
--ssl-domain=rancher.xxxxx.com \
--ssl-trusted-ip=100.0.0.100,100.0.0.101 \ # 建议写所有节点
--ssl-trusted-domain=rancher.xxxxx.com \
--ssl-size=2048 \
--ssl-date=3650

kubectl create ns cattle-system
kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key
kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem=./cacerts.pem

安装Rancher

helm install rancher ./rancher-2.8.5.tgz \
  --namespace cattle-system \
  --set hostname=rancher.xxxxx.com \
  --set rancherImage=172.16.8.204:11111/rancher/rancher \
  --set ingress.tls.source=secret \
  --set privateCA=true \
  --set systemDefaultRegistry=172.16.8.204:11111 \
  --set replicas=2 \
  --set bootstrapPassword="xxxxx@123456" \
  --set useBundledSystemChart=true

ksd · 2024 年7 月 26 日 02:59

从安装步骤没看出来什么问题，但日志报错：

Server certificate is not valid, please check if the host has the correct time configured and if the server certificate has a notAfter date and time in the future.  Certificate information is displayed above.  error: Get \"https://xxx.com\": x509: certificate has expired or is not yet valid: current time 2024-07-25T09:40:24Z is before 2024-07-25T10:28:38Z

感觉问题还是出在了证书上，你可以进行如下检查：

检查相关服务器的系统时间，看是否和当前时间一致
使用 openssl x509 -in tls.crt -noout -text | grep Not 来分别检查 tls.crt/tls.key/cacerts.pem 的证书时间
通过浏览数来确认 rancher 上的证书，是否为你创建的证书，例如：

image2630×1660 249 KB

LT · 2024 年7 月 26 日 06:46

确实是服务器时间问题，但是现在出现了这样的报错

Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error loading x509 client cert/key for probe kube-apiserver (/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt//var/lib/rancher/rke2/server/tls/client-kube-apiserver.key): open /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: no such file or directory"
Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error loading CA cert for probe (kube-apiserver) /var/lib/rancher/rke2/server/tls/server-ca.crt: open /var/lib/rancher/rke2/server/tls/server-ca.crt: no such file or directory"
Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error while appending ca cert to pool for probe kube-apiserver"
Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory"
Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such file or directory"
Jul 26 14:41:57 master02 rancher-system-agent: time="2024-07-26T14:41:57+08:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error loading x509 client cert/key for probe kube-apiserver (/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt//var/lib/rancher/rke2/server/tls/client-kube-apiserver.key): open /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt: no such file or directory"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such file or directory"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error loading CA cert for probe (kube-apiserver) /var/lib/rancher/rke2/server/tls/server-ca.crt: open /var/lib/rancher/rke2/server/tls/server-ca.crt: no such file or directory"
Jul 26 14:42:02 master02 rancher-system-agent: time="2024-07-26T14:42:02+08:00" level=error msg="error while appending ca cert to pool for probe kube-apiserver"

netlqh · 2024 年8 月 22 日 09:25

这个有解决不，兄弟

ksd · 2024 年8 月 22 日 13:57

这个是镜像拉下来前报的错，如果镜像都拉下来，容器都运行的话，应该就可以了