Rke2 私有镜像仓库配置验证

前言

Rancher 中文论坛中有社区用户提问: 使用 system-default-registry: "192.168.101.30" 指定 http 的私有镜像仓库,启动 RKE2 时,还是会使用 https 拉取镜像。

话题连接:使用私有镜像仓库部署rke2失败

随后进行一系列验证,避免日后遗忘,所以整理出文档。

环境准备

配置 Harbor,禁用 https,使用 http

参考:https://goharbor.io/docs/1.10/install-config

注释掉 https 的配置,设置 hostname,其他参数默认值即可

root@ip-172-31-8-132:~# cat harbor.yml
#  # https port for harbor, default is 443
#  port: 443
#  # The path of cert and key files for nginx
#  certificate: /your/certificate/path
#  private_key: /your/private/key/path

hostname: harbor2.kingsd.top

上传镜像

可以根据安装的 rke2 版本到 rke2 release 中查找对应的 tall 包的 txt 文件,然后下载到本地。

至于上传到到 harbor 有个简便方法,就是下载 rancher release 中的 rancher-load-images.sh 和 rancher-save-images.sh 进行上传。

验证 system-default-registry 指定 http 的镜像仓库

准备配置文件:

root@ip-172-31-8-132:~# mkdir -p /etc/rancher/rke2
root@ip-172-31-8-132:~# vi /etc/rancher/rke2/config.yaml
system-default-registry: "harbor2.kingsd.top"
root@ip-172-31-8-132:~# vi /etc/rancher/rke2/registries.yaml
mirrors:
  "harbor2.kingsd.top":
    endpoint:
      - "http://harbor2.kingsd.top"

上面配置文件,通过 system-default-registry 设置 RKE2 的默认系统镜像仓库地址为: harbor2.kingsd.top。然后在通过 registries.yaml 设置使用 http 类型的镜像仓库。

安装 rke2

root@ip-172-31-8-132:~# curl -sfL https://get.rke2.io | sh -

[INFO]  finding release for channel stable
[INFO]  using v1.22.8+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.22.8+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.22.8+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@ip-172-31-8-132:~#
root@ip-172-31-8-132:~# systemctl enable rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@ip-172-31-8-132:~#
root@ip-172-31-8-132:~# systemctl start rke2-server.service
Job for rke2-server.service failed because the control process exited with error code.
See "systemctl status rke2-server.service" and "journalctl -xe" for details.

可以看到已经报错,继续查看日志:

root@ip-172-31-8-132:~# journalctl -f -u rke2-server
-- Logs begin at Tue 2022-04-19 01:51:19 UTC. --
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=info msg="Checking local image archives in /var/lib/rancher/rke2/agent/images for harbor2.kingsd.top/rancher/rke2-runtime:v1.22.8-rke2r1"
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=warning msg="Failed to load runtime image harbor2.kingsd.top/rancher/rke2-runtime:v1.22.8-rke2r1 from tarball: no local image available for harbor2.kingsd.top/rancher/rke2-runtime:v1.22.8-rke2r1: not found in any file in /var/lib/rancher/rke2/agent/images: image not found"
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=info msg="Using private registry config file at /etc/rancher/rke2/registries.yaml"
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=info msg="Pulling runtime image harbor2.kingsd.top/rancher/rke2-runtime:v1.22.8-rke2r1"
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=warning msg="Failed to get image from endpoint: Get \"https://harbor2.kingsd.top/service/token?scope=repository%3Arancher%2Frke2-runtime%3Apull&service=harbor-registry\": dial tcp 3.98.138.181:443: connect: connection refused"
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=warning msg="Failed to get image from endpoint: Get \"https://harbor2.kingsd.top/v2/\": dial tcp 3.98.138.181:443: connect: connection refused"
Apr 19 05:31:13 ip-172-31-8-132 rke2[20972]: time="2022-04-19T05:31:13Z" level=fatal msg="failed to get runtime image harbor2.kingsd.top/rancher/rke2-runtime:v1.22.8-rke2r1: all endpoints failed: Get \"https://harbor2.kingsd.top/service/token?scope=repository%3Arancher%2Frke2-runtime%3Apull&service=harbor-registry\": dial tcp 3.98.138.181:443: connect: connection refused; Get \"https://harbor2.kingsd.top/v2/\": dial tcp 3.98.138.181:443: connect: connection refused"
Apr 19 05:31:13 ip-172-31-8-132 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Apr 19 05:31:13 ip-172-31-8-132 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Apr 19 05:31:13 ip-172-31-8-132 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).

从日志可以看出,虽然刚才已经在 registries.yaml 中设置了使用 http 类型的 镜像仓库,但是连接的依然是 https:https://harbor2.kingsd.top/v2

初步判断,有可能 system-default-registry 参数不支持 http,只能使用 https 的私有仓库,为了验证这个问题,把 /etc/rancher/rke2/config.yaml 中的 system-default-registry 参数移除,让 rke2 通过默认的 docker.io 去下载镜像。

root@ip-172-31-8-132:~# rm /etc/rancher/rke2/config.yaml

root@ip-172-31-8-132:~# systemctl restart rke2-server.service

稍等片刻,RKE2 集群已经成功启动,接下来再来验证 RKE2 集群能否从 http 的镜像仓库去拉取镜像:

root@ip-172-31-8-132:~# cat nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: harbor2.kingsd.top/rancher/nginx
        ports:
root@ip-172-31-8-132:~# /var/lib/rancher/rke2/bin/kubectl create -f nginx.yaml
deployment.apps/nginx-deployment created
root@ip-172-31-8-132:~# /var/lib/rancher/rke2/bin/kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-54cbd467d4-57nm2   1/1     Running   0          18s
root@ip-172-31-8-132:~# root@ip-172-31-8-132:~# /var/lib/rancher/rke2/bin/kubectl describe pods nginx-deployment-54cbd467d4-57nm2

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  46s   default-scheduler  Successfully assigned default/nginx-deployment-54cbd467d4-57nm2 to ip-172-31-8-132
  Normal  Pulling    45s   kubelet            Pulling image "harbor2.kingsd.top/rancher/nginx"
  Normal  Pulled     42s   kubelet            Successfully pulled image "harbor2.kingsd.top/rancher/nginx" in 3.73656947s
  Normal  Created    42s   kubelet            Created container nginx
  Normal  Started    41s   kubelet            Started container nginx

从测试结果可以看出,可能 RKE2 不支持通过 system-default-registry 指定 http 的私有仓库,但业务集群是可以使用 http 的私有仓库的。

验证 system-default-registry 指定 https(自签名) 的镜像仓库

在内网离线环境,如果 system-default-registry 不支持 http,我们可以使用自签名 https 的私有仓库。

Harbor 配置

参考 https://goharbor.io/docs/1.10/install-config/ 配置 harbor 自签名证书和安装 harbor

安装 RKE2

准备配置文件:

mkdir -p /etc/rancher/rke2
root@ip-172-31-8-132:~# vi /etc/rancher/rke2/config.yaml
system-default-registry: "harbor2.kingsd.top"
root@ip-172-31-8-132:~# vi /etc/rancher/rke2/registries.yaml
mirrors:
  "harbor2.kingsd.top":
    endpoint:
      - "https://harbor2.kingsd.top"
configs:
  harbor2.kingsd.top:
    tls:
      insecure_skip_verify: true

上面配置文件,通过 system-default-registry 设置 RKE2 的默认系统镜像仓库地址为: harbor2.kingsd.top。然后在通过 registries.yaml 设置使用 https 类型的自签名镜像仓库,并且通过 insecure_skip_verify: true 跳过证书验证。

安装 rke2

root@ip-172-31-8-132:~# curl -sfL https://get.rke2.io | sh -
[INFO]  finding release for channel stable
[INFO]  using v1.22.8+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.22.8+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.22.8+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local
root@ip-172-31-8-132:~# systemctl enable rke2-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.
root@ip-172-31-8-132:~#  systemctl restart rke2-server.service

确认 RKE2 系统镜像从 harbor2.kingsd.top 获取

root@ip-172-31-8-132:~# /var/lib/rancher/rke2/bin/kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort

cloud-controller-manager-ip-172-31-8-132:	harbor2.kingsd.top/rancher/rke2-cloud-provider:v0.0.3-build20211118,
etcd-ip-172-31-8-132:	harbor2.kingsd.top/rancher/hardened-etcd:v3.5.1-k3s1-build20220112,
helm-install-rke2-canal--1-xz59r:	harbor2.kingsd.top/rancher/klipper-helm:v0.7.0-build20220315,
helm-install-rke2-coredns--1-tgpmr:	harbor2.kingsd.top/rancher/klipper-helm:v0.7.0-build20220315,
helm-install-rke2-ingress-nginx--1-fqxwk:	harbor2.kingsd.top/rancher/klipper-helm:v0.7.0-build20220315,
helm-install-rke2-metrics-server--1-hh9gp:	harbor2.kingsd.top/rancher/klipper-helm:v0.7.0-build20220315,
kube-apiserver-ip-172-31-8-132:	harbor2.kingsd.top/rancher/hardened-kubernetes:v1.22.8-rke2r1-build20220316,
kube-controller-manager-ip-172-31-8-132:	harbor2.kingsd.top/rancher/hardened-kubernetes:v1.22.8-rke2r1-build20220316,
kube-proxy-ip-172-31-8-132:	harbor2.kingsd.top/rancher/hardened-kubernetes:v1.22.8-rke2r1-build20220316,
kube-scheduler-ip-172-31-8-132:	harbor2.kingsd.top/rancher/hardened-kubernetes:v1.22.8-rke2r1-build20220316,
rke2-canal-tv4d2:	harbor2.kingsd.top/rancher/hardened-calico:v3.21.4-build20220228, harbor2.kingsd.top/rancher/hardened-flannel:v0.17.0-build20220317,
rke2-coredns-rke2-coredns-8f8fcd966-c9lqx:	harbor2.kingsd.top/rancher/hardened-coredns:v1.9.1-build20220318,
rke2-coredns-rke2-coredns-autoscaler-765f78cf97-9x6pb:	harbor2.kingsd.top/rancher/hardened-cluster-autoscaler:v1.8.5-build20211119,
rke2-ingress-nginx-controller-4k8b6:	harbor2.kingsd.top/rancher/nginx-ingress-controller:nginx-1.0.2-hardened4,
rke2-metrics-server-778cf67475-gkrsx:	harbor2.kingsd.top/rancher/hardened-k8s-metrics-server:v0.5.0-build20211119,

后记

从测试结果来看,RKE2 不支持通过 system-default-registry 指定 http 的私有仓库,但业务集群是可以使用 http 的私有仓库的。

如果你在私有环境有需要连接私有镜像仓库,可以将镜像仓库设置为 https。

1 个赞

我的问题跟这个一样,我是用rancher ui安装rke2集群的,,insecure_skip_verify: true也勾选了,harbor也改成了https,还是报错

口说无凭