私有仓库不能使用http访问下载镜像

kinbod · 2023 年5 月 24 日 08:22

Rancher Server 设置

Rancher 版本：2.7.3
安装选项 (Docker install/Helm Chart):
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
在线或离线部署：
离线

下游集群信息

Kubernetes 版本: 1.24.11
Cluster Type (Local/Downstream):
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等):

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
- 如果自定义，自定义权限集：

主机操作系统：

问题描述：
自定义集群
重现步骤：
注册机器的时候
结果：

下载镜像还是跳到了https，没有使用http下载。
预期结果：
正常注册
截图：

其他上下文信息：
使用本地harbor，harbor没有开启https，使用http访问。

日志

ksd · 2023 年5 月 24 日 10:08

关于镜像仓库配置的截图，你可以把完整的配置截图发过来

kinbod · 2023 年5 月 25 日 02:00

记得之前没有这么复杂啊，现在怎么越来越复杂了，不好用了。

ksd · 2023 年5 月 25 日 02:02

如果我没看错的话，你这个 harbor 是禁用了 http，并且启用了 https 啊

kinbod · 2023 年5 月 25 日 02:09

我看之前的文章说使用rancher自定义安装集群的时候必须使用https，我之前就是http但是每次cattle-system-agent拉取镜像的时候都是跳到https，哎。。。太复杂了

kinbod · 2023 年5 月 25 日 02:10

上面用的就是http，但是报错。所有开启了https，但是还是不行

ksd · 2023 年5 月 25 日 02:34

如果你要在创建下游集群的时候使用 http 的 harbor，你可以按照如下的方式去配置：

ksd · 2023 年5 月 25 日 02:39

你感觉现在比以前负责，是因为你以前创建的是 rke 集群，现在创建的是 rke2 集群，这俩配置方式是不一样的

kinbod · 2023 年5 月 25 日 02:49

奥，好的，，我再试试

kinbod · 2023 年5 月 25 日 03:28

May 25 11:28:48 k8stest-10.novalocal rancher-system-agent[4918]: time=“2023-05-25T11:28:48+08:00” level=warning msg=“Failed to get image from endpoint: Get “https://194.1.1.107:8888/service/token?scope=repository%3Arancher%2Fsystem-agent-installer-rke2%3Apull&service=harbor-registry”: http: server gave HTTP response to HTTPS client”

May 25 11:28:48 k8stest-10.novalocal rancher-system-agent[4918]: time=“2023-05-25T11:28:48+08:00” level=warning msg=“Failed to get image from endpoint: Get “https://194.1.1.107:8888/v2/”: http: server gave HTTP response to HTTPS client”

May 25 11:28:48 k8stest-10.novalocal rancher-system-agent[4918]: time=“2023-05-25T11:28:48+08:00” level=error msg=“error while staging: all endpoints failed: Get “https://194.1.1.107:8888/service/token?scope=repository%3Arancher%2Fsystem-agent-installer-rke2%3Apull&service=harbor-registry”: http: server gave HTTP response to HTTPS client; Get “https://194.1.1.107:8888/v2/”: http: server gave HTTP response to HTTPS client: failed to get image 194.1.1.107:8888/rancher/system-agent-installer-rke2:v1.25.7-rke2r1”

May 25 11:28:48 k8stest-10.novalocal rancher-system-agent[4918]: time=“2023-05-25T11:28:48+08:00” level=error msg=“error executing instruction 0: all endpoints failed: Get “https://194.1.1.107:8888/service/token?scope=repository%3Arancher%2Fsystem-agent-installer-rke2%3Apull&service=harbor-registry”: http: server gave HTTP response to HTTPS client; Get “https://194.1.1.107:8888/v2/”: http: server gave HTTP response to HTTPS client: failed to get image 194.1.1.107:8888/rancher/system-agent-installer-rke2:v1.25.7-rke2r1”

May 25 11:28:48 k8stest-10.novalocal rancher-system-agent[4918]: time=“2023-05-25T11:28:48+08:00” level=error msg=“error encountered during parsing of last run time: parsing time “” as “Mon Jan _2 15:04:05 MST 2006”: cannot parse “” as “Mon””

May 25 11:28:48 k8stest-10.novalocal rancher-system-agent[4918]: time=“2023-05-25T11:28:48+08:00” level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20230525-112848/6db820479a9f44aa4371f932dd5c7f95719abd0cb2fc6da8a72fc9d9aaea10f0_0"

还是报错 http: server gave HTTP response to HTTPS client

ksd · 2023 年5 月 25 日 05:57

你到 rke2 节点上执行：

cat /etc/rancher/rke2/registries.yaml | jq .

cat /etc/rancher/rke2/config.yaml.d/50-rancher.yaml

这样能查看到最终落实到 rke2 中的配置

在 rancher ui 通过 yaml 编辑集群，然后提供关于容器镜像仓库的配置信息

kinbod · 2023 年5 月 25 日 06:22

1.root@k8stest-10 ~]# cat /etc/rancher/rke2/registries.yaml
{“configs”:{“194.1.1.107:8888”:{“auth”:{“username”:“admin”,“password”:“Wj123456”,“auth”:"",“identity_token”:""},“tls”:null}},“mirrors”:{“194.1.1.107:8888”:{“endpoint”:[“http://194.1.1.107:8888”]}}}[root@k8stest-10 ~]#
[root@k8stest-10 ~]# cat /etc/rancher/rke2/config.yaml.d/50-rancher.yaml
{
“agent-token”: “8hjw9d7h55vsm2d6cjv4lr7dvpktcnf25bhkjr5mpql6tw4zr8ml8k”,
“cni”: “calico”,
“disable-kube-proxy”: false,
“etcd-expose-metrics”: false,
“etcd-snapshot-retention”: 5,
“etcd-snapshot-schedule-cron”: “0 */5 * * *”,
“kube-controller-manager-arg”: [
“cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager”,
“secure-port=10257”
],
“kube-controller-manager-extra-mount”: [
“/var/lib/rancher/rke2/server/tls/kube-controller-manager:/var/lib/rancher/rke2/server/tls/kube-controller-manager”
],
“kube-scheduler-arg”: [
“cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler”,
“secure-port=10259”
],
“kube-scheduler-extra-mount”: [
“/var/lib/rancher/rke2/server/tls/kube-scheduler:/var/lib/rancher/rke2/server/tls/kube-scheduler”
],
“node-label”: [
“cattle.io/os=linux”,
“rke.cattle.io/machine=5e17205d-2638-4f38-8c84-edcd3cd2cae5”
],
“node-taint”: [
“node-role.kubernetes.io/control-plane:NoSchedule”,
“node-role.kubernetes.io/etcd:NoExecute”
],
“private-registry”: “/etc/rancher/rke2/registries.yaml”,
“protect-kernel-defaults”: false,
“system-default-registry”: “194.1.1.107:8888”,
“token”: “cm8pn6csdh976mpjnvl5gmc5cn2k7tnxd8b7wgz5wdz4r92mnlxrsm”
}

2.apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
annotations:
field.cattle.io/creatorId: user-n29vw
field.cattle.io/description: mc
creationTimestamp: ‘2023-05-25T03:28:42Z’
finalizers:
- wrangler.cattle.io/cloud-config-secret-remover
- wrangler.cattle.io/provisioning-cluster-remove
- wrangler.cattle.io/rke-cluster-remove
generation: 1
managedFields:
- apiVersion: provisioning.cattle.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.: {}
v:“wrangler.cattle.io/cloud-config-secret-remover”: {}
manager: rancher-v2.7.3-secret-migrator
operation: Update
time: ‘2023-05-25T03:28:42Z’
- apiVersion: provisioning.cattle.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:field.cattle.io/description: {}
f:finalizers:
v:“wrangler.cattle.io/provisioning-cluster-remove”: {}
v:“wrangler.cattle.io/rke-cluster-remove”: {}
f:spec:
.: {}
f:defaultPodSecurityAdmissionConfigurationTemplateName: {}
f:defaultPodSecurityPolicyTemplateName: {}
f:kubernetesVersion: {}
f:localClusterAuthEndpoint:
.: {}
f:caCerts: {}
f:enabled: {}
f:fqdn: {}
f:rkeConfig:
.: {}
f:chartValues:
.: {}
f:rke2-calico: {}
f:etcd:
.: {}
f:disableSnapshots: {}
f:s3: {}
f:snapshotRetention: {}
f:snapshotScheduleCron: {}
f:machineGlobalConfig:
.: {}
f:cni: {}
f:disable-kube-proxy: {}
f:etcd-expose-metrics: {}
f:machinePools: {}
f:machineSelectorConfig: {}
f:registries:
.: {}
f:configs:
.: {}
f:194.1.1.107:8888:
.: {}
f:authConfigSecretName: {}
f:caBundle: {}
f:insecureSkipVerify: {}
f:tlsSecretName: {}
f:mirrors:
.: {}
f:194.1.1.107:8888:
.: {}
f:endpoint: {}
f:rewrite: {}
f:upgradeStrategy:
.: {}
f:controlPlaneConcurrency: {}
f:controlPlaneDrainOptions:
.: {}
f:deleteEmptyDirData: {}
f:disableEviction: {}
f:enabled: {}
f:force: {}
f:gracePeriod: {}
f:ignoreDaemonSets: {}
f:skipWaitForDeleteTimeoutSeconds: {}
f:timeout: {}
f:workerConcurrency: {}
f:workerDrainOptions:
.: {}
f:deleteEmptyDirData: {}
f:disableEviction: {}
f:enabled: {}
f:force: {}
f:gracePeriod: {}
f:ignoreDaemonSets: {}
f:skipWaitForDeleteTimeoutSeconds: {}
f:timeout: {}
manager: rancher
operation: Update
time: ‘2023-05-25T03:28:43Z’
- apiVersion: provisioning.cattle.io/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:clusterName: {}
f:conditions: {}
f:observedGeneration: {}
manager: rancher
operation: Update
subresource: status
time: ‘2023-05-25T03:28:43Z’
name: miniocluster
namespace: fleet-default
resourceVersion: ‘11387319’
uid: eb6c2e66-bb87-459c-a8bc-2ebe3022a813
spec:
kubernetesVersion: v1.25.7+rke2r1
localClusterAuthEndpoint: {}
rkeConfig:
chartValues:
rke2-calico: {}
etcd:
snapshotRetention: 5
snapshotScheduleCron: 0 */5 * * *
machineGlobalConfig:
cni: calico
disable-kube-proxy: false
etcd-expose-metrics: false
machineSelectorConfig:
- config:
protect-kernel-defaults: false
registries:
configs:
194.1.1.107:8888:
authConfigSecretName: registryconfig-auth-6ndzv
mirrors:
194.1.1.107:8888:
endpoint:
- http://194.1.1.107:8888
upgradeStrategy:
controlPlaneConcurrency: ‘1’
controlPlaneDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
workerConcurrency: ‘1’
workerDrainOptions:
deleteEmptyDirData: true
disableEviction: false
enabled: false
force: false
gracePeriod: -1
ignoreDaemonSets: true
ignoreErrors: false
postDrainHooks: null
preDrainHooks: null
skipWaitForDeleteTimeoutSeconds: 0
timeout: 120
status:
clusterName: c-m-sxlsgtbs
conditions:
- lastUpdateTime: ‘2023-05-25T03:27:38Z’
reason: Reconciling
status: ‘True’
type: Reconciling
- lastUpdateTime: ‘2023-05-25T03:27:38Z’
status: ‘False’
type: Stalled
- lastUpdateTime: ‘2023-05-25T03:28:07Z’
status: ‘True’
type: Created
- lastUpdateTime: ‘2023-05-25T05:38:11Z’
status: ‘True’
type: RKECluster
- lastUpdateTime: ‘2023-05-25T03:27:38Z’
status: ‘True’
type: BackingNamespaceCreated
- lastUpdateTime: ‘2023-05-25T03:27:39Z’
status: ‘True’
type: DefaultProjectCreated
- lastUpdateTime: ‘2023-05-25T03:27:39Z’
status: ‘True’
type: SystemProjectCreated
- lastUpdateTime: ‘2023-05-25T03:27:39Z’
status: ‘True’
type: InitialRolesPopulated
- lastUpdateTime: ‘2023-05-25T03:27:40Z’
status: ‘True’
type: CreatorMadeOwner
- lastUpdateTime: ‘2023-05-25T03:27:41Z’
status: ‘True’
type: NoDiskPressure
- lastUpdateTime: ‘2023-05-25T03:27:41Z’
status: ‘True’
type: NoMemoryPressure
- lastUpdateTime: ‘2023-05-25T03:27:42Z’
status: ‘True’
type: SecretsMigrated
- lastUpdateTime: ‘2023-05-25T03:27:42Z’
status: ‘True’
type: ServiceAccountSecretsMigrated
- lastUpdateTime: ‘2023-05-25T03:27:42Z’
status: ‘True’
type: RKESecretsMigrated
- lastUpdateTime: ‘2023-05-25T03:27:53Z’
status: ‘True’
type: Provisioned
- lastUpdateTime: ‘2023-05-25T03:27:52Z’
status: ‘False’
type: Connected
- lastUpdateTime: ‘2023-05-25T05:38:11Z’
message: waiting for viable init node
reason: Waiting
status: Unknown
type: Updated
- lastUpdateTime: ‘2023-05-25T03:28:07Z’
message: Cluster agent is not connected
reason: Disconnected
status: ‘False’
type: Ready
observedGeneration: 1

ksd · 2023 年5 月 25 日 06:46

看你的配置真是一点问题都没有，你确定你是在修改了集群配置之后，重新执行的注册节点的命令？

kinbod · 2023 年5 月 25 日 06:49

是的，以前很丝滑，2.2的版本好像，最近想再试试，结果一步一个坑。docker24.0.1单机安装rancher:v2.6.12也是不行 Get “https://127.0.0.1:6444/version?timeout=15m0s”: dial tcp 127.0.0.1:6444: connect: connection refused，，redhat8.5

kinbod · 2023 年5 月 25 日 06:51

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher:v2.6.12

INFO: Running k3s server --cluster-init --cluster-reset 2023/05/25 06:25:37 [INFO] Rancher version v2.6.12 (4e6a3cbed) is starting 2023/05/25 06:25:37 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:} 2023/05/25 06:25:37 [INFO] Listening on /tmp/log.sock 2023/05/25 06:25:37 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:39 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:41 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:43 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:45 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:47 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:49 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:51 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused 2023/05/25 06:25:58 [FATAL] k3s exited with: exit status 2

ksd · 2023 年5 月 25 日 06:55

哎，你这是另外的问题，没必要和镜像仓库的这个问题混在一起，这不乱套了么，如果你需要，你可以重新创建一个话题

ksd · 2023 年5 月 25 日 06:57

再给个建议，下次提交问题的时候，像日志啊，命令行或者输出啊，最好用 markdown 语法，论坛是支持 markdown 语法的，方便别人的同时也在方便自己。

kinbod · 2023 年5 月 25 日 07:01

好的

那这个问题算是bug么？？？还是有解决办法？

ksd · 2023 年5 月 25 日 08:13

算是 bug，有两个解决方法：

你可以将 harbor 的 http 端口改成 80，然后再按照类似下面图片中的方式去配置：

image2974×1256 267 KB
你可以将 harbor 改成 https，然后在链接 harbor 的时候选中跳过证书验证即可。

kinbod · 2023 年5 月 25 日 09:25

感谢大佬支持，目前rke2已经安装成功，感谢