K3s中registries.yaml配置的镜像仓库不生效

fshkqhj · 2025 年9 月 7 日 11:50

我有类似的问题，

K3S 版本号是: v1.33.3+k3s1, Helm的版本号是:v3.17.3
就是在pull的时候能够正常

image730×87 3.26 KB
但是helm install, 以及kubectl apply -f , kind 为 HelmChart的时候会不正常。不知道怎么排查

image1204×87 4.12 KB

ksd · 2025 年9 月 8 日 01:37

k3s crictl pull bitnami/kafka 试试

fshkqhj · 2025 年9 月 8 日 03:52

感谢回复。
这个命令是做过,能够正常拉取镜像。

另外我的 kafka.yaml是这样子的

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: kafka
  namespace: kube-system
spec:
  chart: bitnami/kafka
  targetNamespace: infrastucture
  set:
  valuesContent: |
    image:
      tag: latest
    env:
      KAFKA_ADVERTISED_HOST_NAME: kafka-service
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    persistence:
      enabled: true
    sidecar:
      datasources:
        enabled: true

kubectl appy -f ./kafka.yaml的日志. 错误是 repo bitnami NOT FOUND

if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
        echo "KUBERNETES_SERVICE_HOST is using IPv6"
        CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
        CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ false == \t\r\u\e ]]
+ [[ false == \t\r\u\e ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/kafka.tgz.base64
+ CHART_PATH=/tmp/kafka.tgz
+ [[ ! -f /chart/kafka.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ bitnami/kafka == stable/* ]]
+ [[ -n '' ]]
+ helm_update install --namespace infrastucture
++ helm ls --all -f '^kafka$' --namespace infrastucture --output json
++ ++ jq -r tr '"\(.[0].chart),\(.[0].status)"'
'[:upper:]' '[:lower:]'
+ LINE=null,null
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-0-000-HelmChart-ValuesContent.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ null =~ ^(|null)$ ]]
+ [[ null =~ ^(|null)$ ]]
+ echo 'Installing helm chart'
+ helm install --namespace infrastucture kafka bitnami/kafka --values /config/values-0-000-HelmChart-ValuesContent.yaml
Error: INSTALLATION FAILED: repo bitnami not found

这个是 helm 里面的 repo

ksd · 2025 年9 月 8 日 07:05

K3s/RKE2 自带的 Helm controller 在管理的。它的行为和你平时在命令行直接 helm install 有点不一样，相互是不共享的。

针对你的 yaml，需要修改成如下的配置：

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: kafka
  namespace: kube-system
spec:
  repo: https://charts.bitnami.com/bitnami
  chart: kafka
  targetNamespace: infrastucture
  set:
  valuesContent: |
    image:
      tag: latest
    env:
      KAFKA_ADVERTISED_HOST_NAME: kafka-service
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    persistence:
      enabled: true
    sidecar:
      datasources:
        enabled: true

添加了 spec.repo 字段，来指定 bitnami 的地址
修改了 spec.chart 取消了前缀

fshkqhj · 2025 年9 月 8 日 07:12

感谢。我试试看

试过了，还是mirror 的问题

这是logs

root@VM-12-15-ubuntu:~/k3s-charts# kubectl logs -n kube-system -f job/helm-install-kafka
if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
        echo "KUBERNETES_SERVICE_HOST is using IPv6"
        CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
        CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ false == \t\r\u\e ]]
+ [[ false == \t\r\u\e ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/kafka.tgz.base64
+ CHART_PATH=/tmp/kafka.tgz
+ [[ ! -f /chart/kafka.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ kafka/kafka == stable/* ]]
+ [[ -n https://charts.bitnami.com/bitnami ]]
+ [[ -f /auth/username ]]
+ [[ -f /auth/tls.crt ]]
+ helm repo add kafka https://charts.bitnami.com/bitnami
"kafka" already exists with the same configuration, skipping
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kafka" chart repository
Update Complete. ⎈Happy Helming!⎈
+ helm_update install --namespace infrastucture
++ helm ls --all -f '^kafka$' --namespace infrastucture --output json
++ tr '[:upper:]' '[:lower:]'
++ jq -r '"\(.[0].chart),\(.[0].status)"'
+ LINE=null,null
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-0-000-HelmChart-ValuesContent.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ null =~ ^(|null)$ ]]
+ [[ null =~ ^(|null)$ ]]
+ echo 'Installing helm chart'
+ helm install --namespace infrastucture kafka kafka/kafka --values /config/values-0-000-HelmChart-ValuesContent.yaml
Error: INSTALLATION FAILED: failed to perform "FetchReference" on source: Get "https://registry-1.docker.io/v2/bitnamicharts/kafka/manifests/32.4.3": dial tcp 154.85.102.32:443: i/o timeout

这个是 host.toml

root@VM-12-15-ubuntu:~/k3s-charts#  cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]

[host."https://mirror.ccs.tencentyun.com/v2"]
  capabilities = ["pull", "resolve"]

[host."https://registry.cn-hangzhou.aliyuncs.com/v2"]
  capabilities = ["pull", "resolve"]

fshkqhj · 2025 年9 月 8 日 07:26

尝试了一下，还是mirror 的问题

这是logs

root@VM-12-15-ubuntu:~/k3s-charts# kubectl logs -n kube-system -f job/helm-install-kafka
if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
        echo "KUBERNETES_SERVICE_HOST is using IPv6"
        CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
        CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ false == \t\r\u\e ]]
+ [[ false == \t\r\u\e ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/kafka.tgz.base64
+ CHART_PATH=/tmp/kafka.tgz
+ [[ ! -f /chart/kafka.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ kafka/kafka == stable/* ]]
+ [[ -n https://charts.bitnami.com/bitnami ]]
+ [[ -f /auth/username ]]
+ [[ -f /auth/tls.crt ]]
+ helm repo add kafka https://charts.bitnami.com/bitnami
"kafka" already exists with the same configuration, skipping
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kafka" chart repository
Update Complete. ⎈Happy Helming!⎈
+ helm_update install --namespace infrastucture
++ helm ls --all -f '^kafka$' --namespace infrastucture --output json
++ tr '[:upper:]' '[:lower:]'
++ jq -r '"\(.[0].chart),\(.[0].status)"'
+ LINE=null,null
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-0-000-HelmChart-ValuesContent.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ null =~ ^(|null)$ ]]
+ [[ null =~ ^(|null)$ ]]
+ echo 'Installing helm chart'
+ helm install --namespace infrastucture kafka kafka/kafka --values /config/values-0-000-HelmChart-ValuesContent.yaml
Error: INSTALLATION FAILED: failed to perform "FetchReference" on source: Get "https://registry-1.docker.io/v2/bitnamicharts/kafka/manifests/32.4.3": dial tcp 154.85.102.32:443: i/o timeout

这个是 host.toml

root@VM-12-15-ubuntu:~/k3s-charts#  cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml
# File generated by k3s. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]

[host]

[host."https://mirror.ccs.tencentyun.com/v2"]
  capabilities = ["pull", "resolve"]

[host."https://registry.cn-hangzhou.aliyuncs.com/v2"]
  capabilities = ["pull", "resolve"]

ksd · 2025 年9 月 8 日 08:01

我看你mirror 的配置是没问题的，如果mirror 可用，理论上是可以拉取镜像的，我也帮不上啥忙了

如果你看见可以用 crictl 去拉取镜像，放到 chart 里不可以，那你完全可以在主机拉取镜像，然后修改 chart 的 value 优先使用本地镜像。

如果你感觉你 mirror 好用，但是 K3s 里拉取不到，那你可以确认下你的 K3s 集群中是不是所有节点都在 /etc/rancher/k3s/registries.yaml 配置了 mirror。如果某一个节点没配置，pod 正好在这个节点上创建，那肯定也是拉不下来

fshkqhj · 2025 年9 月 8 日 08:10

好的，会不会是mirror本身失败了？有没有靠谱的mirror.我现在只有1太服务器装了k3s

ksd · 2025 年9 月 8 日 08:26

我这没有