K3s中registries.yaml配置的镜像仓库不生效

ncqvyp28970 · 2024 年3 月 4 日 07:13

1.25.0+k3s1生效

离线安装v1.25.0+k3s1

mv k3s /usr/local/bin
chmod +x /usr/local/bin/k3s
mkdir -p /var/lib/rancher/k3s/agent/images/
cp ./k3s-airgap-images-amd64.tar.gz /var/lib/rancher/k3s/agent/images/
chmod +x install.sh
INSTALL_K3S_SKIP_DOWNLOAD=true ./install.sh

配置镜像仓库v1.25.0+k3s1-生效

ps -ef | grep containerd
cat > /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  docker.io:
    endpoint:
      - "https://fsp2sfpr.mirror.aliyuncs.com/"
      - "https://registry.cn-hangzhou.aliyuncs.com/"
EOF
systemctl restart k3s
crictl info | grep -A 5 "registry"
/usr/local/bin/k3s-uninstall.sh

v1.28.7k3s1失败

在线安装 v1.28.7k3s1

curl –sfL \
     https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | \
     sed 's/rancher.cn/oss-cn-beijing.aliyuncs.com/g' | \
     INSTALL_K3S_MIRROR=cn sh -

配置镜像仓库v1.28.7k3s1-失败1

ps -ef | grep containerd
cat > /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  docker.io:
    endpoint:
      - "https://fsp2sfpr.mirror.aliyuncs.com/"
      - "https://registry.cn-hangzhou.aliyuncs.com/"
EOF
systemctl restart k3s
crictl info | grep -A 5 "registry"

配置镜像仓库v1.28.7k3s1-失败2

cp /var/lib/rancher/k3s/agent/etc/containerd/config.toml /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
vi /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl #追加
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://fsp2sfpr.mirror.aliyuncs.com/","https://registry.cn-hangzhou.aliyuncs.com/"]
systemctl restart k3s
crictl info | grep -A 5 registry

配置镜像仓库v1.28.7k3s1-configs可以 mirros不行

rm -rf /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
cat > /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  docker.io:
    endpoint:
      - "https://fsp2sfpr.mirror.aliyuncs.com/"
      - "https://registry.cn-hangzhou.aliyuncs.com/"
  registry.my.io: # 这里并不是像官方写的那样，docker.io 为 key endpoint 为私有仓库其实是一种镜像加速的意思
    endpoint:
      - "http://registry.my.io"
configs:
  "registry.my.io": # 如果仓库是完全开放没有私有项目的则不需要配置 configs 
    auth:
      username: admin
      password: Harbor12345
EOF
systemctl restart k3s
crictl info | grep -A 5 registry

参考文档：

在K3s中配置镜像仓库

创建pod, 拉取registry 私有库镜像报错:x509

k3s 私有仓库问题

K3s Server 配置参考

Private Registry Configuration

ksd · 2024 年3 月 4 日 07:23

貌似一个 bug，我在确认下，多谢反馈

ncqvyp28970 · 2024 年3 月 10 日 13:20

[root@k8s-worker01 ~]# cat > /etc/rancher/k3s/registries.yaml <<EOF
> mirrors:
>   docker.io:
>     endpoint:
>       - "https://fsp2sfpr.mirror.aliyuncs.com/"
>       - "https://registry.cn-hangzhou.aliyuncs.com/"
>   192.168.46.170:5000:
>     endpoint:
>       - "http://192.168.46.170:5000"
> EOF
[root@k8s-worker01 ~]# systemctl restart k3s
[root@k8s-worker01 ~]# crictl info | grep -A 5 "registry"
    "registry": {
      "configPath": "/var/lib/rancher/k3s/agent/etc/containerd/certs.d",
      "mirrors": null,
      "configs": null,
      "auths": null,
      "headers": null
[root@k8s-worker01 ~]# crictl pull 192.168.46.170:5000/ruoyi-ui:v3.8
Image is up to date for sha256:bd94927f11e2779dde2e0ee8112880f5b8f966891734c2b01697f456978ef70e
[root@k8s-worker01 ~]# crictl images
IMAGE                                        TAG                    IMAGE ID            SIZE
192.168.46.170:5000/ruoyi-ui                 v3.8                   bd94927f11e27       60.4MB
[root@k8s-worker01 ~]# uname -a
Linux k8s-worker01 5.4.271-1.el7.elrepo.x86_64 #1 SMP Wed Mar 6 14:47:25 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

虽然不生效，但是确能从私有仓库中拉取镜像。

ksd · 2024 年3 月 10 日 23:25

可参考：Mirrors configured in /etc/rancher/k3s/registries.yaml do not take effect · Issue #9626 · k3s-io/k3s · GitHub

Alonso · 2024 年5 月 28 日 13:17

hello，我有点笨，没看明白新版应该怎么设置，您能再给说详细点吗，谢谢您

xingxing122 · 2024 年7 月 15 日 14:39

我也遇到这个问题，大家是如何解决的

372572571 · 2024 年8 月 5 日 12:33

类似的问题，但是我好像是配置的Auth:nil 无法被识别

asdfghhong · 2025 年7 月 18 日 05:20

目前我已经配置我registries.yaml：
[root@k8s01 k3s]# cat registries.yaml
mirrors:
docker.io:
endpoint:
- “https://registry.cn-hangzhou.aliyuncs.com”
- “https://docker.mirrors.ustc.edu.cn”
- “https://registry-1.docker.io”
但是执行这个命令：crictl info|grep regist，什么都没有输出

asdfghhong · 2025 年7 月 18 日 06:05

关于github解决方案，也没有看到具体是什么解决方案

ksd · 2025 年7 月 18 日 06:26

简单来说较新版本的 containerd 已弃用 config.toml 来配置 registry 端点和 TLS 配置，转而采用 docker 风格的目录结构。所以不能再通过 crictl 来去查询镜像仓库的配置，可以直接查看：/var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml

asdfghhong · 2025 年7 月 18 日 06:36

但是，我配置了
[root@k8s01 docker.io]# cat /etc/rancher/k3s/registries.yaml
mirrors:
“docker.io”:
endpoint:
- “https://registry.cn-hangzhou.aliyuncs.com”
- “https://docker.mirrors.ustc.edu.cn” # 可根据需求替换 mirror 站点
- “https://registry-1.docker.io”
[root@k8s01 docker.io]# systemctl restart k3s
[root@k8s01 docker.io]# ll
总用量 0
[root@k8s01 docker.io]# ls
[root@k8s01 docker.io]# pwd
/var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io
发现这个文件夹下面没有你说的hosts.toml文件，而且测试了
sudo crictl pull docker.io/library/nginx，是无法拉取的

ksd · 2025 年7 月 18 日 06:41

kubectl get node -o wide 看看结果

asdfghhong · 2025 年7 月 18 日 06:46

不好意思，下面存在hosts.toml,内容如下：
[root@k8s01 docker.io]# cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml

File generated by k3s. DO NOT EDIT.

server = “https://registry-1.docker.io/v2”
capabilities = [“pull”, “resolve”, “push”]

[host]

[host.“https://registry.cn-hangzhou.aliyuncs.com/v2”]
capabilities = [“pull”, “resolve”]

[host.“https://docker.mirrors.ustc.edu.cn/v2”]
capabilities = [“pull”, “resolve”]
但拉去镜像，貌似没有通过mirros，

ksd · 2025 年7 月 18 日 06:49

那你通过 k3s crictl pull docker.io/library/nginx，然后再观察日志

asdfghhong · 2025 年7 月 18 日 06:56

[root@k8s01 docker.io]# k3s crictl pull docker.io/library/nginx
E0718 11:12:12.598226 10727 log.go:32] “PullImage from image service failed” err=“rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed” image=“docker.io/library/nginx”
FATA[0024] pulling image: failed to pull and unpack image “docker.io/library/nginx:latest”: failed to resolve reference “docker.io/library/nginx:latest”: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

ksd · 2025 年7 月 18 日 07:10

/var/lib/rancher/k3s/agent/containerd/containerd.log

可以从这里去观察对应的 endpoint 的日志

asdfghhong · 2025 年7 月 18 日 07:13

time=“2025-07-18T11:40:31.193794738+08:00” level=info msg=“PullImage "docker.io/library/nginx"”
time=“2025-07-18T11:40:32.949350745+08:00” level=info msg=“trying next host” error=“pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed” host=registry.cn-hangzhou.aliyuncs.com

registry.cn-hangzhou.aliyuncs.com不需要配置认证吧

ksd · 2025 年7 月 18 日 07:27

你应该少截取了一些日志，日志里会挨个到你设置的 endpoint 里去pull 镜像，如果都没拉去到，就会拉去失败。

至于 registry.cn-hangzhou.aliyuncs.com 这个仓库，你可能是通过 system-default-registry 去设置了，所以到这里面去尝试拉去，但是配置不配置不重要，如果所有仓库没有这个镜像，就会拉去失败

asdfghhong · 2025 年7 月 18 日 07:46

可能没办法验证，只能在私有仓库上验证了，感谢。

ksd · 2025 年7 月 18 日 09:23

也不用验证，你从日志就能看得出来啊，我按照你的配置模拟了一下，日志可以看出来是到每个 endpoint 去拉取镜像，然后没找到：

time="2025-07-18T15:09:11.128700696+08:00" level=info msg="PullImage \"nginx\""
time="2025-07-18T15:09:11.790733708+08:00" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed" host=registry.cn-hangzhou.aliyuncs.com
time="2025-07-18T15:09:12.192376323+08:00" level=info msg="trying next host" error="failed to do request: Head \"https://docker.mirrors.ustc.edu.cn/v2/library/nginx/manifests/latest?ns=docker.io\": dial tcp: lookup docker.mirrors.ustc.edu.cn: no such host" host=docker.mirrors.ustc.edu.cn
time="2025-07-18T15:09:42.201614210+08:00" level=info msg="fetch failed" error="failed to do request: Head \"https://registry-1.docker.io/v2/library/nginx/manifests/latest\": dial tcp 157.240.13.8:443: i/o timeout" host=registry-1.docker.io
time="2025-07-18T15:09:42.204514215+08:00" level=error msg="PullImage \"nginx\" failed" error="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/library/nginx:latest\": failed to resolve reference \"docker.io/library/nginx:latest\": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
time="2025-07-18T15:09:42.204734215+08:00" level=info msg="stop pulling image docker.io/library/nginx:latest: active requests=0, bytes read=1508"