在 k3s.service.env 配置代理后，私有仓库无法正常拉取镜像

Gem · 2023 年7 月 2 日 06:06

环境信息:
K3s 版本: v1.24.7+k3s1

/etc/systemd/system/k3s.service.env

HTTP_PROXY=http://192.168.63.21:7890
HTTPS_PROXY=http://192.168.63.21:7890
NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,.fleur.cn
CONTAINERD_HTTP_PROXY=http://192.168.63.21:7890
CONTAINERD_HTTPS_PROXY=http://192.168.63.21:7890
CONTAINERD_NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.fleur.cn

/etc/rancher/k3s/registries.yaml

mirrors:
  "harbor.fleur.cn":
    endpoint:
    - "https://harbor.fleur.cn"

configs:
  "harbor.fleur.cn":
    auth:
      username: admin
      password: **********
    tls:
      insecure_skip_verify: true

pod 内的 nslookup

crictl pull 可以正常

报错信息：

怀疑 k3s.service.env 没生效

cola@k3g-master-6391:~$ ps -aux | grep  containerd
root         966  0.3  0.5 1791384 40680 ?       Ssl  15:29   0:01 /usr/bin/containerd
root        1176  0.6  0.7 764352 64348 ?        Sl   15:29   0:02 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root        1598  0.0  0.1 712196 10072 ?        Sl   15:29   0:00 /var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin/containerd-shim-runc-v2 -namespace k8s.io -id 246d64365c7603cc82932997d01e0860d66aa3e88dd73af1406c6b0aaa2d1add -address /run/k3s/containerd/containerd.sock
root        1648  0.0  0.1 712452 10192 ?        Sl   15:29   0:00 /var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin/containerd-shim-runc-v2 -namespace k8s.io -id f7c16dfa39a8cf29e5c937e24208a39ca70bdc603e705c0a070a96405b1f9d94 -address /run/k3s/containerd/containerd.sock
root        1682  0.0  0.1 712452  9812 ?        Sl   15:29   0:00 /var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin/containerd-shim-runc-v2 -namespace k8s.io -id 062ca6eccc3fc36866c6a5b13ce0ab71d26fe31e1e8046a700c64ee89dd55ec7 -address /run/k3s/containerd/containerd.sock
root        1914  0.0  0.1 712452  9628 ?        Sl   15:29   0:00 /var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin/containerd-shim-runc-v2 -namespace k8s.io -id fff846d468cfe648660d4bf5785e6526555b469b100561de0da3ac6780f4c313 -address /run/k3s/containerd/containerd.sock
root        2005  0.0  0.1 712452  9356 ?        Sl   15:29   0:00 /var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin/containerd-shim-runc-v2 -namespace k8s.io -id e7c5da220f03fd0396fe6183bf121e3a73a983e49646340ff6bec89eccdf5bcc -address /run/k3s/containerd/containerd.sock
cola        9298  0.0  0.0   6608  2344 pts/0    S+   15:34   0:00 grep --color=auto containerd
cola@k3g-master-6391:~$ sudo cat /proc/1176/environ
LANG=en_US.UTF-8PATH=/var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9/bin/auxINVOCATION_ID=8ff819b6eebe4236b0ad1889cd2204e3JOURNAL_STREAM=8:13067SYSTEMD_EXEC_PID=1018RES_OPTIONS= K3S_DATA_DIR=/var/lib/rancher/k3s/data/e430ebdc9a66bbba77e21922f682788d68338a4eb3c035740aeeef38be42b6f9NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16NODE_NAME=k3g-master-6391c

Gem · 2023 年7 月 2 日 09:12

问题已经解决，为了更好地帮助遇到类似问题的人，留个痕。

如果是 k3s 1.22 及以前的版本，遇到 CONTAINERD_NO_PROXY 不生效，可以升级到 1.24.2-rc1+k3s1 及更高版本。
参考以下 issue

github.com/k3s-io/k3s

Setting CONTAINERD_NO_PROXY does not work

opened 06:27AM - 05 Jun 22 UTC

closed 10:41PM - 24 Jun 22 UTC

aussielunix

**Environmental Info:** K3s Version: ```bash # k3s -v k3s version v1.23.6+k3…s1 (418c3fa8) go version go1.17.5 ``` Node(s) CPU architecture, OS, and Version: ```bash # uname -a Linux k3s-01 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ``` Cluster Configuration: Tested with single node with embedded sqlite and 3 node HA with embedded etcd **Describe the bug:** Setting CONTAINERD_NO_PROXY does not work. As mentioned in https://github.com/k3s-io/k3s/issues/2026, https://github.com/k3s-io/k3s/issues/3552 and https://github.com/k3s-io/k3s/pull/4009, a new feature has been added to configure HTTP proxy settings, for Containerd. I can confirm that any variable prefixed with `CONTAINERD_` is passed through with the exception of `CONTAINERD_NO_PROXY` **Steps To Reproduce:** - Installed K3s: ```bash curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh - ``` - create /etc/systemd/system/k3s.service.env ``` CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/ CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/ CONTAINERD_NO_PROXY=".docker.io" ``` - start k3s ```bash systemctl start k3s.service ``` **Expected behavior:** containerd should have the following environment variables set which should allow it to pull containers from the Docker hub without going via the HTTP_PROXY. ``` HTTP_PROXY=http://this.is.my.fake.proxy/ HTTPS_PROXY=https://this.is.my.fake.proxy/ NO_PROXY=".docker.io" ``` **Actual behavior:** Containers can not be pulled. ``` Failed to pull image "busybox": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/busybox:latest": failed to resolve reference "docker.io/library/busybox:latest": failed to do request: Head "https://registry-1.docker.io/v2/library/busybox/manifests/latest": proxyconnect tcp: dial tcp: lookup this.is.my.fake.proxy on 127.0.0.53:53: no such host ``` **Additional context / logs:** [This](https://github.com/k3s-io/k3s/blob/master/pkg/agent/containerd/containerd.go#L62-L82) is where vars prefixed with `CONTAINERD_` are passed through. What I think is happening is https://github.com/k3s-io/k3s/blob/master/pkg/server/server.go#L440-L455 is always setting `NO_PROXY` which is overriding the `CONTAINERD_NO_PROXY` An inspection of the environment for the containerd process shows `NO_PROXY` is always set regardless if `CONTAINERD_NO_PROXY` is set or not. ```bash # cat /proc/29698/environ LANG=C.UTF-8PATH=/var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/auxINVOCATION_ID=a3956a06a8284ebc968c16ac89e3cb65JOURNAL_STREAM=8:298854SYSTEMD_EXEC_PID=29687HTTP_PROXY=http://this.is.my.fake.proxy/HTTPS_PROXY=https://this.is.my.fake.proxy/NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16RES_OPTIONS= K3S_DATA_DIR=/var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8NODE_NAME=k3s-01.example.com ``` - change /etc/systemd/system/k3s.service.env and it works (but not ideal). ``` CONTAINERD_HTTP_PROXY=http://this.is.my.fake.proxy/ CONTAINERD_HTTPS_PROXY=https://this.is.my.fake.proxy/ NO_PROXY=".docker.io" ```

我遇到的问题不同，我使用的是 GitHub - techno-tim/k3s-ansible: The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more 这个项目通过ansible部署，其 agent 节点的位于 /etc/systemd/system/ 下的文件名从 k3s-agent.service 修改成了 k3s-node.service，如果你也和我一样使用该项目部署，请在 agent 节点创建 /etc/systemd/system/k3s-node.service.env 而不是官方写的 /etc/systemd/system/k3s-agent.service.env 。

最后在 v1.24.7+k3s1 正常运行~