Kubectl logs 报错no such domain

K3s
1

环境信息:
K3s 版本:
k3s version v1.23.8+k3s1 (53f2d4e7)
go version go1.17.5

节点 CPU 架构、操作系统和版本：:
Linux SERVER3 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

集群配置:
2server

问题描述:
kubectl logs 获取pod日志时候报错 No such domain

复现步骤:

预期结果:
kubectl logs calico-node-lq4wj -n kube-system

显示pod日志

实际结果:
Error from server: Get “https://server3:10250/containerLogs/kube-system/calico-node-lq4wj/calico-node”: No such domain
附加上下文/日志:
etc/hosts文件已经修改 可以ping 通server3

(base) yxuer@SERVER3:~$ kubectl logs calico-node-lq4wj -n kube-system -v=6
I0713 11:30:54.263750  760894 loader.go:372] Config loaded from file:  /etc/rancher/k3s/k3s.yaml
I0713 11:30:54.271128  760894 round_trippers.go:553] GET https://127.0.0.1:6443/api?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.279931  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis?timeout=32s 200 OK in 1 milliseconds
I0713 11:30:54.293950  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/metrics.k8s.io/v1beta1?timeout=32s 503 Service Unavailable in 6 milliseconds
I0713 11:30:54.293955  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/scheduling.k8s.io/v1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.293965  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/autoscaling/v2?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.294144  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.295019  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.295381  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/autoscaling/v2beta1?timeout=32s 200 OK in 2 milliseconds
I0713 11:30:54.295421  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/events.k8s.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.295599  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1?timeout=32s 200 OK in 5 milliseconds
I0713 11:30:54.295658  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/networking.k8s.io/v1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.295722  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/discovery.k8s.io/v1beta1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.295960  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/policy/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.296242  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/ui.cattle.io/v1?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.296453  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/catalog.cattle.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.296809  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/crd.projectcalico.org/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.297013  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/node.k8s.io/v1beta1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.297035  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/autoscaling/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.297366  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/discovery.k8s.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.297376  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/storage.k8s.io/v1beta1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.297532  760894 round_trippers.go:553] GET https://127.0.0.1:6443/api/v1?timeout=32s 200 OK in 9 milliseconds
I0713 11:30:54.297672  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/events.k8s.io/v1beta1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.298147  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/node.k8s.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.298150  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.298530  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.298549  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/flowcontrol.apiserver.k8s.io/v1beta1?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.298695  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/k3s.cattle.io/v1?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.298875  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/management.cattle.io/v3?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.298891  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/kubeapps.com/v1alpha1?timeout=32s 200 OK in 5 milliseconds
I0713 11:30:54.298894  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s 200 OK in 9 milliseconds
I0713 11:30:54.299132  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/batch/v1beta1?timeout=32s 200 OK in 8 milliseconds
I0713 11:30:54.299452  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/policy/v1beta1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.299632  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/authorization.k8s.io/v1?timeout=32s 200 OK in 9 milliseconds
I0713 11:30:54.299648  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/helm.cattle.io/v1?timeout=32s 200 OK in 9 milliseconds
I0713 11:30:54.299667  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s 200 OK in 9 milliseconds
I0713 11:30:54.299704  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.299822  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/storage.k8s.io/v1?timeout=32s 200 OK in 6 milliseconds
I0713 11:30:54.300203  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/apiextensions.k8s.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.300231  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/autoscaling/v2beta2?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.300231  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/certificates.k8s.io/v1?timeout=32s 200 OK in 7 milliseconds
I0713 11:30:54.300233  760894 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0713 11:30:54.301581  760894 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0713 11:30:54.534667  760894 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0713 11:30:54.538022  760894 round_trippers.go:553] GET https://127.0.0.1:6443/apis/metrics.k8s.io/v1beta1?timeout=32s 503 Service Unavailable in 2 milliseconds
I0713 11:30:54.544735  760894 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0713 11:30:54.544771  760894 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0713 11:30:54.552122  760894 round_trippers.go:553] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/calico-node-lq4wj 200 OK in 5 milliseconds
I0713 11:30:54.603822  760894 round_trippers.go:553] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/calico-node-lq4wj/log?container=calico-node 500 Internal Server Error in 46 milliseconds
I0713 11:30:54.604697  760894 helpers.go:219] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://server3:10250/containerLogs/kube-system/calico-node-lq4wj/calico-node\": No such domain",
  "code": 500
}]
F0713 11:30:54.604997  760894 helpers.go:118] Error from server: Get "https://server3:10250/containerLogs/kube-system/calico-node-lq4wj/calico-node": No such domain
panic: F0713 11:30:54.604997  760894 helpers.go:118] Error from server: Get "https://server3:10250/containerLogs/kube-system/calico-node-lq4wj/calico-node": No such domain


goroutine 1 [running]:
k8s.io/klog/v2.(*loggingT).output(0x7d46660, 0x3, 0x0, 0xc000fee230, 0x2, {0x61581c6, 0x10}, 0xc0006a8800, 0x0)
        /go/pkg/mod/github.com/k3s-io/klog/v2@v2.30.0-k3s1/klog.go:982 +0x625
k8s.io/klog/v2.(*loggingT).printDepth(0xc002b9a980, 0x76, 0x0, {0x0, 0x0}, 0xc001ebd2f0, {0xc001656620, 0x1, 0x1})
        /go/pkg/mod/github.com/k3s-io/klog/v2@v2.30.0-k3s1/klog.go:735 +0x1ae
k8s.io/klog/v2.FatalDepth(...)
        /go/pkg/mod/github.com/k3s-io/klog/v2@v2.30.0-k3s1/klog.go:1499
k8s.io/kubectl/pkg/cmd/util.fatal({0xc002b9a980, 0x76}, 0xc0019c1180)
        /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/kubectl@v1.23.8-k3s1/pkg/cmd/util/helpers.go:96 +0xc5
k8s.io/kubectl/pkg/cmd/util.checkErr({0x520de40, 0xc0019c1180}, 0x4c78838)
        /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/kubectl@v1.23.8-k3s1/pkg/cmd/util/helpers.go:191 +0x7d7
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
        /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/kubectl@v1.23.8-k3s1/pkg/cmd/util/helpers.go:118
k8s.io/kubectl/pkg/cmd/logs.NewCmdLogs.func1(0xc001d92280, {0xc001de6bc0, 0x1, 0x4})
        /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/kubectl@v1.23.8-k3s1/pkg/cmd/logs/logs.go:159 +0x125
github.com/spf13/cobra.(*Command).execute(0xc001d92280, {0xc001de6b80, 0x4, 0x4})
        /go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc001267b80)
        /go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
k8s.io/component-base/cli.run(0xc001267b80)
        /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/component-base@v1.23.8-k3s1/cli/run.go:146 +0x325
k8s.io/component-base/cli.RunNoErrOutput(...)
        /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/component-base@v1.23.8-k3s1/cli/run.go:84
github.com/k3s-io/k3s/pkg/kubectl.main()
        /go/src/github.com/k3s-io/k3s/pkg/kubectl/main.go:43 +0x8f
github.com/k3s-io/k3s/pkg/kubectl.Main()
        /go/src/github.com/k3s-io/k3s/pkg/kubectl/main.go:36 +0xf6
github.com/docker/docker/pkg/reexec.Init(...)
        /go/pkg/mod/github.com/docker/docker@v20.10.7+incompatible/pkg/reexec/reexec.go:26
main.main()
        /go/src/github.com/k3s-io/k3s/cmd/server/main.go:39 +0xb9a

其他：
本来怀疑是kubectl版本问题 切换kubectl·1.24-windows 问题依旧

2

执行一下 kubectl get nodes -o wide，同时把k3s每个节点的 /etc/hosts 文件也描述一下。

3

非常感谢回复，这里是我执行的结果：

(base) yxuer@SERVER3:~$ kubectl get nodes -o wide
NAME      STATUS   ROLES                       AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
server3   Ready    control-plane,etcd,master   6d2h   v1.23.8+k3s1   192.168.3.240   <none>        Ubuntu 20.04.3 LTS   5.4.0-121-generic   containerd://1.5.13-k3s1
server4   Ready    control-plane,etcd,master   35h    v1.23.8+k3s1   192.168.3.245   <none>        Ubuntu 20.04.3 LTS   5.13.0-51-generic   containerd://1.5.13-k3s1

host文件是发现问题后 加的两个条目 两边是一样的

(base) yxuer@SERVER3:~$ cat /etc/hosts
127.0.0.1       localhost
192.168.3.240   server3
192.168.3.245   server4

另外考虑到coreDns内部是否能解析 我也试过dnstool

(base) yxuer@SERVER3:~$ kubectl exec -i -t dnsutils -- nslookup server3
Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   server3
Address: 192.168.3.240

(base) yxuer@SERVER3:~$ kubectl exec -i -t dnsutils -- nslookup server4
Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   server4
Address: 192.168.3.245
4

！！！！石锤了
由于忽略了一点 两台均配置了http代理…被作为代理服务器的主机hosts文件加入两条dns记录 立马就好了 :rofl: :rofl: :rofl:
谢谢大佬们帮忙排查 乌龙了