K3s创建3节点高可用集群，首先安装的节点可以访问其他两个节点的80、443端口，后安装的两个节点不能访问其他节点的80、443端口

K3s
1

环境信息:
K3s 版本:
k3s version v1.20.15+k3s1 (c7a91e1c)
go version go1.15.15

节点 CPU 架构、操作系统和版本：:
Linux lqlvdocker-ap03 5.4.190-1.el7.elrepo.x86_64 #1 SMP Tue Apr 19 11:42:30 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

集群配置:
3 servers,
172.20.100.12
172.20.100.13
172.20.100.14

问题描述:
k3s创建3节点高可用集群，首先安装的节点可以访问其他两个节点的80、443端口，后安装的两个节点不能访问其他节点的80、443端口

复现步骤:

  • 安装 K3s 的命令:
    三个节点都执行以下命令：
    export K3S_TOKEN=“fc343f3adbaba781058bf5addeed15fe”
    export K3S_DATASTORE_ENDPOINT=“mysql://root:adddddd@tcp(mysql.k3s.aaa.com:3306)/k3s”
    export INSTALL_K3S_SKIP_DOWNLOAD=true
    export INSTALL_K3S_EXEC=“server --tls-san service.k3s.aaa.com --private-registry /etc/rancher/k3s/registries.yaml”
    /usr/local/bin/install-k3s.sh

预期结果:
在各个几点上都能执行以下命令，并正常返回结果
curl http://172.20.100.62
curl http://172.20.100.63
curl http://172.20.100.64

实际结果:
只有在172.20.100.12这台第一个安装的节点上能执行以下命令，并正常返回结果，其他两个节点显示连接超时
curl http://172.20.100.62
curl http://172.20.100.63
curl http://172.20.100.64

2

可以查下后面两个节点的 k3s 日志

3

请帮助分析以下日志，是什么原因引起的，谢谢！

日志

E0428 14:20:01.023942 46853 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.83.63:443/apis/metrics.k8s.io/v1beta1: Get “https://10.43.83.63:443/apis/metrics.k8s.io/v1beta1”: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0428 14:20:06.028620 46853 available_controller.go:508] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io “v1beta1.metrics.k8s.io”: the object has been modified; please apply your changes to the latest version and try again
E0428 14:20:13.250867 46853 controller.go:116] loading OpenAPI spec for “v1beta1.metrics.k8s.io” failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 10.43.83.63:443: i/o timeout
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]

日志2

E0428 12:25:27.899904 35121 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.43.83.63:443/apis/metrics.k8s.io/v1beta1: Get “https://10.43.83.63:443/apis/metrics.k8s.io/v1beta1”: dial tcp 10.43.83.63:443: connect: no route to host

E0428 12:25:28.233043 35121 kuberuntime_manager.go:755] createPodSandbox for pod “svclb-traefik-cfgfd_kube-system(2ddadfec-76cb-45b3-9108-3c5458520db1)” failed: rpc error: code = Unknown desc = failed to setup network for sandbox “912a0699ebd24a0b69d15b2ac0482e3d3508394c4f7f8db01bd0f2215da92cb4”: open /run/flannel/subnet.env: no such file or directory

4月 28 12:25:28 lqlvdocker-ap04 k3s[35121]: E0428 12:25:28.233101 35121 pod_workers.go:191] Error syncing pod 2ddadfec-76cb-45b3-9108-3c5458520db1 (“svclb-traefik-cfgfd_kube-system(2ddadfec-76cb-45b3-9108-3c5458520db1)”), skipping: failed to “CreatePodSandbox” for “svclb-traefik-cfgfd_kube-system(2ddadfec-76cb-45b3-9108-3c5458520db1)” with CreatePodSandboxError: “CreatePodSandbox for pod “svclb-traefik-cfgfd_kube-system(2ddadfec-76cb-45b3-9108-3c5458520db1)” failed: rpc error: code = Unknown desc = failed to setup network for sandbox “912a0699ebd24a0b69d15b2ac0482e3d3508394c4f7f8db01bd0f2215da92cb4”: open /run/flannel/subnet.env: no such file or directory”

4

看上去像是集群网络有问题，你可以用kubectl检查一下各组件pod日志跟状态。