正常运行一段时间内网不互通问题

Rancher 2.x
1

rancher：2.5.12
Kubernetes Version: v1.20.15
网络：flannel:v0.15.1

我用rancher创建的k8s集群，正式跑一段时间后内网就不互通了 然后我删除掉rancher/mirrored-coreos-flannel:v0.15.1 的pod 重新部署pod 后就好了

2

应该抓一些flanneld的logs来看一下，才能展开讨论。否则基本石沉大海了。

3

这是其一：
I0510 05:29:59.904419 1 main.go:217] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[nm-k8s-pro] ifaceRegex: ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}

2022/5/10 13:29:59 W0510 05:29:59.905812 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

2022/5/10 13:30:00 I0510 05:30:00.116443 1 kube.go:120] Waiting 10m0s for node controller to sync

2022/5/10 13:30:00 I0510 05:30:00.116596 1 kube.go:378] Starting kube subnet manager

2022/5/10 13:30:01 I0510 05:30:01.118115 1 kube.go:127] Node controller sync successful

2022/5/10 13:30:01 I0510 05:30:01.118142 1 main.go:237] Created subnet manager: Kubernetes Subnet Manager - k8s-master

2022/5/10 13:30:01 I0510 05:30:01.118148 1 main.go:240] Installing signal handlers

2022/5/10 13:30:01 I0510 05:30:01.118951 1 main.go:459] Found network config - Backend type: vxlan

2022/5/10 13:30:01 I0510 05:30:01.122401 1 main.go:698] Using interface with name nm-k8s-pro and address 10.8.8.2

2022/5/10 13:30:01 I0510 05:30:01.122495 1 main.go:720] Defaulting external address to interface address (10.8.8.2)

2022/5/10 13:30:01 I0510 05:30:01.122510 1 main.go:733] Defaulting external v6 address to interface address ()

2022/5/10 13:30:01 I0510 05:30:01.122762 1 vxlan.go:137] VXLAN config: VNI=1 Port=8472 GBP=false Learning=false DirectRouting=false

2022/5/10 13:30:01 I0510 05:30:01.304971 1 kube.go:339] Setting NodeNetworkUnavailable

2022/5/10 13:30:01 I0510 05:30:01.318968 1 main.go:340] Setting up masking rules

2022/5/10 13:30:01 I0510 05:30:01.506864 1 main.go:361] Changing default FORWARD chain policy to ACCEPT

2022/5/10 13:30:01 I0510 05:30:01.507076 1 main.go:374] Wrote subnet file to /run/flannel/subnet.env

2022/5/10 13:30:01 I0510 05:30:01.507167 1 main.go:378] Running backend.

2022/5/10 13:30:01 I0510 05:30:01.507247 1 main.go:396] Waiting for all goroutines to exit

2022/5/10 13:30:01 I0510 05:30:01.507327 1 vxlan_network.go:60] watching for new subnet leases

其二：
I0510 05:30:02.038049 1 main.go:217] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[nm-k8s-pro] ifaceRegex: ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}

2022/5/10 13:30:02 W0510 05:30:02.038137 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

2022/5/10 13:30:02 I0510 05:30:02.231485 1 kube.go:120] Waiting 10m0s for node controller to sync

2022/5/10 13:30:02 I0510 05:30:02.233709 1 kube.go:378] Starting kube subnet manager

2022/5/10 13:30:03 I0510 05:30:03.232637 1 kube.go:127] Node controller sync successful

2022/5/10 13:30:03 I0510 05:30:03.232665 1 main.go:237] Created subnet manager: Kubernetes Subnet Manager - k8s-node01-hk

2022/5/10 13:30:03 I0510 05:30:03.232671 1 main.go:240] Installing signal handlers

2022/5/10 13:30:03 I0510 05:30:03.232732 1 main.go:459] Found network config - Backend type: vxlan

2022/5/10 13:30:03 I0510 05:30:03.234155 1 main.go:698] Using interface with name nm-k8s-pro and address 10.8.8.3

2022/5/10 13:30:03 I0510 05:30:03.234191 1 main.go:720] Defaulting external address to interface address (10.8.8.3)

2022/5/10 13:30:03 I0510 05:30:03.234199 1 main.go:733] Defaulting external v6 address to interface address ()

2022/5/10 13:30:03 I0510 05:30:03.234263 1 vxlan.go:137] VXLAN config: VNI=1 Port=8472 GBP=false Learning=false DirectRouting=false

2022/5/10 13:30:03 I0510 05:30:03.235966 1 kube.go:339] Setting NodeNetworkUnavailable

2022/5/10 13:30:03 I0510 05:30:03.251394 1 main.go:340] Setting up masking rules

2022/5/10 13:30:03 I0510 05:30:03.521068 1 main.go:361] Changing default FORWARD chain policy to ACCEPT

2022/5/10 13:30:03 I0510 05:30:03.521174 1 main.go:374] Wrote subnet file to /run/flannel/subnet.env

2022/5/10 13:30:03 I0510 05:30:03.521183 1 main.go:378] Running backend.

2022/5/10 13:30:03 I0510 05:30:03.521194 1 main.go:396] Waiting for all goroutines to exit

2022/5/10 13:30:03 I0510 05:30:03.521217 1 vxlan_network.go:60] watching for new subnet leases

其三：
I0510 05:30:05.099896 1 main.go:217] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[nm-k8s-pro] ifaceRegex: ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}

2022/5/10 13:30:05 W0510 05:30:05.101697 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

2022/5/10 13:30:05 I0510 05:30:05.190914 1 kube.go:120] Waiting 10m0s for node controller to sync

2022/5/10 13:30:05 I0510 05:30:05.209344 1 kube.go:378] Starting kube subnet manager

2022/5/10 13:30:06 I0510 05:30:06.210914 1 kube.go:127] Node controller sync successful

2022/5/10 13:30:06 I0510 05:30:06.210955 1 main.go:237] Created subnet manager: Kubernetes Subnet Manager - k8s-node4-hk

2022/5/10 13:30:06 I0510 05:30:06.210960 1 main.go:240] Installing signal handlers

2022/5/10 13:30:06 I0510 05:30:06.211132 1 main.go:459] Found network config - Backend type: vxlan

2022/5/10 13:30:06 I0510 05:30:06.212524 1 main.go:698] Using interface with name nm-k8s-pro and address 10.8.8.1

2022/5/10 13:30:06 I0510 05:30:06.212556 1 main.go:720] Defaulting external address to interface address (10.8.8.1)

2022/5/10 13:30:06 I0510 05:30:06.212560 1 main.go:733] Defaulting external v6 address to interface address ()

2022/5/10 13:30:06 I0510 05:30:06.212644 1 vxlan.go:137] VXLAN config: VNI=1 Port=8472 GBP=false Learning=false DirectRouting=false

2022/5/10 13:30:06 I0510 05:30:06.216571 1 kube.go:339] Setting NodeNetworkUnavailable

2022/5/10 13:30:06 I0510 05:30:06.233072 1 main.go:340] Setting up masking rules

2022/5/10 13:30:06 I0510 05:30:06.475157 1 main.go:361] Changing default FORWARD chain policy to ACCEPT

2022/5/10 13:30:06 I0510 05:30:06.475442 1 main.go:374] Wrote subnet file to /run/flannel/subnet.env

2022/5/10 13:30:06 I0510 05:30:06.475537 1 main.go:378] Running backend.

2022/5/10 13:30:06 I0510 05:30:06.475612 1 main.go:396] Waiting for all goroutines to exit

2022/5/10 13:30:06 I0510 05:30:06.475704 1 vxlan_network.go:60] watching for new subnet leases

4

flannal 配置：
bash-5.1# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.42.0.0/16
FLANNEL_SUBNET=10.42.0.1/24
FLANNEL_MTU=1230
FLANNEL_IPMASQ=true