K8S的1.24.0以上kube-proxy不监听nodeport端口问题

victory · 2024 年5 月 8 日 03:12

RKE2 版本: 2.8 操作系统和版本：Rocky 9 集群配置:v1.28.8+rke2r1 kube-proxy 模式是IPVS,

Service 创建了一个 nodeport ，外部业务能正常访问，但宿主机上使用losf和netstat 都查不到这个nodeport端口在监听?（使用nmap扫描宿主机端口是开启的）

github.com/kubernetes/kubernetes

kube-proxy: remove port opener

kubernetes:master ← khenidak:remove-port-opener

opened 06:05PM - 03 Mar 22 UTC

khenidak

+3 -209

#### What type of PR is this? /kind bug /kind cleanup #### What this PR doe…s / why we need it: kube-proxy holds service node ports open (`Listen()` without `Read()/Receive()` pump). The original idea was to stop users from mistakenly create a listener on the node that listens to a node port which would have created debugging problems. It is important to note that if the opener failed we just log and keep going. However port-opener introduced more problems than value. The problems can be be briefly described as the following: 1. We open the port before we commit the forwarding/LB rules, the rules typically take much longer to process before committing. That means if the user connected before rules were set they will indeed connect to kube-proxy not the actual pod backing the service. This usually require restarting the client (be it pod or external proce) or the proxy itself. This is more visible as more nodes get introduced to existing clusters with large number of service (the time between open-port and fwd/lb rules is longer). 2. Port opener also leaves many dangling tcp connections due to this behavior as described here: https://github.com/kubernetes/kubernetes/issues/106713 further discussions can be found here: https://github.com/kubernetes/kubernetes/issues/100643 > a script or a stand alone tool can be built (runs on node) to scan services+node-ports and performs a bind(2) on address for debugging purposes. But the scan loop will not be part of kube-proxy. #### Which issue(s) this PR fixes: Fixes # https://github.com/kubernetes/kubernetes/issues/100643 https://github.com/kubernetes/kubernetes/issues/106713 #### Special notes for your reviewer: N/A #### Does this PR introduce a user-facing change? ```release-note kube-proxy will no longer hold service node ports open on the node. Users are still advised not to run any listener on node ports range used by kube-proxy. ``` #### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: ```docs N/A ``` /sig network

github的回答显示，在新版本中已经删除了kube-proxy打开端口套接字部分，但是会在iptables中进行转发，故不影响访问。

service配置
![image|628x209](upload://1[root@Cluster01-k8s-master-01 ~]# kubectl get svc |grep netty

netty NodePort 192.168.10.169 10012:10012/TCP 11d
[root@Cluster01-k8s-master-01 ~]# iptables -S -t nat | grep 10012

Warning: iptables-legacy tables present, use iptables-legacy to see them

[root@Cluster01-k8s-master-01 ~]#
OINyqTjxtyfSbsYSrQo9zFm6QV.png)

[root@Cluster01-k8s-worker-02 ~]# iptables -t nat -nvL OUTPUT

Warning: iptables-legacy tables present, use iptables-legacy to see them

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
5256K 351M cali-OUTPUT all – * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 /
5256K 351M KUBE-SERVICES all – * * 0.0.0.0/0 0.0.0.0/0 / kubernetes service portals */
1351K 81M CNI-HOSTPORT-DNAT all – * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL

是这一条吗？
KUBE-SERVICES all – * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */