在server节点使用kubectl logs无法查看agent节点pod的日志

Rancher Server 设置

  • Rancher 版本:v1.30.7+rke2r1

  • 在线或离线部署:在线

**主机操作系统:Linux server1 5.15.0-126-generic #136-Ubuntu SMP Wed Nov 6 10:38:22 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

问题描述:
kubectl logs rancher-xxx-xxx -n cattle-system
Error from server: Get “https://agent节点IP:10250/containerLogs/cattle-system/rancher-xxx-xxx/rancher”: proxy error from 127.0.0.1:9345 while dialing agent节点IP:10250, code 502: 502 Bad Gateway

查看agent节点上的pod日志错误,这个如何处理

重现步骤:
kubectl logs rancher-xxx-xxx -n cattle-system
结果:
Error from server: Get “https://agent节点IP:10250/containerLogs/cattle-system/rancher-xxx-xxx/rancher”: proxy error from 127.0.0.1:9345 while dialing agent节点IP:10250, code 502: 502 Bad Gateway
预期结果:
能查看到agent 节点中pod的日志
截图:

其他上下文信息:

日志

journalctl -u rke2-server -f
Mar 22 09:08:40 server1 rke2[1004]: time=“2025-03-22T09:08:40+08:00” level=info msg=“certificate CN=ecs-xx signed by CN=rke2-server-ca@1734082336: notBefore=2024-12-13 09:32:16 +0000 UTC notAfter=2026-03-22 01:08:40 +0000 UTC”
Mar 22 09:08:46 server1 rke2[1004]: time=“2025-03-22T09:08:46+08:00” level=info msg=“Handling backend connection request [ecs-xx]”
Mar 22 09:51:48 server1 rke2[1004]: time=“2025-03-22T09:51:48+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:29946: failed to find Session for client agent6”
Mar 22 09:53:51 server1 rke2[1004]: time=“2025-03-22T09:53:51+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:50446: failed to find Session for client agent6”
Mar 22 10:01:01 server1 rke2[1004]: time=“2025-03-22T10:01:01+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:9646: failed to find Session for client agent6”
Mar 22 10:04:45 server1 rke2[1004]: time=“2025-03-22T10:04:45+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:12678: failed to find Session for client agent6”
Mar 22 10:04:57 server1 rke2[1004]: time=“2025-03-22T10:04:57+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:52286: failed to find Session for client agent6”
Mar 22 10:10:57 server1 rke2[1004]: time=“2025-03-22T10:10:57+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:50438: failed to find Session for client agent6”
Mar 22 10:11:20 server1 rke2[1004]: time=“2025-03-22T10:11:20+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:45394: failed to find Session for client agent6”
Mar 22 10:11:21 server1 rke2[1004]: time=“2025-03-22T10:11:21+08:00” level=error msg=“Sending HTTP 502 response to 127.0.0.1:45398: failed to find Session for client agent6”

估计是权限问题,我访问用户命名空间pod日志正常,但访问其他命名空间pod日志存在问题。

我执行systemctl restart rke2-agent就好了?

已经好了?

重新启动只是临时好了,我查看了lsof值,1584892 calico-no,然后我调整了ulimit值,现在可以查看日志了,但calico-no过于大了,你们的calico-no值有这么大吗?