Rancher Server 设置
- Rancher 版本:v2.3.11 / v2.5.15
- 安装选项 (Docker install/Helm Chart): 单节点安装
- 在线或离线部署:在线
下游集群信息
- Kubernetes 版本: 1.19
- Cluster Type (Local/Downstream):
- 如果 Downstream,是什么类型的集群?(自定义/导入或为托管 等):
用户信息
- 登录用户的角色是什么? (管理员/集群所有者/集群成员/项目所有者/项目成员/自定义):
- 如果自定义,自定义权限集:
主机操作系统:
问题描述:
最近在使用应用商店(个人搭建的私人应用仓库,helm2)的时候发现了一个问题,最开始是创建应用的时候发现创建和更新配置很慢。之后出现应用无法创建的情况,具体为:应用商店界面显示应用已经存在,但是应用相关的资源(ingress/deployment/secret/configmap等资源)都没有创建。
之后排查了一下发现运行rancher的容器中出现大量僵尸进程:[git-remote-http]
推测应该是僵尸进程过多占用pid资源,导致新的应用无法创建,于是重启后,应用成功创建。
但是依旧会出现每创建一个应用,就会新增几个僵尸进程的情况。
另外发现使用的时候会调用chartmuseum的接口/info/refs,但是会返回404错误,并且计算了下404数量与僵尸进程新增数量基本相符合
helmchart使用镜像:chartmuseum/chartmuseum:v0.8.2
想问一下具体该怎么解决这个问题?
重现步骤:
结果:
预期结果:
rancher容器ps aux
root 2837 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2840 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2843 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2846 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2849 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2852 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2855 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2858 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
root 2861 0.0 0.0 0 0 ? Z 08:52 0:00 [git-remote-http]
截图:
其他上下文信息:
日志
chartmuseum日志
{"L":"INFO","T":"2022-08-23T09:51:27.670Z","M":"[21449393] Request served","path":"/index.yaml","comment":"","latency":"2.778513ms","clientIP":"10.248.142.97","method":"GET","statusCode":200,"reqID":"27bd69980a59930ffc0eb4c85f6f5611"}
{"L":"WARN","T":"2022-08-23T09:51:27.691Z","M":"[21449394] Request served","path":"/info/refs","comment":"","latency":"31.703µs","clientIP":"10.248.138.33","method":"GET","statusCode":404,"reqID":"500a3e3075bcd6d421d8df9c398d57c4"}
{"L":"INFO","T":"2022-08-23T09:51:27.698Z","M":"[21449395] Request served","path":"/index.yaml","comment":"","latency":"2.782421ms","clientIP":"10.248.142.97","method":"GET","statusCode":200,"reqID":"91df3d304014f471ab9de084c2c42666"}
{"L":"WARN","T":"2022-08-23T09:51:27.716Z","M":"[21449396] Request served","path":"/info/refs","comment":"","latency":"31.521µs","clientIP":"10.248.137.161","method":"GET","statusCode":404,"reqID":"da7ebbfe2551f6f6ff3e774eaea3e201"}
{"L":"INFO","T":"2022-08-23T09:51:27.722Z","M":"[21449397] Request served","path":"/index.yaml","comment":"","latency":"2.728031ms","clientIP":"10.248.142.97","method":"GET","statusCode":200,"reqID":"ba73590f5523a5a446114671f80e2324"}
{"L":"WARN","T":"2022-08-23T09:53:46.187Z","M":"[21449426] Request served","path":"/info/refs","comment":"","latency":"49.47µs","clientIP":"10.248.142.241","method":"GET","statusCode":404,"reqID":"3a76c810d28958a528defc8e091d5cff"}
{"L":"INFO","T":"2022-08-23T09:53:46.269Z","M":"[21449427] Request served","path":"/index.yaml","comment":"","latency":"75.357594ms","clientIP":"10.248.139.113","method":"GET","statusCode":200,"reqID":"b1e0dc0a83a6e96eea053a0556fdfacf"}
{"L":"WARN","T":"2022-08-23T09:53:46.395Z","M":"[21449428] Request served","path":"/info/refs","comment":"","latency":"40.504µs","clientIP":"10.248.137.161","method":"GET","statusCode":404,"reqID":"78095a51e3c1d18379dc5b7e074a76ed"}
{"L":"INFO","T":"2022-08-23T09:53:46.411Z","M":"[21449429] Request served","path":"/index.yaml","comment":"","latency":"11.96207ms","clientIP":"10.248.139.113","method":"GET","statusCode":200,"reqID":"e0df1839d767cba7d79d4916f4355a82"}
{"L":"WARN","T":"2022-08-23T09:53:46.435Z","M":"[21449430] Request served","path":"/info/refs","comment":"","latency":"52.439µs","clientIP":"10.248.140.225","method":"GET","statusCode":404,"reqID":"80df0329c9c4fe2f2f6ef9dccaafdf52"}
{"L":"INFO","T":"2022-08-23T09:53:46.442Z","M":"[21449431] Request served","path":"/index.yaml","comment":"","latency":"2.819727ms","clientIP":"10.248.139.113","method":"GET","statusCode":200,"reqID":"11a41836a66dab420aee5224fa064091"}
{"L":"WARN","T":"2022-08-23T09:53:46.465Z","M":"[21449432] Request served","path":"/info/refs","comment":"","latency":"40.424µs","clientIP":"10.248.141.65","method":"GET","statusCode":404,"reqID":"851380681019f31a5c37c92cb5c9a33e"}
{"L":"INFO","T":"2022-08-23T09:53:46.471Z","M":"[21449433] Request served","path":"/index.yaml","comment":"","latency":"2.925226ms","clientIP":"10.248.139.113","method":"GET","statusCode":200,"reqID":"a6818cbc8e7050f9259b084c90ad8341"}
{"L":"WARN","T":"2022-08-23T09:53:46.489Z","M":"[21449434] Request served","path":"/info/refs","comment":"","latency":"38.57µs","clientIP":"10.248.141.81","method":"GET","statusCode":404,"reqID":"fab669a0c3d089e65da98892ccd214bb"}
{"L":"INFO","T":"2022-08-23T09:53:46.495Z","M":"[21449435] Request served","path":"/index.yaml","comment":"","latency":"2.816889ms","clientIP":"10.248.139.113","method":"GET","statusCode":200,"reqID":"6729a8e9c626959ab02b7ec8bd28af00"}