Rancher2.6.3一直重启

Rancher 2.x
1

Rancher Server 设置

  • Rancher 版本：rancher2.6.3
  • 安装选项 (Docker install/Helm Chart): docker install
    • 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
  • 在线或离线部署：在线

主机操作系统： centos 7.9

问题描述：
rancher2.6.3容器一直重启

重现步骤：
不清楚怎么重现
结果：

预期结果：

截图：

其他上下文信息：

日志 
2022/08/24 07:38:49 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:38:49 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:38:49 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:38:50 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-06-30 02:50:51 +0000 UTC notAfter=2023-0C
2022/08/24 07:38:50 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:38:50 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-06-30 02:50:51 +0000 UTC notAfter=2023-0C
2022/08/24 07:38:50 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-06-30 02:50:51 +0000 UTC notAfter=2023-0C
2022/08/24 07:38:50 [INFO] Updating TLS secret for tls-rancher-internal (count: 2): map[listener.cattle.io/cn-10.43.133.95:10.43.133.95 listener.cattle.io/cn-172.17.0.2:1]
2022/08/24 07:38:50 [INFO] Active TLS secret tls-rancher-internal (ver=10865) (count 2): map[listener.cattle.io/cn-10.43.133.95:10.43.133.95 listener.cattle.io/cn-172.17.]
2022/08/24 07:39:12 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:12 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:13 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:13 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:13 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:14 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:14 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:15 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:25 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:25 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:26 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:26 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:27 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:29 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting for agent to check in and apply initial plan
2022/08/24 07:39:34 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting on probes: calico, etcd, kube-apiserver, kube-contrt
2022/08/24 07:39:39 httputil: ReverseProxy read error during body copy: unexpected EOF
2022/08/24 07:39:39 httputil: ReverseProxy read error during body copy: unexpected EOF
2022/08/24 07:39:39 [ERROR] Error fetching user attribute to trigger refresh: Get "https://127.0.0.1:6443/apis/management.cattle.io/v3/userattributes/u-6zfepgiryq": dial d
2022/08/24 07:39:39 [ERROR] Reconciler error: failed to retrieve CustomMachine external object "fleet-default"/"custom-ec3eccdf2d6b": Get "https://127.0.0.1:6443/apis/rked
2022/08/24 07:39:39 [ERROR] failed to check connectivity of cluster [c-m-l4fz9wxh]
E0824 07:39:39.578264     144 leaderelection.go:367] Failed to update lock: Put "https://127.0.0.1:6443/api/v1/namespaces/kube-system/configmaps/cattle-controllers?timeouF
2022/08/24 07:39:39 [ERROR] error syncing 'fleet-default/cloud': handler planner: Put "https://127.0.0.1:6443/apis/rke.cattle.io/v1/namespaces/fleet-default/rkecontrolplag
2022/08/24 07:39:39 [INFO] rkecluster fleet-default/cloud: provisioning bootstrap node(s) custom-ec3eccdf2d6b: waiting on probes: calico, etcd, kube-apiserver, kube-contrt
2022/08/24 07:39:39 [ERROR] error syncing 'cluster/fleet-default/cloud': handler auth-prov-v2-roletemplate: failed to update fleet-default/crt-cloud-nodes-view rbac.authog
2022/08/24 07:39:39 [ERROR] error syncing 'fleet-default/cloud': handler planner: Put "https://127.0.0.1:6443/apis/rke.cattle.io/v1/namespaces/fleet-default/rkecontrolplag
2022/08/24 07:39:39 [ERROR] Error fetching user attribute to trigger refresh: Get "https://127.0.0.1:6443/apis/management.cattle.io/v3/userattributes/u-6zfepgiryq": dial d
2022/08/24 07:39:39 [FATAL] k3s exited with: exit status 255
2

exec 到 rancher 容器里，查看 K3s 的日志 /var/lib/rancher/k3s.log

3

k3s.log
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:736 +0x46
github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc009ec7440, 0xc0079fde58, 0xc0079fde28, 0x203002, 0xc00344ad80)
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:590 +0x3df
github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc009ec7440, 0x4a7c020, 0xc00b423a90, 0x50, 0x49f44c0)
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/rancher/k3s/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredClientStream).RecvMsg(0xc0009fa390, 0x4a7c020, 0xc00b423a90, 0x2, 0x2)
/go/src/github.com/rancher/k3s/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client_metrics.go:160 +0x4b
github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc00992bf00, 0xc0079fdf50, 0x0, 0x1)
/go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3704 +0x62
github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc00761b380, 0x54eec20, 0xc00992bf00)
/go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:764 +0x5c
created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).newWatchClient
/go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:919 +0x556

goroutine 4303 [select, 5 minutes]:
github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream.func5(0xc006d7c700, 0xc009ec7440, 0x54bf1a0, 0xc00a572330)
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:319 +0xaa
created by github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:318 +0xbf0

goroutine 4304 [select, 5 minutes]:
context.propagateCancel.func1(0x54c06a0, 0xc00992bf50, 0x546e810, 0xc0098c7340)
/usr/local/go/src/context/context.go:279 +0xab
created by context.propagateCancel
/usr/local/go/src/context/context.go:278 +0x1fb

goroutine 4305 [select, 5 minutes]:
github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).run(0xc00761b520)
/go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:530 +0x3c5
created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream
/go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:283 +0x2ff

goroutine 4372 [select, 5 minutes]:
github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00a76c3c0, 0xc0009fa4f0, 0x5, 0x5, 0x25, 0x63f, 0x23)
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:186 +0xb1
github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00a76c3c0, 0xc0009fa4f0, 0x5, 0x5, 0xc006b0b9e0, 0xc00a2a68d0, 0x4e929)
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:166 +0x190
github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00a5726c0, 0xc0009fa4f0, 0x5, 0x5, 0x28, 0xc006b0b9b8, 0xa2185c)
/go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x55

4

这个日志是最终的日志？ 没看出来啥原因导致 K3s 退出了。

最好找到导致 K3s 退出的日志，或者你可以在支持矩阵比对下安装要求：Support matrix | SUSE

5

导致k3s退出的日志有什么方法可以查看吗