环境信息:
K3s 版本:
v1.20.5+k3s1
节点 CPU 架构、操作系统和版本：:
Linux xx 5.11.6-1.el7.elrepo.x86_64 #1 SMP Thu Mar 11 08:51:58 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
集群配置:
1 master 14 worker
问题描述:
执行更新证书操作后k3s服务不断重启
复现步骤:
-
安装 K3s 的命令:
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh - -
更新证书操作步骤：
timedatectl set-ntp no
for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
date -s 20220306
kubeclt get nodes
kubectl --insecure-skip-tls-verify -n kube-system delete secrets k3s-serving
rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json
systemctl restart k3s
for i in `ls /var/lib/rancher/k3s/server/tls/*.crt`; do echo $i; openssl x509 -enddate -noout -in $i; done
timedatectl set-ntp yes
ntpdate ntp1.aliyun.com
date
systemctl status k3s
预期结果:
实际结果:
k3s服务启动失败且自动重启
附加上下文/日志:
日志
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.368744 242109 server.go:1177] Started kubelet
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.374167 242109 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.403704 242109 server.go:148] Starting to listen on 0.0.0.0:10250
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.440319 242109 volume_manager.go:271] Starting Kubelet Volume Manager
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.449449 242109 desired_state_of_world_populator.go:142] Desired state populator starts to run
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.471894 242109 scope.go:111] [topologymanager] RemoveContainer - Container ID: dfe1feeb9769a6a04ddba06e92b78b59d246515b5eaaed9c735079f2c69f4305
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.472050 242109 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint “/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs”: unable to find data in memory cache.
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.472072 242109 kubelet.go:1296] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.474415 242109 server.go:410] Adding debug handlers to kubelet server.
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.506543169+08:00” level=info msg=“Node CIDR assigned for: newcloudgx-k8smaster1”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.514310 242109 flannel.go:92] Determining IP address of default interface
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.529555 242109 kubelet_network_linux.go:56] Initialized IPv4 iptables rules.
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.529590 242109 status_manager.go:158] Starting to sync pod status with apiserver
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.529609 242109 kubelet.go:1833] Starting kubelet main sync loop.
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.529654 242109 kubelet.go:1857] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.541306 242109 kuberuntime_manager.go:1006] updating runtime config through cri with podcidr 10.42.0.0/24
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.552748548+08:00” level=info msg=“Handling backend connection request [newcloudgx05]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.554193707+08:00” level=info msg=“Handling backend connection request [newcloudgx02]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.555214049+08:00” level=info msg=“Handling backend connection request [newcloudgx07]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.557271132+08:00” level=info msg=“Handling backend connection request [newcloudgx01]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.566255298+08:00” level=info msg=“Handling backend connection request [newcloudgx06]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.567277339+08:00” level=info msg=“Handling backend connection request [newcloudgx08]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.568182276+08:00” level=info msg=“Handling backend connection request [newcloudgx09]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.569516330+08:00” level=info msg=“Handling backend connection request [newcloudgx04]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.570503270+08:00” level=info msg=“Handling backend connection request [newcloudgx10]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.571719120+08:00” level=info msg=“Handling backend connection request [newcloudgx12]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.575726783+08:00” level=info msg=“Handling backend connection request [newcloudgx13]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.576690222+08:00” level=info msg=“Handling backend connection request [newcloudgx14]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.588425 242109 flannel.go:105] Using interface with name eth0 and address 172.30.26.51
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.597877 242109 kube.go:117] Waiting 10m0s for node controller to sync
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.598023 242109 kube.go:300] Starting kube subnet manager
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.629683 242109 kubelet.go:1857] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.630686 242109 kubelet_network.go:77] Setting Pod CIDR: → 10.42.0.0/24
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.771788157+08:00” level=info msg=“Cluster-Http-Server 2022/04/23 15:40:40 http: TLS handshake error from 172.30.23.11:54410: EOF”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.771985 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.772229 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.814450 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:40.826719 242109 kubelet_node_status.go:71] Attempting to register node newcloudgx-k8smaster1
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.829724 242109 kubelet.go:1857] skipping pod synchronization - container runtime status check may not have completed yet
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.841750 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:40.972445317+08:00” level=info msg=“Handling backend connection request [newcloudgx03]”
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.976433 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.976652 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.976933 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.977122 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.977338 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.977591 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:40 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:40.980286 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:40+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.052483 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.052773 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.059712 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.233214 242109 kubelet.go:1857] skipping pod synchronization - container runtime status check may not have completed yet
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.330065 242109 node.go:161] Failed to retrieve node info: nodes “newcloudgx-k8smaster1” is forbidden: User “system:kube-proxy” cannot get resource “nodes” in API group “” at the cluster scope
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.367114 242109 trace.go:205] Trace[180034593]: “Get” url:/api/v1/nodes/newcloudgx-k8smaster1,user-agent:k3s/v1.20.5+k3s1 (linux/amd64) kubernetes/355fff3,client:127.0.0.1 (23-Apr-2022 15:40:40.550) (total time: 816ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[180034593]: —“About to write a response” 816ms (15:40:00.366)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[180034593]: [816.658614ms] [816.658614ms] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: time=“2022-04-23T15:40:41.445769767+08:00” level=info msg=“labels have already set on node: newcloudgx-k8smaster1”
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.504208 242109 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User “system:k3s-controller” cannot list resource “pods” in API group “” at the cluster scope
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.510147 242109 trace.go:205] Trace[541731064]: “Get” url:/apis/storage.k8s.io/v1/csinodes/newcloudgx-k8smaster1,user-agent:k3s/v1.20.5+k3s1 (linux/amd64) kubernetes/355fff3,client:127.0.0.1 (23-Apr-2022 15:40:40.551) (total time: 958ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[541731064]: —“About to write a response” 958ms (15:40:00.510)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[541731064]: [958.386477ms] [958.386477ms] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.510662 242109 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User “system:k3s-controller” cannot list resource “namespaces” in API group “” at the cluster scope
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.548654 242109 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User “system:k3s-controller” cannot list resource “networkpolicies” in API group “networking.k8s.io” at the cluster scope
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.583759 242109 trace.go:205] Trace[58591903]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx04,user-agent:k3s/v1.20.6+k3s1 (linux/amd64) kubernetes/8d04328,client:172.30.23.4 (23-Apr-2022 15:40:40.623) (total time: 960ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[58591903]: —“About to write a response” 960ms (15:40:00.583)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[58591903]: [960.15655ms] [960.15655ms] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.607274 242109 kube.go:124] Node controller sync successful
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.607366 242109 vxlan.go:121] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:41.712835 242109 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=1990804503106573970, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:40:41+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.714292 242109 trace.go:205] Trace[1255614433]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx06,user-agent:k3s/v1.20.6+k3s1 (linux/amd64) kubernetes/8d04328,client:172.30.23.6 (23-Apr-2022 15:40:40.631) (total time: 1082ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1255614433]: —“About to write a response” 1082ms (15:40:00.714)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1255614433]: [1.08285594s] [1.08285594s] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.714621 242109 trace.go:205] Trace[1686391243]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx-k8smaster1,user-agent:k3s/v1.20.5+k3s1 (linux/amd64) kubernetes/355fff3,client:127.0.0.1 (23-Apr-2022 15:40:40.631) (total time: 1082ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1686391243]: —“About to write a response” 1082ms (15:40:00.714)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1686391243]: [1.082899042s] [1.082899042s] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.714845 242109 trace.go:205] Trace[675565911]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx14,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.14 (23-Apr-2022 15:40:40.630) (total time: 1084ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[675565911]: —“About to write a response” 1084ms (15:40:00.714)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[675565911]: [1.084192294s] [1.084192294s] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.715006 242109 trace.go:205] Trace[1396079528]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx10,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.10 (23-Apr-2022 15:40:40.622) (total time: 1092ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1396079528]: —“About to write a response” 1092ms (15:40:00.714)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1396079528]: [1.092258922s] [1.092258922s] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.804995 242109 trace.go:205] Trace[1039436680]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx11,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.11 (23-Apr-2022 15:40:40.816) (total time: 988ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1039436680]: —“About to write a response” 988ms (15:40:00.804)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1039436680]: [988.67441ms] [988.67441ms] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.806187 242109 trace.go:205] Trace[1404261242]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx02,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.2 (23-Apr-2022 15:40:40.815) (total time: 990ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1404261242]: —“About to write a response” 990ms (15:40:00.806)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1404261242]: [990.762294ms] [990.762294ms] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.807590 242109 trace.go:205] Trace[1496054331]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx03,user-agent:k3s/v1.20.6+k3s1 (linux/amd64) kubernetes/8d04328,client:172.30.23.3 (23-Apr-2022 15:40:40.818) (total time: 989ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1496054331]: —“About to write a response” 989ms (15:40:00.807)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[1496054331]: [989.532045ms] [989.532045ms] END
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:41.807785 242109 trace.go:205] Trace[2019295902]: “Get” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx07,user-agent:k3s/v1.20.5+k3s1 (linux/amd64) kubernetes/355fff3,client:172.30.23.7 (23-Apr-2022 15:40:40.817) (total time: 990ms):
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[2019295902]: —“About to write a response” 990ms (15:40:00.807)
4月 23 15:40:41 newcloudgx-k8smaster1 k3s[242109]: Trace[2019295902]: [990.607388ms] [990.607388ms] END
4月 23 15:40:42 newcloudgx-k8smaster1 k3s[242109]: E0423 15:40:42.052404 242109 kubelet.go:1857] skipping pod synchronization - container runtime status check may not have completed yet
4月 23 15:40:42 newcloudgx-k8smaster1 k3s[242109]: I0423 15:40:42.052582 242109 trace.go:205] Trace[1213048090]: “Get” url:/apis/storage.k8s.io/v1/csinodes/newcloudgx-k8smaster1,user-agent:k3s/v1.20.5+k3s1 (linux/amd64) kubernetes/355fff3,client:127.0.0.1 (23-Apr-2022 15:40:41.548) (total time: 504ms):
4月 23 15:40:42 newcloudgx-k8smaster1 k3s[242109]: Trace[1213048090]: —“About to write a response” 504ms (15:40:00.052)
4月 23 15:40:42 newcloudgx-k8smaster1 k3s[242109]: Trace[1213048090]: [504.192205ms] [504.192205ms] END
…
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: E0423 15:53:18.245604 2721 customresource_handler.go:669] error building openapi models for thanosrulers.monitoring.coreos.com: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.lifecycle.properties.postStart.properties.httpGet.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.lifecycle.properties.postStart.properties.tcpSocket.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.lifecycle.properties.preStop.properties.httpGet.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.lifecycle.properties.preStop.properties.tcpSocket.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.livenessProbe.properties.httpGet.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.livenessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.readinessProbe.properties.httpGet.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: ERROR $root.definitions.com.coreos.monitoring.v1.ThanosRuler.properties.spec.properties.containers.items..properties.readinessProbe.properties.tcpSocket.properties.port has invalid property: anyOf
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.453505 2721 trace.go:205] Trace[198255426]: “Get” url:/api/v1/namespaces/cattle-system/pods/cattle-cluster-agent-6c4fdc8f4b-xb8qq,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.14 (23-Apr-2022 15:53:16.006) (total time: 2446ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[198255426]: —“About to write a response” 2446ms (15:53:00.453)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[198255426]: [2.446591338s] [2.446591338s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.536661 2721 trace.go:205] Trace[699226133]: “GuaranteedUpdate etcd3” type:*coordination.Lease (23-Apr-2022 15:53:13.302) (total time: 5233ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[699226133]: —“Transaction prepared” 3524ms (15:53:00.826)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[699226133]: —“Transaction committed” 1709ms (15:53:00.536)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[699226133]: [5.233831953s] [5.233831953s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.536872 2721 trace.go:205] Trace[1266186514]: “Update” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx01,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.1 (23-Apr-2022 15:53:13.302) (total time: 5234ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1266186514]: —“Object stored in database” 5233ms (15:53:00.536)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1266186514]: [5.234199869s] [5.234199869s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.580783 2721 trace.go:205] Trace[1541653532]: “List etcd3” key:/cronjobs,resourceVersion:,resourceVersionMatch:,limit:500,continue: (23-Apr-2022 15:53:09.009) (total time: 9570ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1541653532]: [9.5709133s] [9.5709133s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.580883 2721 trace.go:205] Trace[454988642]: “List” url:/apis/batch/v1beta1/cronjobs,user-agent:kube-state-metrics/v1.9.7 (linux/amd64) kube-state-metrics/b3fa5852,client:172.30.23.10 (23-Apr-2022 15:53:09.009) (total time: 9571ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[454988642]: —“Listing from storage done” 9570ms (15:53:00.580)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[454988642]: [9.571041805s] [9.571041805s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.826365 2721 trace.go:205] Trace[2066812351]: “List etcd3” key:/volumeattachments,resourceVersion:,resourceVersionMatch:,limit:500,continue: (23-Apr-2022 15:53:10.902) (total time: 7923ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[2066812351]: [7.923343246s] [7.923343246s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.826463 2721 trace.go:205] Trace[196747006]: “List” url:/apis/storage.k8s.io/v1/volumeattachments,user-agent:kube-state-metrics/v1.9.7 (linux/amd64) kube-state-metrics/b3fa5852,client:172.30.23.10 (23-Apr-2022 15:53:10.902) (total time: 7923ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[196747006]: —“Listing from storage done” 7923ms (15:53:00.826)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[196747006]: [7.923467051s] [7.923467051s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: E0423 15:53:18.879104 2721 authentication.go:53] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid: current time 2022-04-23T15:53:18+08:00 is after 2022-04-06T13:44:44Z, verifying certificate SN=2383589142424457143, SKID=, AKID=1F:AF:49:49:4C:68:73:D1:17:16:FE:6D:EA:E8:3D:BE:BD:58:02:F0 failed: x509: certificate has expired or is not yet valid: current time 2022-04-23T15:53:18+08:00 is after 2022-04-06T13:44:44Z]
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.888501 2721 trace.go:205] Trace[1420323369]: “GuaranteedUpdate etcd3” type:*coordination.Lease (23-Apr-2022 15:53:13.722) (total time: 5165ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1420323369]: —“Transaction prepared” 3037ms (15:53:00.760)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1420323369]: —“Transaction committed” 2128ms (15:53:00.888)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1420323369]: [5.165553312s] [5.165553312s] END
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: I0423 15:53:18.888685 2721 trace.go:205] Trace[1089135028]: “Update” url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/newcloudgx12,user-agent:k3s/v1.21.1+k3s1 (linux/amd64) kubernetes/75dba57,client:172.30.23.12 (23-Apr-2022 15:53:13.722) (total time: 5165ms):
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1089135028]: —“Object stored in database” 5165ms (15:53:00.888)
4月 23 15:53:18 newcloudgx-k8smaster1 k3s[2721]: Trace[1089135028]: [5.165847424s] [5.165847424s] END
