Rancher Server 设置
Rancher 版本：2.7.9
安装选项 (Docker install/Helm Chart): Helm Chart
k3s版本：v1.26.9+k3s1
在线或离线部署：离线部署
**主机操作系统：三台ubtuntu22.04.6，两台server，一台agent，都已关闭防火墙
问题描述:
长期未关注进来发现无法进入 rancher的web界面
执行：kubectl logs -f rancher-webhook-7dc5857799-pttpr -n cattle-system --since=1h
查看发现错误如下：
failed to sync schemas: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
发现错误如下：
Failed to read API for groups map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
W0326 01:25:05.584424 33 warnings.go:80] metallb.io v1alpha1 AddressPool is deprecated
E0326 01:25:09.702902 33 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
复现步骤:
- 安装 K3s 的命令:
预期结果:
可以进入rancher的web页面
实际结果:
无法进入web页面
附加上下文/日志:
日志
执行：kubectl logs -f rancher-6b6b974475-cj7jd -n cattle-system --since=1h
k3s日志如下：
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: I0326 09:25:49.325417 3618643 trace.go:236] Trace[820661032]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:39.462) (total time: 129862ms):
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: Trace[820661032]: [2m9.862672246s] [2m9.862672246s] END
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: I0326 09:25:49.325451 3618643 trace.go:236] Trace[872936332]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:39.465) (total time: 129860ms):
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: Trace[872936332]: [2m9.860352557s] [2m9.860352557s] END
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: I0326 09:25:49.325501 3618643 trace.go:236] Trace[1818664435]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:39.462) (total time: 129862ms):
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: Trace[1818664435]: [2m9.862777022s] [2m9.862777022s] END
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: I0326 09:25:49.325555 3618643 trace.go:236] Trace[1467372908]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:39.462) (total time: 129862ms):
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: Trace[1467372908]: [2m9.862705516s] [2m9.862705516s] END
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: I0326 09:25:49.325603 3618643 trace.go:236] Trace[1612127707]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:39.462) (total time: 129862ms):
Mar 26 09:25:49 k3s-agent2 k3s[3618643]: Trace[1612127707]: [2m9.862872094s] [2m9.862872094s] END
Mar 26 09:25:51 k3s-agent2 systemd[3917928]: run-containerd-runc-k8s.io-f43ded8d76dd5b3be9a9e332b8f9aa607e6fe6efeeb46262b5bac82351b232f6-runc.DFAKJG.mount: Succeeded.
Mar 26 09:25:51 k3s-agent2 systemd[1]: run-containerd-runc-k8s.io-f43ded8d76dd5b3be9a9e332b8f9aa607e6fe6efeeb46262b5bac82351b232f6-runc.DFAKJG.mount: Succeeded.
Mar 26 09:25:53 k3s-agent2 k3s[3618643]: E0326 09:25:53.973528 3618643 available_controller.go:456] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.1.205:10250/apis/metrics.k8s.io/v1beta1: Get “https://10.42.1.205:10250/apis/metrics.k8s.io/v1beta1”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: W0326 09:25:54.976870 3618643 handler_proxy.go:100] no RequestInfo found in the context
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: E0326 09:25:54.976909 3618643 controller.go:113] loading OpenAPI spec for “v1beta1.metrics.k8s.io” failed with: Error, could not get list of group versions for APIService
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: I0326 09:25:54.976920 3618643 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: W0326 09:25:54.977773 3618643 handler_proxy.go:100] no RequestInfo found in the context
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: E0326 09:25:54.977955 3618643 controller.go:116] loading OpenAPI spec for “v1beta1.metrics.k8s.io” failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Mar 26 09:25:54 k3s-agent2 k3s[3618643]: I0326 09:25:54.978123 3618643 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: I0326 09:25:56.084703 3618643 trace.go:236] Trace[442220912]: “Update” accept:application/vnd.kubernetes.protobuf,application/json,audit-id:97437460-1ebf-40fb-9b53-fc0547344760,client:127.0.0.1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k3s-agent2,user-agent:k3s/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728,verb:PUT (26-Mar-2024 09:25:55.514) (total time: 570ms):
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: Trace[442220912]: [“GuaranteedUpdate etcd3” audit-id:97437460-1ebf-40fb-9b53-fc0547344760,key:/leases/kube-node-lease/k3s-agent2,type:*coordination.Lease,resource:leases.coordination.k8s.io 570ms (09:25:55.514)
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: Trace[442220912]: —“Txn call completed” 569ms (09:25:56.084)]
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: Trace[442220912]: [570.105861ms] [570.105861ms] END
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: I0326 09:25:56.086095 3618643 trace.go:236] Trace[219366074]: “Update” accept:application/vnd.kubernetes.protobuf, /,audit-id:85f34305-a200-4cdc-80e3-ebb82b03ae1c,client:127.0.0.1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-apiserver-ddcyb2xyq7mzvvrxbvqfwtrkca,user-agent:k3s/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728,verb:PUT (26-Mar-2024 09:25:55.548) (total time: 537ms):
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: Trace[219366074]: [“GuaranteedUpdate etcd3” audit-id:85f34305-a200-4cdc-80e3-ebb82b03ae1c,key:/leases/kube-system/kube-apiserver-ddcyb2xyq7mzvvrxbvqfwtrkca,type:*coordination.Lease,resource:leases.coordination.k8s.io 537ms (09:25:55.548)
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: Trace[219366074]: —“Txn call completed” 536ms (09:25:56.085)]
Mar 26 09:25:56 k3s-agent2 k3s[3618643]: Trace[219366074]: [537.961163ms] [537.961163ms] END
Mar 26 09:25:56 k3s-agent2 systemd[3917928]: run-containerd-runc-k8s.io-f43ded8d76dd5b3be9a9e332b8f9aa607e6fe6efeeb46262b5bac82351b232f6-runc.PmIiiG.mount: Succeeded.
Mar 26 09:25:56 k3s-agent2 systemd[1]: run-containerd-runc-k8s.io-f43ded8d76dd5b3be9a9e332b8f9aa607e6fe6efeeb46262b5bac82351b232f6-runc.PmIiiG.mount: Succeeded.
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: time=“2024-03-26T09:25:57+08:00” level=info msg=“Slow SQL (started: 2024-03-26 09:25:56.330526904 +0800 CST m=+30193.852891040) (total time: 1.127471406s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value) values(?, ?, ?, ?, ?, ?, ?, ?) : [[/registry/configmaps/kube-system/k3s 0 0 215 55246343 0 [107 56 115 0 10 15 10 2 118 49 18 9 67 111 110 102 105 103 77 97 112 18 146 4 10 143 4 10 3 107 51 115 18 0 26 11 107 117 98 101 45 115 121 115 116 101 109 34 0 42 36 101 56 56 98 48 53 53 53 45 99 52 100 102 45 52 57 97 102 45 57 100 55 97 45 101 56 48 54 49 51 99 98 99 57 55 98 50 0 56 0 66 8 8 227 198 237 172 6 16 0 98 196 1 10 40 99 111 110 116 114 111 108 45 112 108 97 110 101 46 97 108 112 104 97 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 101 97 100 101 114 18 151 1 123 34 104 111 108 100 101 114 73 100 101 110 116 105 116 121 34 58 34 107 51 115 45 97 103 101 110 116 50 34 44 34 108 101 97 115 101 68 117 114 97 116 105 111 110 83 101 99 111 110 100 115 34 58 52 53 44 34 97 99 113 117 105 114 101 84 105 109 101 34 58 34 50 48 50 52 45 48 49 45 51 48 84 48 53 58 49 55 58 52 50 90 34 44 34 114 101 110 101 119 84 105 109 101 34 58 34 50 48 50 52 45 48 51 45 50 54 84 48 49 58 50 53 58 53 54 90 34 44 34 108 101 97 100 101 114 84 114 97 110 115 105 116 105 111 110 115 34 58 53 125 138 1 100 10 25 107 51 115 45 115 117 112 101 114 118 105 115 111 114 64 107 51 115 45 109 97 115 116 101 114 18 6 85 112 100 97 116 101 26 2 118 49 34 8 8 193 140 226 173 6 16 0 50 8 70 105 101 108 100 115 86 49 58 37 10 35 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 125 125 125 66 0 138 1 147 1 10 25 107 51 115 45 115 117 112 101 114 118 105 115 111 114 64 107 51 115 45 97 103 101 110 116 50 18 6 85 112 100 97 116 101 26 2 118 49 34 8 8 164 200 136 176 6 16 0 50 8 70 105 101 108 100 115 86 49 58 84 10 82 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 102 58 99 111 110 116 114 111 108 45 112 108 97 110 101 46 97 108 112 104 97 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 101 97 100 101 114 34 58 123 125 125 125 125 66 0 26 0 34 0] [107 56 115 0 10 15 10 2 118 49 18 9 67 111 110 102 105 103 77 97 112 18 146 4 10 143 4 10 3 107 51 115 18 0 26 11 107 117 98 101 45 115 121 115 116 101 109 34 0 42 36 101 56 56 98 48 53 53 53 45 99 52 100 102 45 52 57 97 102 45 57 100 55 97 45 101 56 48 54 49 51 99 98 99 57 55 98 50 0 56 0 66 8 8 227 198 237 172 6 16 0 98 196 1 10 40 99 111 110 116 114 111 108 45 112 108 97 110 101 46 97 108 112 104 97 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 101 97 100 101 114 18 151 1 123 34 104 111 108 100 101 114 73 100 101 110 116 105 116 121 34 58 34 107 51 115 45 97 103 101 110 116 50 34 44 34 108 101 97 115 101 68 117 114 97 116 105 111 110 83 101 99 111 110 100 115 34 58 52 53 44 34 97 99 113 117 105 114 101 84 105 109 101 34 58 34 50 48 50 52 45 48 49 45 51 48 84 48 53 58 49 55 58 52 50 90 34 44 34 114 101 110 101 119 84 105 109 101 34 58 34 50 48 50 52 45 48 51 45 50 54 84 48 49 58 50 53 58 53 52 90 34 44 34 108 101 97 100 101 114 84 114 97 110 115 105 116 105 111 110 115 34 58 53 125 138 1 100 10 25 107 51 115 45 115 117 112 101 114 118 105 115 111 114 64 107 51 115 45 109 97 115 116 101 114 18 6 85 112 100 97 116 101 26 2 118 49 34 8 8 193 140 226 173 6 16 0 50 8 70 105 101 108 100 115 86 49 58 37 10 35 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 125 125 125 66 0 138 1 147 1 10 25 107 51 115 45 115 117 112 101 114 118 105 115 111 114 64 107 51 115 45 97 103 101 110 116 50 18 6 85 112 100 97 116 101 26 2 118 49 34 8 8 162 200 136 176 6 16 0 50 8 70 105 101 108 100 115 86 49 58 84 10 82 123 34 102 58 109 101 116 97 100 97 116 97 34 58 123 34 102 58 97 110 110 111 116 97 116 105 111 110 115 34 58 123 34 102 58 99 111 110 116 114 111 108 45 112 108 97 110 101 46 97 108 112 104 97 46 107 117 98 101 114 110 101 116 101 115 46 105 111 47 108 101 97 100 101 114 34 58 123 125 125 125 125 66 0 26 0 34 0]]]”
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: I0326 09:25:57.459478 3618643 trace.go:236] Trace[1940459196]: “Update” accept:application/json, /,audit-id:04fdb77f-8fc3-4cac-a562-cf9ec93c56c7,client:127.0.0.1,protocol:HTTP/2.0,resource:configmaps,scope:resource,url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:k3s-supervisor@k3s-agent2/v1.26.9+k3s1 (linux/amd64) k3s/4e217286,verb:PUT (26-Mar-2024 09:25:56.262) (total time: 1197ms):
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: Trace[1940459196]: [“GuaranteedUpdate etcd3” audit-id:04fdb77f-8fc3-4cac-a562-cf9ec93c56c7,key:/configmaps/kube-system/k3s,type:*core.ConfigMap,resource:configmaps 1197ms (09:25:56.262)
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: Trace[1940459196]: —“Txn call completed” 1196ms (09:25:57.459)]
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: Trace[1940459196]: [1.197308905s] [1.197308905s] END
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: time=“2024-03-26T09:25:57+08:00” level=info msg=“Slow SQL (started: 2024-03-26 09:25:56.417840519 +0800 CST m=+30193.940204661) (total time: 1.050409744s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = ‘compact_rev_key’), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC : [[/registry/leases/kube-system/kube-controller-manager false]]”
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: I0326 09:25:57.468666 3618643 trace.go:236] Trace[1852146819]: “Get” accept:application/vnd.kubernetes.protobuf, /,audit-id:d6f42cfb-9df4-4bae-b340-8c358cb3a433,client:127.0.0.1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:k3s/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728/leader-election,verb:GET (26-Mar-2024 09:25:56.417) (total time: 1051ms):
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: Trace[1852146819]: —“About to write a response” 1050ms (09:25:57.468)
Mar 26 09:25:57 k3s-agent2 k3s[3618643]: Trace[1852146819]: [1.051052382s] [1.051052382s] END
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: I0326 09:25:59.545483 3618643 trace.go:236] Trace[774911169]: “Update” accept:application/vnd.kubernetes.protobuf, /,audit-id:d51d665e-c23f-444f-813e-8806927f519f,client:127.0.0.1,protocol:HTTP/2.0,resource:apiservices,scope:resource,url:/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io/status,user-agent:k3s/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728,verb:PUT (26-Mar-2024 09:25:58.975) (total time: 570ms):
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[774911169]: [“GuaranteedUpdate etcd3” audit-id:d51d665e-c23f-444f-813e-8806927f519f,key:/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io,type:*apiregistration.APIService,resource:apiservices.apiregistration.k8s.io 570ms (09:25:58.975)]
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[774911169]: [570.367938ms] [570.367938ms] END
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: E0326 09:25:59.545846 3618643 available_controller.go:456] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io “v1beta1.metrics.k8s.io”: the object has been modified; please apply your changes to the latest version and try again
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: I0326 09:25:59.565296 3618643 trace.go:236] Trace[762783504]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:48.958) (total time: 130606ms):
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[762783504]: [2m10.606806874s] [2m10.606806874s] END
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: I0326 09:25:59.565303 3618643 trace.go:236] Trace[1330164676]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:48.958) (total time: 130606ms):
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[1330164676]: [2m10.606898876s] [2m10.606898876s] END
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: I0326 09:25:59.565305 3618643 trace.go:236] Trace[474826034]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:48.958) (total time: 130606ms):
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[474826034]: [2m10.606683696s] [2m10.606683696s] END
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: I0326 09:25:59.565334 3618643 trace.go:236] Trace[458771640]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:48.958) (total time: 130606ms):
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[458771640]: [2m10.606784701s] [2m10.606784701s] END
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: I0326 09:25:59.565369 3618643 trace.go:236] Trace[28370712]: “Proxy via http_connect protocol over tcp” address:10.42.1.205:10250 (26-Mar-2024 09:23:48.958) (total time: 130607ms):
Mar 26 09:25:59 k3s-agent2 k3s[3618643]: Trace[28370712]: [2m10.607193684s] [2m10.607193684s] END
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: W0326 09:26:00.644428 3618643 handler_proxy.go:100] no RequestInfo found in the context
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: W0326 09:26:00.644452 3618643 handler_proxy.go:100] no RequestInfo found in the context
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: E0326 09:26:00.645177 3618643 controller.go:116] loading OpenAPI spec for “v1beta1.metrics.k8s.io” failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: I0326 09:26:00.645190 3618643 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: E0326 09:26:00.645583 3618643 controller.go:113] loading OpenAPI spec for “v1beta1.metrics.k8s.io” failed with: Error, could not get list of group versions for APIService
Mar 26 09:26:00 k3s-agent2 k3s[3618643]: I0326 09:26:00.646937 3618643 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: time=“2024-03-26T09:26:06+08:00” level=info msg=“Slow SQL (started: 2024-03-26 09:26:04.549026651 +0800 CST m=+30202.071390797) (total time: 1.707652963s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = ‘compact_rev_key’), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC : [[/registry/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io false]]”
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: time=“2024-03-26T09:26:06+08:00” level=info msg=“Slow SQL (started: 2024-03-26 09:26:04.361819226 +0800 CST m=+30201.884183368) (total time: 1.895001667s): SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), COUNT(c.theid) FROM ( SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = ‘compact_rev_key’), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC ) c : [[/registry/cronjobs/% false]]”
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: time=“2024-03-26T09:26:06+08:00” level=info msg=“Slow SQL (started: 2024-03-26 09:26:05.0604668 +0800 CST m=+30202.582830940) (total time: 1.196668803s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = ‘compact_rev_key’), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC LIMIT 1 : [[/registry/events/longhorn-system/pvc-681718e4-e582-4299-8d64-aac951017607-e-375330c6.17b96bc8af6f56e3 true]]”
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: time=“2024-03-26T09:26:06+08:00” level=info msg=“Slow SQL (started: 2024-03-26 09:26:04.234076578 +0800 CST m=+30201.756440716) (total time: 2.023854447s): SELECT * FROM ( SELECT ( SELECT MAX(rkv.id) AS id FROM kine AS rkv), ( SELECT MAX(crkv.prev_revision) AS prev_revision FROM kine AS crkv WHERE crkv.name = ‘compact_rev_key’), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value FROM kine AS kv JOIN ( SELECT MAX(mkv.id) AS id FROM kine AS mkv WHERE mkv.name LIKE ? GROUP BY mkv.name) AS maxkv ON maxkv.id = kv.id WHERE kv.deleted = 0 OR ? ) AS lkv ORDER BY lkv.theid ASC : [[/registry/leases/kube-system/k3s false]]”
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: I0326 09:26:06.258344 3618643 trace.go:236] Trace[1237440331]: “Get” accept:application/json, /,audit-id:016b7aed-3edb-414d-914a-81c6e17132ce,client:127.0.0.1,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/k3s,user-agent:k3s-supervisor@k3s-agent2/v1.26.9+k3s1 (linux/amd64) k3s/4e217286,verb:GET (26-Mar-2024 09:26:04.233) (total time: 2024ms):
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: Trace[1237440331]: —“About to write a response” 2024ms (09:26:06.258)
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: Trace[1237440331]: [2.024457764s] [2.024457764s] END
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: I0326 09:26:06.259529 3618643 trace.go:236] Trace[1956190757]: “Update” accept:application/vnd.kubernetes.protobuf, /,audit-id:d1b7df56-1b24-481f-993b-a4d6a370303c,client:127.0.0.1,protocol:HTTP/2.0,resource:apiservices,scope:resource,url:/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io/status,user-agent:k3s/v1.26.9+k3s1 (linux/amd64) kubernetes/4e21728,verb:PUT (26-Mar-2024 09:26:04.548) (total time: 1711ms):
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: Trace[1956190757]: [“GuaranteedUpdate etcd3” audit-id:d1b7df56-1b24-481f-993b-a4d6a370303c,key:/apiregistration.k8s.io/apiservices/v1beta1.metrics.k8s.io,type:*apiregistration.APIService,resource:apiservices.apiregistration.k8s.io 1711ms (09:26:04.548)]
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: Trace[1956190757]: [1.711468605s] [1.711468605s] END
Mar 26 09:26:06 k3s-agent2 k3s[3618643]: E0326 09:26:06.260094 3618643 available_controller.go:456] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io “v1beta1.metrics.k8s.io”: the object has been modified; please apply your changes to the latest version and try again