Rancher Server启动失败

absoult · 2023 年8 月 16 日 07:07

Rancher Server版本：v2.5.15
在手工更新Rancher Server的证书，重启Rancher Server后，发现重启失败，检查日志如下：
2023/08/16 06:58:59 [INFO] Rancher version f10640542-dirty (f10640542) is starting
2023/08/16 06:58:59 [INFO] Rancher arguments {ACMEDomains: AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2023/08/16 06:58:59 [INFO] Listening on /tmp/log.sock
2023/08/16 06:58:59 [INFO] Running etcd --data-dir=management-state/etcd --heartbeat-interval=500 --election-timeout=5000
2023-08-16 06:58:59.933727 W | pkg/flags: unrecognized environment variable ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz
2023-08-16 06:58:59.933788 W | pkg/flags: unrecognized environment variable ETCD_URL_amd64=https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz
2023-08-16 06:58:59.933798 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64
2023-08-16 06:58:59.933809 W | pkg/flags: unrecognized environment variable ETCD_URL=ETCD_URL_amd64
[WARNING] Deprecated ‘–logger=capnslog’ flag is set; use ‘–logger=zap’ flag instead
2023-08-16 06:58:59.933895 I | etcdmain: etcd Version: 3.4.3
2023-08-16 06:58:59.933906 I | etcdmain: Git SHA: 3cf2f69b5
2023-08-16 06:58:59.933912 I | etcdmain: Go Version: go1.12.12
2023-08-16 06:58:59.933919 I | etcdmain: Go OS/Arch: linux/amd64
2023-08-16 06:58:59.933927 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2023-08-16 06:58:59.934009 N | etcdmain: the server is already initialized as member before, starting as etcd member…
[WARNING] Deprecated ‘–logger=capnslog’ flag is set; use ‘–logger=zap’ flag instead
2023-08-16 06:58:59.934629 I | embed: name = default
2023-08-16 06:58:59.934647 I | embed: data dir = management-state/etcd
2023-08-16 06:58:59.934652 I | embed: member dir = management-state/etcd/member
2023-08-16 06:58:59.934657 I | embed: heartbeat = 500ms
2023-08-16 06:58:59.934661 I | embed: election = 5000ms
2023-08-16 06:58:59.934666 I | embed: snapshot count = 100000
2023-08-16 06:58:59.934679 I | embed: advertise client URLs = http://localhost:2379
2023-08-16 06:58:59.934693 I | embed: initial advertise peer URLs = http://localhost:2380
2023-08-16 06:58:59.934703 I | embed: initial cluster =
2023-08-16 06:58:59.948074 I | etcdserver: recovered store from snapshot at index 151001553
2023-08-16 06:58:59.948878 I | mvcc: restore compact to 141695447
2023-08-16 06:59:00.524227 I | etcdserver: restarting member 8e9e05c52164694d in cluster cdf818194e3a8c32 at commit index 151076070
raft2023/08/16 06:59:00 INFO: 8e9e05c52164694d switched to configuration voters=(10276657743932975437)
raft2023/08/16 06:59:00 INFO: 8e9e05c52164694d became follower at term 37
raft2023/08/16 06:59:00 INFO: newRaft 8e9e05c52164694d [peers: [8e9e05c52164694d], term: 37, commit: 151076070, applied: 151001553, lastindex: 151076070, lastterm: 37]
2023-08-16 06:59:00.529844 I | etcdserver/api: enabled capabilities for version 3.4
2023-08-16 06:59:00.529869 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32 from store
2023-08-16 06:59:00.529877 I | etcdserver/membership: set the cluster version to 3.4 from store
2023-08-16 06:59:00.531952 I | mvcc: restore compact to 141695447
2023-08-16 06:59:00.542367 W | auth: simple token is not cryptographically signed
2023-08-16 06:59:00.543929 I | etcdserver: starting server… [version: 3.4.3, cluster version: 3.4]
2023-08-16 06:59:00.544323 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2023-08-16 06:59:00.548409 I | embed: listening for peers on 127.0.0.1:2380
raft2023/08/16 06:59:05 INFO: 8e9e05c52164694d is starting a new election at term 37
raft2023/08/16 06:59:05 INFO: 8e9e05c52164694d became candidate at term 38
raft2023/08/16 06:59:05 INFO: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 38
raft2023/08/16 06:59:05 INFO: 8e9e05c52164694d became leader at term 38
raft2023/08/16 06:59:05 INFO: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 38
2023-08-16 06:59:05.531632 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
2023-08-16 06:59:05.531762 I | embed: ready to serve client requests
2023-08-16 06:59:05.532745 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2023/08/16 06:59:05 [INFO] Waiting for server to become available: Get “https://127.0.0.1:6443/version?timeout=15m0s”: dial tcp 127.0.0.1:6443: connect: connection refused
W0816 06:59:07.571112 7 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0816 06:59:07.590858 7 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
W0816 06:59:07.613251 7 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
exit status 255
2023/08/16 06:59:13 [FATAL] k3s exited with: exit status 255

ksd · 2023 年8 月 16 日 07:18

你是咋更新的证书啊

absoult · 2023 年8 月 16 日 07:19

根据官网的步骤来更新的

absoult · 2023 年8 月 16 日 07:28

下面是k3s的日志：

NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0816 07:16:12.941527 42 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I0816 07:16:12.977725 42 master.go:271] Using reconciler: lease
W0816 07:16:13.361198 42 genericapiserver.go:418] Skipping API batch/v2alpha1 because it has no resources.
W0816 07:16:13.383114 42 genericapiserver.go:418] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0816 07:16:13.407596 42 genericapiserver.go:418] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0816 07:16:13.433785 42 genericapiserver.go:418] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0816 07:16:13.438804 42 genericapiserver.go:418] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0816 07:16:13.467055 42 genericapiserver.go:418] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0816 07:16:13.522195 42 genericapiserver.go:418] Skipping API apps/v1beta2 because it has no resources.
W0816 07:16:13.522228 42 genericapiserver.go:418] Skipping API apps/v1beta1 because it has no resources.
I0816 07:16:13.537229 42 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0816 07:16:13.537253 42 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
time=“2023-08-16T07:16:13.557029263Z” level=info msg=“Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --port=10251 --profiling=false --secure-port=0”
I0816 07:16:13.557478 42 registry.go:173] Registering SelectorSpread plugin
I0816 07:16:13.557537 42 registry.go:173] Registering SelectorSpread plugin
time=“2023-08-16T07:16:13.557900169Z” level=info msg=“Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true”
time=“2023-08-16T07:16:13.557920173Z” level=info msg=“Waiting for API server to become available”
time=“2023-08-16T07:16:13.560308762Z” level=info msg=“Node token is available at /var/lib/rancher/k3s/server/token”
time=“2023-08-16T07:16:13.560351825Z” level=info msg=“To join node to cluster: k3s agent -s https://172.17.0.2:6443 -t ${NODE_TOKEN}”
time=“2023-08-16T07:16:13.561693330Z” level=info msg=“Wrote kubeconfig /etc/rancher/k3s/k3s.yaml”
time=“2023-08-16T07:16:13.561725631Z” level=info msg=“Run: k3s kubectl”
time=“2023-08-16T07:16:13.578451778Z” level=info msg=“Cluster-Http-Server 2023/08/16 07:16:13 http: TLS handshake error from 127.0.0.1:38194: remote error: tls: bad certificate”
time=“2023-08-16T07:16:13.584271540Z” level=info msg=“Cluster-Http-Server 2023/08/16 07:16:13 http: TLS handshake error from 127.0.0.1:38202: remote error: tls: bad certificate”
time=“2023-08-16T07:16:13.595237935Z” level=info msg=“certificate CN=local-node signed by CN=k3s-server-ca@1661420316: notBefore=2022-08-25 09:38:36 +0000 UTC notAfter=2024-08-15 07:16:13 +0000 UTC”
time=“2023-08-16T07:16:13.598333813Z” level=info msg=“certificate CN=system:node:local-node,O=system:nodes signed by CN=k3s-client-ca@1661420316: notBefore=2022-08-25 09:38:36 +0000 UTC notAfter=2024-08-15 07:16:13 +0000 UTC”
time=“2023-08-16T07:16:13.604580555Z” level=info msg=“Module overlay was already loaded”
time=“2023-08-16T07:16:13.604624762Z” level=info msg=“Module nf_conntrack was already loaded”
time=“2023-08-16T07:16:13.604648092Z” level=info msg=“Module br_netfilter was already loaded”
time=“2023-08-16T07:16:13.604669612Z” level=info msg=“Module iptable_nat was already loaded”
time=“2023-08-16T07:16:13.607551711Z” level=info msg=“Set sysctl ‘net/ipv6/conf/all/forwarding’ to 1”
time=“2023-08-16T07:16:13.607692443Z” level=info msg=“Set sysctl ‘net/bridge/bridge-nf-call-ip6tables’ to 1”
time=“2023-08-16T07:16:13.607732565Z” level=error msg=“Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-ip6tables: no such file or directory”
time=“2023-08-16T07:16:13.607809504Z” level=info msg=“Set sysctl ‘net/bridge/bridge-nf-call-iptables’ to 1”
time=“2023-08-16T07:16:13.607870141Z” level=error msg=“Failed to set sysctl: open /proc/sys/net/bridge/bridge-nf-call-iptables: no such file or directory”
time=“2023-08-16T07:16:13.607932494Z” level=info msg=“Set sysctl ‘net/netfilter/nf_conntrack_tcp_timeout_established’ to 86400”
time=“2023-08-16T07:16:13.608026293Z” level=info msg=“Set sysctl ‘net/netfilter/nf_conntrack_tcp_timeout_close_wait’ to 3600”
time=“2023-08-16T07:16:13.609642849Z” level=info msg=“Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log”
time=“2023-08-16T07:16:13.609921341Z” level=info msg=“Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd”
time=“2023-08-16T07:16:14.612234878Z” level=info msg=“Containerd is now running”
time=“2023-08-16T07:16:14.619892969Z” level=info msg=“Connecting to proxy” url=“wss://127.0.0.1:6443/v1-k3s/connect”
time=“2023-08-16T07:16:14.622181180Z” level=info msg=“Handling backend connection request [local-node]”
time=“2023-08-16T07:16:14.622815087Z” level=warning msg=“Disabling CPU quotas due to missing cpu.cfs_period_us”
time=“2023-08-16T07:16:14.622927512Z” level=info msg=“Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/2d753699589478b1821bd86b3efed6baafd0388c616e89c9d32f1842d4f31eb6/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=/run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --cpu-cfs-quota=false --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=local-node --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --kubelet-cgroups=/k3s --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/etc/resolv.conf --runtime-cgroups=/k3s --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key”
time=“2023-08-16T07:16:14.623606577Z” level=info msg=“Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=local-node --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables”
Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
W0816 07:16:14.624417 42 server.go:226] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0816 07:16:14.624600 42 server.go:407] Version: v1.19.13+k3s1
W0816 07:16:14.625334 42 proxier.go:639] Failed to read file /lib/modules/3.10.0-1160.76.1.el7.x86_64/modules.builtin with error open /lib/modules/3.10.0-1160.76.1.el7.x86_64/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0816 07:16:14.626092 42 proxier.go:649] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0816 07:16:14.626695 42 proxier.go:649] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0816 07:16:14.627326 42 proxier.go:649] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0816 07:16:14.628098 42 proxier.go:649] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0816 07:16:14.628682 42 proxier.go:649] Failed to load kernel module nf_conntrack_ipv4 with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
time=“2023-08-16T07:16:14.637804260Z” level=info msg=“Node CIDR assigned for: local-node”
I0816 07:16:14.638234 42 flannel.go:92] Determining IP address of default interface
I0816 07:16:14.638523 42 flannel.go:105] Using interface with name eth0 and address 172.17.0.2
time=“2023-08-16T07:16:14.639296572Z” level=warning msg=“Running modprobe ip_vs failed with message: modprobe: can't change directory to '/lib/modules': No such file or directory, error: exit status 1”
time=“2023-08-16T07:16:14.645032100Z” level=info msg=“labels have already set on node: local-node”
I0816 07:16:14.645140 42 kube.go:117] Waiting 10m0s for node controller to sync
I0816 07:16:14.645190 42 kube.go:300] Starting kube subnet manager
I0816 07:16:14.659777 42 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
E0816 07:16:14.660737 42 node.go:125] Failed to retrieve node info: nodes “local-node” is forbidden: User “system:kube-proxy” cannot get resource “nodes” in API group “” at the cluster scope
I0816 07:16:15.645397 42 kube.go:124] Node controller sync successful
I0816 07:16:15.645535 42 vxlan.go:121] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E0816 07:16:15.797149 42 node.go:125] Failed to retrieve node info: nodes “local-node” is forbidden: User “system:kube-proxy” cannot get resource “nodes” in API group “” at the cluster scope
I0816 07:16:16.031858 42 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
I0816 07:16:16.031888 42 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I0816 07:16:16.032185 42 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
I0816 07:16:16.032477 42 secure_serving.go:197] Serving securely on 127.0.0.1:6444
I0816 07:16:16.032510 42 tlsconfig.go:240] Starting DynamicServingCertificateController
I0816 07:16:16.032611 42 controller.go:83] Starting OpenAPI AggregationController
I0816 07:16:16.032669 42 available_controller.go:475] Starting AvailableConditionController
I0816 07:16:16.032679 42 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0816 07:16:16.032611 42 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
I0816 07:16:16.033111 42 autoregister_controller.go:141] Starting autoregister controller
I0816 07:16:16.033122 42 cache.go:32] Waiting for caches to sync for autoregister controller
I0816 07:16:16.033089 42 apiservice_controller.go:97] Starting APIServiceRegistrationController
I0816 07:16:16.033164 42 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0816 07:16:16.033448 42 customresource_discovery_controller.go:209] Starting DiscoveryController
I0816 07:16:16.034253 42 crdregistration_controller.go:111] Starting crd-autoregister controller
I0816 07:16:16.034272 42 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
I0816 07:16:16.034403 42 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0816 07:16:16.034416 42 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
I0816 07:16:16.034984 42 controller.go:86] Starting OpenAPI controller
I0816 07:16:16.035020 42 naming_controller.go:291] Starting NamingConditionController
I0816 07:16:16.035055 42 establishing_controller.go:76] Starting EstablishingController
I0816 07:16:16.035085 42 nonstructuralschema_controller.go:186] Starting NonStructuralSchemaConditionController
I0816 07:16:16.035110 42 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
I0816 07:16:16.035138 42 crd_finalizer.go:266] Starting CRDFinalizer
I0816 07:16:16.035183 42 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
I0816 07:16:16.035216 42 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
time=“2023-08-16T07:16:16.065022439Z” level=info msg=“Running cloud-controller-manager --allocate-node-cidrs=true --allow-untagged-cloud=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --node-status-update-frequency=1m --profiling=false --secure-port=0”
Flag --allow-untagged-cloud has been deprecated, This flag is deprecated and will be removed in a future release. A cluster-id will be required on cloud instances.
I0816 07:16:16.074416 42 controllermanager.go:127] Version: v1.19.13+k3s1
W0816 07:16:16.074442 42 controllermanager.go:139] detected a cluster without a ClusterID. A ClusterID will be required in the future. Please tag your cluster to avoid any future issues
I0816 07:16:16.074492 42 leaderelection.go:243] attempting to acquire leader lease kube-system/cloud-controller-manager…
E0816 07:16:16.080120 42 leaderelection.go:325] error retrieving resource lock kube-system/cloud-controller-manager: endpoints “cloud-controller-manager” is forbidden: User “cloud-controller-manager” cannot get resource “endpoints” in API group “” in the namespace “kube-system”
E0816 07:16:16.103259 42 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
I0816 07:16:16.132800 42 cache.go:39] Caches are synced for AvailableConditionController controller
I0816 07:16:16.133186 42 cache.go:39] Caches are synced for autoregister controller
I0816 07:16:16.133209 42 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0816 07:16:16.134642 42 shared_informer.go:247] Caches are synced for crd-autoregister
I0816 07:16:16.136382 42 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
I0816 07:16:16.165018 42 trace.go:205] Trace[195979236]: “GuaranteedUpdate etcd3” type:*core.Node (16-Aug-2023 07:16:15.654) (total time: 510ms):
Trace[195979236]: —“Transaction prepared” 509ms (07:16:00.163)
Trace[195979236]: [510.864769ms] [510.864769ms] END
I0816 07:16:16.165382 42 trace.go:205] Trace[31577409]: “Patch” url:/api/v1/nodes/local-node/status,user-agent:k3s/v1.19.13+k3s1 (linux/amd64) kubernetes/99eadcc,client:127.0.0.1 (16-Aug-2023 07:16:15.653) (total time: 511ms):
Trace[31577409]: —“Object stored in database” 508ms (07:16:00.165)
Trace[31577409]: [511.406589ms] [511.406589ms] END
I0816 07:16:16.176248 42 flannel.go:78] Wrote subnet file to /run/flannel/subnet.env
I0816 07:16:16.176267 42 flannel.go:82] Running backend.
I0816 07:16:16.176277 42 vxlan_network.go:60] watching for new subnet leases
I0816 07:16:16.178446 42 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0816 07:16:16.178467 42 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 -j ACCEPT
I0816 07:16:16.178627 42 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0816 07:16:16.178647 42 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
I0816 07:16:16.179308 42 iptables.go:167] Deleting iptables rule: -d 10.42.0.0/16 -j ACCEPT
I0816 07:16:16.180047 42 iptables.go:167] Deleting iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0816 07:16:16.180282 42 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 -j ACCEPT
I0816 07:16:16.191582 42 iptables.go:167] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/24 -j RETURN
I0816 07:16:16.193294 42 iptables.go:155] Adding iptables rule: -d 10.42.0.0/16 -j ACCEPT
I0816 07:16:16.193296 42 iptables.go:167] Deleting iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully
I0816 07:16:16.194550 42 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 -d 10.42.0.0/16 -j RETURN
I0816 07:16:16.196828 42 iptables.go:155] Adding iptables rule: -s 10.42.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0816 07:16:16.198939 42 iptables.go:155] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/24 -j RETURN
I0816 07:16:16.200552 42 iptables.go:155] Adding iptables rule: ! -s 10.42.0.0/16 -d 10.42.0.0/16 -j MASQUERADE --random-fully
I0816 07:16:17.031690 42 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0816 07:16:17.044889 42 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
I0816 07:16:18.018455 42 node.go:136] Successfully retrieved node IP: 172.17.0.2
I0816 07:16:18.018506 42 server_others.go:143] kube-proxy node IP is an IPv4 address (172.17.0.2), assume IPv4 operation
I0816 07:16:18.020800 42 server_others.go:186] Using iptables Proxier.
I0816 07:16:18.021176 42 server.go:650] Version: v1.19.13+k3s1
I0816 07:16:18.023634 42 config.go:315] Starting service config controller
I0816 07:16:18.023658 42 shared_informer.go:240] Waiting for caches to sync for service config
I0816 07:16:18.023698 42 config.go:224] Starting endpoint slice config controller
I0816 07:16:18.023710 42 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
time=“2023-08-16T07:16:18.058483753Z” level=info msg=“Kube API server is now running”
time=“2023-08-16T07:16:18.058566515Z” level=info msg=“k3s is up and running”
Flag --address has been deprecated, see --bind-address instead.
I0816 07:16:18.075309 42 controllermanager.go:175] Version: v1.19.13+k3s1
I0816 07:16:18.075948 42 deprecated_insecure_serving.go:53] Serving insecurely on 127.0.0.1:10252
I0816 07:16:18.076006 42 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-controller-manager…
I0816 07:16:18.089636 42 registry.go:173] Registering SelectorSpread plugin
I0816 07:16:18.089668 42 registry.go:173] Registering SelectorSpread plugin
W0816 07:16:18.093466 42 authorization.go:47] Authorization is disabled
W0816 07:16:18.093485 42 authentication.go:40] Authentication is disabled
I0816 07:16:18.093508 42 deprecated_insecure_serving.go:51] Serving healthz insecurely on 127.0.0.1:10251
I0816 07:16:18.124213 42 shared_informer.go:247] Caches are synced for endpoint slice config
I0816 07:16:18.124270 42 shared_informer.go:247] Caches are synced for service config
time=“2023-08-16T07:16:18.190395490Z” level=info msg=“Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-1.81.0.tgz”
time=“2023-08-16T07:16:18.190757287Z” level=info msg=“Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml”
time=“2023-08-16T07:16:18.191255482Z” level=info msg=“Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml”
time=“2023-08-16T07:16:18.191556116Z” level=info msg=“Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml”
I0816 07:16:18.193639 42 leaderelection.go:243] attempting to acquire leader lease kube-system/kube-scheduler…
time=“2023-08-16T07:16:18.293008864Z” level=info msg=“Starting k3s.cattle.io/v1, Kind=Addon controller”
I0816 07:16:18.293056 42 leaderelection.go:243] attempting to acquire leader lease kube-system/k3s…
time=“2023-08-16T07:16:18.298140053Z” level=info msg=“Cluster dns configmap already exists”
I0816 07:16:18.303089 42 leaderelection.go:253] successfully acquired lease kube-system/k3s
I0816 07:16:18.390003 42 controller.go:609] quota admission added evaluator for: addons.k3s.cattle.io
I0816 07:16:18.458759 42 controller.go:609] quota admission added evaluator for: deployments.apps
time=“2023-08-16T07:16:18.505866414Z” level=info msg=“Starting helm.cattle.io/v1, Kind=HelmChart controller”
time=“2023-08-16T07:16:18.505919730Z” level=info msg=“Starting /v1, Kind=Endpoints controller”
time=“2023-08-16T07:16:18.505891601Z” level=info msg=“Starting helm.cattle.io/v1, Kind=HelmChartConfig controller”
time=“2023-08-16T07:16:18.505905195Z” level=info msg=“Starting /v1, Kind=Pod controller”
time=“2023-08-16T07:16:18.505907093Z” level=info msg=“Starting batch/v1, Kind=Job controller”
time=“2023-08-16T07:16:18.505926064Z” level=info msg=“Starting /v1, Kind=Service controller”
time=“2023-08-16T07:16:18.505944917Z” level=info msg=“Starting /v1, Kind=Node controller”
I0816 07:16:18.771483 42 network_policy_controller.go:149] Starting network policy controller
I0816 07:16:18.849569 42 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
time=“2023-08-16T07:16:19.040163744Z” level=info msg=“Starting /v1, Kind=Secret controller”
time=“2023-08-16T07:16:19.622428721Z” level=info msg=“Stopped tunnel to 127.0.0.1:6443”
time=“2023-08-16T07:16:19.622465602Z” level=info msg=“Connecting to proxy” url=“wss://172.17.0.2:6443/v1-k3s/connect”
time=“2023-08-16T07:16:19.622470336Z” level=info msg=“Proxy done” err=“context canceled” url=“wss://127.0.0.1:6443/v1-k3s/connect”
time=“2023-08-16T07:16:19.622731356Z” level=info msg=“error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF”
time=“2023-08-16T07:16:19.625243686Z” level=info msg=“Handling backend connection request [local-node]”
I0816 07:16:19.668283 42 server.go:640] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
I0816 07:16:19.668639 42 container_manager_linux.go:281] container manager verified user specified cgroup-root exists:
I0816 07:16:19.668670 42 container_manager_linux.go:286] Creating Container Manager object based on Node Config: {RuntimeCgroupsName:/k3s SystemCgroupsName: KubeletCgroupsName:/k3s ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map SystemReserved:map HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:} {Signal:nodefs.available Operator:LessThan Value:{Quantity: Percentage:0.05} GracePeriod:0s MinReclaim:}]} QOSReserved:map ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:false CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
I0816 07:16:19.668813 42 topology_manager.go:126] [topologymanager] Creating topology manager with none policy
I0816 07:16:19.668847 42 container_manager_linux.go:316] [topologymanager] Initializing Topology Manager with none policy
I0816 07:16:19.668857 42 container_manager_linux.go:321] Creating device plugin manager: true
W0816 07:16:19.668974 42 util_unix.go:103] Using “/run/k3s/containerd/containerd.sock” as endpoint is deprecated, please consider using full url format “unix:///run/k3s/containerd/containerd.sock”.
W0816 07:16:19.669087 42 util_unix.go:103] Using “/run/k3s/containerd/containerd.sock” as endpoint is deprecated, please consider using full url format “unix:///run/k3s/containerd/containerd.sock”.
I0816 07:16:19.669200 42 kubelet.go:394] Attempting to sync node with API server
I0816 07:16:19.669233 42 kubelet.go:261] Adding pod path: /var/lib/rancher/k3s/agent/pod-manifests
I0816 07:16:19.669276 42 kubelet.go:273] Adding apiserver pod source
I0816 07:16:19.669300 42 apiserver.go:43] Waiting for node sync before watching apiserver pods
I0816 07:16:19.671733 42 kuberuntime_manager.go:214] Container runtime containerd initialized, version: v1.4.8-k3s1, apiVersion: v1alpha2
I0816 07:16:19.672251 42 server.go:1148] Started kubelet
I0816 07:16:19.674573 42 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
E0816 07:16:19.675100 42 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint “/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs”: unable to find data in memory cache.
E0816 07:16:19.675133 42 kubelet.go:1238] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
I0816 07:16:19.675927 42 server.go:152] Starting to listen on 0.0.0.0:10250
I0816 07:16:19.677167 42 server.go:425] Adding debug handlers to kubelet server.
I0816 07:16:19.678745 42 volume_manager.go:265] Starting Kubelet Volume Manager
I0816 07:16:19.679334 42 desired_state_of_world_populator.go:139] Desired state populator starts to run
I0816 07:16:19.694154 42 controller.go:609] quota admission added evaluator for: leases.coordination.k8s.io
I0816 07:16:19.707471 42 cpu_manager.go:184] [cpumanager] starting with none policy
I0816 07:16:19.707489 42 cpu_manager.go:185] [cpumanager] reconciling every 10s
I0816 07:16:19.707523 42 state_mem.go:36] [cpumanager] initializing new in-memory state store
I0816 07:16:19.707794 42 state_mem.go:88] [cpumanager] updated default cpuset: “”
I0816 07:16:19.707808 42 state_mem.go:96] [cpumanager] updated cpuset assignments: “map”
I0816 07:16:19.707864 42 policy_none.go:43] [cpumanager] none policy: Start
F0816 07:16:19.722535 42 kubelet.go:1316] Failed to start ContainerManager failed to initialize top level QOS containers: failed to create top level Burstable QOS cgroup : mkdir /sys/fs/cgroup/memory/kubepods/burstable: cannot allocate memory
goroutine 6535 [running]:
github.com/rancher/k3s/vendor/k8s.io/klog/v2.stacks(0xc000010001, 0xc011c0c000, 0xf8, 0x251)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/v2/klog.go:996 +0xb9
github.com/rancher/k3s/vendor/k8s.io/klog/v2.(*loggingT).output(0x70ff1e0, 0xc000000003, 0x0, 0x0, 0xc0087de540, 0x6da9e41, 0xa, 0x524, 0x0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/v2/klog.go:945 +0x191
github.com/rancher/k3s/vendor/k8s.io/klog/v2.(*loggingT).printf(0x70ff1e0, 0xc000000003, 0x0, 0x0, 0x46b9ded, 0x23, 0xc00c697cb0, 0x1, 0x1)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/v2/klog.go:733 +0x17a
github.com/rancher/k3s/vendor/k8s.io/klog/v2.Fatalf(...)
/go/src/github.com/rancher/k3s/vendor/k8s.io/klog/v2/klog.go:1456
github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet.(*Kubelet).initializeRuntimeDependentModules(0xc00df75500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1316 +0x385
sync.(*Once).doSlow(0xc00df75d40, 0xc001313dd0)
/usr/local/go/src/sync/once.go:66 +0xec
sync.(*Once).Do(…)
/usr/local/go/src/sync/once.go:57
github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet.(*Kubelet).updateRuntimeUp(0xc00df75500)
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:2145 +0x554
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc005d6d4b0)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc005d6d4b0, 0x4d30ea0, 0xc01164d0e0, 0x6444615139784101, 0xc00007e780)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc005d6d4b0, 0x12a05f200, 0x0, 0x68774d4554315501, 0xc00007e780)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc005d6d4b0, 0x12a05f200, 0xc00007e780)
/go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet.(*Kubelet).Run
/go/src/github.com/rancher/k3s/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1363 +0x16a

goroutine 1 [chan receive]:
github.com/rancher/k3s/pkg/agent.run(0x4df2660, 0xc0009b8b80, 0xc006576fc0, 0x6a, 0x0, 0x0, 0x0, 0x0, 0xc002ea7fa0, 0x16, …)
/go/src/github.com/rancher/k3s/pkg/agent/run.go:126 +0x2eb
github.com/rancher/k3s/pkg/agent.Run(0x4df2660, 0xc0009b8b80, 0xc006576fc0, 0x6a, 0x0, 0x0, 0x0, 0x0, 0xc002ea7fa0, 0x16, …)
/go/src/github.com/rancher/k3s/pkg/agent/run.go:218 +0x438
github.com/rancher/k3s/pkg/cli/server.run(0xc0006ea420, 0x71001a0, 0xc0011f0d58, 0x0, 0x0, 0xc0011f0d58, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/k3s/pkg/cli/server/server.go:347 +0x1b98
github.com/rancher/k3s/pkg/cli/server.Run(0xc0006ea420, 0x0, 0x0)
/go/src/github.com/rancher/k3s/pkg/cli/server/server.go:45 +0x85
github.com/rancher/k3s/vendor/github.com/urfave/cli.HandleAction(0x3ed21c0, 0x4842738, 0xc0006ea420, 0xc0006ea420, 0x0)
/go/src/github.com/rancher/k3s/vendor/github.com/urfave/cli/app.go:523 +0xfd
github.com/rancher/k3s/vendor/github.com/urfave/cli.Command.Run(0x463349b, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x466a5cc, 0x15, 0xc000645980, …)
/go/src/github.com/rancher/k3s/vendor/github.com/urfave/cli/command.go:174 +0x58e
github.com/rancher/k3s/vendor/github.com/urfave/cli.(*App).Run(0xc0003ab6c0, 0xc000070d80, 0x9, 0x9, 0x0, 0x0)
/go/src/github.com/rancher/k3s/vendor/github.com/urfave/cli/app.go:276 +0x7d4
main.main()
/go/src/github.com/rancher/k3s/cmd/server/main.go:49 +0x69a

ksd · 2023 年8 月 16 日 08:38

你还是提供下你轮换证书的具体步骤，就是在主机上怎么执行的，就把命令复制下来，我怀疑你删东西了

absoult · 2023 年8 月 16 日 08:52

kubectl --insecure-skip-tls-verify -n kube-system delete secrets k3s-serving
kubectl --insecure-skip-tls-verify -n cattle-system delete secret serving-cert
rm -f /var/lib/rancher/k3s/server/tls/dynamic-cert.json
然后重启Rancher Server
查看/var/log/messages，可以看下面的日志信息：
Aug 16 16:29:14 xxxxx containerd: time=“2023-08-16T16:29:14.852121612+08:00” level=warning msg=“cleanup warnings time=“2023-08-16T16:29:14+08:00” level=info msg=“starting signal loop” namespace=moby pid=23367 runtime=io.containerd.runc.v2\ntime=“2023-08-16T16:29:14+08:00” level=warning msg=“failed to read init pid file” error=“open /run/containerd/io.containerd.runtime.v2.task/moby/44ff7b4f0f5a816a85508d4b6d7d04adfd1a1f95b8d84a7957eead56094660c2/init.pid: no such file or directory” runtime=io.containerd.runc.v2\n”
Aug 16 16:29:14 xxxxx containerd: time=“2023-08-16T16:29:14.852419378+08:00” level=error msg=“copy shim log” error=“read /proc/self/fd/29: file already closed”
Aug 16 16:29:14 xxxxx dockerd: time=“2023-08-16T16:29:14.852877158+08:00” level=error msg=“stream copy error: reading from a closed fifo”
Aug 16 16:29:14 xxxxx dockerd: time=“2023-08-16T16:29:14.852971344+08:00” level=error msg=“stream copy error: reading from a closed fifo”
Aug 16 16:29:14 xxxxx kernel: docker0: port 1(veth9f0db71) entered disabled state
Aug 16 16:29:14 xxxxx kernel: device veth9f0db71 left promiscuous mode
Aug 16 16:29:14 xxxxx kernel: docker0: port 1(veth9f0db71) entered disabled state
Aug 16 16:29:14 xxxxx dockerd: time=“2023-08-16T16:29:14.974553137+08:00” level=error msg=“44ff7b4f0f5a816a85508d4b6d7d04adfd1a1f95b8d84a7957eead56094660c2 cleanup: failed to delete container from containerd: no such container”
Aug 16 16:29:14 xxxxx dockerd: time=“2023-08-16T16:29:14.985710732+08:00” level=error msg=“restartmanger wait error: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/memory/docker/44ff7b4f0f5a816a85508d4b6d7d04adfd1a1f95b8d84a7957eead56094660c2: cannot allocate memory: unknown”