K3S+MySQL部署高可用时，第二个server节点报错

风起枫落 · 2022 年10 月 10 日 08:39

环境信息:
K3s 版本:
k3s version v1.23.12+k3s1 (66309a8e)
go version go1.17.13

节点 CPU 架构、操作系统和版本：:
Linux t1 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

集群配置: t1 (server1 + mysql ),t2 (server2)

问题描述:

K3S+MySQL 方式部署高可用集群， server1节点部署成功后，复制server1的token并部署server2，server2部署报错。journalctl -xe 显示： starting kubernetes: preparing server: bootstrap data already found and encrypted with different token

预期结果:

离线安装方式：
步骤1-1：在192.168.130.101 主机上部署server1节点(192.168.130.101中部署中部署有mysql)。
INSTALL_K3S_SKIP_DOWNLOAD=true
K3S_DATASTORE_ENDPOINT=‘mysql://user:passswd@tcp(192.168.130.101:3306)/k3s’
INSTALL_K3S_EXEC=‘server --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 644 --docker’ ./install.sh

步骤1-2：安装成功后查看token
cat /var/lib/rancher/k3s/server/node-token
K102caa4a6e0876d11b31cc9c5d53d022d2e656dc2b59f8a0e75189e3a427cc6a08::server:c3e82193a62b706298581bc63e88e4f4

步骤2-1：拷贝步骤1-2中的token，并在192.168.130.102主机上安装server2
INSTALL_K3S_SKIP_DOWNLOAD=true
K3S_DATASTORE_ENDPOINT=‘mysql://user:passswd@tcp(192.168.130.101:3306)/k3s’
INSTALL_K3S_EXEC=‘server --token K102caa4a6e0876d11b31cc9c5d53d022d2e656dc2b59f8a0e75189e3a427cc6a08::server:c3e82193a62b706298581bc63e88e4f4 --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 644 --docker’ ./install.sh

安装报错，journalctl -xe 查看日志：

10月 10 16:05:13 t2 k3s[11955]: time=“2022-10-10T16:05:13+08:00” level=fatal msg=“starting kubernetes: preparing server: bootstrap data already found and encrypted with different token”
10月 10 16:05:13 t2 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
10月 10 16:05:13 t2 systemd[1]: Failed to start Lightweight Kubernetes.

实际结果:

[/details]

ksd · 2022 年10 月 10 日 09:17

日志报错说是 token 不匹配，但从你的操作来看，应该是没问题的。

你可以把变量和安装命令放在同一行在试试，然后再确认下 token 是否一致。

以下是我的测试步骤：

k3s1:

INSTALL_K3S_SKIP_DOWNLOAD=true K3S_DATASTORE_ENDPOINT='mysql://root:password@tcp(192.168.205.30)/k3s' INSTALL_K3S_EXEC='server --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 644 ' ./install.sh

cat /var/lib/rancher/k3s/server/node-token
K102caa4a6e0876d11b31cc9c5d53d022d2e656dc2b59f8a0e75189e3a427cc6a08::server:c3e82193a62b706298581bc63e88e4f4

k3s2:

INSTALL_K3S_SKIP_DOWNLOAD=true K3S_DATASTORE_ENDPOINT='mysql://root:password@tcp(192.168.205.30:3306)/k3s' INSTALL_K3S_EXEC='server --token K106454712eee94385ca2b33efb8f61e6c90baddbc5a115aebc09e8ceb21a0779b0::server:8677b12465eceec8aa1971854a746f28 --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 644' ./install.sh

k3s1 或 k3s2:

root@k3s1:~# kubectl get nodes
NAME   STATUS   ROLES                  AGE     VERSION
k3s1   Ready    control-plane,master   8m19s   v1.23.12+k3s1
k3s2   Ready    control-plane,master   15s     v1.23.12+k3s1

风起枫落 · 2022 年10 月 12 日 07:52

我重新安装了，还是没有解决问题。

server1 节点上的运行状态：
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
Active: active (running) since 三 2022-10-12 15:25:11 CST; 25min ago
Docs: https://k3s.io
Process: 3501 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 3497 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 3494 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Main PID: 3504 (k3s-server)
Tasks: 108
Memory: 1.0G
CGroup: /system.slice/k3s.service
├─3504 /usr/local/bin/k3s server
├─3539 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
├─3930 /var/lib/rancher/k3s/data/6950a87734e6d9778dbbd420aaa1e94ab68976f86a57c868b5ebacd971ab664f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4cc74ca55fa2dc3ecf027b283bc94c8a9991ac1bb43993a0b11a40d7…
├─4327 /var/lib/rancher/k3s/data/6950a87734e6d9778dbbd420aaa1e94ab68976f86a57c868b5ebacd971ab664f/bin/containerd-shim-runc-v2 -namespace k8s.io -id b8b29052d17664b2014904684e5bf0501747a74a3815ec0dee645983…
├─4589 /var/lib/rancher/k3s/data/6950a87734e6d9778dbbd420aaa1e94ab68976f86a57c868b5ebacd971ab664f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 33882553057c68eeb2ece6fda5fbde4a554111a3deb98d8b71326651…
├─5677 /var/lib/rancher/k3s/data/6950a87734e6d9778dbbd420aaa1e94ab68976f86a57c868b5ebacd971ab664f/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4c1474fe4f22b9dbade6573c08d07036e335140823d6e5e4829edad7…
└─5898 /var/lib/rancher/k3s/data/6950a87734e6d9778dbbd420aaa1e94ab68976f86a57c868b5ebacd971ab664f/bin/containerd-shim-runc-v2 -namespace k8s.io -id cb6f66b159bb6a2b79001e1e30913e0f433bebea3344eaa4b6a7c051…

10月 12 15:47:30 t2 k3s[3504]: I1012 15:47:30.269687 3504 trace.go:205] Trace[2124489019]: “Get” url:/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/workload-low,user-agent:k3s/v1.23.12+k3…
10月 12 15:47:30 t2 k3s[3504]: Trace[2124489019]: —“About to write a response” 20029ms (15:47:30.269)
10月 12 15:47:30 t2 k3s[3504]: Trace[2124489019]: [20.029104063s] [20.029104063s] END
10月 12 15:48:24 t2 k3s[3504]: time=“2022-10-12T15:48:24+08:00” level=info msg="Slow SQL (started: 2022-10-12 15:48:04.592460639 +0800 CST m=+1536.251662093) (total time: 20.021820948s): SELECT * F…ROM kine AS crkv
10月 12 15:48:24 t2 k3s[3504]: I1012 15:48:24.616633 3504 trace.go:205] Trace[923180384]: “GuaranteedUpdate etcd3” type:*core.ConfigMap (12-Oct-2022 15:48:04.592) (total time: 20024ms):
10月 12 15:48:24 t2 k3s[3504]: Trace[923180384]: —“Transaction committed” 20024ms (15:48:24.616)
10月 12 15:48:24 t2 k3s[3504]: Trace[923180384]: [20.024503694s] [20.024503694s] END
10月 12 15:48:24 t2 k3s[3504]: I1012 15:48:24.616730 3504 trace.go:205] Trace[267223063]: “Update” url:/api/v1/namespaces/kube-system/configmaps/k3s,user-agent:deploy@t2/v1.23.12+k3s1 (linux/amd64) k3s/66309a8e,…
10月 12 15:48:24 t2 k3s[3504]: Trace[267223063]: —“Object stored in database” 20024ms (15:48:24.616)
10月 12 15:48:24 t2 k3s[3504]: Trace[267223063]: [20.024718822s] [20.024718822s] END
Hint: Some lines were ellipsized, use -l to show in full.

查看token:
[root@t2 ~]# cat /var/lib/rancher/k3s/server/node-token
K1042d8c761e72e5751be82d3d3004aac4f34bd8d2086cabdeb69fcc968b1de38b8::server:045fcf88bd7bbf2f78f99e7beb5bab11

在server2服务器上执行操作：
[root@t3 k3s-v1.23.12]# INSTALL_K3S_SKIP_DOWNLOAD=true K3S_DATASTORE_ENDPOINT=‘mysql://root:123456@tcp(192.168.130.107:13306)/k3s’ INSTALL_K3S_EXEC=‘server --token K1042d8c761e72e5751be82d3d3004aac4f34bd8d2086cabdeb69fcc968b1de38b8::server:045fcf88bd7bbf2f78f99e7beb5bab11 --write-kubeconfig ~/.kube/config --write-kubeconfig-mode 644 --docker’ ./install.sh

执行结果：
[INFO] Skipping k3s download and verify
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/rancher/k3s/k3s.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink from /etc/systemd/system/multi-user.target.wants/k3s.service to /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code. See “systemctl status k3s.service” and “journalctl -xe” for details.

风起枫落 · 2022 年10 月 12 日 07:53

[root@t3 k3s-v1.23.12]# systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since 三 2022-10-12 15:52:57 CST; 2s ago
Docs: https://k3s.io
Process: 3167 ExecStart=/usr/local/bin/k3s server $K3S_SERVER_ARGS (code=exited, status=1/FAILURE)
Process: 3165 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 3161 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 3157 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Main PID: 3167 (code=exited, status=1/FAILURE)

10月 12 15:52:57 t3 systemd[1]: Failed to start Lightweight Kubernetes.
10月 12 15:52:57 t3 systemd[1]: Unit k3s.service entered failed state.
10月 12 15:52:57 t3 systemd[1]: k3s.service failed.
[root@t3 k3s-v1.23.12]#

风起枫落 · 2022 年10 月 12 日 07:55

我操作系统是在虚拟机中安装的最小版本 centos7.9 。
firewall打开状态，打开了6443/tcp 8442/udp这两个端口

ksd · 2022 年10 月 12 日 09:38

那你可以先把 firewall 啊、selinux 啊、networkmanager 都关闭，然后试试是否可用。

ksd · 2022 年10 月 12 日 09:39

或者你也可以看 /var/log/message，查看 K3s 的详细报错信息，从你上面的日志，没看见具体的原因