rancher版本v2.8.0
RKE2版本v1.27.7+rke2r2
我先删除了所有节点,
然后添加一个节点,
mkdir -p /var/lib/rancher/rke2/server/db/snapshots/
cp <snapshots_file> /var/lib/rancher/rke2/server/db/snapshots/
接着修改了spec.rkeConfig.etcdSnapshotRestore.name
每次执行到error while staging: stream error: stream ID 7; PROTOCOL_ERROR; received from peer这个位置报错
下面是日志文件,能帮忙看看怎么处理?
Jan 19 14:46:57 debian sshd[2710]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="Detected first start, force-applying one-time instruction set"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240119-144801/88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] Running command: /bin/sh [-x /var/lib/rancher/capr/idempotence/idempotent.sh etcd-restore/restore-kill-all e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f 77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3 /bin/sh -c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi]"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + currentHash="
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + key=etcd-restore/restore-kill-all"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + targetHash=e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + hashedCmd=77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + cmd=/bin/sh"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + shift 4"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + dataRoot=/var/lib/rancher/capr/idempotence/etcd-restore/restore-kill-all/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + attemptFile=/var/lib/rancher/capr/idempotence/etcd-restore/restore-kill-all/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f/last-attempt"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + cat /var/lib/rancher/capr/idempotence/etcd-restore/restore-kill-all/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f/last-attempt"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: cat: /var/lib/rancher/capr/idempotence/etcd-restore/restore-kill-all/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f/last-attempt: No such file or directory"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + echo -1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + currentAttempt=-1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + [ -1 != 1 ]"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + mkdir -p /var/lib/rancher/capr/idempotence/etcd-restore/restore-kill-all/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + echo 1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stderr]: + exec /bin/sh -c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_0:stdout]: rke2 does not appear to be installed"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] Command /bin/sh [-x /var/lib/rancher/capr/idempotence/idempotent.sh etcd-restore/restore-kill-all e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f 77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3 /bin/sh -c if [ -z $(command -v rke2) ] && [ -z $(command -v rke2-killall.sh) ]; then echo rke2 does not appear to be installed; exit 0; else rke2-killall.sh; fi] finished with err: <nil> and exit code: 0"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240119-144801/88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] Running command: /bin/sh [-x /var/lib/rancher/capr/idempotence/idempotent.sh etcd-restore/restore-manifest-removal e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f 77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3 /bin/sh -c rm -rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml]"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + currentHash="
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + key=etcd-restore/restore-manifest-removal"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + targetHash=e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + hashedCmd=77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + cmd=/bin/sh"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + shift 4"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + dataRoot=/var/lib/rancher/capr/idempotence/etcd-restore/restore-manifest-removal/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + attemptFile=/var/lib/rancher/capr/idempotence/etcd-restore/restore-manifest-removal/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f/last-attempt"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + cat /var/lib/rancher/capr/idempotence/etcd-restore/restore-manifest-removal/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f/last-attempt"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: cat: /var/lib/rancher/capr/idempotence/etcd-restore/restore-manifest-removal/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f/last-attempt: No such file or directory"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + echo -1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + currentAttempt=-1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + [ -1 != 1 ]"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + mkdir -p /var/lib/rancher/capr/idempotence/etcd-restore/restore-manifest-removal/77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3/e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + echo 1"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_1:stderr]: + exec /bin/sh -c rm -rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] Command /bin/sh [-x /var/lib/rancher/capr/idempotence/idempotent.sh etcd-restore/restore-manifest-removal e9a86864cd61016c7bee26d2ab5a379e4e6fd8b1cc6389b9578d1b351c77575f 77bafa9e3a8a092afc4f1dd84e6e1cc58b68f42eb934bf0f6b73deb2518f78a3 /bin/sh -c rm -rf /var/lib/rancher/rke2/server/manifests/rke2-*.yaml] finished with err: <nil> and exit code: 0"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="[Applyinator] Extracting image repo.yousen.plus/rancher/system-agent-installer-rke2:v1.27.7-rke2r2 to directory /var/lib/rancher/agent/work/20240119-144801/88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_2"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="Pulling image repo.yousen.plus/rancher/system-agent-installer-rke2:v1.27.7-rke2r2"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20240119-144801/88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_2/installer.sh"
Jan 19 14:48:01 debian rancher-system-agent[2643]: time="2024-01-19T14:48:01+08:00" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20240119-144801/88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39_2/rke2.linux-amd64.tar.gz"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=error msg="error while staging: stream error: stream ID 7; PROTOCOL_ERROR; received from peer"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=error msg="error executing instruction 2: stream error: stream ID 7; PROTOCOL_ERROR; received from peer"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=info msg="[K8s] updated plan secret fleet-default/custom-ea54025c6f96-machine-plan with feedback"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (572814 vs 573340)"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=error msg="error syncing 'fleet-default/custom-ea54025c6f96-machine-plan': handler secret-watch: secret received was too old, requeuing"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=error msg="[K8s] Maximum failure threshold exceeded for plan with checksum value of 88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39, (failures: 1, threshold: 1)"
Jan 19 14:49:03 debian rancher-system-agent[2643]: time="2024-01-19T14:49:03+08:00" level=error msg="[K8s] Maximum failure threshold exceeded for plan with checksum value of 88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39, (failures: 1, threshold: 1)"
Jan 19 14:49:08 debian rancher-system-agent[2643]: time="2024-01-19T14:49:08+08:00" level=error msg="[K8s] Maximum failure threshold exceeded for plan with checksum value of 88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39, (failures: 1, threshold: 1)"
Jan 19 14:49:13 debian rancher-system-agent[2643]: time="2024-01-19T14:49:13+08:00" level=error msg="[K8s] Maximum failure threshold exceeded for plan with checksum value of 88750b29d1b049ac28ee5025ac1041f16ece590d31443c7f15a2b4191527ca39, (failures: 1, threshold: 1)"