服务器硬件故障,恢复后导致 rancher 内置的 etcd 丢数据或者其他原因

Rancher Server 设置

  • Rancher 版本:2.3.11
  • 安装选项 (Docker install/Helm Chart): Docker
    • 如果是 Helm Chart 安装,需要提供 Local 集群的类型(RKE1, RKE2, k3s, EKS, 等)和版本:
  • 在线或离线部署:离线

下游集群信息

  • Kubernetes 版本: v1.17.17-rancher1-1
  • Cluster Type (Local/Downstream):
    • 如果 Downstream,是什么类型的集群?(自定义/导入或为托管 等): 自定义

主机操作系统:
CentOS Linux release 7.5.1804 (Core)

问题描述:
设备故障修复后,虚拟机上的Rancher UI 无法访问。
经重启查看日志如下:

panic: runtime error: slice bounds out of range
致命错误推出。

重现步骤:

结果:

预期结果:

截图:

其他上下文信息:

日志
{"log":"2024/05/16 03:38:49 [INFO] Rancher version v2.3.11 is starting\n","stream":"stdout","time":"2024-05-16T03:38:49.730244293Z"}
{"log":"2024/05/16 03:38:49 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false KubeConfig: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false NoCACerts:false ListenConfig:\u003cnil\u003e AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}\n","stream":"stdout","time":"2024-05-16T03:38:49.730264422Z"}
{"log":"2024/05/16 03:38:49 [INFO] Listening on /tmp/log.sock\n","stream":"stdout","time":"2024-05-16T03:38:49.730290851Z"}
{"log":"2024/05/16 03:38:49 [INFO] Running etcd --data-dir=management-state/etcd\n","stream":"stdout","time":"2024-05-16T03:38:49.730641402Z"}
{"log":"2024-05-16 03:38:49.741264 W | pkg/flags: unrecognized environment variable ETCD_URL_arm64=https://github.com/etcd-io/etcd/releases/download/v3.3.14/etcd-v3.3.14-linux-arm64.tar.gz\n","stream":"stderr","time":"2024-05-16T03:38:49.741398481Z"}
{"log":"2024-05-16 03:38:49.741305 W | pkg/flags: unrecognized environment variable ETCD_URL_amd64=https://github.com/etcd-io/etcd/releases/download/v3.3.14/etcd-v3.3.14-linux-amd64.tar.gz\n","stream":"stderr","time":"2024-05-16T03:38:49.741422575Z"}
{"log":"2024-05-16 03:38:49.741310 W | pkg/flags: unrecognized environment variable ETCD_UNSUPPORTED_ARCH=amd64\n","stream":"stderr","time":"2024-05-16T03:38:49.741427159Z"}
{"log":"2024-05-16 03:38:49.741314 W | pkg/flags: unrecognized environment variable ETCD_URL=ETCD_URL_amd64\n","stream":"stderr","time":"2024-05-16T03:38:49.741430864Z"}
{"log":"2024-05-16 03:38:49.741333 I | etcdmain: etcd Version: 3.3.14\n","stream":"stderr","time":"2024-05-16T03:38:49.74144219Z"}
{"log":"2024-05-16 03:38:49.741337 I | etcdmain: Git SHA: 5cf5d88a1\n","stream":"stderr","time":"2024-05-16T03:38:49.741445731Z"}
{"log":"2024-05-16 03:38:49.741340 I | etcdmain: Go Version: go1.12.9\n","stream":"stderr","time":"2024-05-16T03:38:49.741449051Z"}
{"log":"2024-05-16 03:38:49.741343 I | etcdmain: Go OS/Arch: linux/amd64\n","stream":"stderr","time":"2024-05-16T03:38:49.741452639Z"}
{"log":"2024-05-16 03:38:49.741347 I | etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8\n","stream":"stderr","time":"2024-05-16T03:38:49.741456009Z"}
{"log":"2024-05-16 03:38:49.741400 N | etcdmain: the server is already initialized as member before, starting as etcd member...\n","stream":"stderr","time":"2024-05-16T03:38:49.741461105Z"}
{"log":"2024-05-16 03:38:49.741817 I | embed: listening for peers on http://localhost:2380\n","stream":"stderr","time":"2024-05-16T03:38:49.741875319Z"}
{"log":"2024-05-16 03:38:49.741951 I | embed: listening for client requests on localhost:2379\n","stream":"stderr","time":"2024-05-16T03:38:49.742005537Z"}
{"log":"2024-05-16 03:38:49.756301 I | etcdserver: recovered store from snapshot at index 221802322\n","stream":"stderr","time":"2024-05-16T03:38:49.756369204Z"}
{"log":"2024-05-16 03:38:49.756768 I | mvcc: restore compact to 184040025\n","stream":"stderr","time":"2024-05-16T03:38:49.756842764Z"}
{"log":"panic: runtime error: slice bounds out of range\n","stream":"stderr","time":"2024-05-16T03:38:49.767780637Z"}
{"log":"\n","stream":"stderr","time":"2024-05-16T03:38:49.767792572Z"}
{"log":"goroutine 133 [running]:\n","stream":"stderr","time":"2024-05-16T03:38:49.767796175Z"}
{"log":"github.com/coreos/etcd/mvcc.bytesToRev(0x7f81e42760b0, 0x5, 0x5, 0x0, 0x192be00)\n","stream":"stderr","time":"2024-05-16T03:38:49.767799689Z"}
{"log":"\u0009/tmp/etcd-release-3.3.14/etcd/release/etcd/mvcc/revision.go:58 +0x75\n","stream":"stderr","time":"2024-05-16T03:38:49.767803254Z"}
{"log":"github.com/coreos/etcd/mvcc.restoreIntoIndex.func1(0xc0001bc1c0, 0xc00001a480, 0x11320c0, 0xc00000cc60)\n","stream":"stderr","time":"2024-05-16T03:38:49.767822304Z"}
{"log":"\u0009/tmp/etcd-release-3.3.14/etcd/release/etcd/mvcc/kvstore.go:428 +0x227\n","stream":"stderr","time":"2024-05-16T03:38:49.767833801Z"}
{"log":"created by github.com/coreos/etcd/mvcc.restoreIntoIndex\n","stream":"stderr","time":"2024-05-16T03:38:49.767837983Z"}
{"log":"\u0009/tmp/etcd-release-3.3.14/etcd/release/etcd/mvcc/kvstore.go:403 +0xa0\n","stream":"stderr","time":"2024-05-16T03:38:49.767841439Z"}
{"log":"2024/05/16 03:38:49 [FATAL] etcd exited\n","stream":"stdout","time":"2024-05-16T03:38:49.769027042Z"}


参考:Rancher单节点容器无法启动