Rancher 无原故一直自动重启

jzyuan2008 · 2025 年5 月 28 日 07:28

Rancher Server 设置

Rancher 版本： 2.6.9
安装选项 (Docker install/Helm Chart): docker install
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
在线或离线部署：离线

下游集群信息

Kubernetes 版本: v1.23.6
Cluster Type (Local/Downstream): Downstream
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等): 自定义

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：admin
- 如果自定义，自定义权限集：

主机操作系统：
centos7.9
问题描述：
rancher正常运行大概一年多，这两天无端自动重启，一直在自动重启，机器负载正常，没有做过其它的的操作。
重现步骤：

结果：
无法正常运行rancher

预期结果：
希望可以正常跑起来

截图：

其他上下文信息：

日志

INFO: Running k3s server --cluster-init --cluster-reset
ERROR:
time="2025-05-28T14:34:09+08:00" level=warning msg="remove /var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json: no such file or directory"
time="2025-05-28T14:34:09+08:00" level=warning msg="remove /var/lib/rancher/k3s/agent/etc/k3s-api-server-agent-load-balancer.json: no such file or directory"
time="2025-05-28T14:34:09+08:00" level=info msg="Starting k3s v1.24.4+k3s1 (c3f830e9)"
time="2025-05-28T14:34:09+08:00" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
time="2025-05-28T14:34:09+08:00" level=info msg="Starting temporary etcd to reconcile with datastore"
{"level":"info","ts":"2025-05-28T14:34:10.277+0800","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2400"]}
{"level":"info","ts":"2025-05-28T14:34:10.277+0800","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"]}
{"level":"info","ts":"2025-05-28T14:34:10.277+0800","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.18.1","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"b7baa90125cd-1c6fd1d9","data-dir":"/var/lib/rancher/k3s/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/k3s/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2025-05-28T14:34:10.282+0800","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","took":"4.759823ms"}
{"level":"info","ts":"2025-05-28T14:34:10.753+0800","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":277849591,"snapshot-size":"401 kB"}
{"level":"warn","ts":"2025-05-28T14:34:10.754+0800","caller":"snap/db.go:88","msg":"failed to find [SNAPSHOT-INDEX].snap.db","snapshot-index":277849591,"snapshot-file-path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/00000000108fa5f7.snap.db","error":"snap: snapshot file doesn't exist"}
{"level":"panic","ts":"2025-05-28T14:34:10.754+0800","caller":"etcdserver/server.go:515","msg":"failed to recover v3 backend from snapshot","error":"failed to find database snapshot file (snap: snapshot file doesn't exist)","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.NewServer\n\t/go/pkg/mod/github.com/k3s-io/etcd/server/v3@v3.5.3-k3s1/etcdserver/server.go:515\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/go/pkg/mod/github.com/k3s-io/etcd/server/v3@v3.5.3-k3s1/embed/etcd.go:245\ngithub.com/k3s-io/k3s/pkg/daemons/executor.(*Embedded).ETCD\n\t/go/src/github.com/k3s-io/k3s/pkg/daemons/executor/etcd.go:30\ngithub.com/k3s-io/k3s/pkg/etcd.(*ETCD).StartEmbeddedTemporary\n\t/go/src/github.com/k3s-io/k3s/pkg/etcd/etcd.go:862\ngithub.com/k3s-io/k3s/pkg/cluster.(*Cluster).reconcileEtcd\n\t/go/src/github.com/k3s-io/k3s/pkg/cluster/bootstrap.go:500\ngithub.com/k3s-io/k3s/pkg/cluster.(*Cluster).Bootstrap\n\t/go/src/github.com/k3s-io/k3s/pkg/cluster/bootstrap.go:63\ngithub.com/k3s-io/k3s/pkg/daemons/control.prepare\n\t/go/src/github.com/k3s-io/k3s/pkg/daemons/control/server.go:264\ngithub.com/k3s-io/k3s/pkg/daemons/control.Server\n\t/go/src/github.com/k3s-io/k3s/pkg/daemons/control/server.go:37\ngithub.com/k3s-io/k3s/pkg/server.StartServer\n\t/go/src/github.com/k3s-io/k3s/pkg/server/server.go:59\ngithub.com/k3s-io/k3s/pkg/cli/server.run\n\t/go/src/github.com/k3s-io/k3s/pkg/cli/server/server.go:454\ngithub.com/k3s-io/k3s/pkg/cli/server.Run\n\t/go/src/github.com/k3s-io/k3s/pkg/cli/server/server.go:42\ngithub.com/urfave/cli.HandleAction\n\t/go/pkg/mod/github.com/urfave/cli@v1.22.9/app.go:524\ngithub.com/urfave/cli.Command.Run\n\t/go/pkg/mod/github.com/urfave/cli@v1.22.9/command.go:173\ngithub.com/urfave/cli.(*App).Run\n\t/go/pkg/mod/github.com/urfave/cli@v1.22.9/app.go:277\nmain.main\n\t/go/src/github.com/k3s-io/k3s/cmd/server/main.go:74\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
panic: failed to recover v3 backend from snapshot

goroutine 1 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00089c000, {0xc00039b000, 0x1, 0x1})
        /go/pkg/mod/go.uber.org/zap@v1.19.0/zapcore/entry.go:232 +0x44c
go.uber.org/zap.(*Logger).Panic(0xc00007d8a8?, {0x4c5ed46?, 0x0?}, {0xc00039b000, 0x1, 0x1})
        /go/pkg/mod/go.uber.org/zap@v1.19.0/logger.go:230 +0x59
go.etcd.io/etcd/server/v3/etcdserver.NewServer({{0xc00007d8a8, 0x15}, {0x0, 0x0}, {0x0, 0x0}, {0xc000c4dcb0, 0x1, 0x1}, {0xc000c4db90, ...}, ...})
        /go/pkg/mod/github.com/k3s-io/etcd/server/v3@v3.5.3-k3s1/etcdserver/server.go:515 +0x1d85
go.etcd.io/etcd/server/v3/embed.StartEtcd(0xc0003a7100)
        /go/pkg/mod/github.com/k3s-io/etcd/server/v3@v3.5.3-k3s1/embed/etcd.go:245 +0x1158
github.com/k3s-io/k3s/pkg/daemons/executor.(*Embedded).ETCD(_, {_, _}, {{{0xc000312e28, 0x15}, {0x0, 0x0}, {0x0, 0x0}}, {0xc000312c78, ...}, ...}, ...)
        /go/src/github.com/k3s-io/k3s/pkg/daemons/executor/etcd.go:30 +0x10e
github.com/k3s-io/k3s/pkg/etcd.(*ETCD).StartEmbeddedTemporary(0xc00035c1e0, {0x565d380, 0xc00012b6c0})
        /go/src/github.com/k3s-io/k3s/pkg/etcd/etcd.go:862 +0x53c
github.com/k3s-io/k3s/pkg/cluster.(*Cluster).reconcileEtcd(0xc0002808d0, {0x565d380, 0xc00039b140})
        /go/src/github.com/k3s-io/k3s/pkg/cluster/bootstrap.go:500 +0x2e5
github.com/k3s-io/k3s/pkg/cluster.(*Cluster).Bootstrap(0xc0002808d0, {0x565d380, 0xc00039b140}, 0x0)
        /go/src/github.com/k3s-io/k3s/pkg/cluster/bootstrap.go:63 +0x1e5
github.com/k3s-io/k3s/pkg/daemons/control.prepare({0x565d380, 0xc00039b140}, 0xc000c36a08)
        /go/src/github.com/k3s-io/k3s/pkg/daemons/control/server.go:264 +0x245
github.com/k3s-io/k3s/pkg/daemons/control.Server({0x565d380?, 0xc00039b140}, 0xc000c36a08)
        /go/src/github.com/k3s-io/k3s/pkg/daemons/control/server.go:37 +0xff
github.com/k3s-io/k3s/pkg/server.StartServer({0x565d380?, 0xc00039b140}, 0xc000c36a00, 0x1?)
        /go/src/github.com/k3s-io/k3s/pkg/server/server.go:59 +0x93
github.com/k3s-io/k3s/pkg/cli/server.run(0xc000c98160, 0x7eb7b80, {0xc000d9eb60, 0x0, 0xf?}, {0xc000d9eb60, 0x0, 0x7d27760?})
        /go/src/github.com/k3s-io/k3s/pkg/cli/server/server.go:454 +0x29d1
github.com/k3s-io/k3s/pkg/cli/server.Run(0xc000c98160?)
        /go/src/github.com/k3s-io/k3s/pkg/cli/server/server.go:42 +0x35
github.com/urfave/cli.HandleAction({0x42d9fe0?, 0x4fa8cc0?}, 0x6?)
        /go/pkg/mod/github.com/urfave/cli@v1.22.9/app.go:524 +0xa8
github.com/urfave/cli.Command.Run({{0x4b97c3d, 0x6}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x4bd4411, 0x15}, {0xc000cf40a8, ...}, ...}, ...)
        /go/pkg/mod/github.com/urfave/cli@v1.22.9/command.go:173 +0x652
github.com/urfave/cli.(*App).Run(0xc0007ec540, {0xc00012a040, 0x4, 0x4})
        /go/pkg/mod/github.com/urfave/cli@v1.22.9/app.go:277 +0x8a7
main.main()
        /go/src/github.com/k3s-io/k3s/cmd/server/main.go:74 +0xafb

ksd · 2025 年5 月 28 日 08:35

看日志，rancher 内置的 K3s 在启动时尝试恢复 etcd 快照失败。

有可能是主机异常断电导致的。针对这种情况，如果有备份就用备份恢复下 rancher

如果您在生产环境中使用 Rancher，希望获得更专业、及时的技术支持，也欢迎了解一下我们的商业订阅服务。可以点击论坛右上角聊天（）图标，私信联系我了解详细信息，我们有中文支持团队为您服务

jzyuan2008 · 2025 年5 月 28 日 08:49

没有断过电，机器已经开机200天了，主要是没有备份，还有其它办法吗？

jzyuan2008 · 2025 年5 月 28 日 10:04

这个文件是否为etcd的备份文件？能否通过这个文件恢复？如何恢复？