Rancher升级至2.6.8后,运行一段时间后便会崩溃

Rancher 版本:2.6.8

安装选项: Helm Chart

Helm Chart 版本:v3.8.1

在线或离线部署:在线部署

Kubernetes 版本:1.19.16

Cluster Type (Local/Downstream): Local

问题描述:升级Rancher2.6.8后,运行一段时间便会报错,错误信息如下:
E0901 03:21:00.615480 32 runtime.go:79] Observed a panic: runtime.boundsError{x:3, y:0, signed:true, code:0x2} (runtime error: slice bounds out of range [:3] with capacity 0)
goroutine 5509 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3f478c0, 0xc006d51800})
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc014365400})
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:49 +0x75
panic({0x3f478c0, 0xc006d51800})
/usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/catalogv2/helm.decodeHelm3({0x0, 0xb6806d})
/go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:124 +0x1b1
github.com/rancher/rancher/pkg/catalogv2/helm.fromHelm3Data({0x0, 0xc004a23280}, 0x4238387)
/go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:23 +0x25
github.com/rancher/rancher/pkg/catalogv2/helm.ToRelease({0x4b3ce90, 0xc004a23280}, 0xc00bddfa78)
/go/src/github.com/rancher/rancher/pkg/catalogv2/helm/release.go:74 +0x3eb
github.com/rancher/rancher/pkg/controllers/dashboard/helm.(*appHandler).OnSecretChange(0xc001662640, {0xc0051a8e70, 0x24}, 0xc004a23280)
/go/src/github.com/rancher/rancher/pkg/controllers/dashboard/helm/apps.go:170 +0xa5
github.com/rancher/wrangler/pkg/generated/controllers/core/v1.FromSecretHandlerToHandler.func1({0xc0051a8e70, 0x5adffb3f7b2dbc}, {0x4b3ce90, 0xc004a23280})
/go/pkg/mod/github.com/rancher/wrangler@v1.0.1-0.20220520195731-8eeded9bae2a/pkg/generated/controllers/core/v1/secret.go:102 +0x44
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc001a930e0, {0xc0051a8e70, 0x430807b}, {0x4b3ce90, 0xc004a23280})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc001716b90, {0xc0051a8e70, 0x24}, {0x4b3ce90, 0xc004a23280})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000e1c160, {0xc0051a8e70, 0x24})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000e1c160, {0x37c5e40, 0xc014365400})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000e1c160)
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(…)
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f9b89642cc8)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x4b09600, 0xc007b0b830}, 0x1, 0xc002806060)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6
panic: runtime error: slice bounds out of range [:3] with capacity 0 [recovered]
panic: runtime error: slice bounds out of range [:3] with capacity 0
goroutine 5509 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc014365400})
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/runtime/runtime.go:56 +0xd8
panic({0x3f478c0, 0xc006d51800})
/usr/lib64/go/1.17/src/runtime/panic.go:1038 +0x215
github.com/rancher/rancher/pkg/catalogv2/helm.decodeHelm3({0x0, 0xb6806d})
/go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:124 +0x1b1
github.com/rancher/rancher/pkg/catalogv2/helm.fromHelm3Data({0x0, 0xc004a23280}, 0x4238387)
/go/src/github.com/rancher/rancher/pkg/catalogv2/helm/helm3.go:23 +0x25
github.com/rancher/rancher/pkg/catalogv2/helm.ToRelease({0x4b3ce90, 0xc004a23280}, 0xc00bddfa78)
/go/src/github.com/rancher/rancher/pkg/catalogv2/helm/release.go:74 +0x3eb
github.com/rancher/rancher/pkg/controllers/dashboard/helm.(*appHandler).OnSecretChange(0xc001662640, {0xc0051a8e70, 0x24}, 0xc004a23280)
/go/src/github.com/rancher/rancher/pkg/controllers/dashboard/helm/apps.go:170 +0xa5
github.com/rancher/wrangler/pkg/generated/controllers/core/v1.FromSecretHandlerToHandler.func1({0xc0051a8e70, 0x5adffb3f7b2dbc}, {0x4b3ce90, 0xc004a23280})
/go/pkg/mod/github.com/rancher/wrangler@v1.0.1-0.20220520195731-8eeded9bae2a/pkg/generated/controllers/core/v1/secret.go:102 +0x44
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc001a930e0, {0xc0051a8e70, 0x430807b}, {0x4b3ce90, 0xc004a23280})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedcontroller.go:29 +0x38
github.com/rancher/lasso/pkg/controller.(*SharedHandler).OnChange(0xc001716b90, {0xc0051a8e70, 0x24}, {0x4b3ce90, 0xc004a23280})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/sharedhandler.go:75 +0x23f
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc000e1c160, {0xc0051a8e70, 0x24})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:233 +0x93
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc000e1c160, {0x37c5e40, 0xc014365400})
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:214 +0x10e
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc000e1c160)
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:191 +0x46
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(…)
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:180
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f9b89642cc8)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:155 +0x67
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0, {0x4b09600, 0xc007b0b830}, 0x1, 0xc002806060)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.24.2/pkg/util/wait/wait.go:90 +0x25
created by github.com/rancher/lasso/pkg/controller.(*controller).run
/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20220627205005-00d9c8e9dda6/pkg/controller/controller.go:148 +0x2c6

重现步骤:rancher启动后10分钟左右出现

在删除了类型为Helm 版本的密文后,便不再报错和重启了。

没错,可能是你的版本是一直升级上来的,带有一些脏数据。
这个issue的workaround可以帮助解决:Dirty data of helm release causes cluster-agent to crash · Issue #35971 · rancher/rancher · GitHub

你已经解决了。不过,我把关联方案贴一下,方便其他群众。