Cattle-impersonation-system這個namespace會不斷產生 Secrets

luo · 2024 年4 月 10 日 03:59

问题描述：
這是一個下游集群的問題
cattle-impersonation-system這個namespace會不斷產生 Secrets，導致上游集群kube-apiserver的pod會一直出現錯誤

截图：
此圖為下游及群

日志

024-04-10T11:53:15.181780031+08:00 E0410 03:53:15.181310       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:19.970954836+08:00 E0410 03:53:19.970546       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:20.020637105+08:00 E0410 03:53:20.020499       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:20.381181309+08:00 E0410 03:53:20.380808       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:25.368476706+08:00 E0410 03:53:25.368191       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:30.155023701+08:00 E0410 03:53:30.154415       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:35.730861297+08:00 E0410 03:53:35.729632       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:35.774153476+08:00 E0410 03:53:35.771225       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:35.873098352+08:00 E0410 03:53:35.872659       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"
2024-04-10T11:53:40.960611563+08:00 E0410 03:53:40.960136       1 authentication.go:73] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"

jacie · 2024 年4 月 11 日 02:32

也许与这个问题类似

github.com/rancher/rancher

[BUG] Stale secrets for impersonation user in downstream cluster

opened 11:26AM - 06 Feb 24 UTC

vatsalparekh

kind/bug priority/0 area/rbac JIRA team/collie

**Rancher Server Setup** - Rancher version: 2.7.9, 2.8.2 **Information about… the Cluster** - Kubernetes version: v1.26.11+rke2r1 - Cluster Type (Local/Downstream): Downstream, created from Rancher on EC2 **User Information** - What is the role of the user logged in? `Admin` **Describe the bug** - Multiple stale `Secrets` created for a single impersonation user/ServiceAccount in the downstream cluster - SA shows 1 secret and single reference but multiple secrets with same SA reference found - The stale secrets don't get reconciled even if we delete the current referenced secret in the ServiceAccount ``` > k get secrets -n cattle-impersonation-system NAME TYPE DATA AGE cattle-impersonation-u-hjxlz-token-24kwk kubernetes.io/service-account-token 3 22h cattle-impersonation-u-l767yhswxf-token-lnzc2 kubernetes.io/service-account-token 3 3d23h cattle-impersonation-u-l767yhswxf-token-rdlkx kubernetes.io/service-account-token 3 4d22h cattle-impersonation-u-l767yhswxf-token-w5klt kubernetes.io/service-account-token 3 3d23h cattle-impersonation-u-oxfisb4qb2-token-gtjln kubernetes.io/service-account-token 3 4d23h cattle-impersonation-user-ptj5z-token-gzjnc kubernetes.io/service-account-token 3 4d23h > k get sa -n cattle-impersonation-system -o wide NAME SECRETS AGE cattle-impersonation-u-hjxlz 1 22h cattle-impersonation-u-l767yhswxf 1 4d22h cattle-impersonation-u-oxfisb4qb2 1 4d23h cattle-impersonation-user-ptj5z 1 4d23h default 0 4d23h > k get secrets -A -o yaml | grep 'cattle-impersonation-u-l767yhswxf' kubernetes.io/service-account.name: cattle-impersonation-u-l767yhswxf generateName: cattle-impersonation-u-l767yhswxf-token- name: cattle-impersonation-u-l767yhswxf-token-lnzc2 name: cattle-impersonation-u-l767yhswxf kubernetes.io/service-account.name: cattle-impersonation-u-l767yhswxf generateName: cattle-impersonation-u-l767yhswxf-token- name: cattle-impersonation-u-l767yhswxf-token-rdlkx name: cattle-impersonation-u-l767yhswxf kubernetes.io/service-account.name: cattle-impersonation-u-l767yhswxf generateName: cattle-impersonation-u-l767yhswxf-token- name: cattle-impersonation-u-l767yhswxf-token-w5klt name: cattle-impersonation-u-l767yhswxf ``` A snippet from the Rancher pod log shows that the multiple creation happens usually quickly after each other, hinting at lack of concurency while the Ensure func is called for the same ServiceAccount ``` 2024/02/13 11:29:43 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-user-kgnqk-token-cd6fl] to be populated with token 2024/02/13 11:29:43 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-user-kgnqk-token-v7r6z] to be populated with token 2024/02/13 11:29:44 [INFO] EnsureSecretForServiceAccount: waiting for secret [cattle-impersonation-user-kgnqk-token-sg9sb] to be populated with token 2024/02/13 11:29:44 [INFO] Creating roleBinding User user-kgnqk Role admin in cattle-system 2024/02/13 11:29:44 [INFO] Updating clusterRole project-owner-promoted for project access to global resource. 2024/02/13 11:29:44 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-xpeyhn3op3 with role cluster-owner in namespace p-g8bfg 2024/02/13 11:29:44 [INFO] Creating roleBinding User user-kgnqk Role project-owner in default 2024/02/13 11:29:44 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-xpeyhn3op3 with role cluster-owner in namespace p-s2mp7 2024/02/13 11:29:44 [INFO] Creating roleBinding User user-kgnqk Role admin in kube-system ``` **To Reproduce** This seems random at this point, on a Rancher with a downstream cluster, this can be seen out of the box in the downstream cluster. A customer has seen this for a custom user, in my reproducer this is happening for a fleet user. There's a time difference between the stale secrets creation, so this might not occur at the first try