Rancher 创建rke2集群开启授权端点访问，直连下游集群报错 the server has asked for the client to provide credentials

GGbond · 2024 年4 月 2 日 03:20

Rancher Server 设置

Rancher 版本：2.8.2
安装选项 (Docker install/Helm Chart): helm rke1
- 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
在线或离线部署：在线

下游集群信息

Kubernetes 版本: v1.27.8 +rke2r1
Cluster Type (Local/Downstream): Downstream
- 如果 Downstream，是什么类型的集群?(自定义/导入或为托管等): 自定义

用户信息

登录用户的角色是什么？（管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：管理员
- 如果自定义，自定义权限集：

**主机操作系统：**centos7.9

**问题描述：**Rancher ui 开启集群授权端点访问，通过kube-api-auth 认证token直连下游集群。下载kubeconfig文件，使用kubectl 命令测试

**重现步骤：**编辑集群

结果：

预期结果：

截图：

其他上下文信息：

日志

kube-api-auth 日志
W0402 03:19:13.786501       1 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v3.ClusterUserAttribute: the server could not find the requested resource (get clusteruserattributes.meta.k8s.io)
2024-04-02T11:19:13.786689192+08:00 E0402 03:19:13.786528       1 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v3.ClusterUserAttribute: failed to list *v3.ClusterUserAttribute: the server could not find the requested resource (get clusteruserattributes.meta.k8s.io)

GGbond · 2024 年4 月 2 日 05:55

rke2集群没有clusterauthtokens 资源
kubectl --kubeconfig=dev.yaml get clusterauthtokens -A
error: the server doesn’t have a resource type “clusterauthtokens”
kubectl get crd |grep clusterauthtokens

ksd · 2024 年4 月 2 日 06:42

我使用相同的步骤并没有重现：

docker run 安装的 rancher v2.8.2，操作系统为 Ubuntu 20.04.6
通过创建下游集群 rke2 v1.27.12+rke2r1，并手动选择
Authorized Endpoint 选择 true

创建成功后，可以通过 rancher api 连接下游集群：

root@ip-172-31-5-166:~# kubectl --kubeconfig=demo.yaml config get-contexts
CURRENT   NAME                    CLUSTER                 AUTHINFO   NAMESPACE
*         demo                    demo                    demo
          demo-ip-172-31-13-146   demo-ip-172-31-13-146   demo
root@ip-172-31-5-166:~# kubectl --kubeconfig=demo.yaml get  clusterauthtokens -A
NAMESPACE       NAME                         AGE
cattle-system   kubeconfig-user-sg2zhj6rn8   14m
cattle-system   kubeconfig-user-sg2zhtkwkq   14m

ksd · 2024 年4 月 2 日 06:54

弱弱问一句，你这是有需求必须从 rancher api 访问下游集群么？

GGbond · 2024 年4 月 2 日 06:57

首次创建集群的时候没有选择 * Authorized Endpoint 选择 true
集群运行了一段时间，然后才开启这个参数，不会自动创建clusterauthtokens 资源

GGbond · 2024 年4 月 2 日 07:03

通过rancher 代理很多命令都会出问题，比如用helm的时候，或者
kubectl rollout status deployment 命令都会有问题

ksd · 2024 年4 月 2 日 07:10

那你为啥不描述清楚，测试环境都铲掉了……

ksd · 2024 年4 月 2 日 07:10

这是两个问题，应该和是否开启 Authorized Endpoint 没关系

GGbond · 2024 年4 月 2 日 07:17

现在主要问题是，没有在首次创建集群的时候选择开启 Authorized Endpoint ，之后再开启就有问题，不会创建clusterauthtokens 资源，kube-api-auth 就没办法校验token

GGbond · 2024 年4 月 2 日 08:04

这个问题有办法处理不，集群创建成功后才开启 Authorized Endpoint，就会有问题，不会创建clusterauthtokens

ksd · 2024 年4 月 2 日 08:39

我刚才按照你的步骤又进行了重现，虽然确实没有创建 clusterauthtokens，但不影响使用：

root@ip-172-31-8-202:~# kubectl --kubeconfig=demo.yaml get nodes
NAME              STATUS   ROLES                              AGE   VERSION
ip-172-31-9-222   Ready    control-plane,etcd,master,worker   12m   v1.27.12+rke2r1
root@ip-172-31-8-202:~# kubectl --kubeconfig=demo.yaml config get-contexts
CURRENT   NAME                   CLUSTER                AUTHINFO   NAMESPACE
*         demo                   demo                   demo
          demo-ip-172-31-9-222   demo-ip-172-31-9-222   demo
root@ip-172-31-8-202:~#
root@ip-172-31-8-202:~# kubectl --kubeconfig=demo.yaml get  clusterauthtokens -A

error: the server doesn't have a resource type "clusterauthtokens"

GGbond · 2024 年4 月 2 日 08:43

切换到demo-ip-172-31-9-222 contexts 无法使用，我需要直接通过集群的节点连接，不走rancher的代理
E0402 16:21:35.703402 29732 memcache.go:265] couldn’t get current server API group list: the server has asked for the client to provide credentials
E0402 16:21:35.705065 29732 memcache.go:265] couldn’t get current server API group list: the server has asked for the client to provide credentials
E0402 16:21:35.706148 29732 memcache.go:265] couldn’t get current server API group list: the server has asked for the client to provide credentials
E0402 16:21:35.707172 29732 memcache.go:265] couldn’t get current server API group list: the server has asked for the client to provide credentials
E0402 16:21:35.708027 29732 memcache.go:265] couldn’t get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

GGbond · 2024 年4 月 2 日 08:55

使用demo-ip-172-31-9-222 直接和下游集群通信会报错的
kube-api-auth 都没办法验证token了

ksd · 2024 年4 月 2 日 08:59

确实重现了，我们后续会排查。

另外，有个备选方案看看是否满足你的需求，就是你直接从下游 rke2 集群的 /etc/rancher/rke2/rke2.yaml 直接访问 rke2 ，我这边测试是没问题的，但不知道是否满足你的需求。

GGbond · 2024 年4 月 2 日 09:00

这个确实可以，权限太大了。我就想用rancher创建好的权限。用于cicd 构建的，只允许更新自己的项目

GGbond · 2024 年4 月 2 日 09:00

自己创建rbac就是有点麻烦，想直接用rancher配置好的角色

GGbond · 2024 年4 月 2 日 09:02

这个问题如果修复了，或者有方案，回复一下咯！感谢

tbag · 2024 年4 月 17 日 04:34

这个问题我也遇到了，今天在测试ACE的时候，如下：

CURRENT   NAME                                  CLUSTER                               AUTHINFO   NAMESPACE
          yk-dev                                yk-dev                                yk-dev     
          yk-dev-sg-dev-yk-k8s-master-01-rke2   yk-dev-sg-dev-yk-k8s-master-01-rke2   yk-dev     
          yk-dev-sg-dev-yk-k8s-master-02-rke2   yk-dev-sg-dev-yk-k8s-master-02-rke2   yk-dev     
*         yk-dev-sg-dev-yk-k8s-master-03-rke2   yk-dev-sg-dev-yk-k8s-master-03-rke2   yk-dev

[root@sg-dev-yk-k8s-master-01-rke2 spadm]# /var/lib/rancher/rke2/bin/kubectl --kubeconfig yk-dev.yaml --context yk-dev-sg-dev-yk-k8s-master-03-rke2 get node
E0417 12:13:24.416990 1133467 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0417 12:13:24.419895 1133467 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0417 12:13:24.422547 1133467 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0417 12:13:24.425318 1133467 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
E0417 12:13:24.428502 1133467 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server (the server has asked for the client to provide credentials)

上面是我没有使用 FQDN；

当我配置了 fqdn 的时候，并解析到其中一台 master 节点：

[root@sg-dev-yk-k8s-master-01-rke2 spadm]# /var/lib/rancher/rke2/bin/kubectl --kubeconfig yk-dev-fqdn.yaml config get-contexts
CURRENT   NAME          CLUSTER       AUTHINFO   NAMESPACE
          yk-dev        yk-dev        yk-dev     
*         yk-dev-fqdn   yk-dev-fqdn   yk-dev     

[root@sg-dev-yk-k8s-master-01-rke2 spadm]# /var/lib/rancher/rke2/bin/kubectl --kubeconfig yk-dev.yaml get node
E0417 11:25:19.482585 1080477 memcache.go:265] couldn't get current server API group list: Get "https://rancher-ace-yk-dev.ab.aaa/api?timeout=32s": dial tcp 10.65.23.14:443: connect: connection refused
E0417 11:25:19.490030 1080477 memcache.go:265] couldn't get current server API group list: Get "https://rancher-ace-yk-dev.ab.aaa/api?timeout=32s": dial tcp 10.65.23.14:443: connect: connection refused
E0417 11:25:19.496368 1080477 memcache.go:265] couldn't get current server API group list: Get "https://rancher-ace-yk-dev.ab.aaa/api?timeout=32s": dial tcp 10.65.23.14:443: connect: connection refused
E0417 11:25:19.541515 1080477 memcache.go:265] couldn't get current server API group list: Get "https://rancher-ace-yk-dev.ab.aaa/api?timeout=32s": dial tcp 10.65.23.14:443: connect: connection refused
E0417 11:25:19.547848 1080477 memcache.go:265] couldn't get current server API group list: Get "https://rancher-ace-yk-dev.ab.aaa/api?timeout=32s": dial tcp 10.65.23.14:443: connect: connection refused
The connection to the server rancher-ace-yk-dev.ab.aaa was refused - did you specify the right host or port?

Rancher	v2.8.3
仪表板	v2.8.3
Helm	v2.16.8-rancher2
Machine	v0.15.0-rancher110

这是 bug 还是什么？？？？？有谁知道吗？

@ksd

GGbond · 2024 年4 月 23 日 07:45

github.com/rancher/rancher

[BUG] Enabling ACE after cluster provisioning results in unusable kubeconfig contexts

opened 09:16PM - 13 Jun 23 UTC

thatmidwesterncoder

kind/bug internal area/ace team/hostbusters JIRA

**Rancher Server Setup** - Rancher version: v2.7.3+, reported on 2.7.3 but pers…ists to newest version - Installation option (Docker install/Helm Chart): Docker and running locally from source - If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): k3s latest via k3d - Proxy/Cert Details: **Information about the Cluster** - Kubernetes version: 1.25.9, but any version most likely will be effected - Cluster Type (Local/Downstream): Downstream - If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): Custom and DigitalOcean  **User Information** - What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom): Admin - If custom, define the set of permissions: all **Describe the bug** When you create a RKE2 cluster on existing nodes or provisioned through <Cloud Provider> and activate "Authorized Cluster Endpoint" after the cluster is available; connecting to the ACE endpoint results in a "401 Unauthorized" even though there is a cert/token present. The ACE works fine _if it is enabled when creating the cluster_. This behavior only occurs when enabling ACE after creation. **To Reproduce** 1. Create a RKE2 cluster without ACE, can leave everything default 2. Once it is provisioned, edit the cluster and activate ACE 3. Download a kubeconfig and try to use an ACE context **Result** The ACE context does not work. **Expected Result** The ACE should work exactly as it does if it is enabled during provisioning -> e.g. exactly like a regular k8s context hitting the infrastructure node(s). **Screenshots** n/a **Additional context** - The kubeconfig looks fine - cert is present for ACE, and token is used for relay context - I dug a bit into it due to the fact that it appears CRDs for `clusterauthtoken` aren't present, thus the kube-api-proxy pod barks about not being able to fetch the token when the user comes in on the ACE endpoint - applying the diff below seems to fix it. But I am not sure of the other repercussions that could be caused by having the agent watching more CRDs. Maybe these CRDs are harder/heavier to watch and thats why we wouldn't want them installed on every cluster. - this also makes rancher be the part that says "install X CRDs", I'm not sure if this should be moved somewhere else. <details> <summary>always_install_token_crds.diff</summary> ```diff diff --git a/pkg/controllers/managementuser/controllers.go b/pkg/controllers/managementuser/controllers.go index f08e9909a..7a9129214 100644 --- a/pkg/controllers/managementuser/controllers.go +++ b/pkg/controllers/managementuser/controllers.go @@ -48,13 +48,12 @@ func Register(ctx context.Context, mgmt *config.ScaledContext, cluster *config.U // register secrets controller for impersonation cluster.Core.Secrets("").Controller() - if clusterRec.Spec.LocalClusterAuthEndpoint.Enabled { - err := clusterauthtoken.CRDSetup(ctx, cluster.UserOnlyContext()) - if err != nil { - return err - } - clusterauthtoken.Register(ctx, cluster) + // create the auth CRDs + err := clusterauthtoken.CRDSetup(ctx, cluster.UserOnlyContext()) + if err != nil { + return err } + clusterauthtoken.Register(ctx, cluster) // Ensure these caches are started cluster.Core.Namespaces("").Controller() ``` </details> SURE-6359 SURE-6353

tbag · 2024 年4 月 23 日 07:48

这个我看到了