Rancher在离线的环境下， 重启后无法启动了

Rancher 2.x
1

Rancher Server 设置

  • Rancher 版本：2.5.17
  • 安装选项 (Docker install/Helm Chart):
    • 如果是 Helm Chart 安装，需要提供 Local 集群的类型（RKE1, RKE2, k3s, EKS, 等）和版本：
  • 在线或离线部署： Docker方式部署

下游集群信息

  • Kubernetes 版本:
  • Cluster Type (Local/Downstream):
    • 如果 Downstream，是什么类型的集群?(自定义/导入或为托管 等):

用户信息

  • 登录用户的角色是什么？ （管理员/集群所有者/集群成员/项目所有者/项目成员/自定义）：
    • 如果自定义，自定义权限集：

主机操作系统：

问题描述：

启动命令：

#!/bin/bash
# 运行rancher

docker run -itd --name rancher \
           -p 80:80 \
           -p 443:443 \
           --restart=unless-stopped \
           -v ${PWD}/rancher-data:/var/lib/rancher \
           -v /var/log/rancher/auditlog:/var/log/auditlog \
           -v ${PWD}/antiy.crt:/etc/rancher/ssl/cert.pem \
           -v ${PWD}/antiy.key:/etc/rancher/ssl/key.pem \
           -e AUDIT_LEVEL=3 \
           --privileged \
           rancher/rancher:v2.5.17 --no-cacerts 


所在的机器属于内网， 没有外网。 之前一直是正常的。后来重启了后，就报错了。

重现步骤：


[root@rancher-server-demo01 rancher]# docker restart rancher

结果：

预期结果：

截图：

其他上下文信息：

日志 
[root@rancher-server-demo01 rancher]# docker logs -f  rancher --tail 100 
2023/11/21 08:05:08 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:45788->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:08 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:45788->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:08 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:45788->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:28 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:47426->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:28 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:47426->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:28 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:47426->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:48 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:50900->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:48 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:50900->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:05:48 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:50900->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:09 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:51215->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:09 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:51215->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:09 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:51215->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:29 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:58845->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:29 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:58845->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:29 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:58845->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:49 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:33960->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:49 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:33960->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:06:49 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:33960->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:09 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:57109->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:09 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:57109->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:09 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:57109->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:15 [INFO] Stopping cluster agent for c-sd55f
2023/11/21 08:07:15 [ERROR] failed to start cluster controllers c-sd55f: context canceled
W1121 08:07:21.700032       8 warnings.go:80] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
2023/11/21 08:07:29 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:54564->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:29 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:54564->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:29 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:54564->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:49 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:35200->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:49 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:35200->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:07:49 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:35200->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:09 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:44788->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:09 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:44788->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:09 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:44788->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:29 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:35269->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:29 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:35269->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:29 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:35269->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:49 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:38947->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:49 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:38947->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:08:49 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:38947->10.255.0.2:53: i/o timeout, requeuing
W1121 08:08:55.189907       8 warnings.go:80] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
2023/11/21 08:09:09 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:43204->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:09 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:43204->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:09 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:43204->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:17 [INFO] Stopping cluster agent for c-sd55f
2023/11/21 08:09:17 [ERROR] failed to start cluster controllers c-sd55f: context canceled
2023/11/21 08:09:29 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:58092->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:29 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:58092->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:29 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:58092->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:49 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:39689->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:49 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:39689->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:09:49 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:39689->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:10:09 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:36016->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:10:09 [ERROR] error syncing 'system-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/system-charts] is not accessible: Get "https://git.rancher.io/repos/system-charts/commits/release-v2.5": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:36016->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:10:09 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:36016->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:10:29 [ERROR] error syncing 'helm3-library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/helm3-charts] is not accessible: Get "https://git.rancher.io/repos/helm3-charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:36770->10.255.0.2:53: i/o timeout, requeuing
2023/11/21 08:10:29 [ERROR] error syncing 'library': handler catalog: Remote commit check failed: Repo [https://git.rancher.io/charts] is not accessible: Get "https://git.rancher.io/repos/charts/commits/master": dial tcp: lookup git.rancher.io on 10.255.0.2:53: read udp 172.17.0.2:36770->10.255.0.2:53: i/o timeout, requeuing
2

handler catalog 的报错没啥大影响，因为你是离线环境，但 failed to start cluster controllers c-sd55f: context canceled 是有问题的，你应该检查下游集群的 cluster agent 日志，看看是什么原因没启动成功